Re: autopkgtest/sbuild environment variables: LC_ALL, HOME, XDG_RUNTIME_DIR etc

[Simon McVittie <smcv@debian.org>]

On Fri, 29 Apr 2022 at 10:43:40 +0800, Paul Wise wrote:
> Does that mean that autopkgtests that directly or indirectly use the
> dbus-run-session tool to setup a temporary D-Bus session are buggy?

There's no single answer to that. For unit-test-style testing, it's
often more convenient to start a session bus that's fully under the control
of the test suite, particularly if the tests need special configuration
(rejecting certain D-Bus messages, or setting limits, or making mock
D-Bus services activatable, or something like that).

For tests of the overall system (closer to an integration test), the
fact that the service is genuinely activatable on a real system seems
like something that the autopkgtest framework should not prevent you from
testing. You don't necessarily need to actually test this - perhaps your
priorities are elsewhere, and 100% test coverage is rarely realistic -
but I think it would be a design flaw if autopkgtest prevented you from
testing it, by forcing you to use a mock XDG_RUNTIME_DIR or a mock home
directory or anything else that would interfere with your tests' access
to our real system integration layer.

If you run dbus-run-session without the --config-file option, it does
use the real /usr/share/dbus-1/services, so you can test traditional
(non-systemd) session service activation like this. You can't test
systemd service activation this way, because that only works for the
dbus-daemon that was launched by systemd and told to integrate with it.

dbus has debian/tests/system-bus which does a brief integration test to
check that the system bus works, but it doesn't yet have an equivalent for
dbus-user-session - contributions welcome, if you are interested in
expanding coverage.

autopkgtest tries to set up a "real" system login session by
running su, even if it's already running as the target uid, to make
sure that some of the side-effects of login (in particular a PAM
stack) have already happened; but I suspect this doesn't have full
coverage. autopkgtest-virt-ssh is probably the most "realistic" of the
backends, by running test commands under a ssh session that is genuinely
a login entry point (meaning processes have a loginuid and so on),
and one of the things on my increasingly intractable to-do list is to
add a way to launch qemu VMs on-demand and use ssh to log in to them,
rather than using a privileged serial console like virt-qemu does.

In some backends it isn't really feasible to run the tests in a "real"
system login session, so integration tests for things that rely on
the system's session infrastructure might have to have the 'skippable'
restriction and detect whether they are running in a login session at
runtime. This is also very likely to be something that some autopkgtest
backends just cannot provide: -chroot and -schroot don't have their own
init systems (and neither would a Docker backend, review/finishing of
which is also on my to-do list), so they cannot be expected to run their
own system services.

> The other thing is testing graphical tools; should that be done in a
> mock X11/Wayland session (using xvfb-run), or a real desktop session?

At the moment a mock X11 session using xvfb-run or a mock
Wayland session using weston --backend=headless-backend.so (see
debian/tests/run-with-display in src:gtk4) are the only realistic options
for this.

Ideally it'd be possible to test graphical tools under a real desktop
session, but that's not straightforward to set up even under something
like autopkgtest-virt-qemu (the test framework would have to script
a login to a display manager using techniques similar to openQA, or
configure the display manager to autologin and then reboot to make the
autologin happen), and I'm not sure whether this would be feasible to
set up under other backends at all. There's a cost/benefit tradeoff here:
running tests under a real desktop session is likely to be more difficult,
slower, and less robust than xvfb-run, so we should only do it if the
benefit is worth those costs. In particular, we can are not going to
be able to use this as a QA gate unless someone puts a lot of time and
effort into making it completely reliable.

GNOME's installed-tests convention[1], which is a frequent source of
autopkgtests for Debian, *does* call for tests to be run in a real,
active GUI desktop session, although in practice most packages'
installed-tests do not actually need this.

[1] https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests

    smcv