Re: autopkgtest/sbuild environment variables: LC_ALL, HOME, XDG_RUNTIME_DIR etc
On Thu, 28 Apr 2022 at 08:33:02 +0100, Julian Gilbey wrote:
> When I run autopkgtest using sbuild on my own machine, I sometimes get
> warnings or breakages because certain environment variables are either
> unset or have "broken" values, whereas I see that on ci.debian.net and
> the buildds, the builds run without problem.
What autopkgtest backend are you using? (autopkgtest-virt-schroot?
autopkgtest-virt-qemu? ...) There are several, and their behaviour can be
very different, depending on the level of isolation they provide.
Having a concrete example of the command-line and configuration you use,
and a package that has warnings or breakage in this configuration, would
be useful.
> It would be really useful to be able to set up my local sbuild
> environment in the same way as the Debian machines (buildd and
> ci.debian.net) for testing purposes.
buildds do not run autopkgtest, only sbuild, so they run build-time tests
(dh_auto_test or similar) but do not interact with debian/tests/.
ci.debian.net currently uses autopkgtest-virt-lxc via the debci package.
> HOME (pointing to where?)
sbuild
------
There is no guarantee about $HOME while a Debian package is being built.
Policy §4.9 [0] says it is a bug for the build process of a package to
write into the real $HOME.
[0] https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules
In sbuild, $HOME is set to the home directory of the user running the
build, but that directory is not guaranteed to exist inside the chroot.
Most official buildds set it to /nonexistent, specifically so that builds
that rely on $HOME will fail.
debhelper compat levels >= 13 set a temporary $HOME for the duration
of the build, to make it easier for packages to comply with the Policy
§4.9 requirement.
autopkgtest
-----------
According to the autopkgtest spec[1],
Tests can expect that the $HOME environment variable to be set to
a directory that exists and is writeable by the user running the test.
Depending on backend, that directory might be your real home directory,
or it might be a transient/expendable home directory that it is OK for
the test to damage. It would probably be good to have a restriction[2]
that has the effect of guaranteeing that an expendable home directory
is used, as it is in autopkgtest-virt-qemu.
[1] https://salsa.debian.org/ci-team/autopkgtest/blob/HEAD/doc/README.package-tests.rst
[2] https://salsa.debian.org/ci-team/autopkgtest/blob/HEAD/doc/README.package-tests.rst#defined-restrictions
> LC_ALL (or something similar; what is its setting?)
LC_ALL is not normally set on "real" Debian systems. Usually, only $LANG
and $LANGUAGE are set, even on a full desktop system (for example my GNOME
desktop has LANG=en_GB.utf8 and LANGUAGE=en_GB:en, but no LC_ALL).
LC_ALL is intended to be the "big hammer" that overrides everything else.
sbuild
------
There are no guarantees about the locale with which a Debian package
will be built. The reproducible-builds infrastructure regularly builds
packages with non-English locales, which in practice often causes test
failures or non-reproducibility.
debhelper sets LC_ALL=C.UTF-8 for Meson/Ninja packages. debian/rules can
have `export LC_ALL=C.UTF-8`, if you want to (I would recommend this).
Even if you set the locale explicitly, locales other than C (aka
POSIX) and C.UTF-8 (aka C.utf8) are not guaranteed to be available
inside the test environment, unless your test has a dependency on
locales-all. Several source packages (for example gtk4) have a script
debian/tests/run-with-locales which generates additional locales
on-demand.
autopkgtest
-----------
The autopkgtest spec does not specify whether these variables will be set,
and if they are set, what their values will be.
Test scripts can set LC_ALL=C or LC_ALL=C.UTF-8 or individual locale
environment variables, if you want them to.
Even if you set the locale explicitly, locales other than C and C.UTF-8
are not guaranteed to be available inside the test environment, unless
your test has a dependency on locales-all. Again,
debian/tests/run-with-locales from the gtk4 package might be useful.
> and XDG_RUNTIME_DIR
sbuild
------
There is no guarantee about $XDG_RUNTIME_DIR while a Debian package is
being built. Policy §4.9 [0] indirectly implies that it is a bug for
the build process of a package to write into the real $XDG_RUNTIME_DIR.
sbuild does whatever schroot does. If I remember correctly, schroot
currently inherits XDG_RUNTIME_DIR from the host system, without doing
the setup to make it *work* inside the chroot. If true, that's a bug; but
schroot is orphaned in Debian and seems mostly dead upstream (its upstream
maintainer has left Debian, the most recent upstream release was in 2014,
and there was no response to a merge request that I sent a while ago) so
it is a bug that seems unlikely to be fixed unless someone in Debian forks
schroot. I personally think making sbuild default to --chroot-mode=unshare
or --chroot-mode=autopkgtest, and using one of those on official buildds,
would be a better way forward than forking and fixing schroot.
debhelper compat levels >= 13 set a temporary $XDG_RUNTIME_DIR for the
duration of `dh_auto_test`, and unset it at all other times, to make it
easier for packages to comply with the Policy §4.9 requirement.
autopkgtest
-----------
The autopkgtest spec does not specify whether this will be set or not.
In practice, on "real" systems, XDG_RUNTIME_DIR is only available if
libpam-systemd or libpam-elogind is installed and working. In practice
this means it will only be available in full-system containers, virtual
machines or physical machines (Requirements: isolation-container or
isolation-machine); only if they run a full init system like systemd-sysv
or sysvinit-core; and only if libpam-systemd or libpam-elogind is
installed and working (your test should have a Depends on libpam-systemd,
or similar, if you need this).
In simple chroot-based environments like schroot, or in non-full-system
containers like Docker/Podman, the infrastructure to set a correct
XDG_RUNTIME_DIR with its documented semantics[3] is not there, so ideally
it should be unset.
However, I think schroot currently inherits XDG_RUNTIME_DIR from the host
system (see sbuild, above), so you might get a wrong XDG_RUNTIME_DIR
for tests whose Restrictions allow them to be run in a simple chroot.
Test scripts can work around this by setting it to a temporary directory.
smcv
Reply to: