Re: autopkgtest/sbuild environment variables: LC_ALL, HOME, XDG_RUNTIME_DIR etc
Thanks Paul and Simon for your very thoughtful and helpful replies!
Comments on excerpted text below.
On Thu, Apr 28, 2022 at 10:16:30AM +0100, Simon McVittie wrote:
> On Thu, 28 Apr 2022 at 08:33:02 +0100, Julian Gilbey wrote:
> > When I run autopkgtest using sbuild on my own machine, I sometimes get
> > warnings or breakages because certain environment variables are either
> > unset or have "broken" values, whereas I see that on ci.debian.net and
> > the buildds, the builds run without problem.
>
> What autopkgtest backend are you using? (autopkgtest-virt-schroot?
> autopkgtest-virt-qemu? ...) There are several, and their behaviour can be
> very different, depending on the level of isolation they provide.
I had been using the schroot backend. I hadn't realised that schroot
is abandoned upstream, though it seems to work quite nicely. I
haven't looked at lxc or qemu, though they sound like interesting
alternatives. (Happy to receive recommendations for what would be
simplest for an individual developer's machine!)
> Having a concrete example of the command-line and configuration you use,
> and a package that has warnings or breakage in this configuration, would
> be useful.
I was trying to apply a patch to pymca to fix bug#1010258; see version
5.7.1+dfsg-2 of the package.
When building the package with sbuild, using the command line:
gbp buildpackage --git-builder=sbuild -d unstable -A --source-only-changes
the tests during package build time were fine, but the autopkgtest
failed. I fixed the autopkgtest by specifying:
LC_ALL=C HOME="$AUTOPKGTEST_TMP" xvfb-run ...
This makes perfect sense given what you wrote later.
> ci.debian.net currently uses autopkgtest-virt-lxc via the debci package.
Ah, that would explain at least some of the difference with what I am
seeing.
> > HOME (pointing to where?)
>
> sbuild
> ------
>
> There is no guarantee about $HOME while a Debian package is being built.
> Policy §4.9 [0] says it is a bug for the build process of a package to
> write into the real $HOME.
It may well be that these autopkgtests don't write to home but attempt
to access it to read from it. But I don't know.
> debhelper compat levels >= 13 set a temporary $HOME for the duration
> of the build, to make it easier for packages to comply with the Policy
> §4.9 requirement.
That would explain why the package build part worked smoothly.
> autopkgtest
> -----------
>
> According to the autopkgtest spec[1],
>
> Tests can expect that the $HOME environment variable to be set to
> a directory that exists and is writeable by the user running the test.
That's interesting; I don't find any code that sets HOME in
autopkgtest. And I just found this bug report, which points out that
this doesn't actually happen:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986665
Now I understand that better, it seems that changing the autopkgtest
spec to say something more like the following, if it's true:
The HOME environment variable is inherited from the calling
process. It is up to the container manager to ensure that this
directory exists and is writeable by the user running the test.
> Depending on backend, that directory might be your real home directory,
> or it might be a transient/expendable home directory that it is OK for
> the test to damage. It would probably be good to have a restriction[2]
> that has the effect of guaranteeing that an expendable home directory
> is used, as it is in autopkgtest-virt-qemu.
That sounds like a really good idea!
> > LC_ALL (or something similar; what is its setting?)
>
> LC_ALL is not normally set on "real" Debian systems. Usually, only $LANG
> and $LANGUAGE are set, even on a full desktop system (for example my GNOME
> desktop has LANG=en_GB.utf8 and LANGUAGE=en_GB:en, but no LC_ALL).
>
> LC_ALL is intended to be the "big hammer" that overrides everything else.
Yes, indeed. But for some reason, pymca tries to set the locale at
some point in the tests. Setting LC_ALL=C, though it is a big hammer,
certainly helped! I haven't tried using a more restricted thing like
LANG.
> > and XDG_RUNTIME_DIR
>
> [...]
>
> sbuild does whatever schroot does. If I remember correctly, schroot
> currently inherits XDG_RUNTIME_DIR from the host system, [...]
It doesn't appear to do so (at least not by default).
> debhelper compat levels >= 13 set a temporary $XDG_RUNTIME_DIR for the
> duration of `dh_auto_test`, and unset it at all other times, to make it
> easier for packages to comply with the Policy §4.9 requirement.
Ah, that would explain why the tests work during build time but give
warning messages with autopkgtest! (Incidentally, given that
typically XDG_RUNTIME_DIR is in the ephemeral /run filesystem, writing
to it would not violate policy 4.9.) I wonder whether autopkgtest
could consider setting up a temporary directory for this purpose like
dh_auto_test does, rather than using the heavy machinery of
libpam-systemd?
So at the end of it all, it seems that setting up a temporary home
directory for HOME and runtime directory for XDG_RUNTIME_DIR in
autopkgtests that rely on these is the right way to go.
Best wishes,
Julian
Reply to: