On Tue, 06 Jun 2023 at 20:40:52 -0700, Russ Allbery wrote:
> Luca Boccassi <bluca@debian.org> writes:
> > +Packages might need additional files or directories to implement their
> > +functionality. Directories that are located under ``/var/`` or
> > +``/etc/``, and files that are located under ``/var/``, must not be
> > +created manually via maintainer scripts, but instead be declaratively
> > +defined via the `tmpfiles.d
> > +<https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html>`_
> > +interface.
>
> This is an oddly specific list of directories and not at all the
> directories that I would have expected to be handled by tmpfiles.d.
Sorry, in my previous reading of this bug I had been concentrating on
the mechanics of how to make tmpfiles.d(5) something that maintainers can
rely on if it's convenient/helpful, and I'd missed that Luca is asking
for its use to be mandatory in some cases.
I would personally be inclined to concentrate on making tmpfiles.d(5)
something that we can rely on and encourage the use of where appropriate,
even on non-systemd systems, so that (upstream and downstream) maintainers
can move towards it of their own accord because it's more convenient
than other options, and put aside the question of making it generally a
"should" or "must" for the moment.
I believe (please correct me if I'm wrong) that Luca's intention here
is that this is drawing a line between:
- the static files of the OS: /usr and the /usr-like top-level directories
(the ones that get merged by the /usr merge), which should be statically
shipped in packages and managed by dpkg (or on "immutable" systems
that use image-based/tree-based upgrades, maybe by ostree or casync
or similar, from an tree originally constructed from dpkg packages)
- this specific system's persistent state: /var and parts of /etc
with the intention of eventually enabling functionality like being
able to do a "factory reset" to the equivalent of a freshly installed
system by deleting (most of) /etc and /var, rebooting, and letting the
OS re-create them from a template below /usr; or doing the equivalent
for individual packages by deleting only their part of /etc and /var.
/etc is somewhere between static files and state, because traditionally
it has been a mixture of files that the sysadmin or installer must provide
(like /etc/passwd); configuration files that are shipped by a package and
can be edited by the sysadmin (like /etc/systemd/logind.conf); and
integration glue that links up one package with another, can in principle
be edited by the sysadmin, but in practice is rarely edited
(like /etc/profile.d/flatpak.sh).
Various upstream projects including systemd have been trying to reduce
the extent to which /etc and /var are included in the data.tar.* of a .deb
or other packaging systems' equivalents, by moving the integration glue
to a /usr-like directory (/lib/udev/rules.d, /usr/share/dbus-1/system.d),
reserving the corresponding /etc directory for sysadmin configuration
(/etc/udev/rules.d, /etc/dbus-1/system.d), and providing a way for the
sysadmin to "mask" any integration files they want the system to ignore.
If we disregard conffiles and configuration files in /etc for the
moment, there are basically three ways for a package to get a file onto
the running system:
- ship it in the data.tar.* of a .deb
- create it from a maintainer script and also during boot
- maybe via tmpfiles.d(5)
- or maybe open-coded
- have the package create it at runtime, on-demand
- this clearly doesn't work if the package's code runs unprivileged
and relies on root having created a directory for it already
For /usr and the /usr-like directories, shipping files in the .deb is by
far the most common, although a few packages need to create files here
via maintainer scripts or triggers (for example
/usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache which is a summary
of files created by multiple packages, and is updated whenever those
packages are added, removed or changed).
For /run, /tmp and /var/tmp, I think there's consensus that shipping files
in those directories in the .deb is a bug, because at the next reboot,
the file will be deleted, leaving the files that dpkg thinks it's managing
out of sync with the files that actually exist. At the moment, these are
variously created by maintainer scripts, systemd units/init scripts, or the
daemons themselves, with some duplication, and no good way to get an
overview of which packages "own" which locations: dpkg doesn't know anything
about them, and systemd knows about some but not all of them.
tmpfiles.d seems like a good way to keep track of who "owns" those
transient files and directories. I think a Policy "must" is probably too
strong here, but a "should" might be reasonable?
For the persistent parts of /var, several packages ship regular files
(as opposed to directories) in the .deb. I think there might be rough
consensus that doing so is at least a "code smell", but quite a lot of
packages do this, so a Policy "must" certainly seems too strong at this
stage. Legacy policy files for polkitd (<< 0.106) are a notable example.
They're no longer necessary with bookworm's polkitd, and I'm hoping to
get rid of them during the trixie cycle; but historically /var/lib was
the only place supported by polkitd other than /etc, so it was functionally
necessary to ship these files.
Looking at my /var, the TeX family of packages seem to be the heaviest
users of regular files in /var, with /var/lib/tex-common/**/*.{cfg,cnf}.
I think we can safely say that creating the top-level directory
of a package's /var, /run, etc. subdirectory declaratively is more
self-documenting than creating it imperatively, and tmpfiles.d seems
like a perfectly good way to achieve that.
I don't think it's desirable to have wording that could be taken to imply
that packages would be wrong to create files inside those top-level
directories at runtime - that would seem silly (for example I don't
want anyone arguing that because the mariadb maintainer script starts
the server, it's somehow a bug for mariadb to create arbitrary database
tables below /var/lib/mysql).
I also don't think Policy should forbid maintainer scripts using files
inside those tmpfiles-managed directories for their own state. For example,
maintainer scripts should be allowed to create files like
/var/lib/mysql/debian-10.11.flag and
/var/lib/systemd/deb-systemd-helper-enabled/*, even if we want to require
/var/lib/mysql and /var/lib/systemd/deb-systemd-helper-enabled to be
registered in tmpfiles.d(5) so that the sysadmin can discover where they
come from.
> Packages that need to create files or directories in file systems that
> may be deleted on each reboot (for example, ``/run``, ``/tmp``, and
> ``/var/tmp``) should do so via configuration files in the
> ``/usr/lib/tmpfiles.d`` directory. The syntax of those files is
> defined by the `systemd tmpfiles.d documentation
> <https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html>`__.
>
> However, reading that documentation, it sounds like most of the cases for
> other directories are handled by other systemd unit configuration
> directives. We should say that explicitly here and reproduce the list of
> other directories that should be handled directly by the unit file if
> that's what we want people to do.
Hmm, yes. Which of these two policies do the systemd maintainers want?
- If a service has RuntimeDirectory= etc. in its unit, then redundantly
registering those directories in tmpfiles.d(5) is not required unless
there is some technical reason to do so
- If a service has RuntimeDirectory= etc. in its unit, then it must
redundantly register those same directories with tmpfiles.d(5)
Reasons we might want the first of those: "don't repeat yourself", and
letting systemd create the directories as late as possible before starting
the service, and clean them up as early as possible after stopping it.
Reasons we might want the second: if we had the first policy, a sysadmin
wanting to find out who "owns" /var/lib/mystery will need to check both
tmpfiles.d and systemd units; if we had the second, in principle they only
need to check tmpfiles.d. Also, non-systemd init systems won't read
RuntimeDirectory= etc., so if the directory is functionally required for
an LSB init script, it needs to be created some other way, and tmpfiles.d
is a reasonable choice for that other way.
> What's the plan for creating directories in /run on non-systemd systems?
> I had assumed that tmpfiles.d would be used for those directories so that
> we can use the same mechanism for both systemd and non-systemd systems,
> but it looks like that's not ideal for systemd systems. Is it safe to use
> *both* (e.g.) RuntimeDirectory= *and* tmpfiles.d for the same directory so
> that tmpfiles.d is used for non-systemd systems?
A good question. I would hope that systemd's quality-of-implementation is
sufficient that this is safe for maintainers to do, if there's some reason
why it's useful to do so.
> > +``tmpfiles.d`` snippets should be detected at package build time by
> > +tools such as ``debhelper``, packaged, and the appropriate snippet to
> > +call them on installation, upgrade, removal, purge and other steps as
> > +required, should be automatically added by helpers such as
> > +``dh_installtmpfiles``.
>
> Also, isn't the most obvious way to implement this to use triggers?
Many packages that want to create directories with tmpfiles.d will
want to start a service (systemd unit or init script), and the service
is quite likely to rely on the tmpfiles.d having been run *first* -
"noawait" triggers are certainly not going to be enough. Are "await"
triggers sufficient to make this happen?
If we are going to do the equivalent thing with declarative system user
creation via sysusers.d, then the sequence we will need in general is:
1. sysusers.d to create system user _foo
2. tmpfiles.d to create /var/lib/foo owned by _foo
3. systemd units/init scripts that run a daemon as uid _foo and require
/var/lib/foo to exist already
in exactly that order. Can triggers guarantee that?
> Can't we just say
> specifically what init systems need to do? I assume it's something like
> "call systemd-tmpfiles --create --boot on boot and systemd-tmpfiles
> --clean periodically on some schedule" (what schedule?).
I think the --create side is the key thing here, and --clean can be
left to quality-of-implementation: it's common for services to require
a directory to exist and have specified ownership, and (I suspect) much
less common for them to require it to have old files cleaned up in a
timely fashion.
smcv