Bug#945269: debian-policy: packages should use tmpfiles.d(5) to create directories below /var
On Wed, 7 Jun 2023 at 11:46, Luca Boccassi <bluca@debian.org> wrote:
>
> On Wed, 7 Jun 2023 at 11:29, Simon McVittie <smcv@debian.org> wrote:
> >
> > On Tue, 06 Jun 2023 at 20:40:52 -0700, Russ Allbery wrote:
> > > Luca Boccassi <bluca@debian.org> writes:
> > > > +Packages might need additional files or directories to implement their
> > > > +functionality. Directories that are located under ``/var/`` or
> > > > +``/etc/``, and files that are located under ``/var/``, must not be
> > > > +created manually via maintainer scripts, but instead be declaratively
> > > > +defined via the `tmpfiles.d
> > > > +<https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html>`_
> > > > +interface.
> > >
> > > This is an oddly specific list of directories and not at all the
> > > directories that I would have expected to be handled by tmpfiles.d.
> >
> > Sorry, in my previous reading of this bug I had been concentrating on
> > the mechanics of how to make tmpfiles.d(5) something that maintainers can
> > rely on if it's convenient/helpful, and I'd missed that Luca is asking
> > for its use to be mandatory in some cases.
> >
> > I would personally be inclined to concentrate on making tmpfiles.d(5)
> > something that we can rely on and encourage the use of where appropriate,
> > even on non-systemd systems, so that (upstream and downstream) maintainers
> > can move towards it of their own accord because it's more convenient
> > than other options, and put aside the question of making it generally a
> > "should" or "must" for the moment.
> >
> > I believe (please correct me if I'm wrong) that Luca's intention here
> > is that this is drawing a line between:
> >
> > - the static files of the OS: /usr and the /usr-like top-level directories
> > (the ones that get merged by the /usr merge), which should be statically
> > shipped in packages and managed by dpkg (or on "immutable" systems
> > that use image-based/tree-based upgrades, maybe by ostree or casync
> > or similar, from an tree originally constructed from dpkg packages)
> >
> > - this specific system's persistent state: /var and parts of /etc
> >
> > with the intention of eventually enabling functionality like being
> > able to do a "factory reset" to the equivalent of a freshly installed
> > system by deleting (most of) /etc and /var, rebooting, and letting the
> > OS re-create them from a template below /usr; or doing the equivalent
> > for individual packages by deleting only their part of /etc and /var.
> >
> > /etc is somewhere between static files and state, because traditionally
> > it has been a mixture of files that the sysadmin or installer must provide
> > (like /etc/passwd); configuration files that are shipped by a package and
> > can be edited by the sysadmin (like /etc/systemd/logind.conf); and
> > integration glue that links up one package with another, can in principle
> > be edited by the sysadmin, but in practice is rarely edited
> > (like /etc/profile.d/flatpak.sh).
> >
> > Various upstream projects including systemd have been trying to reduce
> > the extent to which /etc and /var are included in the data.tar.* of a .deb
> > or other packaging systems' equivalents, by moving the integration glue
> > to a /usr-like directory (/lib/udev/rules.d, /usr/share/dbus-1/system.d),
> > reserving the corresponding /etc directory for sysadmin configuration
> > (/etc/udev/rules.d, /etc/dbus-1/system.d), and providing a way for the
> > sysadmin to "mask" any integration files they want the system to ignore.
> >
> > If we disregard conffiles and configuration files in /etc for the
> > moment, there are basically three ways for a package to get a file onto
> > the running system:
> >
> > - ship it in the data.tar.* of a .deb
> > - create it from a maintainer script and also during boot
> > - maybe via tmpfiles.d(5)
> > - or maybe open-coded
> > - have the package create it at runtime, on-demand
> > - this clearly doesn't work if the package's code runs unprivileged
> > and relies on root having created a directory for it already
> >
> > For /usr and the /usr-like directories, shipping files in the .deb is by
> > far the most common, although a few packages need to create files here
> > via maintainer scripts or triggers (for example
> > /usr/lib/x86_64-linux-gnu/gio/modules/giomodule.cache which is a summary
> > of files created by multiple packages, and is updated whenever those
> > packages are added, removed or changed).
> >
> > For /run, /tmp and /var/tmp, I think there's consensus that shipping files
> > in those directories in the .deb is a bug, because at the next reboot,
> > the file will be deleted, leaving the files that dpkg thinks it's managing
> > out of sync with the files that actually exist. At the moment, these are
> > variously created by maintainer scripts, systemd units/init scripts, or the
> > daemons themselves, with some duplication, and no good way to get an
> > overview of which packages "own" which locations: dpkg doesn't know anything
> > about them, and systemd knows about some but not all of them.
> >
> > tmpfiles.d seems like a good way to keep track of who "owns" those
> > transient files and directories. I think a Policy "must" is probably too
> > strong here, but a "should" might be reasonable?
> >
> > For the persistent parts of /var, several packages ship regular files
> > (as opposed to directories) in the .deb. I think there might be rough
> > consensus that doing so is at least a "code smell", but quite a lot of
> > packages do this, so a Policy "must" certainly seems too strong at this
> > stage. Legacy policy files for polkitd (<< 0.106) are a notable example.
> > They're no longer necessary with bookworm's polkitd, and I'm hoping to
> > get rid of them during the trixie cycle; but historically /var/lib was
> > the only place supported by polkitd other than /etc, so it was functionally
> > necessary to ship these files.
> >
> > Looking at my /var, the TeX family of packages seem to be the heaviest
> > users of regular files in /var, with /var/lib/tex-common/**/*.{cfg,cnf}.
> >
> > I think we can safely say that creating the top-level directory
> > of a package's /var, /run, etc. subdirectory declaratively is more
> > self-documenting than creating it imperatively, and tmpfiles.d seems
> > like a perfectly good way to achieve that.
> >
> > I don't think it's desirable to have wording that could be taken to imply
> > that packages would be wrong to create files inside those top-level
> > directories at runtime - that would seem silly (for example I don't
> > want anyone arguing that because the mariadb maintainer script starts
> > the server, it's somehow a bug for mariadb to create arbitrary database
> > tables below /var/lib/mysql).
> >
> > I also don't think Policy should forbid maintainer scripts using files
> > inside those tmpfiles-managed directories for their own state. For example,
> > maintainer scripts should be allowed to create files like
> > /var/lib/mysql/debian-10.11.flag and
> > /var/lib/systemd/deb-systemd-helper-enabled/*, even if we want to require
> > /var/lib/mysql and /var/lib/systemd/deb-systemd-helper-enabled to be
> > registered in tmpfiles.d(5) so that the sysadmin can discover where they
> > come from.
>
> Yes I agree, it was not my intention to forbid or discourage such
> cases, quite the opposite. What I wanted to say is: if you are
> creating directories/files by hand in a maintainer script, please use
> tmpfiles.d instead, or if your service/tool/code can do it instead,
> that's fine too. Among the goals that you correctly described above,
> there's also the desire to remove as much custom maintainer script
> code as possible, and that's what I was aiming at.
>
> > > Packages that need to create files or directories in file systems that
> > > may be deleted on each reboot (for example, ``/run``, ``/tmp``, and
> > > ``/var/tmp``) should do so via configuration files in the
> > > ``/usr/lib/tmpfiles.d`` directory. The syntax of those files is
> > > defined by the `systemd tmpfiles.d documentation
> > > <https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html>`__.
> > >
> > > However, reading that documentation, it sounds like most of the cases for
> > > other directories are handled by other systemd unit configuration
> > > directives. We should say that explicitly here and reproduce the list of
> > > other directories that should be handled directly by the unit file if
> > > that's what we want people to do.
> >
> > Hmm, yes. Which of these two policies do the systemd maintainers want?
> >
> > - If a service has RuntimeDirectory= etc. in its unit, then redundantly
> > registering those directories in tmpfiles.d(5) is not required unless
> > there is some technical reason to do so
> >
> > - If a service has RuntimeDirectory= etc. in its unit, then it must
> > redundantly register those same directories with tmpfiles.d(5)
> >
> > Reasons we might want the first of those: "don't repeat yourself", and
> > letting systemd create the directories as late as possible before starting
> > the service, and clean them up as early as possible after stopping it.
> >
> > Reasons we might want the second: if we had the first policy, a sysadmin
> > wanting to find out who "owns" /var/lib/mystery will need to check both
> > tmpfiles.d and systemd units; if we had the second, in principle they only
> > need to check tmpfiles.d. Also, non-systemd init systems won't read
> > RuntimeDirectory= etc., so if the directory is functionally required for
> > an LSB init script, it needs to be created some other way, and tmpfiles.d
> > is a reasonable choice for that other way.
>
> I would tend toward the first one, for the DynamicUser case - for
> clarity, this currently means RuntimeDir and friends are recursively
> chowned on the fly. I hope for Trixie we'll get uid mapping done
> instead and it will be moot, but it's not implemented yet so trying to
> be careful.
>
> Also, note that as mentioned earlier these settings work well when
> there is a clear and obvious owner, to whose lifecycle the directory
> can be tied to, but there are cases where there's either no service at
> all or no clear owner.
New revision attached, reworded a bunch of stuff following
suggestions, and also I've added a section specifically about the
service directories settings.
From a9437eef805b24bd26fbc95e01c52cb19b8ef8ba Mon Sep 17 00:00:00 2001
From: Luca Boccassi <bluca@debian.org>
Date: Tue, 9 May 2023 01:38:13 +0100
Subject: [PATCH] Define service directories and tmpfiles.d interfaces and
usage
---
policy/ch-files.rst | 64 +++++++++++++++++++++++++++++++++
policy/ch-maintainerscripts.rst | 7 ++++
2 files changed, 71 insertions(+)
diff --git a/policy/ch-files.rst b/policy/ch-files.rst
index b34c183..073677d 100644
--- a/policy/ch-files.rst
+++ b/policy/ch-files.rst
@@ -722,6 +722,70 @@ The name of the files and directories installed by binary packages
outside the system PATH must be encoded in UTF-8 and should be
restricted to ASCII when it is possible to do so.
+.. _s-service-directories:
+
+Service Directories
+-------------------
+
+Services might need auxiliary directories under ``/var/``, ``/run/`` or
+``/etc``. Instead of shipping empty directories in packages, or creating them
+with custom code in maintainer scripts, services should use ``systemd``'s native
+settings to ensure the required directories are created regardless of the
+privilege level under which the services are running. The relevant settings are
+`RuntimeDirectory=, StateDirectory=, CacheDirectory=, LogsDirectory= and
+ConfigurationDirectory=
+<https://freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=>`_,
+covering respectively ``/run/``, ``/var/lib/``, ``/var/cache/``, ``/var/log/``
+and ``/etc/`` for system services, and the equivalent XDG-defined location for
+user services.
+
+Init systems other than ``systemd`` should allow providing the same
+functionality as appropriate for each system, for example managing the
+directories from the init script shipped by the package.
+
+.. _s-tmpfiles.d:
+
+tmpfiles.d
+----------
+
+Packages might need additional files or directories to implement their
+functionality. Directories that are located under ``/var/`` or ``/etc/``, and
+files that are located under ``/var/``, should not be created manually via
+maintainer scripts, but instead be declaratively defined via the `tmpfiles.d
+<https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html>`_ interface.
+Files and directories under ephemeral filesystems such as ``/tmp/`` may also be
+created and managed via ``tmpfiles.d`` snippets.
+
+When ownership of a directory can be clearly tied to a specific service,
+``Service Directories`` should be preferred to ``tmpfiles.d`` snippets.
+
+The ``tmpfiles.d`` file format is defined by the ``systemd`` project, and is
+guaranteed to be stable. Details about the syntax and installation paths are
+defined by its `reference implementation's documentation,
+<https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html>`_ and will
+not be redefined here.
+
+``tmpfiles.d`` snippets should be usable on systems that do not boot (such as a
+very minimal chroot image), and also on systems booting with init systems other
+than ``systemd``.
+
+``tmpfiles.d`` snippets should be integrated in packages using automated shared
+tooling rather than by manually writing package-specific code in maintainers
+scripts. For example, packages built using ``debhelper`` should make use of the
+``dh_installtmpfiles`` addon. Packages shipping ``tmpfiles.d`` snippets should
+depend on the appropriate virtual packages in the following order:
+``default-systemd-tmpfiles | systemd-tmpfiles``.
+
+Init systems are required to integrate with ``tmpfiles.d`` and run the service
+that applies them on boot, and regularly for cleanup purposes, depending on the
+appropriate package providing the appropriate implementation that best
+integrates with each system. For example, ``systemd`` will make the reference
+implementation available when its main package is installed. The documentation
+for the reference implementation, `systemd-tmpfiles,
+<https://www.freedesktop.org/software/systemd/man/systemd-tmpfiles.html>`_
+explains how to call the program so that the appropriate ``tmpfiles.d`` snippets
+are applied at the appropriate time.
+
.. [#]
If you are using GCC, ``-fPIC`` produces code with relocatable
position independent code, which is required for most architectures
diff --git a/policy/ch-maintainerscripts.rst b/policy/ch-maintainerscripts.rst
index 724074c..3734deb 100644
--- a/policy/ch-maintainerscripts.rst
+++ b/policy/ch-maintainerscripts.rst
@@ -50,6 +50,13 @@ absolute pathname. Maintainer scripts should also not reset the
appending package-specific directories. These considerations really
apply to all shell scripts.
+Maintainer scripts should not be used to create or remove auxiliary files and/or
+directories that packages may need, such as those in ``/var/`` or ``/etc/``.
+Instead, :ref:`s-tmpfiles.d` snippets should be shipped, with the ones provided
+by the upstream sources, if any, to be preferred over Debian-specific ones when
+possible. For more details about the ``tmpfiles.d`` interface, see
+:ref:`s-tmpfiles.d`.
+
.. _s-idempotency:
Maintainer scripts idempotency
--
2.39.2
Reply to: