Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)
On Sat, 17 Jul 2021 at 20:34:41 +0000, Johannes Drexl wrote:
> /usr is allowed to be not only on a separate partition, but even on a
> network device
This has been discussed at considerable length before, but I'll try to
recap:
A separate or network-mounted /usr is possible in Debian, whether
merged-/usr is used or not, but only if /usr is already mounted
by the time the system hands over from the initramfs (usually generated by
initramfs-tools, or sometimes dracut or another alternative) to the main
system (which starts systemd, sysvinit or some other init system as the
main pid 1 and proceeds from there). Merged /usr is actually better for
this configuration than unmerged /usr in some ways: it lets you share
*all* the non-modifiable files between machines, not just the ones that
are in /usr (as opposed to /lib, /bin, /sbin).
This is not a new requirement: Debian >= 9 (2017) officially did not support
mounting /usr halfway through the main system boot process. Older Debian
releases aimed to support /usr being mounted during boot (for example
during the rcS sequence in sysv-rc) but it frequently didn't actually
work in practice, particularly if combined with network mounts.
The reason why mounting /usr during main-system boot is no longer
supported is that it often didn't work. A major reason why it often didn't
work is that depending on how the system is configured, mounting /usr can
require networking; but bringing up networking can require arbitrarily
many programs and libraries (networking infrastructure like ifupdown or
NetworkManager often has either a plugin architecture, or an arbitrary
hooks mechanism that executes programs of the sysadmin's choice that
will often have dependencies in /usr, or both).
Similarly, udevd needs to run early in the main system boot to bring up
/dev, but udev rules can run arbitrary programs, some of them in /usr;
and for the main system, those programs often need data from /usr/share.
More generally, if we took every library and every program that could
conceivably be a dependency of /usr and moved them from /usr into the
root filesystem, then the root filesystem would become increasingly large
over time, negating any benefit that you hoped to gain by mounting /usr
separately. Over time, this approach would tend towards the layout that
the Debian Hurd port briefly tried to use, which was the opposite of
merged-/usr (you could call it "merged rootfs"), with /usr as either a
symlink to /, or containing only symlinks to /lib, /bin and so on.
Instead, we require everything that is needed in *your* configuration
(not necessarily in *anyone's* configuration, just yours!) to be part
of the initramfs, which is generated per-machine.
Effectively, the requirement to mount /usr before pivoting from initramfs
to main system means that instead of a small rootfs (which can be used for
recovery) and a larger /usr, we have a small initramfs (which can be used
for recovery) and a large main system.
One key advantage of this is that decisions about what to include in
the rootfs (of non-merged-/usr systems) have to be made globally for all
of Debian, but decisions about what to include in the initramfs can be
made per-machine, allowing some otherwise impossible situations to be
resolved. If your /usr is mounted via NFS, but bringing up my network
requires /usr (perhaps for a VPN), in the old model with a small rootfs
and a larger /usr it was impossible for us both to have what we needed:
we could not bring up networking both before /usr (as you would have
needed) and after /usr (as I would have needed).
> /usr is allowed to be [...] on a network device shared by other machines
The FHS may allow this, but it has significant practical problems,
unrelated to merged-/usr, on machines that receive security/stable updates
(which I would hope by now should mean all machines). Updates to Debian
packages can touch both mutable per-machine files (/etc, /var) and
immutable/shareable per-(package,version) files (/usr, /lib*, /bin, /sbin).
If a machine's /usr is not in sync with its /etc and /var, then it is likely
to work incorrectly: at a minimum, asking dpkg which packages and versions
are installed will give you an answer that does not match what is actually
in /usr.
On non-merged-/usr systems, the machine's /usr and {/lib*,/bin,/sbin}
must also be kept in sync: for example, there is no guarantee that
/usr/bin/dbus-daemon (in /usr) will work correctly with a mismatched
version of /lib/x86_64-linux-gnu/libdbus-1.so.3 (on the rootfs of a
non-merged-/usr system). On merged-/usr systems, it is impossible for
those directories to become out-of-sync, so this particular requirement
becomes trivial to achieve.
A more robust approach to sharing /usr between machines (or containers)
might be to give each machine (or container) its own private /usr or even
its own private root filesystem, and then carry out file- or block-level
deduplication on the storage backend (for example, if /usr is NFS-shared
and is stored on btrfs on the NFS server, then reflinks could be used to
make files with identical content share storage).
If the FHS allows something, that also does not mean "all FHS-compliant
operating systems must allow this"; it just means "programs designed to
run on FHS-compliant operating systems must not assume this can't happen".
For example, the FHS allows /run and /var/run to be separate, but on
Debian they are always synonymous (because making them distinct would
only cause problems, without providing any benefit). If Debian did not
support a network-mounted /usr, then asking for network-mounted /usr to be
supported would be a reasonable feature request, but the lack of that
feature would not make Debian any less FHS-compliant.
smcv
Reply to: