Hello,
On Wed 15 Sep 2021 at 11:46AM +01, Simon McVittie wrote:
> As discussed in our last meeting, I think we should issue more specific
> advice about merged-/usr, and in particular about what #978636 means for
> maintainers right now.
With five votes cast in favour of the text by Simon, Niko, Gunnar, Marga
and myself, the outcome is no longer in doubt. Thus, in accordance with
the Debian Constitution, the voting period has now concluded.
Therefore, using its powers under section 6.1.5 of the Debian
Constitution, the Technical Committee issues the following advice:
Summary
=======
There are currently Debian 11 installations with both the merged-/usr
and non-merged-/usr filesystem layouts. All of these installations
should successfully upgrade via normal upgrade paths to a merged-/usr
Debian 12. Only after the release of Debian 12 can packages assume that
all installations will be merged-/usr.
Main points:
- We have recommended merged-/usr for Debian 12.
- Moving individual files is not merged-/usr.
- "Symlink farms" are not merged-/usr.
- Upgrading a non-merged-/usr system to Debian 12 needs to work.
- Upgrading a merged-/usr system to Debian 12 needs to work.
- Packages cannot assume all systems are merged-/usr until the Debian 13
development cycle begins.
- Upgrading via apt in the usual way should work.
- Testing and QA systems should be able to avoid this transition, but if
they do, they cannot be upgraded beyond Debian 12.
- A package building incorrectly on a non-merged-/usr system is a bug.
- A package building incorrectly on a merged-/usr system is a bug.
- Please stop moving individual packages' files from the root filesystem
into /usr, at least for now.
Definitions and current status
==============================
libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
such as lib64 on the amd64 architecture.
Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
/libQUAL that exists are symbolic links to the corresponding directories
below /usr. This results in aliasing between /bin and /usr/bin, and so
on.
In the merged-/usr layout, files whose canonical logical location is in
one of the affected directories on the root filesystem, such as
/bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
the corresponding path below /usr, such as /usr/bin/bash. Each file in
one of the affected directories is accessible via two paths: its
canonical logical location (such as /bin/bash or /usr/bin/env), and the
other path implied by the aliasing (such as /usr/bin/bash or /bin/env).
There are two supported categories of Debian 11 installation, which are
currently considered equally-supported:
- Merged-/usr installations. These were installed with debian-installer
from Debian 10 or later, or installed with debootstrap --merged-usr,
or converted from the non-merged-/usr layout by installing the
usrmerge package, or installed or converted by any similar
procedure. They have the merged-/usr layout.
- Non-merged-/usr installations. These were installed with
debian-installer from Debian 9 or earlier and subsequently upgraded
without converting to merged-/usr, or installed with debootstrap
--no-merged-usr, or converted from the merged-/usr layout with dpkg's
"dpkg-fsys-usrunmess" utility or any similar procedure. They have the
traditional, non-merged-/usr layout in which /bin/bash and
/usr/bin/env have exactly those physical paths, and /usr/bin/bash and
/bin/env do not exist.
Merged-/usr is not the only filesystem layout that has been proposed for
unifying the root filesystem with /usr. For avoidance of doubt, we do
not consider other filesystem layouts to be implementations of
merged-/usr. In particular, we do not consider these to be
implementations of merged-/usr:
- all affected files physically located in /bin, /sbin, /lib and
/libQUAL, with /usr/bin as a symlink to /bin, etc. (this is the
reverse of merged-/usr, and was historically used in the hurd-i386
port)
- a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with
individual symbolic links such as /bin/bash -> /usr/bin/bash for only
those files that historically had their canonical logical location on
the root filesystem
- a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with
individual symbolic links such as /bin/env -> /usr/bin/env for all
files in the affected directories, regardless of whether they
historically had their canonical logical location on the root
filesystem
[1]: https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
Upgrade path from Debian 11 to Debian 12
========================================
The technical committee resolved in #978636 [2] that Debian 12
'bookworm' should only support the merged-/usr layout. This resolution
describes the implications of that decision in more detail.
Debian installations have traditionally had a straightforward upgrade
path between consecutive releases, and the technical committee expects
maintainers to continue this. In the case of #978636, the upgrades we
are interested in are:
- Upgrading from Debian 11 (stable release) to Debian 12 (stable release)
- Upgrading from Debian 11 (stable release) to testing/unstable during
the development cycle that will lead to Debian 12, and subsequently
upgrading from testing/unstable to Debian 12 (stable release)
What we understand this to imply is as follows:
- Because Debian 11 installations with the merged-/usr layout already
exist, and because Debian 12 should only support the merged-/usr
layout, all packages in Debian 12 should be installable onto a
merged-/usr system along with their dependencies, and work correctly
on the resulting system.
- Because Debian 11 installations with the non-merged-/usr layout
already exist, all packages in Debian 12 should be installable onto a
non-merged-/usr system along with their dependencies, and work
correctly on the resulting system.
+ The key reason for this is that apt is not required to perform the
upgrade between stable releases in any particular order, so long
as package dependency relationships are respected; therefore the
upgrade can happen in whatever order apt chooses, which can vary
between machines. Debian has not traditionally required use of a
special upgrade tool similar to Ubuntu's do-release-upgrade(8) and
we believe the upgrade to Debian 12 should be no different (see
below for more details on this topic).
+ Another reason for this is that during the development of Debian
12, testing/unstable systems undergo a series of partial upgrades,
which similarly will happen in an undefined order.
+ We do not require that the resulting system remains
non-merged-/usr: if the packages involved in this installation
transaction are part of the implementation of a transition to
merged-/usr, then installing them might result in the system
becoming merged-/usr.
- The same expectations apply to packages uploaded to testing/unstable
during the development cycle that will lead to Debian 12.
- Debian contributors who are interested in merged-/usr are invited to
implement a straightforward migration path from non-merged-/usr to
merged-/usr. The Technical Committee will not design this migration
path in detail. If disputes arise related to this migration path, or
if advice on this migration path is requested, we will resolve those
by following our usual procedures.
+ One example of a migration path that might be used is for an
Essential package to add a dependency on the usrmerge package, so
that it will be installed automatically during upgrades. We do not
require this to be the migration path that is chosen; it is
presented here merely to demonstrate that such a migration path
can exist.
- After Debian 12 is released, the transition to merged-/usr is
considered to be complete. Because we do not support "skipping a
release" when upgrading systems, new packages uploaded to
testing/unstable during the development cycle that will lead to Debian
13 may assume and require that the system onto which they are
installed is merged-/usr.
If a suitable transition mechanism is not available by the time the
Debian 12 freeze is reached, the Technical Committee will rescind our
advice in #978636 and modify our advice in this resolution to reflect
the situation that has been reached. We hope that this will not be
necessary.
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978636
Use of a special upgrade path
=============================
Some developers have argued that we should deploy merged /usr by means
of a special upgrade path, with all other upgrade paths being declared
to be unsupported. Some examples of special upgrade paths that might be
proposed:
- upgrading dpkg before carrying out the rest of the upgrade
- installing the usrmerge package before carrying out the upgrade
- using a special tool similar to Ubuntu's do-release-upgrade(8) which
encapsulates whatever steps are necessary
This would imply that upgrade paths other than the recommended one are
not expected to work.
We are aware that many Debian users do not read the release notes before
upgrading from oldstable to stable, and will expect to be able to
upgrade via whatever apt commands they are used to using, regardless of
whether the release notes recommend something different. Given that
there are alternatives available, we do not think that merged-/usr is
sufficiently good cause to interfere with this.
During the Debian 12 development cycle, users of testing/unstable will
also need an upgrade path from Debian 11 to testing/unstable, or from
older to newer snapshots of testing/unstable. In general this will be
done using apt, and it needs to continue to work if at all possible,
even before a special upgrade tool has been prepared; introducing a
"flag day" at which all testing/unstable users are expected to carry out
additional non-automatic upgrade steps would be disruptive, and in
practice many testing/unstable users are likely to skip those steps.
For these reasons, we make the simple, conservative recommendation that
a special upgrade path should not be required, and upgrading via apt in
approximately the same way as previous Debian releases should continue
to work in general.
Testing and QA
==============
We anticipate that during the development cycle that will lead to Debian
12, it is likely to be useful for testing and QA infrastructure (such as
autopkgtest, piuparts and/or reproducible-builds) to be able to produce
an installation of Debian testing/unstable that is not merged-/usr, in
order to be able to verify that packages targeted for inclusion in
Debian 12 can still be installed and/or built successfully in a
non-merged-/usr environment during partial upgrades.
As a result, we recommend that if there is an automatic migration from
non-merged-/usr to merged-/usr, it should be possible to prevent that
migration. However, systems where that migration has been prevented are
not required to be fully-supported, and in particular, upgrading them to
Debian 13 or to the state of testing/unstable during development of
Debian 13 should be considered unsupported.
Building packages
=================
In #914897 [3] the Technical Committee resolved that for Debian 11,
packages can validly be built in either a merged-/usr or non-merged-/usr
environment. We understand this to imply that packages built in either
a merged-/usr or non-merged-/usr environment should work as intended in
either a merged-/usr or non-merged-/usr environment.
There is a class of bugs in which a package embeds the absolute path to
some executable or other file, in such a way that if it is built in an
environment with a unified /usr (either via merged-/usr or symlink
farms), this can result in embedding a non-canonical path such as
/usr/bin/sh or /bin/env which will not work correctly in a
non-merged-/usr environment. The Technical Committee reminds
maintainers that packages in Debian 12 are expected to work correctly in
a non-merged-/usr environment, and therefore this class of bugs needs to
be resolved. As a result, we recommend that these bugs should generally
be treated as release-critical for Debian 12 by maintainers and the
release team.
The Reproducible Builds effort tracks this class of bugs as
"paths_vary_due_to_usrmerge" [4] and many bugs in this class are tracked
with the usertag "usrmerge" [5]. In Autotools-based build systems, they
can often be avoided by passing arguments such as `SED=/bin/sed` to the
configure script.
Note that although the name of the usertag mentions usrmerge, this class
of bugs is not specific to the usrmerge package, and not completely
specific to merged-/usr; many of these bugs would also manifest if
unified /usr was achieved via symlink farms, or if a local executable
such as /usr/local/bin/sed existed on the build system.
#993675 [6] is a typical example of this class of bugs.
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914897
[4]:
https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
[5]:
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org&tag=usrmerge
[6]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993675
Moratorium on moving files' logical locations into /usr
=======================================================
In the past, some package maintainers have taken advantage of the fact
that /usr is now required to be mounted during early boot (since Debian
9) to move individual files from the root filesystem to /usr. Examples
of this being done during previous development cycles include the
libglib-2.0.so.0 and libgcrypt.so.20 shared libraries, and the
command-line utilities in the iptables package.
The Technical Committee recommends that during the Debian 12 development
cycle, the maintainers of individual packages should not proactively
move files from the root filesystem to the corresponding locations in
/usr in the data.tar.* of packages. Files that were in /usr in the
Debian 11 release should remain in /usr, while files that were in /bin,
/lib* or /sbin in the Debian 11 release should remain in those
directories. If files were moved from /bin, /lib* or /sbin into /usr
since the Debian 11 release, they should be moved back to their Debian
11 locations.
Similarly, during the Debian 12 development cycle, we recommend that
maintainers of tools such as debhelper should not move files from the
root filesystem into /usr.
For example, debhelper 13.4 started moving systemd units from
/lib/systemd into /usr/lib/systemd, but this was subsequently reverted
in 13.5.2. We consider the revert that was done in 13.5.2 to have been
an appropriate course of action.
We issue this recommendation for several reasons:
- The transition to merged-/usr will make the logical locations of these
files irrelevant to their physical locations.
- The transitional mechanisms necessary to prevent such moves from
breaking other packages that hard-code specific paths are error-prone,
and can themselves interfere with the transition to merged-/usr.
- After the transition to merged-/usr has completed, during the Debian
13 development cycle, we expect that maintainers will be able to move
these files without needing transitional mechanisms.
- On merged-/usr systems, there is a possible failure mode involving
files being moved between packages (with Replaces) during the same
release cycle that their logical location is changed from the root
filesystem to the corresponding aliased directory in /usr, which can
result in the affected file disappearing. This can be avoided by not
changing the file's logical location until the beginning of the Debian
13 development cycle, after the transition to merged-/usr is complete.
- On non-merged-/usr systems, a failure mode has been observed in which
older shared libraries in /lib/MULTIARCH are not always deleted as
intended, and interfere with correct loading of the newer shared
library in /usr/lib/MULTIARCH. This can be avoided by not changing the
file's logical location until the beginning of the Debian 13
development cycle, after the transition to merged-/usr is complete, at
which point ldconfig(8) will choose the newer library even if this
occurs.
This recommendation applies until Debian 12 is released, or until a
subsequent Technical Committee resolution rescinds it, whichever is
sooner. We intend to rescind this recommendation if mechanisms are
developed to avoid the undesired side-effects of moving files from the
root filesystem into /usr.
For the Technical Committee:
--
Sean Whitton
Attachment:
signature.asc
Description: PGP signature