[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#994388: tech-ctte: More specific advice regarding merged-/usr and implications of #978636



> I'm calling for votes on the following resolution as formal advice from
> the Technical Committee (Constitution §6.1.5). There are no non-accepted
> amendments, so the options to vote on are "yes" or "further discussion".

My vote on the text quoted below:
  yes > further discussion

> 
> ~~~~ begin text to be voted on ~~~~
> 
> Summary
> =======
> 
> There are currently Debian 11 installations with both the merged-/usr and
> non-merged-/usr filesystem layouts. All of these installations should
> successfully upgrade via normal upgrade paths to a merged-/usr Debian 12.
> Only after the release of Debian 12 can packages assume that all
> installations will be merged-/usr.
> 
> Main points:
> 
> - We have recommended merged-/usr for Debian 12.
> - Moving individual files is not merged-/usr.
> - "Symlink farms" are not merged-/usr.
> - Upgrading a non-merged-/usr system to Debian 12 needs to work.
> - Upgrading a merged-/usr system to Debian 12 needs to work.
> - Packages cannot assume all systems are merged-/usr until the Debian 13
>   development cycle begins.
> - Upgrading via apt in the usual way should work.
> - Testing and QA systems should be able to avoid this transition, but if
>   they do, they cannot be upgraded beyond Debian 12.
> - A package building incorrectly on a non-merged-/usr system is a bug.
> - A package building incorrectly on a merged-/usr system is a bug.
> - Please stop moving individual packages' files from the root filesystem
>   into /usr, at least for now.
> 
> Definitions and current status
> ==============================
> 
> libQUAL refers to the directories described in FHS 3.0 section 3.10 [1],
> such as lib64 on the amd64 architecture.
> 
> Merged /usr is the filesystem layout in which /bin, /sbin, /lib and each
> /libQUAL that exists are symbolic links to the corresponding directories
> below /usr. This results in aliasing between /bin and /usr/bin, and
> so on.
> 
> In the merged-/usr layout, files whose canonical logical location is
> in one of the affected directories on the root filesystem, such as
> /bin/bash, /sbin/fsck and /lib/ld-linux.so.2, are physically located at
> the corresponding path below /usr, such as /usr/bin/bash. Each file in
> one of the affected directories is accessible via two paths: its canonical
> logical location (such as /bin/bash or /usr/bin/env), and the other path
> implied by the aliasing (such as /usr/bin/bash or /bin/env).
> 
> There are two supported categories of Debian 11 installation, which are
> currently considered equally-supported:
> 
> - Merged-/usr installations. These were installed with debian-installer
>   from Debian 10 or later, or installed with debootstrap --merged-usr,
>   or converted from the non-merged-/usr layout by installing the usrmerge
>   package, or installed or converted by any similar procedure. They have
>   the merged-/usr layout.
> 
> - Non-merged-/usr installations. These were installed with debian-installer
>   from Debian 9 or earlier and subsequently upgraded without converting
>   to merged-/usr, or installed with debootstrap --no-merged-usr, or
>   converted from the merged-/usr layout with dpkg's "dpkg-fsys-usrunmess"
>   utility or any similar procedure. They have the traditional,
>   non-merged-/usr layout in which /bin/bash and /usr/bin/env have exactly
>   those physical paths, and /usr/bin/bash and /bin/env do not exist.
> 
> Merged-/usr is not the only filesystem layout that has been proposed for
> unifying the root filesystem with /usr. For avoidance of doubt, we do not
> consider other filesystem layouts to be implementations of merged-/usr.
> In particular, we do not consider these to be implementations of merged-/usr:
> 
> - all affected files physically located in /bin, /sbin, /lib and /libQUAL,
>   with /usr/bin as a symlink to /bin, etc. (this is the reverse of
>   merged-/usr, and was historically used in the hurd-i386 port)
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/bash -> /usr/bin/bash for only those files that
>   historically had their canonical logical location on the root filesystem
> 
> - a "symlink farm" in each of /bin, /sbin, /lib, /libQUAL with individual
>   symbolic links such as /bin/env -> /usr/bin/env for all files in the
>   affected directories, regardless of whether they historically had
>   their canonical logical location on the root filesystem
> 
> [1]: https://www.debian.org/doc/packaging-manuals/fhs/fhs-3.0.html#libltqualgtAlternateFormatEssential
> 
> Upgrade path from Debian 11 to Debian 12
> ========================================
> 
> The technical committee resolved in #978636 [2] that Debian 12 'bookworm'
> should only support the merged-/usr layout. This resolution describes
> the implications of that decision in more detail.
> 
> Debian installations have traditionally had a straightforward upgrade
> path between consecutive releases, and the technical committee expects
> maintainers to continue this. In the case of #978636, the upgrades we
> are interested in are:
> 
> - Upgrading from Debian 11 (stable release) to Debian 12 (stable release)
> 
> - Upgrading from Debian 11 (stable release) to testing/unstable during the
>   development cycle that will lead to Debian 12, and subsequently upgrading
>   from testing/unstable to Debian 12 (stable release)
> 
> What we understand this to imply is as follows:
> 
> - Because Debian 11 installations with the merged-/usr layout already
>   exist, and because Debian 12 should only support the merged-/usr layout,
>   all packages in Debian 12 should be installable onto a merged-/usr system
>   along with their dependencies, and work correctly on the resulting system.
> 
> - Because Debian 11 installations with the non-merged-/usr layout already
>   exist, all packages in Debian 12 should be installable onto a non-merged-/usr
>   system along with their dependencies, and work correctly on the resulting
>   system.
> 
>     + The key reason for this is that apt is not required to perform the
>       upgrade between stable releases in any particular order, so long
>       as package dependency relationships are respected; therefore the
>       upgrade can happen in whatever order apt chooses, which can vary
>       between machines. Debian has not traditionally required use of a
>       special upgrade tool similar to Ubuntu's do-release-upgrade(8) and
>       we believe the upgrade to Debian 12 should be no different (see
>       below for more details on this topic).
> 
>     + Another reason for this is that during the development of Debian 12,
>       testing/unstable systems undergo a series of partial upgrades, which
>       similarly will happen in an undefined order.
> 
>     + We do not require that the resulting system remains non-merged-/usr:
>       if the packages involved in this installation transaction are part of
>       the implementation of a transition to merged-/usr, then installing them
>       might result in the system becoming merged-/usr.
> 
> - The same expectations apply to packages uploaded to testing/unstable
>   during the development cycle that will lead to Debian 12.
> 
> - Debian contributors who are interested in merged-/usr are invited to
>   implement a straightforward migration path from non-merged-/usr to
>   merged-/usr. The Technical Committee will not design this migration path
>   in detail. If disputes arise related to this migration path, or if
>   advice on this migration path is requested, we will resolve those by
>   following our usual procedures.
> 
>     + One example of a migration path that might be used is for an Essential
>       package to add a dependency on the usrmerge package, so that it will
>       be installed automatically during upgrades. We do not require this to
>       be the migration path that is chosen; it is presented here merely to
>       demonstrate that such a migration path can exist.
> 
> - After Debian 12 is released, the transition to merged-/usr is considered
>   to be complete. Because we do not support "skipping a release" when
>   upgrading systems, new packages uploaded to testing/unstable during the
>   development cycle that will lead to Debian 13 may assume and require
>   that the system onto which they are installed is merged-/usr.
> 
> If a suitable transition mechanism is not available by the time the
> Debian 12 freeze is reached, the Technical Committee will rescind our
> advice in #978636 and modify our advice in this resolution to reflect
> the situation that has been reached. We hope that this will not be necessary.
> 
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978636
> 
> Use of a special upgrade path
> =============================
> 
> Some developers have argued that we should deploy merged /usr by means of
> a special upgrade path, with all other upgrade paths being declared to be
> unsupported. Some examples of special upgrade paths that might be proposed:
> 
> - upgrading dpkg before carrying out the rest of the upgrade
> 
> - installing the usrmerge package before carrying out the upgrade
> 
> - using a special tool similar to Ubuntu's do-release-upgrade(8) which
>   encapsulates whatever steps are necessary
> 
> This would imply that upgrade paths other than the recommended one are not
> expected to work.
> 
> We are aware that many Debian users do not read the release notes before
> upgrading from oldstable to stable, and will expect to be able to upgrade
> via whatever apt commands they are used to using, regardless of whether
> the release notes recommend something different. Given that there are
> alternatives available, we do not think that merged-/usr is sufficiently
> good cause to interfere with this.
> 
> During the Debian 12 development cycle, users of testing/unstable will also
> need an upgrade path from Debian 11 to testing/unstable, or from older to
> newer snapshots of testing/unstable. In general this will be done using
> apt, and it needs to continue to work if at all possible, even before a
> special upgrade tool has been prepared; introducing a "flag day" at which
> all testing/unstable users are expected to carry out additional
> non-automatic upgrade steps would be disruptive, and in practice many
> testing/unstable users are likely to skip those steps.
> 
> For these reasons, we make the simple, conservative recommendation that a
> special upgrade path should not be required, and upgrading via apt in
> approximately the same way as previous Debian releases should continue to work
> in general.
> 
> Testing and QA
> ==============
> 
> We anticipate that during the development cycle that will lead to Debian 12,
> it is likely to be useful for testing and QA infrastructure (such as
> autopkgtest, piuparts and/or reproducible-builds) to be able to produce
> an installation of Debian testing/unstable that is not merged-/usr,
> in order to be able to verify that packages targeted for inclusion in
> Debian 12 can still be installed and/or built successfully in a
> non-merged-/usr environment during partial upgrades.
> 
> As a result, we recommend that if there is an automatic migration from
> non-merged-/usr to merged-/usr, it should be possible to prevent that
> migration. However, systems where that migration has been prevented are
> not required to be fully-supported, and in particular, upgrading them
> to Debian 13 or to the state of testing/unstable during development of
> Debian 13 should be considered unsupported.
> 
> Building packages
> =================
> 
> In #914897 [3] the Technical Committee resolved that for Debian 11, packages
> can validly be built in either a merged-/usr or non-merged-/usr environment.
> We understand this to imply that packages built in either a merged-/usr or
> non-merged-/usr environment should work as intended in either a merged-/usr
> or non-merged-/usr environment.
> 
> There is a class of bugs in which a package embeds the absolute path to
> some executable or other file, in such a way that if it is built in an
> environment with a unified /usr (either via merged-/usr or symlink farms),
> this can result in embedding a non-canonical path such as /usr/bin/sh or
> /bin/env which will not work correctly in a non-merged-/usr environment.
> The Technical Committee reminds maintainers that packages in Debian
> 12 are expected to work correctly in a non-merged-/usr environment,
> and therefore this class of bugs needs to be resolved. As a result, we
> recommend that these bugs should generally be treated as release-critical
> for Debian 12 by maintainers and the release team.
> 
> The Reproducible Builds effort tracks this class of bugs as
> "paths_vary_due_to_usrmerge" [4] and many bugs in this class are
> tracked with the usertag "usrmerge" [5]. In Autotools-based
> build systems, they can often be avoided by passing arguments such as
> `SED=/bin/sed` to the configure script.
> 
> Note that although the name of the usertag mentions usrmerge, this class
> of bugs is not specific to the usrmerge package, and not completely specific
> to merged-/usr; many of these bugs would also manifest if unified /usr was
> achieved via symlink farms, or if a local executable such as
> /usr/local/bin/sed existed on the build system.
> 
> #993675 [6] is a typical example of this class of bugs.
> 
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914897
> [4]: https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
> [5]: https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=reproducible-builds%40lists.alioth.debian.org&tag=usrmerge
> [6]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993675
> 
> Moratorium on moving files' logical locations into /usr
> =======================================================
> 
> In the past, some package maintainers have taken advantage of the
> fact that /usr is now required to be mounted during early boot
> (since Debian 9) to move individual files from the root filesystem
> to /usr. Examples of this being done during previous development cycles
> include the libglib-2.0.so.0 and libgcrypt.so.20 shared libraries,
> and the command-line utilities in the iptables package.
> 
> The Technical Committee recommends that during the Debian 12 development
> cycle, the maintainers of individual packages should not proactively move
> files from the root filesystem to the corresponding locations in /usr in
> the data.tar.* of packages. Files that were in /usr in the Debian 11
> release should remain in /usr, while files that were in /bin, /lib* or
> /sbin in the Debian 11 release should remain in those directories.
> If files were moved from /bin, /lib* or /sbin into /usr since the
> Debian 11 release, they should be moved back to their Debian 11
> locations.
> 
> Similarly, during the Debian 12 development cycle, we recommend that
> maintainers of tools such as debhelper should not move files from the
> root filesystem into /usr.
> 
> For example, debhelper 13.4 started moving systemd units from /lib/systemd
> into /usr/lib/systemd, but this was subsequently reverted in 13.5.2. We
> consider the revert that was done in 13.5.2 to have been an appropriate
> course of action.
> 
> We issue this recommendation for several reasons:
> 
> - The transition to merged-/usr will make the logical locations of these
>   files irrelevant to their physical locations.
> 
> - The transitional mechanisms necessary to prevent such moves from breaking
>   other packages that hard-code specific paths are error-prone, and can
>   themselves interfere with the transition to merged-/usr.
> 
> - After the transition to merged-/usr has completed, during the Debian 13
>   development cycle, we expect that maintainers will be able to move these
>   files without needing transitional mechanisms.
> 
> - On merged-/usr systems, there is a possible failure mode involving files
>   being moved between packages (with Replaces) during the same release
>   cycle that their logical location is changed from the root filesystem
>   to the corresponding aliased directory in /usr, which can result in
>   the affected file disappearing. This can be avoided by not changing
>   the file's logical location until the beginning of the Debian 13
>   development cycle, after the transition to merged-/usr is complete.
> 
> - On non-merged-/usr systems, a failure mode has been observed in which
>   older shared libraries in /lib/MULTIARCH are not always deleted as
>   intended, and interfere with correct loading of the newer shared
>   library in /usr/lib/MULTIARCH. This can be avoided by not changing
>   the file's logical location until the beginning of the Debian 13
>   development cycle, after the transition to merged-/usr is complete,
>   at which point ldconfig(8) will choose the newer library even if
>   this occurs.
> 
> This recommendation applies until Debian 12 is released, or until
> a subsequent Technical Committee resolution rescinds it, whichever
> is sooner. We intend to rescind this recommendation if mechanisms are
> developed to avoid the undesired side-effects of moving files from the
> root filesystem into /usr.
> 
> ~~~~ end text to be voted on ~~~~



-- 
Regards,
Marga

Attachment: signature.asc
Description: PGP signature


Reply to: