tearing down the /usr-move project
Hello developers,
in 2022, Freexian ran a survey[1] about developer interests. Some of its
questions were about funding development and the responses indicated a wish for
managing the /usr-merge aftermath. As a result, Freexian tasked me with leading
this work. The initial idea was to enhance dpkg with knowledge about aliasing,
i.e. when two distinct locations refer to the same filesystem object. I
formalized that plan into a Debian Enhancement Proposal assigned number 17[2],
but the ensuing discussion made it clear that this very much was not what the
Debian members wanted. The counterproposal was to move all the files - from a
dpkg point of view - to their actual locations below the /usr hierarchy. On the
filesystem level, that move has already been mandatory since the bookworm
release. It was all but clear that this was going to work out and the proposal
document turned into a collection of subproblems and workarounds that would
arise from pulling this off. While this approach was painful in the interim,
the end state that we now have in forky is that there is no more aliasing in
the packaging database. From start to finish this project took almost three
years of varying work. Freexian compensated around 700 hours of work on this
project. While the project was initially met with resistance due to the prior
experience with the matter, constructive conversations arose and many
volunteers started supporting this on their own time probably matching
Freexian's effort. With trixie having been released, we may mostly call this
done. The release team set up a migration blocker for including aliased files.
This shall conclude my and Freexian's involvement in this transition. In
particular, the dumat[3] monitoring service will cease operation. We ask
package maintainers to take over and finish the cleanup work in their own
packages as we expect it to be uncontroversial.
There is still some cleanup to be done.
* A number of packages are using dh-sequence-movetousr or dh_movetousr.
Ideally, packages stop using these and simply install the affected files
below /usr explicitly. Packages that intend to support backporting beyond
trixie should defer this cleanup.
https://codesearch.debian.net/search?q=dh-sequence-movetousr&literal=1
https://codesearch.debian.net/search?q=dh_movetousr+path%3Adebian%2Frules&literal=1
* Some mitigations incurred temporary maintainer scripts such as protective
diversions that solely exist during a package upgrade or duplicated dpkg
triggers (udev and runit). In most cases, those have been marked up such
that the Debian janitor may propose MRs for removing them. The relevant
sections enclosed in "begin-remove-after: released:trixie" and the next
"end-remove-after" can simply be deleted with little extra thought.
https://codesearch.debian.net/search?q=begin-remove-after&literal=1
* For more important packages (e.g. e2fsprogs, readline, libiio0, ...),
protective diversions exist in a trixie installation. For very important
ones such as base-files, dash or glibc, I recommend keeping them in the
forky release and then removing them in forky+1 to be fully done in forky+2.
For most others, forky will have to carry maintainer scripts that actively
remove such diversions in postinst while dropping the present preinst/postrm
code to create/remove them. Most of the diversions use a .usr-is-merged
suffix. https://codesearch.debian.net/search?q=.usr-is-merged&literal=1
* Few packages (e.g. molly-guard, zutils) had their diversions duplicated. We
may now stop creating the aliased diversions and actively remove them in
postinst.
https://binarycontrol.debian.net/?q=DEP17&path=unstable (might give clues)
* Debian installer packages (udebs) have been deferred as the effects were
minor there. Those packages ideally should be migrated as well and once
that has happened for all udebs, the merging code can be replaced with just
creating the symlinks and bailing otherwise. Some work to this end has
already started.
https://sources.debian.org/src/debian-installer/20250803%2Bdeb13u1/build/util/merge-usr
* Care must be taken with backports before trixie.
* For bookworm-backports, the file move moratorium technically still holds
and therefore all the moves and stuff need to be reverted. This is where
dh_movetousr may come in handy. On a case-by-case evaluation, there might
be exceptions (when it is clear that no other package has a trigger
interest or diversion or anything else).
* For bookworm-backports-sloppy, the story for packages is more difficult.
I recommend not porting packages that installed aliased files.
* For bullseye-backports-sloppy, unmerged-/usr still was vaguely supported.
Therefore, such backports must revert the moves.
* In a distant future, we may be able to remove the merged-/usr support from
debootstrap as the file layout is now defined by packages. For that to
happen, stuff that calls debootstrap should stop using the relevant options
where possible.
* Where Breaks and Replaces have been converted to Conflicts for the /usr-move,
they shall remain Conflicts. Typically, such relations have a version
constraint and can be expired by the Debian janitor.
* The dep17* BTS usertag namespace on my email helmutg@debian.org may be used
by those doing cleanup work. Thus far, there is a generic dep17 usertag as
well as usertags dep17pNN for specific problems according to DEP17[2].
Let me close by saying thank you. I don't want to know how much longer we would
have had to deal with the fallout if it were not Freexian's managers taking the
initiative and funding to make this happen. At the same time, it would not have
worked out so uneventfully if it were not for probably a hundred volunteers
supporting this work. Allow me to single out the effort of Chris Hofstaedler
and Michael Biebl as outstanding. It is the collaboration of many despite their
earlier experience with the matter that now allows us to call this a success.
Helmut
[1] https://debian.pages.debian.net/dd-surveys/dd-survey-analysis-2022.pdf
[2] https://dep-team.pages.debian.net/deps/dep17/
[3] https://salsa.debian.org/helmutg/dumat
Reply to: