/usr-merge status update + next steps
Hi,
Yeah, I know we have too many /usr-merge discussions. Still, there is
reason to continue posting. My last summary/status was
https://lists.debian.org/20230712133438.GA935843@subdivi.de from July
12th and I'm giving an update of what happened since then here and
explain how I want to move forward.
# DEP17
I've updated the DEP17 MR at
https://salsa.debian.org/dep-team/deps/-/merge_requests/5 and continue
to provide a rendered version at https://subdivi.de/~helmut/dep17.html.
Notable changes:
* We learned that in bookworm and beyond, the rootfs of the
debian-installer is not /usr-merged. If we start moving files in
packages, we'd likely break the installer. (This is DEP17-P10.) The
kinda obvious mitigation is performing the merge there (DEP17-M22)
and I've submitted a MR to that end.
https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39
* The heated discussion around debootstrap and the bootstrap protocol
was tracked down to be rooted in a misunderstanding. The proposal
changing debootstrap (DEP17-M19) has gained consensus and has been
uploaded to unstable as debootstrap/1.0.130 by Luca Boccassi. Thanks.
Please test this version of debootstrap and report bugs as we intend
to backport this change and the change merging --variant=buildd in
trixie and beyond to bookworm and bullseye.
* The issue of loosing empty directories (DEP17-P6) has been clarified
to also loose such directories in plain upgrades when they are
canonicalized. Therefore all empty directories in aliased locations
are affected regardless of whether other packages ship files within.
* The issue of loosing empty directories (DEP17-P6) has formalized two
known mitigations.
M20: Restore empty directories in maintainer scripts
M21: Add placeholder files
* I have recorded the perceived consensus in a new "Proposal" section.
# Archive work
* I currently manually file RC bugs for file conflict bugs that affect
the /usr-merge. If you also file such bugs, please add the
debian-qa@lists.debian.org usertag "fileconflict" and ensure that you
set appropriate affects or assign the bug to multiple packages.
* I have filed bugs with patches and MRs for deleting empty directories
(DEP17-P6) that I was as unused. After all, a directory that is not
installed, cannot be lost. In case of lib32lsan0 and libx32lsan0,
this resulted in deletion of the entire package (thanks Matthias).
* I have filed bugs for trigger interests on aliased locations
(DEP17-P2) and asked maintainers to duplicate them (DEP17-M12).
# Continuous monitoring of problems
The dumat service introduced earlier continues to detect problems
related to /usr-merge. Its detection has been slightly improved and it
also associates reported bugs now. If bugs are correctly assigned,
usertagged and have the right affects, dumat associates them and
displays them in the output file https://subdivi.de/~helmut/dumat.yaml.
At the time of this writing 40 of 111 issues are associated with bug
reports. MRs are not associated.
# Next steps
## Continuous MBF
I intend to implement automatic bug filing in dumat for some classes of
bugs. The bug association mentioned above has been implemented to avoid
filing duplicates. I intend to extend the service to automatically (with
no human intervention) file bugs for issues that are not yet associated
with a bug number. If you close such a bug without fixing the cause, a
new one will be filed. I intend to implement this one category at a time
and most of these categories will result in bugs of severity serious. In
general, no bugs will be filed for "risky" issues (52 at the time of
this writing) and bugs will only be filed for versions in unstable or
experimental. In order to limit possible damage, I intend to limit the
service to filing one bug per run (i.e. maximum 4 bugs per day). The
categories that shall receive automatic rc bugs are:
* undeclared file conflicts (all existing ones filed manually)
* P1: ineffective replaces (none at present due to the moratorium)
* P2: ineffective trigger interest (all existing ones filed manually)
* P3: ineffective diversions (none at present due to the moratorium)
* P6: loss of empty directories (some filed)
* P7: loss of m-a:same shared files (none at present due to the
moratorium)
The first is usertagged under the qa umbrella:
https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=debian-qa%40lists.debian.org&tag=fileconflict
The others will be usertagged with my email helmutg@debian.org and tag
name dep17pN.
Why are undeclared file conflicts on this list? They typically are
mitigated with Breaks+Replaces or diversions. Both of these mechanisms
have potential breakage with aliasing. Therefore, an undeclared file
conflict may hide a problem related to /usr-merge.
The empty directory loss issues are currently not filed at RC severity
even though they tend to be reproducible on bookworm. The moratorium
does not prevent these from happening. The first issue of this kind was
discovered by Andreas Beckmann using piuparts and for this reason, I
intend to upgrade these issues to RC severity. It is easier to deal with
them in a well-understood way than in some piuparts report blocking
testing migration of another package.
In general, the use of RC severity is deliberate in order to block
/usr-merge problems from migrating to testing. This approach has been
acked by release team member Paul Gevers.
I don't currently have bug templates for these and hope that you can
trust me with coming up with good templates. Since this is a continuous
MBF, I expect to improve templates based on feedback from recipients.
Given the current rate of problems introduced, I expect that the service
files about 1 bug per week for now. When we lift the moratorium and mass
convert packages to canonical locations, I expect it to file more bugs
and hope that we do not introduce more than 28 bugs per week, which
would exceed the proposed rate limit.
I recognize that this is quite a non-standard way to ask for a MBF. Does
anyone object to me doing it in this way? Do you need more details on
the implementation before I can proceed? In any case, I'm ready to stop
the MBF at any point if it causes problems. I will continuously monitor
the bugs it files and will be ready to respond as people reply to those
bugs. It's not the first time I've raised this MBF, but I go into more
detail that previously here. All of the feedback I've got thus far is
positive.
## Updating debootstrap
Another major piece is updating debootstrap in bookworm and bullseye.
There are two changes that Simon McVittie intends to backport:
* Merging of --variant=buildd chroots in trixie and beyond. Once this
change is deployed to buildds, their chroots will become merged and
this is a precondition to lifting the moratorium.
* Changing the merge implementation to merge after initial unpack
(M19). This change is necessary to be able to ship the aliasing
symlinks in a package (M11). It is backwards compatible.
## dh_usrmerge
I intend to add a new tool dh_usrmerge to debhelper (not yet
implemented). Its purpose is performing the path canonicalization in
binary packages. As long as the moratorium is in effect, this helper
must not be used. It shall be possible to enable this helper via
"Build-Depends: dh-sequence-usrmerge" or "--with=usrmerge". I discussed
the possibility of adding this helper to the default sequence via a
compatibility level. However, Niels Thykier pointed out that a new
compatibility level typically takes 3 years to be adopted and we want
100% adoption before trixie. It will not be mandatory to use this
helper. If dropping e.g. --prefix=/ and relying on the /usr default
works, that's better.
## Informative MBF
I was asked to perform another MBF at minor severity for all (2000)
affected packages (collapsed to one per source package) informing
maintainers about what's going on. Since path canonicalization can
introduce bugs, we want it to happen on an opt-in basis rather than
binNMUs and opt-out. In particular, these bug reports will ask affected
maintainers to perform their canonicalizing upload to experimental and
only proceed to uploading to unstable if they do not receive a bug
within three days. In a similar vein, affected maintainers will be
asked to upload any change that restructures the package to experimental
first (throughout the entire trixie cycle) and also wait three days
prior to proceeding to unstable. If these guides are followed, little
breakage shall enter unstable. Otherwise, the automatic filing of bugs
shall prevent those issues from migrating to trixie. The intention here
is to remove the need of having to remember all these crazy problems
from the average maintainer's duties and instead have them rely on
automatic reporting. I expect that half of the packages can be converted
with no changes beyond enabling dh_usrmerge.
## Lifting the moratorium
I see the following preconditions for having the moratorium lifted and
hope that other CTTE members agree.
* buildd chroots need to be /usr-merged.
* The debian-installer needs to be /usr-merged.
* The automatic RC bug filing service needs to be in operation.
* We need to supply tools (e.g. dh_usrmerge) that enable maintainers to
do their part with minimal effort and communicate this (informative
MBF).
## Mass conversion of packages
With the exception of some essential packages, files are moved from / to
/usr in binary packages with support from maintainers and in some cases
using NMUs. Once most packages are converted, updates to the remaining
essential packages including base-files, bash, dash, glibc, and
util-linux are prepared and coordinated to be uploaded in a single
dinstall with the respective maintainers. When base-files installs the
aliasing symlinks, it will provide usr-is-merged.
## Coordinate with downstreams
Every derivative that also implements the /usr-merge in this way, will
have to monitor their archive in a similar way as we do. An early
version of dumat has been run on Ubuntu, but this effectively affects
every derivative. We also need to update documentation such as release
notes and explain that local packages and addon repositories may be
negatively affected.
Let me close this mail with asking for feedback. I'm quite sure that
this plan will not work out exactly the way it is presented here and it
will need refinement. Still it provides a starting point for resolving
the issues at hand. If you disagree with any of this or have questions,
please speak up.
Helmut
Reply to: