Re: Second take at DEP17 - consensus call on /usr-merge matters
Hi Helmut,
On 6/29/23 04:37, Helmut Grohne wrote:
Consensus proposal #1:
This updated DEP17 is a complete representation of the known and
relevant problems and known mitigations under discussion at the time
of this writing.
Do you miss a related problem important to you? Do you miss your
preferred mitigation? Please speak up, so we can record it.
I think "backports" are missing as a user story.
Most packages should be harmless, and the Contents file for
bullseye-backports doesn't have too much in any of the affected directories.
For /bin and /sbin, the list is short:
admin/brltty
admin/btrfs-progs
admin/e2fsck-static
admin/e2fsprogs
admin/glusterfs-client
admin/logsave
admin/mdadm
admin/moosefs-client
admin/systemd
admin/systemd,admin/systemd-standalone-sysusers
admin/systemd,admin/systemd-standalone-tmpfiles
admin/systemd-container
admin/systemd-resolved
admin/systemd-sysv
admin/udev
net/iproute2
net/wpasupplicant
but the list of packages installing files into /lib is longer and
includes all the kernel backports, so I guess that is another potential
source of problems.
There might be an easy solution here, I have not investigated this very
deeply because it is a workday and 11 hours out of every day are already
spoken for.
Stating a goal has been quite difficult, but I think that most of the
participants agree that we want to eliminate the file move moratorium
without getting problems in return.
I'd even widen that to "no more special handling needed in any place for
the transition", with the moratorium being an example of that.
When we get into mitigations, consensus is hard to come by. My
impression is that we have roughly two camps. One is in favour of major
changes to dpkg and the other is not.
It's difficult to summarize the situation in one sentence, because
neither group is really objecting to dpkg changes, so I'd put the fault
line at whether the transition should be facilitated through dpkg or not.
* Even if dpkg were fixed in trixie, trixie's libc6 would still be
unpacked using bookworm's dpkg. At least for this package, we cannot
rely on dpkg changes in trixie. Therefore we need workarounds or
spread the transition to forky. For other packages, even a
Pre-Depends on dpkg is not a guarantee that a changed dpkg will
perform the unpack.
* Changes to dpkg will not fix bootstrapping.
The dpkg changes will fix bootstrapping, but we can't finish the
transition until forky this way, because we need to be able to bootstrap
with a libc6 package that can be installed from bookworm.
The subset of affected packages is rather small though.
There also is a minority arguing in favour of doing both. I've kinda
ruled out that option already as we get the downsides of both without
any further benefit in return.
I think that was me being pessimistic and expecting that we're going to
end up doing both anyway. The downsides of the combination approach are
more severe than those of either solution in isolation, because dpkg
incurs a permanent and significant performance penalty to verify its
database against reality, and gains additional error states.
So unless there is someone else, I'd call it a stretch to say that
someone is "in favour" of this solution.
Consensus proposal #2:
The primary method for finalizing the /usr-merge transition is
moving files to their canonical locations (i.e. from / to /usr)
according to the dpkg fsys database (i.e. in data.tar of binary
packages). dpkg is not augmented with a general mechanism
supporting aliasing nor do we encode specific aliases into dpkg.
I recognize that this is not unanimous, but I think we still have
sufficient consensus on this. I suspect that maybe Simon Richter and a
few more would disagree here. If consensus fails, we may have to put
this to a vote.
We need to be careful here to not conflate the goal of the transition
with the method for reaching it. We have consensus on the goal
(basically, data.tar and filesystem matching is the definition of "done"
I use for this transition).
We do not have consensus on the technical implementation because there
are people who believe the technical implementation proposed is not
actually feasible. In my opinion, it is a 95% solution, which is very
tempting but we need the remaining 5% as well. To a large extent, we are
having this discussion because the usrmerge package left us with a 90%
solution.
All a vote can achieve in that situation is assign responsibility, and
authorize more drastic measures such as bypassing regular procedures
(e.g. the "flag day" transition proposal for bootstrapping). Unless the
vote clarifies which of those it is, we will have disagreement on the
scope, which would not be much of an improvement over the status quo
where there is disagreement on the implications of the TC decision.
The minimal scope would be one where no actions are authorized, but the
voters assume responsibility for the outcome, which is not very useful,
because that responsibility doesn't translate to anything. We're not a
company, where we'd need to generate a paper trail in the anticipation
of being blamed for a failing project.
Larger scopes should IMO be specific actions that we're willing to take
to get the transition done earlier.
I mean, at the drastic end we could have a usrmerge package that patches
dpkg's database and diverts dpkg to install a wrapper that patches all
packages it encounters, that is technically also a valid implementation
according to this statement, and it would be quick.
If the intention is only to focus the effort, then this should be made
explicit in the wording. I'm all for focusing the effort, even if it is
in a direction I don't believe is the right one, because either I am
wrong, and it works, or I am right and we then know why it doesn't work.
Both of these are vast improvements over "further discussion."
What I'd like to avoid here is locking ourselves into a particular
direction of thought, and then making the following decisions based on
"but we earlier decided that... and this step, although unpopular, is
now an inherent necessity."
I believe we have consensus for focusing our efforts in that way without
the need for a vote.
Others
are in favour of not changing the bootstrap protocol. In that view, some
data.tar will have to ship the symbolic links and all other essential
packages need to have their files canonicalized.
There is a third option: not changing the bootstrapping protocol,
unpacking into an unmerged tree and merging as soon as possible, from
one of the essential packages, so that the result of "debootstrap
--merged-usr" and "debootstrap --no-merged-usr" is the same (which is
also an excellent test).
This would be an intermediate solution for bootstrapping trixie from
bullseye/bookworm, and would allow a solution where we convert almost
all packages but have to leave out a few because something outside of
bootstrapping (most likely upgrades and backports) imposes a constraint.
This would also be the M1 method for fixing bootstrapping, so M1 cannot
finish the transition in trixie.
[...]
An earlier version of this document proposed the addition of an `--add-alias` option for `dpkg` to record aliasing symbolic links after their introduction.
Simon Richter proposed [adding a `control.tar` member to record aliases](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths).
Yes, this has in the meantime been updated to an explicit "dpkg-alias"
tool that would behave similarly to dpkg-divert, and be responsible for
moving files if necessary.
Any package that wants to rely on the new behavior requires a versioned `Pre-Depends` on `dpkg`.
Doing this to `libc6` would introduce a dependency cycle.
That is likely not a big problem, the only packages relying on the new
behaviour are systemd-sysv and any packages that want to move files
between packages and between paths at the same time, so basically we'd
reduce the scope of the file move moratorium to a small set of core
packages that have no reason to move files, and these would be moved in
forky.
M3: Let `dpkg` use the filesystem as source of truth
----------------------------------------------------
Problems P1, P6, P7 and P9 are concerned with unexpected deletion of filesystem resources that are subject to aliasing.
As such, a targeted behavior change to the code dealing with deletion of unused filesystem resources in `dpkg` is plausible.
`dpkg` could be updated to perform a new check prior to deleting filesystem objects.
There is already a check in place, which is why moving files inside the
same package is possible:
https://salsa.debian.org/dpkg-team/dpkg/-/blob/main/src/main/unpack.c#L750
Extending this check is probably a viable path, but because the check
whether a file is known to dpkg requires a hash table lookup, we need a
list of candidate file names, which requires dpkg to know the list of
aliases. This has not been explored in depth yet, so there may be
additional checkmarks in that table.
> [Table]
I'm fairly sure that M1 can also solve P8, but it will require an
interim solution in trixie and thus delay the finalization until forky.
Simon
Reply to: