Re: move to merged-usr-only?
*Sigh*, replying only now because I find this topic exhausting, draining,
tiresome and I'd rather be doing productive stuff instead of rehashing
stuff for the nth time. :( Even though I've also tried to document and
summarize this in the dpkg FAQ and the dpkg MergedUsr wiki page, but I
assume I'm not explaining myself clearly because this is not getting
through… :(
On Fri, 2020-11-20 at 19:35:07 +0000, Simon McVittie wrote:
> On Fri, 20 Nov 2020 at 11:12:20 +0100, Ansgar wrote:
> > On Fri, 2020-11-20 at 10:19 +0100, Adam Borowski wrote:
> > > On Fri, Nov 20, 2020 at 09:35:42AM +0100, Ansgar wrote:
> > > > As far as I know nothing broke catastrophically over the last releases
> > > > with merged-/usr.
> > >
> > > Unless you look at dpkg or attempts at speeding up bootstrap.
> > >
> > > See https://salsa.debian.org/glibc-team/glibc/-/commit/49d137c4392cb1144f2313f78f31466aaa169b75
> > > for an example.
>
> Trying to transition from /lib to /usr/lib in the way that is advocated
> by some people who don't like usrmerge (one package at a time, without
> using usrmerge or something equivalent) can unfortunately *also* cause
> transitional brokenness: see
> <https://salsa.debian.org/gnome-team/glib/-/commit/1f9c505abe2639296615d13c048227a3df378576>.
I mentioned this before, this does not look specific to whatever
method to do merged-/usr, instead this seems like a general dpkg
problem related to missing fsync() on the directories during unpacks:
<https://lists.debian.org/debian-devel/2020/02/msg00477.html>
I don't see why this could not affect also merged-/usr-via-aliased-dirs
systems when moving other pathnames around.
I still need to sit down and fix it, I guess during 1.21.x…
> > The good news here is that shipping bash as /usr/bin/bash (instead of
> > /bin/bash) solves that problems as dpkg will just install it under
> > /usr/bin/bash. No aliasing over symlinks involved.
Err, no, it does definitely not solve any such problem, the aliasing
via directories is still going to be there (the problem will just show
up instead in the other direction). This would just imply trying to
force onto everyone getting a broken system.
> So we need a way to get there from here. The only possibilities I can
> see for that within the framework of a dpkg-based OS are:
>
> * Aliasing via symlinks:
> Something magics /bin -> usr/bin, etc., symlinks into existence,
> and dpkg continues to tolerate unpacking over them, providing
> a per-installation flag day to move to merged /usr. usrmerge is
> an implementation of this approach, debootstrap --merged-usr is
> another. Individual systems can do this transition at any time (many
> have already done so), and they will get the benefit of merged /usr
> (in particular, a class of avoidable bugs can no longer affect them)
> after they have passed the flag day.
Fortunately the breakage stemming from this approach can now be fixed
starting with dpkg 1.20.6, which contains the new dpkg-fsys-usrunmess
tool.
> * Package-by-package migration:
> For every package that ships a file in /bin, /sbin, /lib* whose path
> might have been hard-coded elsewhere, the maintainer of that package
> takes action to move the file into /usr. If the file's full path might
> have been hard-coded elsewhere (/bin/bash, etc.), the maintainer
> scripts must additionally create an unmanaged symlink
> /bin/bash -> /usr/bin/bash, etc. - similar to what was done to move
> /usr/bin/chacl in src:acl to /bin, but in reverse. And then, to get the
> benefits of merged /usr, we have to wait for *every* package to stop
> shipping anything in /bin, /sbin, /lib*, and *then* have a usrmerge-like
> per-installation flag day at which the collection of symlinks /bin/bash
> -> /usr/bin/bash, etc. are replaced by a single symlink /bin -> usr/bin.
We only need the unmanaged symlinks generated by maintscripts due to
the merged-/usr-via-aliased-dirs hack. :( A transition could have been
done automatically instead, as I've mentioned in the past. See below.
But if the intention is to end up with a merged-/usr-via-aliased-dirs
then this approach seems rather pointless TBH.
Instead what I've mentioned multiple times in the past is that this
could have been done (and could still be done), by say a new debhelper
compat level that automatically moves all the pathnames from «/» to
«/usr», during building, and creates the symlink farms under «/». This
would require reverting the merged-/usr-via-aliased-dirs mess (which is
now possible) and banning that method completely, so that we can avoid
the mess with the required maintainer scripts. Which would imply at
least, that:
* there are no directory aliasing issues anymore due to this (so
f.ex. unpacking different things over the same aliased pathname
is not possible, or querying for pathnames in anything that tracks
filesystem pathnames does not break or fails silently, be that
dpkg*, u-a, or other tools, etc)
* let's dpkg track all these pathnames in the system
* let's maintainers transition manually to the new paths when they can
* let's maintainers remove the «/» compat symlinks when no more users,
and for external parties it's not a silent error
* let's everyone know by simply looking at the filesystem, what
pathnames have already been migrated or not, because some are
still required by ABI to be in specific places
> In all three cases, Debian-as-a-project does not get the full benefit of
> merged /usr until all installations that remain supported have had their
> per-installation transition; and in all three cases, we cannot ship
> /bin -> usr/bin, etc. in base-files until after every supported system
> has undergone that transition. So if Debian 12 is the first to make merged
> /usr mandatory during upgrade, as Ansgar proposes, then Debian 13 would be
> the first release where base-files could safely ship those symlinks.
TBH, I'd consider such base-files package broken, and unsupported from
dpkg's perspective, as a breakage inducer.
This also obviates anything external to Debian that ships stuff directly
under the «/» pathnames or both «/» and «/usr», or does queries or
uses any other dpkg command handling pathnames, etc.
> I agree that aliasing via symlinks has a complexity cost for low-level
> packages like dpkg and debootstrap clones. One way to mitigate this
> would be to make the transitional period as short as we can (which is
> essentially what Ansgar is proposing, I think), so that some release of
> Debian (12 in Ansgar's proposal) can guarantee to be merged-/usr, after
> which we have the option to take away the scaffolding that we used to get
> there (Ansgar proposes that this should happen between Debian 12 and 13).
> There's certainly an argument to be made that since we are *already*
> paying this complexity cost, we might as well get the benefit from it
> as soon as we can.
This still is based on the assumption that the end result is not
broken, but that's not the case, this is just a way to get to a broken
end state faster.
And to restate, I consider the merged-/usr-via-aliased-dirs approach
proposed here, broken and unsupported from dpkg's perspective.
Regards,
Guillem
Reply to: