Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)
On Wed, 17 May 2023 at 10:31, Helmut Grohne <helmut@subdivi.de> wrote:
>
> Hi,
>
> This bootstrap aspect got me and I discussed this with a number of
> people and did some research.
>
> On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> > I don't think this is true? At least not in the broader sense: if you
> > compile something on Debian, it will obviously get linked against
> > libraries and dependencies as they are in Debian.
> > Perhaps what you mean is that, given an entire separate sysroot-like
> > tree, passing the appropriate compiler and linker flags and
> > environment variables, you can use the local compiler we ship to build
> > 'foreign' programs. That is true, but again it requires to set up the
> > environment appropriately, including linker flags. And the caller
> > needs to ensure the environment, including linker flags, is
> > appropriate for the target environment (I guess 'host' environment, in
> > GNU parlance). Therefore, I don't think it would be unreasonable to
> > require that if the target environment is split-usr, then the caller
> > also needs to specify an appropriate
> > '-Wl,--dynamic-linker=/lib/ld-whatever' option.
>
> Given the feedback, I am convinced that changing PT_INTERP is a stupid
> idea regardless of whether it is technically feasible. There must be a
> better way. Let's step back a bit.
>
> The underlying problem here is performing the initial filesystem
> bootstrap. The semantics of this are a bit vague as they are not spelled
> out in policy, so we will have to derive them from implementations.
>
> I think the major players are (in descending popularity):
> * debootstrap
> * mmdebstrap
> * cdebootstrap
> * multistrap
>
> multistrap predates mmdebstrap and when there was no mmdebstrap, I used
> it a lot. When attempting to test it, I totally couldn't convince it to
> bootstrap from an unsigned or locally signed repository. The patch in
> #908451 didn't cut it. I also note that it creates a /lib64 -> /lib
> symbolic link which feels quite incompatible with merged-/usr. For
> these reasons, I am dropping multistrap from the tools under
> consideration and recommend removing it from the archive. If you happen
> to use multistrap, now would be a good moment to tell me. Personally,
> all of my use cases of multistrap have been converted to mmdebstrap and
> that made a lot of things simpler.
>
> cdebootstrap vaguely works though unsigned operation seems dysfunctional
> as it runs apt-get update during cdebootstrap-helper-apt.postinst and
> that fails. I happen to not have figured out why and treat this failure
> as a success.
>
> So the most popular implementations quite evidently are debootstrap and
> mmdebstrap and both "just work". I note though that they work quite
> differently:
> * debootstrap (depending on flags including --variant) pre-merges its
> chroot while mmdebstrap relies on packages doing it.
>
> I think that the question whether a distribution is merged is a
> property of the distribution and not the bootstrap tool, so I
> strongly recommend following mmdebstrap's view on this. The
> debootstrap way means that we have to include patches for every
> derivative, which is a process that does not scale well.
>
> * mmdebstrap operates in two phases. It first unpacks and configures a
> rather minimal set of packages and then proceeds to adding packages
> passed to --include in a second phase once essential is fully
> configured while debootstrap immediately unpacks everything.
>
> I think the debootstrap approach is slightly worse here, because it
> means that preinst scripts of non-essential packages cannot rely on
> essential packages having been configured.
>
> In any case, we have to deal with both behaviours.
>
> After this little excursion into bootstrap technology, let's go back to
> the /usr-merge and its effects.
>
> I think at this point, we have quite universal consensus about the goal
> of moving files to their canonical location (i.e. from / to /usr) as a
> solution to the aliasing problems while we do not have consensus on
> precisely how to do this (i.e. with changing dpkg or without). If you
> believe that this is not consensus, please speak up.
>
> So in a distant future our packages will not contain any files in /bin
> or /lib. In particular, this affects /bin/sh and the dynamic loader,
> both of which are required to run maintainer scripts, which are
> currently required for creating the symbolic links. Boom.
>
> Solutions have been proposed to this and I think they all fall into one
> of the following four categories.
>
> 1. Don't move. We just keep those files that require a particular
> location (such as /bin/sh or the dynamic loader) in their
> non-canonical location. As such, maintainer scripts will be able to
> run and perform the conversion to symbolic links afterwards.
>
> 2. Move and ship links. Since we unpack all essential data.tar before
> running the first maintainer script, having one package contain the
> compatibility symlinks is enough to fix the problem.
>
> 3. Move and avoid using non-canonical locations. This is the approach
> where we write maintainer scripts as #!/usr/bin/sh and considered
> changing PT_INTERP.
>
> 4. Change the bootstrap protocol. In essence, this has been attempted
> in debootstrap by creating these symlinks prior to unpack, but no
> consensus has evolved around this approach yet. The category is
> wider though and generally requires changes to all bootstrapping
> tools.
>
> To see that it's just these four we can watch an imaginary bootstrap
> process on an imaginary distribution. If any package contains the
> symlinks, we're in category 2. Otherwise, we see whether these symlinks
> exist prior to running the first maintainer script. If they do, we're in
> category 4. And finally, we see whether any files are left in aliased
> locations (according to the dpkg database). If there are, we're category
> 1 and otherwise category 3.
>
> For instance, Luca proposed changing PT_INTERP in the toolchain, which
> is category 3 here. I proposed a minor adaption where we have debhelper
> change it post build using patchelf, which also amounts to category 3.
> I've started with these to get started with classifying and think we can
> disregard both.
>
> For completeness sake, there is one more entry in category 3: We can run
> the dynamic loader from its canonical location explicitly, so we'd
> modify maintainer scripts to start with:
>
> #!/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/sh
>
> This would only affect preinst scripts participating in bootstrap and
> we'd not bother with changing PT_INTERP in the toolchain nor in
> packages. Unfortunately, this completely breaks the DPKG_ROOT work and
> with that, I see no viable entries in category 3 for moving forward.
Apart from DPKG_ROOT, that is also complicated because of multiarch I
suppose? Ie, installing a random $arch package on !$arch.
> Moving on to category 4 feels rather obvious, especially because work
> has been done there in debootstrap. The approach in debootstrap however
> is one that I see as a dead end, because it causes us to maintain this
> code multiple times. It's the number of derivatives times the number of
> bootstrap tools and that doesn't scale.
>
> Category 4 is wider though and we also have other prior art at
> https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap. In particular,
> making architecture bootstrap become chrootless would likely be a
> generic solution to this and other problems. However, any change needs
> to propagate to a stable release in all bootstrapping tools. Therefore
> we cannot reasonably finish the transition before forky. This makes
> category 4 rather unattractive in a short term, but still worth pursuing
> in a long term.
I'm not sure this is really a show-stopper, and it would really need
to delay till forkie. We are talking about only two packages, needing
updates to bootstrap a future release. We have already updated
debootstrap in stable and oldstable via proposed-updates for this
purpose in the past, so as long as the changes are self-contained and
do not affect building older releases, it sounds to me it would be
doable to do the same here.
https://tracker.debian.org/news/1349768/accepted-debootstrap-10114deb10u1-source-into-oldstable-proposed-updates-oldstable-new-oldstable-proposed-updates/
> Having ruled out categories 3 and 4 maybe category 2 would be good? We
> could just ship those symlinks in base-files and be done, right?
> Unfortunately, we pass -k to tar in debootstrap, so when it extracts
> base-files and tries to unpack the bin -> usr/bin symlink, it sees that
> oh no, there already is a symlink at bin (as debootstrap placed it
> there) and thus fails. So in order to make this work, we also have to
> modify debootstrap (and thus are in a combination of category 3 and 4).
As far as gut feelings go, it doesn't feel like to me that modifying
debootstrap to deal with this would be too difficult.
> Conversely, if we unpack anything else that happens to ship
> /bin/something before base-files, then mmdebstrap will fail to unpack
> base-files. So we can only add this link to base-files after all other
> (essential) packages have moved everything out of bin. That happens to
> include /bin/sh. A possible solution to this is doing all of this in the
> same dinstall. Then mmdebstrap works before and works afterwards.
>
> Unfortunately, we're not yet done here. When (for instance) dash is the
> last package to move /bin/sh and such to /usr and you upgrade dash, dpkg
> notices that no package owns /bin anymore. Thus it helpfully deletes
> /bin (the symlink). You're not happy when this happens. We remember the
> silver bullet: diversions! So dash.preinst could dpkg-divert --no-rename
> --divert /bin.usrmerged -add /bin and dash.postinst could revert that.
> Unfortunately, such a diversion affects every package but dash and we
> want it exactly the other way round. So what we could do is pass
> --package dash-usrmerged (which must not exist). Then it'll actually
> keep /bin safe. Unfortunately, we don't know whether dash or bash will
> be the last package owning /bin, so both of them need this diversion and
> this is a conflict.
>
> And as if that wasn't enough, we also run into issues around hard links.
> As dpkg unpacks a canonicalized gzip, it notices that /bin/gunzip (which
> is scheduled for deletion) has the same inode as the new
> /usr/bin/uncompress (because gunzip and uncompress are hard linked). As
> far as I can see, all we get here is a warning and both files survive
> the unpack. It is not clear to me whether this happens by chance or
> reliably.
>
> dpkg: warning: old file '/bin/uncompress' is the same as several new files! (both '/usr/bin/gunzip' and '/usr/bin/uncompress')^
> dpkg: warning: old file '/bin/gunzip' is the same as several new files! (both '/usr/bin/gunzip' and '/usr/bin/uncompress')
>
> Given what I've seen, I'm fairly convinced that I haven't reached the
> bottom of it and I'm ready to conclude that this approach is fragile - a
> property that is most unwelcome when we deal with the essential set.
>
> So what's left is category 1. I looked into what the minimum set of
> files to be retained could be. To do that end, I moved everything and
> then reverted as much as was needed to make bootstrapping work.
> * /lib64/ld-linux-x86-64.so.2 (hopefully obvious)
> * /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (link target)
> * /bin/sh (hopefully obvious)
> * /bin/dash (link target)
> * /bin/bash (usrmerge runs ldd, which is a #!/bin/bash script)
> * /bin/more (update-alternatives doesn't like its absence)
> * /bin/cp (unless usrmerge stops hard coding its path)
>
> So in principle, we may be able to pull this off if we keep bash,
> dash, libc6, and util-linux in their original location. In this
> scenario, we're also unable to remove usrmerge from the essential set.
> The major benefit of category 1 it allows us to move to any other
> category at a later time and it removes aliasing effects from all but a
> small set of packages.
Is it necessary to avoid canonicalizing bash and util-linux? My
understanding was that the reason for that was to avoid dpkg from
deleting the symlink - wouldn't a single package (eg: dash for /bin,
libc6 for /lib* and something else for /sbin) be enough to achieve
that? Or did I misread the issue?
Overall, this sounds like a fine approach to me - simple, and gets us
99.999% of the way there. It means dash and libc6 cannot move
/bin/dash or /lib/ld to other packages in the Trixie cycle, but
somehow I suspect that won't be an issue in reality :-)
We could start with this, and then update debootstrap and mmdebstrap
to also deal with this, and canonicalize dash/libc only after we are
confident the bootstrapping issue is really solved. That way we can
canonicalize and unblock 99.999% of the distro immediately.
Kind regards,
Luca Boccassi
Reply to: