Hi,
This bootstrap aspect got me and I discussed this with a number of
people and did some research.
On Sun, May 07, 2023 at 12:51:21PM +0100, Luca Boccassi wrote:
> I don't think this is true? At least not in the broader sense: if you
> compile something on Debian, it will obviously get linked against
> libraries and dependencies as they are in Debian.
> Perhaps what you mean is that, given an entire separate sysroot-like
> tree, passing the appropriate compiler and linker flags and
> environment variables, you can use the local compiler we ship to build
> 'foreign' programs. That is true, but again it requires to set up the
> environment appropriately, including linker flags. And the caller
> needs to ensure the environment, including linker flags, is
> appropriate for the target environment (I guess 'host' environment, in
> GNU parlance). Therefore, I don't think it would be unreasonable to
> require that if the target environment is split-usr, then the caller
> also needs to specify an appropriate
> '-Wl,--dynamic-linker=/lib/ld-whatever' option.
Given the feedback, I am convinced that changing PT_INTERP is a stupid
idea regardless of whether it is technically feasible. There must be a
better way. Let's step back a bit.
The underlying problem here is performing the initial filesystem
bootstrap. The semantics of this are a bit vague as they are not spelled
out in policy, so we will have to derive them from implementations.
I think the major players are (in descending popularity):
* debootstrap
* mmdebstrap
* cdebootstrap
* multistrap
multistrap predates mmdebstrap and when there was no mmdebstrap, I used
it a lot. When attempting to test it, I totally couldn't convince it to
bootstrap from an unsigned or locally signed repository. The patch in
#908451 didn't cut it. I also note that it creates a /lib64 -> /lib
symbolic link which feels quite incompatible with merged-/usr. For
these reasons, I am dropping multistrap from the tools under
consideration and recommend removing it from the archive. If you happen
to use multistrap, now would be a good moment to tell me. Personally,
all of my use cases of multistrap have been converted to mmdebstrap and
that made a lot of things simpler.
cdebootstrap vaguely works though unsigned operation seems dysfunctional
as it runs apt-get update during cdebootstrap-helper-apt.postinst and
that fails. I happen to not have figured out why and treat this failure
as a success.
So the most popular implementations quite evidently are debootstrap and
mmdebstrap and both "just work". I note though that they work quite
differently:
* debootstrap (depending on flags including --variant) pre-merges its
chroot while mmdebstrap relies on packages doing it.
I think that the question whether a distribution is merged is a
property of the distribution and not the bootstrap tool, so I
strongly recommend following mmdebstrap's view on this. The
debootstrap way means that we have to include patches for every
derivative, which is a process that does not scale well.
* mmdebstrap operates in two phases. It first unpacks and configures a
rather minimal set of packages and then proceeds to adding packages
passed to --include in a second phase once essential is fully
configured while debootstrap immediately unpacks everything.
I think the debootstrap approach is slightly worse here, because it
means that preinst scripts of non-essential packages cannot rely on
essential packages having been configured.
In any case, we have to deal with both behaviours.
After this little excursion into bootstrap technology, let's go back to
the /usr-merge and its effects.
I think at this point, we have quite universal consensus about the goal
of moving files to their canonical location (i.e. from / to /usr) as a
solution to the aliasing problems while we do not have consensus on
precisely how to do this (i.e. with changing dpkg or without). If you
believe that this is not consensus, please speak up.
So in a distant future our packages will not contain any files in /bin
or /lib. In particular, this affects /bin/sh and the dynamic loader,
both of which are required to run maintainer scripts, which are
currently required for creating the symbolic links. Boom.
Solutions have been proposed to this and I think they all fall into one
of the following four categories.
1. Don't move. We just keep those files that require a particular
location (such as /bin/sh or the dynamic loader) in their
non-canonical location. As such, maintainer scripts will be able to
run and perform the conversion to symbolic links afterwards.
2. Move and ship links. Since we unpack all essential data.tar before
running the first maintainer script, having one package contain the
compatibility symlinks is enough to fix the problem.
3. Move and avoid using non-canonical locations. This is the approach
where we write maintainer scripts as #!/usr/bin/sh and considered
changing PT_INTERP.
4. Change the bootstrap protocol. In essence, this has been attempted
in debootstrap by creating these symlinks prior to unpack, but no
consensus has evolved around this approach yet. The category is
wider though and generally requires changes to all bootstrapping
tools.
To see that it's just these four we can watch an imaginary bootstrap
process on an imaginary distribution. If any package contains the
symlinks, we're in category 2. Otherwise, we see whether these symlinks
exist prior to running the first maintainer script. If they do, we're in
category 4. And finally, we see whether any files are left in aliased
locations (according to the dpkg database). If there are, we're category
1 and otherwise category 3.
For instance, Luca proposed changing PT_INTERP in the toolchain, which
is category 3 here. I proposed a minor adaption where we have debhelper
change it post build using patchelf, which also amounts to category 3.
I've started with these to get started with classifying and think we can
disregard both.
For completeness sake, there is one more entry in category 3: We can run
the dynamic loader from its canonical location explicitly, so we'd
modify maintainer scripts to start with:
#!/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 /usr/bin/sh
This would only affect preinst scripts participating in bootstrap and
we'd not bother with changing PT_INTERP in the toolchain nor in
packages. Unfortunately, this completely breaks the DPKG_ROOT work and
with that, I see no viable entries in category 3 for moving forward.
Moving on to category 4 feels rather obvious, especially because work
has been done there in debootstrap. The approach in debootstrap however
is one that I see as a dead end, because it causes us to maintain this
code multiple times. It's the number of derivatives times the number of
bootstrap tools and that doesn't scale.
Category 4 is wider though and we also have other prior art at
https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap. In particular,
making architecture bootstrap become chrootless would likely be a
generic solution to this and other problems. However, any change needs
to propagate to a stable release in all bootstrapping tools. Therefore
we cannot reasonably finish the transition before forky. This makes
category 4 rather unattractive in a short term, but still worth pursuing
in a long term.
Having ruled out categories 3 and 4 maybe category 2 would be good? We
could just ship those symlinks in base-files and be done, right?
Unfortunately, we pass -k to tar in debootstrap, so when it extracts
base-files and tries to unpack the bin -> usr/bin symlink, it sees that
oh no, there already is a symlink at bin (as debootstrap placed it
there) and thus fails. So in order to make this work, we also have to
modify debootstrap (and thus are in a combination of category 3 and 4).
Conversely, if we unpack anything else that happens to ship
/bin/something before base-files, then mmdebstrap will fail to unpack
base-files. So we can only add this link to base-files after all other
(essential) packages have moved everything out of bin. That happens to
include /bin/sh. A possible solution to this is doing all of this in the
same dinstall. Then mmdebstrap works before and works afterwards.
Unfortunately, we're not yet done here. When (for instance) dash is the
last package to move /bin/sh and such to /usr and you upgrade dash, dpkg
notices that no package owns /bin anymore. Thus it helpfully deletes
/bin (the symlink). You're not happy when this happens. We remember the
silver bullet: diversions! So dash.preinst could dpkg-divert --no-rename
--divert /bin.usrmerged -add /bin and dash.postinst could revert that.
Unfortunately, such a diversion affects every package but dash and we
want it exactly the other way round. So what we could do is pass
--package dash-usrmerged (which must not exist). Then it'll actually
keep /bin safe. Unfortunately, we don't know whether dash or bash will
be the last package owning /bin, so both of them need this diversion and
this is a conflict.
And as if that wasn't enough, we also run into issues around hard links.
As dpkg unpacks a canonicalized gzip, it notices that /bin/gunzip (which
is scheduled for deletion) has the same inode as the new
/usr/bin/uncompress (because gunzip and uncompress are hard linked). As
far as I can see, all we get here is a warning and both files survive
the unpack. It is not clear to me whether this happens by chance or
reliably.
dpkg: warning: old file '/bin/uncompress' is the same as several new files! (both '/usr/bin/gunzip' and '/usr/bin/uncompress')^
dpkg: warning: old file '/bin/gunzip' is the same as several new files! (both '/usr/bin/gunzip' and '/usr/bin/uncompress')
Given what I've seen, I'm fairly convinced that I haven't reached the
bottom of it and I'm ready to conclude that this approach is fragile - a
property that is most unwelcome when we deal with the essential set.
So what's left is category 1. I looked into what the minimum set of
files to be retained could be. To do that end, I moved everything and
then reverted as much as was needed to make bootstrapping work.
* /lib64/ld-linux-x86-64.so.2 (hopefully obvious)
* /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (link target)
* /bin/sh (hopefully obvious)
* /bin/dash (link target)
* /bin/bash (usrmerge runs ldd, which is a #!/bin/bash script)
* /bin/more (update-alternatives doesn't like its absence)
* /bin/cp (unless usrmerge stops hard coding its path)
So in principle, we may be able to pull this off if we keep bash,
dash, libc6, and util-linux in their original location. In this
scenario, we're also unable to remove usrmerge from the essential set.
The major benefit of category 1 it allows us to move to any other
category at a later time and it removes aliasing effects from all but a
small set of packages.
In case you want to reproduce my analysis, I've attached a
test-usrmergebootstrap.sh script. It attempts the following strategies:
* "earlylink": add links to base-files first without converting other
packages
* "movefull": move all files to their canonical locations and add links
to base-files
* "movemost": move most files except those mentioned earlier to their
canonical locations without adding links to base-files
These strategies are tested in the following use cases:
* "cdebootstrap": bootstrap using cdebootstrap
* "debootstrap": bootstrap using debootstrap
* "debootstrap-buildd": bootstrap using debootstrap --variant=buildd
--merged-usr
* "mmdebstrap": bootstrap using mmdebstrap
* "upgrade": perform a regular bootstrap and upgrade packages to the
modified ones
Broken combinations
* *strap-earlylink: unpack error during base-files
* debootstrap*-movefull: unpack error during base-files
* upgrade-movefull: /lib64 missing unless diverted for libc6
And now you probably expect me to write some kind of conclusion, but
really I think none of our options are attractive. In a short term I
strongly recommend not moving files around that are part of the
transitively essential set, because things can (and do) go wrong in so
many surprising ways. The other major takeaway is that a significant
chunk of the problems mentioned in this mail cannot be fixed by
modifying dpkg only.
I sincerely hope that everyone will now point out why this analysis is
incomplete and misses out on covering strategies. I'd love to be wrong
about the dark picture I've drawn here.
Helmut
Attachment:
test-usrmergebootstrap.sh
Description: Bourne shell script