Bug#1061248: glibc: DEP17: move most files but rtld to /usr
Hi,
On 2024-01-23 17:40, Helmut Grohne wrote:
> Hi,
>
> TL;DR: I brainstorm solutions and appreciated feedback, but no action is
> required at this time.
I am not sure I understood everything from that mail, but let me provide
a few answers.
> On Sun, Jan 21, 2024 at 10:32:29PM +0100, Helmut Grohne wrote:
> > This seems pretty much unfxiable to me now.
>
> Unfixable was a bit too strong. With much help from Aurelien and ideas
> from Enrico I looked at this more systematically.
>
> I first tried to understand what kind of file sharing we between glibc
> packages. To that end I wrote a collect.sh (attached) that downlods all
> the glibc packages and a diagnose.py (attached) that tries to scrape
> relevant detail from them. This results in an output.txt (attached).
> This is all quite hacky, but it gets the job done. For the purpose of
> this analysis, I am assuming that files that differ in content also
> differ in size (which of course is not generally correct, but simplifies
> matters). That output.txt lists files (normalized to lack any /usr
> prefix) and the packages shipping them (as a 3-tuple package name,
> architecture, size or symlink target). Assuming that all the packages
> were Multi-Arch: same, files with identical content (i.e. size) are
> discarded. We're left with three kinds of file sharing:
>
> debhelper version on sh4
> ========================
>
> Apparently sh4 has a different debhelper that emits different files in
> /usr/share/doc. This breaks M-A:same, but is otherwise boring in the
> DEP17 context.
>
> Conflicting multilibs
> =====================
>
> Around 300 files in /lib64 conflict between libc6-ppc64:powerpc,
> libc6-amd64:i386 and libc6-amd64:x32. Likewise, around 300 files in
> /lib32 conflict between libc6-s390:s390x, libc6-sparc:sparc64,
> libc6-i386:amd64 libc6-powerpc:ppc64, libc6-i386:x32 and
> libc6-mipsn32:mips64el. These are declared file conflicts.
> Unfortunately, we learned that Conflicts do not reliably prevent
> concurrent unpacks in the presence of aliasing, so when moving these
> libraries, we can construct a file loss, for example:
>
> * apt install libc6:mips64el libc6-i386:amd64
> * Add hypothetical apt sources with a moved glibc.
> * echo libc6-i386:amd64 deinstall | dpkg --set-selections
> * dpkg --auto-deconfigure --install libc6-mipsn32_usrmoved_mips64el.deb
>
> In essence, this will be upgrading from bookworm to trixe while
> simultaneously replacing libc6-i386 with libc6-mipsn32 for some reason.
> On top of that, I guess that apt would prefer removing libc6-i386 before
> unpacking libc6-mipsn32 such that the loss scenario likely requires
> working with dpkg directly.
>
> There is a relatively simple mitigation. For every file in SLIBDIR in
> the trixie, libc-alt.preinst can set up a diversion for the
> corresponding aliased location diverting to some non-existent filename.
> libc-alt.postinst then removes these diversions. The Conflicts require
> the conflicting libc-alt to actually be removed before libc-alt.postinst
> is run. (Breaks is not sufficient as I learned the hard way.) These are
> very temporary diversions, but they're also quite many (300).
>
> At the time of this writing, I have not prototyped this approach. Let me
> already pose the question of how you assess the involved risks here.
> Actually triggering this failure is a rather strange corner case and it
> is not clear to me whether this corner case is worth risking possible
> implementation bugs in the mitigation.
>
> Conflicting runtime dynamic linkers between multiarch packages
> ==============================================================
>
> Some architecture combinations such as s390, powerpc, hppa, m68k,
> mipsn32, and hurd-i386 or alpha, i386, sh4, and sparc have conflicting
> runtime dynamic loaders. In theory this violates Multi-Arch: same, but
> there is basically nothing we can do about this and dropping Multi-Arch:
> same from the packages would completely break any kind of multiarch
> setup. There is little we can do here and this is also unrelated to
> DEP17.
We tried to add conflicts for those, like it's done for the multilib
packages, but at the time the infrastructure exploded (see #745552). I
don't remember the details, but I guess it was either dak or
dose-builddebcheck.
> Conflicting runtime dynamic linkers between multiarch and multilib
> ==================================================================
>
> Runtime dynamic linkers need to be installed both into libc:ARCH and
> libc-ARCH:SIBLINGARCH. An example is libc6:i386 and libc6-i386:amd64
> both containing /lib/ld-linux.so.2. Both packages really need to ensure
> presence of the runtime dynamic linker on installation in the exact
> location so there is little we can do about this. The current situation
> is that the multiarch package Replaces the multilib one such that
> ownership is automatically transferred to the multiarch one when both
> are installed. The multiarch postrm also checks whether a multilib
> package is remaining and restores the symlink if it was stolen.
>
> This way of doing things works badly with the /usr-move. In the DEP17
> document, this amounts to a P1 problem where a file is both moved from
> one package to another and from / to /usr. Since we actually want to
> permit concurrent unpack, we cannot use Conflicts (M7). We will have to
> employ protective diversions one way or another. Unfortunately, the
> /usr-move makes it difficult to safely transition the ownership of the
> runtime dynamic linker from the multilib package to the multiarch one
> via Replaces. I see two routes forward:
>
> We can stop using the Replaces approach and always manage the runtime
> dynamic linker of multilib packages using maintainer scripts. libc-alt
> packages installing a runtime dynamic linker in bookworm would drop this
> symlink from their data.tar in the trixie upgrade. Hence, dpkg would
> attempt removing during the package upgrade and we have to prevent that
> from happening by forging a protective diversion owned by libc6.
> libc-alt.preinst would add it and libc-alt.postinst would remove it.
> >From then on, we manage the runtime dynamic linker symbolic link with
> maintainer scripts only:
> * libc-alt.postinst creates the link unless it exists.
> * libc-alt.postrm checks whether a libc owns it and removes it if not.
> * libc.postrm checks wether there is a libc-alt in need of the link and
> restores it if needed. (This last part already exists.)
> As a side-effect of this change in approach, the target of the symbolic
> link no longer depends on the unpack order and we always prefer the
> multiarch symlink over the multilib symlink.
>
> Otherwise, we can continue shipping the runtime dynamic linker in
> multilib packages as we currently do. In this case, the protective
> diversion must be issued both by libc.preinst and libc-alt.preinst and
> can be owned by neither package as each needs to be protected from the
> other. Beyond that we need to track how "often" it was diverted. The
> libc.postinst or libc-alt.postinst that runs later needs to remove the
> protective diversion. I am not sure how we track this state in a
> reliable way yet.
>
> I'd like to better understand why we are using the
> Replaces-without-Breaks approach here before recommending to change to
> maintain the multilib link using maintainer scripts. Can you shed some
> light on this question?
I guess the symlink in the maintainer script could indeed work. I am not
sure why it has been done like that, it was part of the multiarch
patchset from Steve Langasek more than 10 years ago.
Note however that the those packages are used by cross-toolchain-base
(which builds them from glibc-source) and mangled to install them in the
cross path. See for instance libc6-amd64-x32-cross. For such cases, we
probably do want to keep the dynamic linker symlink, as those packages
do not have maintainer scripts.
Aurelien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurelien@aurel32.net http://aurel32.net
Reply to: