Re: DEP 17: Improve support for directory aliasing in dpkg
On Sun, 7 May 2023 at 06:55, Helmut Grohne <helmut@subdivi.de> wrote:
>
> Hi Luca,
>
> On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
> > Sure, there are some things that need special handling, as you have
> > pointed out. What I meant is that I don't think we need special
> > handling for _all_ affected packages. AFAIK nothing is using
> > diversions for unit files or udev rules, for example (I mean if any
> > package is, please point it out, because I would like a word...). I
>
> I've posted a list in
> https://lists.debian.org/20230428080516.GA203171@subdivi.de and indeed,
> udev rules are being diverted in one case.
*fetching sledgehammer*
> But then, you only capture diversions inside Debian's ecosystem and miss
> out on other kinds of diversions such as local diversions. We currently
> support imposing local diversions on pretty much arbitrary files
> including unit files. And while we've occasionally moved files between /
> and /usr before the transition, doing it for 2000 packages significantly
> grows the risk of it affecting someone. So really, we want all such
> diversions duplicated before unpacking a package the has moved its
> files. The way to achieve that is Pre-Depending on usr-is-merged. To me,
> this sounds like we really want some special handling for all affected
> packages.
The local/external aspect is already covered in Ansgar's reply and subthread.
> I also caution that we've started from a very simple approach and tried
> fixing it up to address the problems that we recognized in analyzing it.
> My impression is that we are not finished with our discovery here and
> won't be for quite some time. This uncertainty of what else we might
> break is the most significant downside I see with your approach.
Sure, but adding changes that are (seemingly) unnecessary for a large
percentage of affected packages also brings uncertainty. Every
software has bugs, thus it follows that injecting more software in the
way of a package being installed will likely also inject bugs. Which
doesn't mean we shouldn't consider it, however, it should be weighted
appropriately.
> > very strongly suspect this will be a small minority out of the total
> > of ~2k would need this treatment, and the vast majority would not. Do
> > you disagree with this gut feeling?
>
> I do disagree indeed. While the special handling may be mostly
> mechanical for the majority of packages, it still see it as required.
>
> Worse, we also need to discuss how this affects backporting of packages.
> Any package enabling the addon needs to have the addon removed for a
> backport to undo the move. Worse, when backporting debhelper, any
> package that uses the new compat level must explicitly disable the
> addon. And then we may need to fix upgrade paths from backports to
> stable.
Packages that need special handling will need special handling for
backporting too. This is nothing new, there was never a project-wide
guarantee that a package uploaded to testing can apply 1:1 to
backports, it is common enough to require changes/reverts/adjustments,
and if it's fine to require that in other cases, it's fine for this
case too.
If the majority of packages are simply converted, with no manual
handling and no diversion, then it should be simple to handle: the
debhelper in stable will not perform the conversion by definition as
the logic won't be present, and any dh upload to backports will have
such logic disabled, so that other packages that get uploaded to
backports and built with either the stable or the backports debhelper
won't have any change performed on them.
> > Curve ball question: is there anything blocking us from canonicalizing
> > PT_INTERP at the source in gcc so that the essential set (and
> > everything else, really) can work without the lib -> usr/lib symlink?
> > Would anything obvious break?
> > This would be in trixie, so even when unpacked on bookworm for the
> > upgrade case, the loader is guaranteed to be accessible from the
> > canonical path. Heck, we could even consider canonicalizing shebangs
> > in scripts shipped in essential packages? I'd imagine /bin/sh would
> > have the same issue as the loader?
>
> Heh! As much as this initially sounds insanely crazy,
I know, right? And yet...
> I was unable to
> spot a reason for why this cannot work at all. It definitely sounds like
> fixing the bootstrap protocol to me. On the flip side, I'd assume the
> loader location (for the purpose of examining ELF objects) to be hard
> coded in quite some places. Changing gcc seems out of question, because
> we also expect binaries built on Debian to run on non-Debian systems.
I don't think this is true? At least not in the broader sense: if you
compile something on Debian, it will obviously get linked against
libraries and dependencies as they are in Debian.
Perhaps what you mean is that, given an entire separate sysroot-like
tree, passing the appropriate compiler and linker flags and
environment variables, you can use the local compiler we ship to build
'foreign' programs. That is true, but again it requires to set up the
environment appropriately, including linker flags. And the caller
needs to ensure the environment, including linker flags, is
appropriate for the target environment (I guess 'host' environment, in
GNU parlance). Therefore, I don't think it would be unreasonable to
require that if the target environment is split-usr, then the caller
also needs to specify an appropriate
'-Wl,--dynamic-linker=/lib/ld-whatever' option.
I do not want to break use cases, but at the same time I don't think
it's fair to impose that any and all possible use cases have to
forever work without any changes - that's not how we approach building
new major releases. We very much allow ABI breaking changes, to pick
the most obvious example, and we rebuild our stuff but leave it up to
the users to rebuild their own local programs to deal with it.
Or to put it in another way: I think our defaults should prioritize
the Debian native use case. Given we ship our loader in /usr/lib/ld*
now, it makes sense to me that the default in GCC is to point to
/usr/lib/ld*. Callers can override that as needed for
third-party/external/foreign use cases.
It is important to note, just to be clear, that we will most likely
forever have the lib -> usr/lib symlinks in the final image when it is
booted/instantiated/etc, so existing software that uses the
non-canonicalized PT_INTERP will continue to work normally.
> Rather than change gcc, I think the interpreter could be changed
> after-the-fact using patchelf. We do have prior art for this since GUIX
> uses this technique to redirect the loader below /gnu/store to implement
> its versioning scheme and allow concurrent installations of glibc. The
> existence of GUIX should also limit the possible breakage. By not doing
> it in gcc, we likely save apparmor's and heaptrack's test suites from
> breaking. Still radare2 an systemtap seem fragile in this regard. (You
> can use codesearch.d.n with lib64/ld-linux-x86 and one of those packages
> to discover the code I see breaking. My search was non-exhaustive.)
>
> How about the long-term vision of this? Elsewhere you indicated that
> you'd like the aliasing symlinks to not be shipped by any data.tar. Does
> that imply that we'd keep patching the interpreter and using /usr/bin/sh
> forever in the essential set? If adding the links to base-files, it
> would be of temporary nature only.
>
> If adding the symlinks to base-files, how about /lib64? Would we ship it
> for all architectures or just for those that need it (e.g. amd64,
> loong64, mips64el, ppc64, ppc64el)?
> https://wiki.debian.org/ArchitectureSpecificsMemo has a list of dynamic
> loaders we also need /libx32 for x32 at least. If making this
> architecture-dependent, would base-files become Multi-Arch: same?
As above, I am pretty sure the running system will forever have the
bin -> usr/bin, lib -> usr/lib etc. symlinks, so the vast majority of
external programs should be unaffected? IE if you try to search
manually for ld, you'll still find it at the old path. And if there is
hard-coded parsing of ELF sections anywhere then yeah it will need to
be rebuilt, but to me it seems like any other ABI changes.
I think we should leave the long term vision for another day, and
focus on your requirements for the essential set unpacking right now.
Kind regards,
Luca Boccassi
Reply to: