Re: DEP 17: Improve support for directory aliasing in dpkg
Hi Luca,
On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
> Sure, there are some things that need special handling, as you have
> pointed out. What I meant is that I don't think we need special
> handling for _all_ affected packages. AFAIK nothing is using
> diversions for unit files or udev rules, for example (I mean if any
> package is, please point it out, because I would like a word...). I
I've posted a list in
https://lists.debian.org/20230428080516.GA203171@subdivi.de and indeed,
udev rules are being diverted in one case.
But then, you only capture diversions inside Debian's ecosystem and miss
out on other kinds of diversions such as local diversions. We currently
support imposing local diversions on pretty much arbitrary files
including unit files. And while we've occasionally moved files between /
and /usr before the transition, doing it for 2000 packages significantly
grows the risk of it affecting someone. So really, we want all such
diversions duplicated before unpacking a package the has moved its
files. The way to achieve that is Pre-Depending on usr-is-merged. To me,
this sounds like we really want some special handling for all affected
packages.
I also caution that we've started from a very simple approach and tried
fixing it up to address the problems that we recognized in analyzing it.
My impression is that we are not finished with our discovery here and
won't be for quite some time. This uncertainty of what else we might
break is the most significant downside I see with your approach.
> very strongly suspect this will be a small minority out of the total
> of ~2k would need this treatment, and the vast majority would not. Do
> you disagree with this gut feeling?
I do disagree indeed. While the special handling may be mostly
mechanical for the majority of packages, it still see it as required.
Worse, we also need to discuss how this affects backporting of packages.
Any package enabling the addon needs to have the addon removed for a
backport to undo the move. Worse, when backporting debhelper, any
package that uses the new compat level must explicitly disable the
addon. And then we may need to fix upgrade paths from backports to
stable.
> Of course, it goes without saying, we should check this before going
> forward in any direction.
The more I try, the more I have the impression that we enumerate the
ways this can go wrong and the more we poke, the more we find.
> Of course the release team needs to be on board, no questions about
> that. But given the idea is to maintain their decision exactly as it
> stands I wouldn't imagine it would be an issue? Once again, the
> moratorium is explicitly about moving between locations _and_
> packages, in combination, not either/or. From that same email you
> linked:
This is evidently ambiguous as RT also reference the CTTE moratorium,
which includes
"Files that were in /usr in the Debian 11 release should remain in /usr,
while files that were in /bin, /lib* or /sbin in the Debian 11 release
should remain in those directories."
Quite evidently, clarification is needed.
> There are already distro-wide upgrade piuparts checks run occasionally
> IIRC, at least I've seen a bug from one being reported this week, so
> we should be most of the way there already?
I examined the piuparts check in
https://lists.debian.org/20230425190728.GA1471384@subdivi.de already, so
no, not at all.
> To be clear, this would be very nice and welcome to have obviously,
> but I don't think it needs to be a blocker. We don't have such checks
Actually, getting the service seems to be the least of our problems.
It's fairly simple to implement and I have written a PoC style
implementation for parts of it already as part of my analysis.
> for vast parts of Policy, including moving files without
> Breaks+Replaces as evidenced by the recent MBF, and yet we managed to
> survive thus far. I don't think it's fair that this workstream should
> be held to higher standards than the rest of the project.
Given the expected breakage and its latency ahead, I think minimizing
risk is prudent.
> > 4. Add canonicalization support to debhelper as an (internal) addon.
> > Enable this addon in the next compat level. This will again populate
> > ${misc:Pre-Depends} with "usr-is-merged (>= fixed version)" as needed.
> > Note that usrmerge's provides is unversioned and doesn't satisfy such
> > a dependency.
>
> As already mentioned, I do not believe this is necessary for _all_
> cases. It is necessary for a certain number (that we should ascertain
> beforehand!) of cases, and we need the machinery implemented for them,
> but I don't think we should impose this workflow with pre-depends and
> diversions for all affected packages. I think it should be mandatory
> for problematic packages such as those you already pointed out, _and_
> for cases where the maintainer wants to move files also between binary
> packages.
Given local diversions, I am now convinced that any package that may
have one of its files (locally) diverted should have this Pre-Depends,
so yeah, that's basically all of them.
But then, this change can still be mechanical and could even be
implemented via the janitor or some other automation. However we do it,
I still see this as a necessity.
> Curve ball question: is there anything blocking us from canonicalizing
> PT_INTERP at the source in gcc so that the essential set (and
> everything else, really) can work without the lib -> usr/lib symlink?
> Would anything obvious break?
> This would be in trixie, so even when unpacked on bookworm for the
> upgrade case, the loader is guaranteed to be accessible from the
> canonical path. Heck, we could even consider canonicalizing shebangs
> in scripts shipped in essential packages? I'd imagine /bin/sh would
> have the same issue as the loader?
Heh! As much as this initially sounds insanely crazy, I was unable to
spot a reason for why this cannot work at all. It definitely sounds like
fixing the bootstrap protocol to me. On the flip side, I'd assume the
loader location (for the purpose of examining ELF objects) to be hard
coded in quite some places. Changing gcc seems out of question, because
we also expect binaries built on Debian to run on non-Debian systems.
Rather than change gcc, I think the interpreter could be changed
after-the-fact using patchelf. We do have prior art for this since GUIX
uses this technique to redirect the loader below /gnu/store to implement
its versioning scheme and allow concurrent installations of glibc. The
existence of GUIX should also limit the possible breakage. By not doing
it in gcc, we likely save apparmor's and heaptrack's test suites from
breaking. Still radare2 an systemtap seem fragile in this regard. (You
can use codesearch.d.n with lib64/ld-linux-x86 and one of those packages
to discover the code I see breaking. My search was non-exhaustive.)
How about the long-term vision of this? Elsewhere you indicated that
you'd like the aliasing symlinks to not be shipped by any data.tar. Does
that imply that we'd keep patching the interpreter and using /usr/bin/sh
forever in the essential set? If adding the links to base-files, it
would be of temporary nature only.
If adding the symlinks to base-files, how about /lib64? Would we ship it
for all architectures or just for those that need it (e.g. amd64,
loong64, mips64el, ppc64, ppc64el)?
https://wiki.debian.org/ArchitectureSpecificsMemo has a list of dynamic
loaders we also need /libx32 for x32 at least. If making this
architecture-dependent, would base-files become Multi-Arch: same?
> I don't think we should lose sleep over third party/proprietary
> packages. Given most of those are autogenerated from dubious scripts
> (I have seen things...) I doubt they are using diverts and such, most
> packages I see from companies are built using cmake or java built-in
> scripts that shove stuff in /opt and manually generate the .ar and
> pack them, so...
> I mean, we were not concerned at all when Canonical switched to zstd
> for deb compression, creating a massive ticking time bomb given the
> vast majority of third party .debs are built on some flavour of Ubuntu
> and ship a single .deb for all consumers, and thus would stop being
> installable on Debian (because for years the dpkg patch was rejected,
> thankfully now it's all sorted and we are good for bookworm), I don't
> think we should spend time trying to pre-empt fixing those. Noting in
> the release notes that in case they do X and Y and Z they will need
> adjustments and pointing to appropriate documentation seems more than
> enough to me, no?
What you write here makes it obvious that we do not have consensus on
this aspect. I caution that those not interested in /usr-merge would
rather have things continue to just work even if dubious, so it seems to
me that worrying a bit more than we would ourselves would increase the
chances of our transition to be socially acceptable.
> The most popular ones (by virtue of the very fair and impartial
> property of being currently installed on my laptop) like chrome,
> vscode, edge, steam and signal do not ship anything in the legacy
> directories anyway.
I vaguely agree that vendor packages are the category with most unlikely
breakage here.
> Same for local modifications, documenting in the release notes what
> corner cases people need to be aware of and how to deal with them
> seems standard practice to me when doing new releases. For example
> there's a bunch of things you were supposed to do on your machine when
> upgrading to Bullseye as documented at:
> https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#upgrade-specific-issues
> and without checking, I'm pretty sure this is normal for all releases.
It is pretty normal that people don't read release-notes even though
they should. The approach of "let's put it in release notes and be done"
also fails on another level. For instance, both live-build and
open-infrastructure-system-tools employ a local diversion of
/bin/hostname as part of their image building implementation.
At this time, I am strongly convinced that this forced file move
strategy to resolving the /usr-merge very much is a game of
whack-a-mole, which is not meant as ruling out the option, because all
known strategies have significant downsides.
Helmut
Reply to: