Re: DEP 17: Improve support for directory aliasing in dpkg
On Sat, 6 May 2023 at 19:51, Helmut Grohne <helmut@subdivi.de> wrote:
>
> Hi Luca,
>
> I fear you are still missing too many relevant details. What sounds like
> a simple plan is severely broken if carried out in this simple form. It
> disappoints me that you continue to present it as this simple given the
> earlier evidence.
>
> On Fri, May 05, 2023 at 11:11:54PM +0100, Luca Boccassi wrote:
> > - every package is forcefully canonicalized as soon as trixie is open
> > for business
>
> I gave you scripts to prototype this (via binary patching) and
> demonstrated that this immediately makes a fair number of packages rc
> buggy due to causing unpack errors. I used zutils as an example. It
> really is not that simple and it absolutely cannot be done forcefully,
> but must be done in an opt-in way. I won't repeat the rationale here.
Sure, there are some things that need special handling, as you have
pointed out. What I meant is that I don't think we need special
handling for _all_ affected packages. AFAIK nothing is using
diversions for unit files or udev rules, for example (I mean if any
package is, please point it out, because I would like a word...). I
very strongly suspect this will be a small minority out of the total
of ~2k would need this treatment, and the vast majority would not. Do
you disagree with this gut feeling?
Of course, it goes without saying, we should check this before going
forward in any direction.
> > - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
> > packages at the same time is maintained from bookworm till trixie, and
> > will lifted after trixie ships, and applies implicitly to all the
> > ~2000 binary pkgs that are affected by the above
>
> While the CTTE placed the moratorium until the release of bookworm, the
> release team extended it beyond, see
> https://lists.debian.org/E1oCdQk-0005ge-Nr@respighi.debian.org. So you
> need explicit agreement from the release team on your plan. Failing
> that, any package that has been forcefully moved is immediately
> rc-buggy due to failing a release team requirement.
Of course the release team needs to be on board, no questions about
that. But given the idea is to maintain their decision exactly as it
stands I wouldn't imagine it would be an issue? Once again, the
moratorium is explicitly about moving between locations _and_
packages, in combination, not either/or. From that same email you
linked:
"Files moving their canonical location between / and /usr (details in
[1]) *and* from one binary package to another binary package within
one release cycle remain an RC issue unless dpkg supports it."
I'm proposing to keep this in place as a general rule, with the new
escape hatch that you devised as the only addition.
> Again, let me try to fix up your plan:
>
> 1. Implement a service that reliably notices missing Breaks + Replaces
> even in the presence of aliasing. Augment it to notice
> canonicalization changes and have it report that Conflicts (or other
> measures) are needed in such cases. This should operate for upgrades
> from stable to testing, stable to unstable and testing to unstable.
There are already distro-wide upgrade piuparts checks run occasionally
IIRC, at least I've seen a bug from one being reported this week, so
we should be most of the way there already?
To be clear, this would be very nice and welcome to have obviously,
but I don't think it needs to be a blocker. We don't have such checks
for vast parts of Policy, including moving files without
Breaks+Replaces as evidenced by the recent MBF, and yet we managed to
survive thus far. I don't think it's fair that this workstream should
be held to higher standards than the rest of the project.
> 2. Add a usr-merge-helper script to usr-is-merged that helps with
> instating temporary diversions for the purpose of avoiding file loss.
>
> 3. Add a dpkg-divert wrapper for duplicating diversions according to
> aliasing to usr-is-merged. Add postinst scripts that duplicate
> existing diversions.
Yep, sounds good
> 4. Add canonicalization support to debhelper as an (internal) addon.
> Enable this addon in the next compat level. This will again populate
> ${misc:Pre-Depends} with "usr-is-merged (>= fixed version)" as needed.
> Note that usrmerge's provides is unversioned and doesn't satisfy such
> a dependency.
As already mentioned, I do not believe this is necessary for _all_
cases. It is necessary for a certain number (that we should ascertain
beforehand!) of cases, and we need the machinery implemented for them,
but I don't think we should impose this workflow with pre-depends and
diversions for all affected packages. I think it should be mandatory
for problematic packages such as those you already pointed out, _and_
for cases where the maintainer wants to move files also between binary
packages.
> 5. Write a policy on how to perform changes and how to move files. Simon
> Richter suggested a policy for dpkg-divert already. This needs to be
> extended to cover other matters and refined.
>
> 6. Convert packages one by one by enabling the addon or bumping the
> compat level. Such a conversion may require:
> * Adding Pre-Depends: ${misc:Pre-Depends}
> * Adding Conflicts
> * Adding invocations of the usr-merge-helper
> * Modifying maintainer scripts to canonicalize diversions
> * ...
As above.
> 7. Continuously monitor the reports from the service and turn that
> feedback into patches.
>
> This plan definitely is incomplete and misses critical steps. I suspect
> that Simon Richter could immediately tell at least two flaws if not
> more.
>
> A rather obvious flaw (that Simon Richter already mentioned) is how this
> breaks filesystem bootstrap. The policy requirement is that all
> essential packages must work in unpacked state provided that they have
> been configured at least once. (Please don't ask who added this
> requirement to policy. I might be embarrassed.) In reality bootstrap
> implementations expect more. In order to perform that initial
> configuration, we expect that they work sufficiently well to be able to
> run maintainer scripts even in unpacked state prior to initial
> configuration. This currently works due to the dynamic linker being
> shipped in the non-canonical path it is referenced as. Moving it will
> break bootstrap tooling unless we install /lib in some binary package
> (as Simon Richter pointed out). However, due to how dpkg handles
> directory vs symlink conflicts, it is not clear whether we can
> reasonably add it as a symlink to any data.tar without causing havoc.
> This definitely needs more research and is one of the areas where
> teaching dpkg about aliasing brings significant benefits.
Curve ball question: is there anything blocking us from canonicalizing
PT_INTERP at the source in gcc so that the essential set (and
everything else, really) can work without the lib -> usr/lib symlink?
Would anything obvious break?
This would be in trixie, so even when unpacked on bookworm for the
upgrade case, the loader is guaranteed to be accessible from the
canonical path. Heck, we could even consider canonicalizing shebangs
in scripts shipped in essential packages? I'd imagine /bin/sh would
have the same issue as the loader?
> Let me restate that I am still not convinced that we can pull this stunt
> without painting us in a very dark corner. Too many moving parts of this
> approach do not have well-understood solutions. And even then, my
> analysis has mainly focused on packages shipped by Debian. It neglected
> other relevant aspects such as:
> * custom Debian packages (e.g. equivs, config-package-dev, private
> packages)
> * vendor packages
> * local diversions
> * local statoverrides
> * probably more
I don't think we should lose sleep over third party/proprietary
packages. Given most of those are autogenerated from dubious scripts
(I have seen things...) I doubt they are using diverts and such, most
packages I see from companies are built using cmake or java built-in
scripts that shove stuff in /opt and manually generate the .ar and
pack them, so...
I mean, we were not concerned at all when Canonical switched to zstd
for deb compression, creating a massive ticking time bomb given the
vast majority of third party .debs are built on some flavour of Ubuntu
and ship a single .deb for all consumers, and thus would stop being
installable on Debian (because for years the dpkg patch was rejected,
thankfully now it's all sorted and we are good for bookworm), I don't
think we should spend time trying to pre-empt fixing those. Noting in
the release notes that in case they do X and Y and Z they will need
adjustments and pointing to appropriate documentation seems more than
enough to me, no?
The most popular ones (by virtue of the very fair and impartial
property of being currently installed on my laptop) like chrome,
vscode, edge, steam and signal do not ship anything in the legacy
directories anyway.
Same for local modifications, documenting in the release notes what
corner cases people need to be aware of and how to deal with them
seems standard practice to me when doing new releases. For example
there's a bunch of things you were supposed to do on your machine when
upgrading to Bullseye as documented at:
https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#upgrade-specific-issues
and without checking, I'm pretty sure this is normal for all releases.
Kind regards,
Luca Boccassi
Reply to: