Re: DEP 17: Improve support for directory aliasing in dpkg

[Helmut Grohne <helmut@subdivi.de>]

Hi Luca,

I fear you are still missing too many relevant details. What sounds like
a simple plan is severely broken if carried out in this simple form. It
disappoints me that you continue to present it as this simple given the
earlier evidence.

On Fri, May 05, 2023 at 11:11:54PM +0100, Luca Boccassi wrote:
> - every package is forcefully canonicalized as soon as trixie is open
> for business

I gave you scripts to prototype this (via binary patching) and
demonstrated that this immediately makes a fair number of packages rc
buggy due to causing unpack errors. I used zutils as an example. It
really is not that simple and it absolutely cannot be done forcefully,
but must be done in an opt-in way. I won't repeat the rationale here.

> - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other
> packages at the same time is maintained from bookworm till trixie, and
> will lifted after trixie ships, and applies implicitly to all the
> ~2000 binary pkgs that are affected by the above

While the CTTE placed the moratorium until the release of bookworm, the
release team extended it beyond, see
https://lists.debian.org/E1oCdQk-0005ge-Nr@respighi.debian.org. So you
need explicit agreement from the release team on your plan. Failing
that, any package that has been forcefully moved is immediately
rc-buggy due to failing a release team requirement.

Again, let me try to fix up your plan:

1. Implement a service that reliably notices missing Breaks + Replaces
   even in the presence of aliasing. Augment it to notice
   canonicalization changes and have it report that Conflicts (or other
   measures) are needed in such cases. This should operate for upgrades
   from stable to testing, stable to unstable and testing to unstable.

2. Add a usr-merge-helper script to usr-is-merged that helps with
   instating temporary diversions for the purpose of avoiding file loss.

3. Add a dpkg-divert wrapper for duplicating diversions according to
   aliasing to usr-is-merged. Add postinst scripts that duplicate
   existing diversions.

4. Add canonicalization support to debhelper as an (internal) addon.
   Enable this addon in the next compat level. This will again populate
   ${misc:Pre-Depends} with "usr-is-merged (>= fixed version)" as needed.
   Note that usrmerge's provides is unversioned and doesn't satisfy such
   a dependency.

5. Write a policy on how to perform changes and how to move files. Simon
   Richter suggested a policy for dpkg-divert already. This needs to be
   extended to cover other matters and refined.

6. Convert packages one by one by enabling the addon or bumping the
   compat level. Such a conversion may require:
    * Adding Pre-Depends: ${misc:Pre-Depends}
    * Adding Conflicts
    * Adding invocations of the usr-merge-helper
    * Modifying maintainer scripts to canonicalize diversions
    * ...

7. Continuously monitor the reports from the service and turn that
   feedback into patches.

This plan definitely is incomplete and misses critical steps. I suspect
that Simon Richter could immediately tell at least two flaws if not
more.

A rather obvious flaw (that Simon Richter already mentioned) is how this
breaks filesystem bootstrap. The policy requirement is that all
essential packages must work in unpacked state provided that they have
been configured at least once. (Please don't ask who added this
requirement to policy. I might be embarrassed.) In reality bootstrap
implementations expect more. In order to perform that initial
configuration, we expect that they work sufficiently well to be able to
run maintainer scripts even in unpacked state prior to initial
configuration.  This currently works due to the dynamic linker being
shipped in the non-canonical path it is referenced as. Moving it will
break bootstrap tooling unless we install /lib in some binary package
(as Simon Richter pointed out). However, due to how dpkg handles
directory vs symlink conflicts, it is not clear whether we can
reasonably add it as a symlink to any data.tar without causing havoc.
This definitely needs more research and is one of the areas where
teaching dpkg about aliasing brings significant benefits.

Let me restate that I am still not convinced that we can pull this stunt
without painting us in a very dark corner. Too many moving parts of this
approach do not have well-understood solutions. And even then, my
analysis has mainly focused on packages shipped by Debian. It neglected
other relevant aspects such as:
 * custom Debian packages (e.g. equivs, config-package-dev, private
   packages)
 * vendor packages
 * local diversions
 * local statoverrides
 * probably more

Helmut