Re: DEP 17: Improve support for directory aliasing in dpkg
On Tue, 25 Apr 2023 at 20:07, Helmut Grohne <helmut@subdivi.de> wrote:
>
> Hi Luca,
>
> On Sat, Apr 22, 2023 at 01:06:18PM +0100, Luca Boccassi wrote:
> > On Sat, 22 Apr 2023 at 11:50, Helmut Grohne <helmut@subdivi.de> wrote:
> > > On Fri, Apr 21, 2023 at 03:29:33PM +0100, Luca Boccassi wrote:
> > > > After Bookworm ships I plan to propose a policy change to the CTTE and
> > > > policy maintainers to forbid shipping files in the legacy directories
> > > > altogether, followed by a debhelper change to adjust any stragglers
> > > > automatically at build time and a mass rebuild, plus MBF for the small
> > > > % that does not use dh and a piuparts test to stop migration for
> > > > anything that is uploaded and doesn't comply. That should bring the
> > > > matter to an end, without needing to modify dpkg.
> > >
> > > I agree with the goal of removing aliases by moving files to their
> > > canonical locations. However, I do not quite see us getting there in the
> > > way you see it, but maybe I am missing something. As long as dpkg does
> > > not understand the effects of aliasing, we cannot safely move those
> > > files and thus the file move moratorium will have to be kept in place.
> > > And while moving the files would bring the matter to an end, we cannot
> > > do so without either modifying dpkg or rolling back the transition and
> > > starting over. I hope that we all agree that rolling back would be too
> > > insane to even consider, but I fail to see how you safely move files
> > > without dpkg being changed. Can you elaborate on that aspect?
> >
> > Moving files within _the same_ package is actually fine as far as I
> > know. It's moving between location _and_ packages within the same
> > upgrade that is problematic. The piuparts test I added is overzealous,
> > but it doesn't need to be.
>
> You got me interested to dig deeper. I looked into that piuparts
> check[1]. From what I understand, it does something differently from
> what you suggest here. It detects files moved between / and /usr (which
> is what is going to happen according to your plan) and it does not
> detect files being moved between packages (which would actually be
> problematic here). It also does not produce an error (merely a warning),
> so it doesn't halt testing migration and in particular, it doesn't
> detect the problematic situation at all. That's kinda disappointing.
It was a while ago, but from my recollection it did check both things,
and it did fail the run. I'll check again when I have a moment (or
better: MRs welcome ;-) ).
Did you run it with any specific cases/packages to check this?
> You also side stepped the question of how to handle the situation where
> we've moved files from / to /usr and then need to move files between
> packages in a safe way, though your other response to Simon McVittie
> suggests you have an idea there.
>
> On Sat, 22 Apr 2023 13:03:01 +0100, Luca Boccassi wrote:
> > We already have piuparts tests detecting files moving, it should be
> > easy enough to extend that to check that the appropriate
> > Breaks/Replaces have been added. Correct me if I'm wrong, but I
> > believe it's already against policy to do this without
> > Breaks/Replaces, so it's not a use case that we need to support, no?
> > If someone does that by mistake, the package will not migrate to
> > testing.
>
> Yeah, we agree that you need Breaks+Replaces. The issue here is that due
> to dpkg not knowing about the aliasing, Breaks+Replaces is insufficient.
> Due to the insufficiency the CTTE enacted the moratorium.
>
> My impression is that you believe that with bookworm, the moratorium is
> being lifted and thus we can start moving files. Unfortunately, the
> underlying problem does not go away just because we've released
> bookworm. It's a problem that is unique to merged installations and
> those are not going to go away in bookworm.
Not quite, from my point of view I don't see a problem with keeping
the moratorium in place for bookworm, personally.
In practice, it would mean it applies to everything that currently
ships files in bin/sbin/lib, as all of those are moved automatically
and thus can no longer move to another package at the same time, which
is of course a non-trivial constraint.
But it sounds like you came up with an even better way:
> So yeah, we all want these files moved to their canonical locations and
> I kinda like the simplicity of your approach, but thus far my
> understanding is that it is plain broken and doesn't work. Well, yeah it
> does work in the sense that we break user installations during upgrade
> and notice so late in the freeze without any good options to fix.
>
> On Sat, Apr 22, 2023 at 01:06:18PM +0100, Luca Boccassi wrote:
> > > I'd also be interested on how you plan to move important files in
> > > essential packages. This is an aspect raised by Simon Richter and where
> > > I do not see an obvious answer yet.
> >
> > Do you have a pointer? Not sure I follow what "important" files means
> > here, doesn't ring a bell.
>
> In <[🔎] 669234b3-555b-4e2a-ccc7-dd5510b6e9c1@debian.org>, Simon Richter
> said:
> > Dpkg already has defined behaviour for directory vs symlink: the directory
> > wins. In principle a future version of dpkg could change that, but
> > /lib/ld-linux.so.2 is just too special, we'd never want to have a package that
> > actually moves it.
>
> This and /bin/sh is the kind of files I'd consider important. And then
> upon thinking further it became more and more difficult for me to make
> sense of the objection. On a merged system, we can just move that file
> to its canonical location without having any trouble even with an
> unmodified dpkg. So from my pov, the question about important files can
> be disregarded. I hope Simon Richter agrees.
>
> Let us circle back to your "broken" approach. It sure is simple (just
> move all the files and be done) and if we could just skip over the
> upgrade issues and have all the files moved without having to modify
> dpkg, that would actually be a better result than DEP 17. Just how do we
> avoid the issue of file loss arising from the aliasing in your scenario?
>
> There kinda is an obvious solution here. We just need to tell dpkg that
> it needs to remove the package containing the file that is being moved
> before unpacking the replacing package. This semantic actually exists
> and we call it "Conflicts". So instead of using Breaks+Replaces, the
> solution is to use Conflicts! Problem solved, right?
>
> Yeah, I think this solves a number of cases, but there are two areas
> where it does not:
> * We generally prefer Breaks over Conflicts for a reason. It gives the
> dependency resolver more freedom and in taking this freedom away, it
> may fail to find solutions to complex upgrades. (Which is why Breaks
> got introduced in the first place.)
> * We cannot use Conflicts inside the transitively essential set.
>
> So if we move all those files, in 90% (number made up) of the cases it
> will go well (since we don't move between packages) and in 90% (number
> made up) of the remaining 10%, we can use Conflicts, but what do we do
> about that remaining 1%?
>
> If we look deeper into the dpkg toolbox, we pause at diversions. What if
> the new package were to add a (--no-rename) diversion for files that are
> at risk of being accidentally deleted in newpkg.preinst and then remove
> that diversion in newpkg.postinst? Any such diversion will cause package
> removal of the oldpkg to skip removal of the diverted file (and instead
> deleted the non-existent path that we diverted to). Thus we retain the
> files we were interested in.
>
> This way, we can just move the files as you suggested.
> * For 90% of the packages, this will just work.
> * For 9% of the packages, we'll need to turn Breaks+Replaces into
> Conflicts.
> * And for 1% of the packages, we'll need to add complex diversions to
> maintainer scripts.
>
> And while this sounds super ugly in the 1% of cases, it's a complexity
> that maybe isn't necessary at all and we can remove it after trixie
> unlike the complexity being added in DEP 17.
>
> In order to better understand the mechanics at hand (and due to Simon
> Richter's call for test cases), I sat down and wrote some. So in this
> mail you find 4 files attached:
> * runtest.sh is a wrapper script to run each case inside a fresh
> chroot created by mmdebstrap (as you don't want to mess your system).
> * case1.sh demonstrates the file loss problem with Breaks+Replaces and
> thus fails.
> * case2.sh demonstrates how Conflicts fix the problem.
> * case3.sh demonstrates how diversions fix the problem.
>
> In sincerely hope that this fixed-up plan doesn't have any serious
> issues. If you find any please tell.
>
> And before closing this mail, I would like to express my gratitude and
> thanks to Emilio Pozuelo Monfort for enduring me in numerous discussions
> on this matter and providing so many of the key insights captured in
> this mail.
Brilliant! Would never have thought of using divert like that.
So, what work would need to happen to make this reality? Do we need
tooling/scripts/build changes to support the divert scheme, or is it
"simply" a matter of documenting and testing?
Kind regards,
Luca Boccassi
Reply to: