Hi Luca, On Sat, Apr 22, 2023 at 01:06:18PM +0100, Luca Boccassi wrote: > On Sat, 22 Apr 2023 at 11:50, Helmut Grohne <helmut@subdivi.de> wrote: > > On Fri, Apr 21, 2023 at 03:29:33PM +0100, Luca Boccassi wrote: > > > After Bookworm ships I plan to propose a policy change to the CTTE and > > > policy maintainers to forbid shipping files in the legacy directories > > > altogether, followed by a debhelper change to adjust any stragglers > > > automatically at build time and a mass rebuild, plus MBF for the small > > > % that does not use dh and a piuparts test to stop migration for > > > anything that is uploaded and doesn't comply. That should bring the > > > matter to an end, without needing to modify dpkg. > > > > I agree with the goal of removing aliases by moving files to their > > canonical locations. However, I do not quite see us getting there in the > > way you see it, but maybe I am missing something. As long as dpkg does > > not understand the effects of aliasing, we cannot safely move those > > files and thus the file move moratorium will have to be kept in place. > > And while moving the files would bring the matter to an end, we cannot > > do so without either modifying dpkg or rolling back the transition and > > starting over. I hope that we all agree that rolling back would be too > > insane to even consider, but I fail to see how you safely move files > > without dpkg being changed. Can you elaborate on that aspect? > > Moving files within _the same_ package is actually fine as far as I > know. It's moving between location _and_ packages within the same > upgrade that is problematic. The piuparts test I added is overzealous, > but it doesn't need to be. You got me interested to dig deeper. I looked into that piuparts check[1]. From what I understand, it does something differently from what you suggest here. It detects files moved between / and /usr (which is what is going to happen according to your plan) and it does not detect files being moved between packages (which would actually be problematic here). It also does not produce an error (merely a warning), so it doesn't halt testing migration and in particular, it doesn't detect the problematic situation at all. That's kinda disappointing. You also side stepped the question of how to handle the situation where we've moved files from / to /usr and then need to move files between packages in a safe way, though your other response to Simon McVittie suggests you have an idea there. On Sat, 22 Apr 2023 13:03:01 +0100, Luca Boccassi wrote: > We already have piuparts tests detecting files moving, it should be > easy enough to extend that to check that the appropriate > Breaks/Replaces have been added. Correct me if I'm wrong, but I > believe it's already against policy to do this without > Breaks/Replaces, so it's not a use case that we need to support, no? > If someone does that by mistake, the package will not migrate to > testing. Yeah, we agree that you need Breaks+Replaces. The issue here is that due to dpkg not knowing about the aliasing, Breaks+Replaces is insufficient. Due to the insufficiency the CTTE enacted the moratorium. My impression is that you believe that with bookworm, the moratorium is being lifted and thus we can start moving files. Unfortunately, the underlying problem does not go away just because we've released bookworm. It's a problem that is unique to merged installations and those are not going to go away in bookworm. So yeah, we all want these files moved to their canonical locations and I kinda like the simplicity of your approach, but thus far my understanding is that it is plain broken and doesn't work. Well, yeah it does work in the sense that we break user installations during upgrade and notice so late in the freeze without any good options to fix. On Sat, Apr 22, 2023 at 01:06:18PM +0100, Luca Boccassi wrote: > > I'd also be interested on how you plan to move important files in > > essential packages. This is an aspect raised by Simon Richter and where > > I do not see an obvious answer yet. > > Do you have a pointer? Not sure I follow what "important" files means > here, doesn't ring a bell. In <[🔎] 669234b3-555b-4e2a-ccc7-dd5510b6e9c1@debian.org>, Simon Richter said: > Dpkg already has defined behaviour for directory vs symlink: the directory > wins. In principle a future version of dpkg could change that, but > /lib/ld-linux.so.2 is just too special, we'd never want to have a package that > actually moves it. This and /bin/sh is the kind of files I'd consider important. And then upon thinking further it became more and more difficult for me to make sense of the objection. On a merged system, we can just move that file to its canonical location without having any trouble even with an unmodified dpkg. So from my pov, the question about important files can be disregarded. I hope Simon Richter agrees. Let us circle back to your "broken" approach. It sure is simple (just move all the files and be done) and if we could just skip over the upgrade issues and have all the files moved without having to modify dpkg, that would actually be a better result than DEP 17. Just how do we avoid the issue of file loss arising from the aliasing in your scenario? There kinda is an obvious solution here. We just need to tell dpkg that it needs to remove the package containing the file that is being moved before unpacking the replacing package. This semantic actually exists and we call it "Conflicts". So instead of using Breaks+Replaces, the solution is to use Conflicts! Problem solved, right? Yeah, I think this solves a number of cases, but there are two areas where it does not: * We generally prefer Breaks over Conflicts for a reason. It gives the dependency resolver more freedom and in taking this freedom away, it may fail to find solutions to complex upgrades. (Which is why Breaks got introduced in the first place.) * We cannot use Conflicts inside the transitively essential set. So if we move all those files, in 90% (number made up) of the cases it will go well (since we don't move between packages) and in 90% (number made up) of the remaining 10%, we can use Conflicts, but what do we do about that remaining 1%? If we look deeper into the dpkg toolbox, we pause at diversions. What if the new package were to add a (--no-rename) diversion for files that are at risk of being accidentally deleted in newpkg.preinst and then remove that diversion in newpkg.postinst? Any such diversion will cause package removal of the oldpkg to skip removal of the diverted file (and instead deleted the non-existent path that we diverted to). Thus we retain the files we were interested in. This way, we can just move the files as you suggested. * For 90% of the packages, this will just work. * For 9% of the packages, we'll need to turn Breaks+Replaces into Conflicts. * And for 1% of the packages, we'll need to add complex diversions to maintainer scripts. And while this sounds super ugly in the 1% of cases, it's a complexity that maybe isn't necessary at all and we can remove it after trixie unlike the complexity being added in DEP 17. In order to better understand the mechanics at hand (and due to Simon Richter's call for test cases), I sat down and wrote some. So in this mail you find 4 files attached: * runtest.sh is a wrapper script to run each case inside a fresh chroot created by mmdebstrap (as you don't want to mess your system). * case1.sh demonstrates the file loss problem with Breaks+Replaces and thus fails. * case2.sh demonstrates how Conflicts fix the problem. * case3.sh demonstrates how diversions fix the problem. In sincerely hope that this fixed-up plan doesn't have any serious issues. If you find any please tell. And before closing this mail, I would like to express my gratitude and thanks to Emilio Pozuelo Monfort for enduring me in numerous discussions on this matter and providing so many of the key insights captured in this mail. Helmut
Attachment:
runcase.sh
Description: Bourne shell script
Attachment:
case1.sh
Description: Bourne shell script
Attachment:
case2.sh
Description: Bourne shell script
Attachment:
case3.sh
Description: Bourne shell script