Re: DEP 17: Improve support for directory aliasing in dpkg
Hi Guillem,
On Sat, Apr 08, 2023 at 04:35:25AM +0200, Guillem Jover wrote:
> I thought my reply was rather clear, and that we had further clarified
> that privately, that at the time I thought there was no other answer
> required as (AFAIR) you stated you'd be digging further on it. And I
> mentioned I'd try to reply to the list, but it didn't feel urgent given
> the clarifications given, neither the timing during the freeze?
I'm sorry for the misunderstanding. Thank you for replying here as it
helps clarify the relevant matters in various areas. The freeze timing
arises because we ideally merge intrusive changes early in a release
cycle.
> Sigh, a DEP(!?), for a dpkg change? It feels more like a way to exhort
> pressure over this than anything else TBH…
I'm sorry if you perceive it as such. The intention was to capture
consensus around a change that is necessary in some sense and
controversial in another.
> I'm unlikely to discuss this topic on debian-devel, given previous
> nastiness and abuse.
I see. I would have appreciated a Cc in that case as I am not subscribed
to debian-dpkg.
> The text includes most (but not all) of what I've been saying publicly,
> and what I've tried to further clarify to you and Emilio in private.
> But I think ignores the essence of what I've been repeating all along.
It's good to see that I captured many of your arguments. That essence
however probably is the sticking point where we fundamentally disagree.
I'll reference this "fundamental disagreement" a few more times and
circle back to it later.
> I already mentioned this in my reply for the thread you reference. So,
> let me repeat and possibly expand to avoid any future doubt. I already
> considered and discarded something like this (except for using a config
> option instead of a new command, but that does not really change the
> substance of the problems).
Thank you for repeating it in such clarity.
> Let's also get back to the very basics. dpkg manages objects shipped
> in binary packages, on the filesystem. It assumes this managing role in
> exclusivity, it will for example overwrite unmanaged files. It preserves
> admin changes with interfaces specifically provided for that (diversions,
> statoverrides, conffile changes) or the unfortunate symlink redirects.
> These shipped objects define the filesystem layout (not the other way
> around). Due to the missing fsys metadata, where it does not have all
> such metadata at hand when necessary (it might only have the one for
> the currently unpacked .deb), it might use heuristics or check the
> filesystem for such metadata, because it does not have anything else,
> but that should not be taken to mean that the filesystem is the source
> of truth, as most of those will be unnecessary once it has such
> metadata at hand.
This captures an insight I previously didn't have in that clarity and
that I find agreeable conceptually.
> So the reason this proposal is still conceptually wrong is manifold:
>
> * dpkg cannot safely and atomically perform such switches (and I don't
> see it ever being able to portably do so, so I don't see ever
> supporting that).
I agree, but the proposal also does not ask dpkg to perform such
switches, so I kinda fail to see how this is a relevant argument.
> * No packages ships those symlinks (and none should! as that would
> currently imply having the same pathname contain different file types
> on the same system, introducing ordering issues and file type
> conflicts).
I disagree with this argument on two levels. For one thing, I think that
the transition only is complete once these symlinks are shipped in a
package. In particular, that notion of complete likely encompasses that
no aliasing occurs anymore as all aliased files have been moved to their
canonical location somehow (<- and this likely will be a quite difficult
thing to do). For another, no package actually ships those symlinks now.
They are created behind dpkg's back in some postinst. This is
unfortunate and I agree with Simon Richter that this kinda is a policy
violation, but at this time, it is an aspect we have to deal with
whether we want to or not.
I suspect that you disagree with the notion the we have to deal with
this situation, which I consider to be our fundamental disagreement.
> * This introduces a series of commands to let dpkg know that a
> filesystem change that was not shipped in any .deb (even though that
> should have been the way to do it), has been done, which:
> - Switches the source of truth from the .deb to the fsys.
While this is correct on some level, the aim of this change is to put
that truth back into dpkg of course.
> - Confuses admin initiated changes from distro initiated ones.
I think we already do this with dpkg-divert, dpkg-statoverride and other
tools. While this may not be nice, it certain has prior art and is
consistent with how we have been doing things in the past.
> * Wants to be a generic change but it is really targeted to this
> specific mess. We have been doing similar aliasing transitions for
> many doc dirs, by stopping shipping files within, shipping that
> pathname as a symlink and then switching the directories to symlinks
> to match (via the dpkg-maintscript-helper hack because we miss fsys
> metadata). This means we'd need to then register all these directories
> too? Meh.
I would love to agree with this, but I believe that this ship has
sailed. This likely is part of our fundamental disagreement.
> * This information can get out of sync with reality, as it adds an
> additional and unconnected with anything source of truth, that dpkg
> cannot do anything about if it diverges (in contrast to diversions
> or statoverrides f.ex.). This can never happen when that information
> comes from the real source of truth (the fsys metadata via the .deb).
I have difficulties accurately capturing the argument. The problem of
information getting out of sync with reality should affect every aspect
of dpkg and indeed, that kinda is the status quo where upgrades can
loose files, because dpkg has an incomplete picture of reality. The aim
of this change is to allow us to re-sync the status quo into dpkg. My
view is that dpkg's information presently is out of sync with reality
and the proposed change partially fixes that.
> * This also adds undue complexity, by supporting those as admin aliases.
> The admin generated redirecting symlinks are already annoying, I'd rather
> not add further to that pile. I don't really want to support admins doing
> this (dpkg-divert does not even support diverting a directory).
I have to agree with this one. This and the fact that this feature is
probably impossible to remove later is my main problem with the proposed
change. I have been proposing it anyway, because my impression is that
it is the least bad option available. The notion of "least bad" quite
obviously is somewhat subjective and is where we need to get consensus.
> [ As an aside, I think ideally eventually nothing distro provided should
> be allowed to be installed within an aliased dir, and dpkg should
> eventually just error out in those cases, which eventually would get
> rid of the aliasing problems and any such complexity (I'm not sure how
> or when that would be feasible though, but obviously in Debian at
> least not until nothing ships files there). ]
It seems to me that this is something everyone agrees on. So our
disagreement resides in the way to get there rather than where to get
to.
> So this still looks like a terrible interface, like it did at the time
> it was discarded; founded on a hack, an interface that seems wants to
> be kind of a file-type override but it cannot be, and cannot even
> properly act as record tracker, etc…
I agree that in a perfect world, we would not need this. Let me circle
back to our fundamental disagreement.
My impression is that at this time basically everyone except you agrees
that we have to deal with the aliasing problems that have been rolled
out to users and will be forced in bookworm. I believe that this is the
state that we have to consider as starting point and that we cannot
magically turn this transition back to perform it in a better way. And
indeed, I believe that there would have been a better way[1] that no
longer is available to us.
On the other hand, my impression is that you continue to see the
transition as fundamentally broken and in a state that we cannot work
from. You appear to believe that if we want to do it, we must start over
in a better way. That better way must not cause aliasing problems to
dpkg.
And sure enough, DEP 17 is a result of having first created these
aliases and now trying to move the files. If we had opted for first
moving the files and then creating the aliases, much of the bad effects
would never have affected anyone and we wouldn't be discussing this
change as it would not be necessary. With DEP 17 (or any similar change
to dpkg), we will be able to actually move files to their canonical
location and thus resolve the aliasing at which point DEP 17 will become
unused.
Did I accurately describe your view on this matter?
> I thought it would be clear that if there is stuff that depends on
> any of this kind of changes to dpkg, relying on those changes in
> Debian would not be possible until after trixie+1. Of course there is
> always the route to further pile up over the Jenga tower of hacks,
> by for example adding huge amounts of Pre-Depends…
I agree that we probably will deal with this until at least trixie+1.
This is precisely why I would like to have a plan to finish it sooner
rather than later.
> So given the above, I don't see why the apparent rush here. And as I've
> mentioned many times now, I'm planning to continue working on the fsys
> metadata stuff for 1.22.x, probably at the cost of database duplication
> if necessary, if current blockers have not adapted by then. But as I've
> mentioned before, that might not guarantee this support is sufficient to
> support fixing this mess. But all other proposed changes I've seen
> flying around for changes to dpkg are just conceptually wrong in one way
> or another.
As I see it, the fsys metadata work may help with the aliasing problems
only if the respective symlinks are actually shipped in a data.tar of a
.deb, which is not the way we currently do things. For that reason, I
fail to see how the fsys metadata work is part of the solution for the
problems we currently experience. I'd appreciate if you could elaborate
on how you see it helping to fix the file loss on upgrade problem
affecting current installations if you have sufficient energy to do so.
Helmut
[1] What follows is to be considered a time travel fix and is purely
academic. Imagine we never had never done the usrmerge and would be
introducing it again in a usrmerge package that works quite
differently. Rather than create those aliases upfront, it would
declare a trigger interest in affected locations (i.e. both /bin and
/usr/bin and so on). It would practically trigger on every package
operation. Whenever triggered, it would scan the non-/usr location
for files. If any are found, it would populate both the /usr
location and the non-/usr location with symlinks to the actual
files. However when the non-/usr location becomes empty, it would
remove the hierarchy and place the symlink directly. In that
approach, we'd end up at the same layout that we currently have, but
use symlink farms until we reach the point that all files have been
moved. There are a number of aspects where this approach is
problematic as well, but I think it is no longer useful to discuss
it as switching to this approach is not an available option anymore.
Reply to: