Re: Second take at DEP17 - consensus call on /usr-merge matters
Helmut Grohne <helmut@subdivi.de> writes:
> To me, this leaves more question marks than earlier. What applications
> of aliasing do you envision that would benefit here? Anything concrete?
So, I do want to stay focused on Sam's point, which is that we should
avoid this specific argument by noting that we're not going to get dpkg
aliasing support within a workable length of time anyway, given where
we're at today. I'm therefore not willing to put in the effort required
to deeply flesh out use cases, and I'm sure my initial thoughts are both
incomplete and inaccurate.
This is more of a high-level design intuition that stems from some basic
architectural principles, such as "dpkg should be the authority for what
Debian installs on the file system so that it can ensure global
consistency."
But to give you a concrete answer, here are the sorts of things that come
to mind even after this transition is complete in Debian:
* Old packages that still use aliased paths installed on current systems.
* Locally-built packages with /bin paths in data.tar.gz.
* Weird local symlinks to move files around in the file system.
* Some unforseen future transition where we want to merge two directories.
More generally, the way I am thinking about it from a high level is that
this transition has uncovered problems with dpkg's understanding of
symlinks. (Well, uncovered for some of us who weren't paying close
atention; I'm sure Guillem has been aware of them for eons.) Its model of
the world is too simple; there are POSIX file system semantics that it
does not understand and therefore mishandles. Ideally we would fix this.
> Why did you see waiting for a dpkg patch as a reasonable approach
> earlier?
Because I thought it would be easier than it turns out to be.
> I think we have roughly three categories of dpkg changes on the table:
> * Minimal hacks that would paper over some of the effects without
> causing a generally applicable aliasing support
> * A mechanism where dpkg is being told about aliasing symlinks
> explicitly and thus would work with them
> * An extension of the dpkg filesystem database wherein it would be able
> to determine the filetype for every filesystem object. In particular,
> this would allow it to tell directories from symlinks apart and
> handle the overlap in a meaningful way.
> Which of these do you consider "aliasing support"?
The second and the third.
> The third category is quite involved. Since it changes the internal
> layout of dpkg's database, a prerequisite for doing this is removing
> direct accesses to the database by other tools. Guillem has been doing
> that work and you can find more information about it at
> https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking. Note that this
> work goes beyond what is needed for aliasing as it also considers
> extending the metadata format in binary packages to be able to represent
> e.g. extended attributes.
Indeed, this is the point. This is exactly what Sam and I are both
saying: a good architectural solution is going to take long enough that we
can't block this transition on it, and therefore debating whether or not
that solution is desirable is beside the point and is just creating more
things to argue about without resolving the problem in front of us.
> Did you expect Guillem to implement this faster?
I think we have an entire project full of people with expert skills,
including some truly impressive C programmers, and I find our inability to
muster those resources to improve *the* foundational program on which the
work of the entire distribution rests on, in a way that meets a very high
quality bar and is sustainable for the project, to be disappointing. I'm
not assigning blame; I'm simply saying that it makes me sad that the
assumption is that only Guillem alone is able to work on this project and
therefore its success is constrained by his available time.
You will note that I am not ramping up on dpkg development despite being
an expert C programmer. I really am not assigning blame. I'm sad about a
lot of things that I could be doing and currently am not.
> The second category seems fairly narrow and tailored to the /usr-merge
> application. Do you consider it to support aliasing in the generic way
> that you had in mind?
I didn't have a specific generic way in mind. I think it would be a
cleaner architecture than what we're going to do instead, but not as clean
as what Guillem is working on.
I have some concerns that Guillem is trying to boil the ocean and he's
missing some viable incremental approach, but this is only a vague concern
and I absolutely do not have the expertise in dpkg to argue that case.
Clearly Guillem has convinced himself that it is the correct approach and
he is an expert in this area, so he is probably correct and I am probably
wrong.
> Since Guillem is working on the third category, whom did you imagine to
> be working on the second one?
The answer to that question that I have been pushing since the beginning
of this debacle is "the people who wanted to do the /usr-merge
transition." But clearly that was never going to happen, so this is just
another thing for me to be sad about.
> Please bear in mind that a general form of aliasing support entails all
> sorts of crazy corner cases.
Yes. The problem that dpkg is trying to solve is inherently full of crazy
corner cases. I don't think aliasing support is at all unusual in that
sense; just about every problem in this space has lots of crazy corner
cases. This is unavoidable.
In my ideal world, we would meet this head-on by making dpkg robust
against crazy corner cases via an excellent architecture and a thoroughly
comprehensive test suite. It is quite likely that in the process we will
discover that some of the existing dpkg concepts are insufficiently robust
and need to be improved so that they're less fragile.
In other words, the fact that packaging problems give rise to crazy corner
caases is, to me, a reason to put *more* effort into dpkg, not less,
because solving those crazy corner cases is what makes our package
management something that we can be deeply proud of. This is something on
which I wholeheartedly agree with Guillem (and, I think, you): the problem
is complex and requires careful design and thorough testing.
--
Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>
Reply to: