Bug#1050001: Unwinding directory aliasing
Hi Ian,
On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> Desired end state
> =================
>
> This is a very good question. I had a very constructive conversation
> with Helmut via video chat. It seems that there's a misunderstanding
> about the desired end state.
I concur. Thank you for taking the time to write this down. It now makes
a lot more sense to me.
> My idea of a desired end state is as follows:
>
> /bin and /lib etc. remain directories (so there is no aliasing). All
> actual files are shipped in /usr. / contains compatibility symlinks
> pointing into /usr, for those files/APIs/programs where this is needed
> (which is far from all of them). Eventualloy, over time, the set of
> compatibility links is reduced to a mere handful.
This end state vaguely makes sense to me in principle. I think it does
have a two noteworthy limitations though.
* Basically every other distro uses aliasing now. I expect that
a few upstreams have stopped paying attention to systems in your end
state. Therefore, they may not only hard code paths in /usr/bin, but
also the other way round assume that everything is to be found in /.
I see that we already use #!/bin/python3 e.g. in
https://sources.debian.org/src/uchardet/0.0.7-1/script/langs/sv.py/?hl=1#L1
and expect more of that in future.
* A key motivation cited for doing the merged-/usr work is called
"hermetic /usr". I'm not a hermetic /usr expert, but roughly speaking
the idea is that the entire OS actually lives below /usr and
everything else is either the user's data or constructed in a
straight forward way. Setting up the aliasing symlinks is easier and
less prone to change over time than setting up the symlink farm that
you are proposing.
> I think this is a more desirable situation than the current planned
> end state, which is that /bin and /lib are symlinks.
What situation is desirable is dependent on whom you ask. We also have a
number of people who'd say that the aliasing approach is more desirable
than your approach. And then we have a large body of people who simply
want this to be over and not having to thing about /usr-merge
consequences anymore.
> Aliasing is EBW, and "Only use canonical names" is not good enough
> ==================================================================
>
> There is basically one underlying technical reason for preferring the
> un-aliased usrmerge approach: aliasing directories in this way leads
> to great complication in file management, especially in package
> management software and in individual packages.
I'm not sure I follow this argument precisely. Ubuntu has shipped this
aliasing approach since quite a while including LTS releases. Debian has
defaulted to it in two stable releases. While there are complications,
my impression is that these complications mostly affect ourselves and
our package management while end users are mostly unaffected. Then once
packages have moved their files to canonical locations (as is proposed
in DEP-17), are there that much complications that remain?
> The DEP-17 problem list is a survey of the aliasing-induced problems
> which have been discovered so far. But we (still!) keep discovering
> new ones.
Indeed, my main worry about this approach is discovering new classes of
problems. (And effects on downstreams and derivatives as you also point
out.)
> The current plan, as I understand it, is that we will fix these
> problems by arranging to *always* name files by their canonical paths,
> ie the ones in /usr.
I confirm.
> Naming files by their canonical names will have to be done everywhere.
> This is because any time a file is named by a non-canonical path, a
> program that tries to manipulate that file might malfunction.
> (Whether it malfunctions in practice depends on the precise details
> and gets very complicated.)
This seems overgeneralized to me. If you look into the list of known
problems from DEP-17, it's not like any program would be affected. It is
more like most of the problems affect dpkg. Also keep in mind that while
we want all files to be named canonically (according to DEP-17), the
problems we may see tend to scale with the number of exceptions to the
rule. So if we canonicalize most files, chances are that most users
don't experience symptoms anymore.
> Spotting and mitigating violations is hard
> ------------------------------------------
>
> We do not currently have good tooling that will spot violations of
> this rule. It's not clear precisley what the right behaviour of our
> tools would be; we need to alert *the right set of users* to the
> mistakes, and *with the right level of severity*. Many of our key
> tools don't have a good way to produce "critical warnings". The
> consequences of violations are unpredicatable and can depend on event
> ordering. But they can be very severe. So we are creating a source
> of bad heisenbugs.
We already have the Debian Usr Merge Analysis Tool available at
https://salsa.debian.org/helmutg/dumat and its output at
https://subdivi.de/~helmut/dumat.yaml. As explained on d-d@l.d.o, I want
to turn those findings into automatic RC bugs. Does that alleviate your
concerns to some extent?
> Also, we only have direct control over the behaviour of our own
> packages, images, etc. that we (Debian) ship. Any time anyone in the
> field (perhaps an invididual sysadmin or user; perhaps a 3rd party
> software supplier; perhaps a downstream distro) violates this rule
> (whether through ignorance, or choice), affected systems will
> malfunction. (I think this means that relying on lintian, for
> example, as a defence against these mistakes, is not good enough.)
This is very true. On the flip side, shipping files in /bin or /lib is
usually done by essential packages or packages required for booting.
Neither of these topics are very popular in 3rd party software.
We may also argue that things that integrate with Debian have to adapt
to Debian as Debian changes. For instance, multiarch is an aspect that
impacted quite a number of upstreams. Maybe this case is similar?
> The answer implied by the current plan seems to be that these people
> are just doing the wrong thing and will have learn not to? But the
> very existence of the directory symlinks implies a recognition that
> confusion over whether to name files in / or in /usr is expected to
> continue for a long time. If it weren't, then there would be no need
> for these symlinks.
That confusion that you describe does not matter on aliased systems as
the promise is that however you choose, both ways will be supported.
Also note that this confusion arises regardless of how Debian chooses,
because other distributions use the aliasing approach already.
> Violations of the "only use canonical names" rule are required
> --------------------------------------------------------------
>
> Worse, violations of the "use only canonical names" rule are not only
> expected, they are *necessary*:
>
> There are quite a few places where we will have to keep naming files
> by their names in /, becaue those things appear in highly stsble
> public APIs/ABIs. For example, we must ship binaries that refer to
> the dynamic loader in /lib; shell scripts must start #!/bin/sh.
I am not sure why you consider this to be a violation. The "only use
canonical names" rule only applies to binary package contents and to
some of the relevant tools (such as triggers, diversions, etc.).
Non-packaging software may use either name at will. That precisely is
the promise of aliasing.
> Now, those references are almost all in "immutable" contexts, where it
> doesn't actually matter, since the file is in fact available by the
> non-canonical name. However, this introduces a new implied rule:
> it becomes a bug to take a filename you see in a place where the file
> is being *read*, and apply it in a context where the file is going to
> be *updated*.
This is accurate and a nice way to describe it. Are there actually that
many tools that need to update files of a Debian installation?
> This reuse of a filename is a very natural approach. It is something
> that is frequently done by humans, but it is also sometimes doen by
> automatic software of many kinds. It's not something we've even had
> to consider before as a thing. But now it is (sometimes) wrong.
> Usually it will work, but sometimes it will make a (perhaps latent or
> unpredictable) bug.
This feels vague to me. Given the general adoption of the aliasing
approach in other distributions, I'd expect that we have at least
anecdotal evidence. Are you aware of any?
> Looking towards the future
> --------------------------
>
> It seems to me that directory aliasing will continue to be a source of
> very annoying bugs indefinitely, well after the transition is fully
> complete. In another 20 years we'll still be debugging strange
> installation breakage that will turn out to be due to directory
> aliasing.
I very much hope that this is not where we're heading. Quite contrary,
over the past 20 years, we've had packages move between / and /usr and
catching up with where stuff was located. I expect that we get rid of
this bug class and trade it for that other bug class that you are
describing. The big question then is the frequency of each class.
> As I understand it the focus of the current technical work is to try
> to figure out how we can get to only-canonical-paths from here, while
> working aorund all of the (potential) bugs which arise during the
> transition period, when necessarily we will be naming files sometimes
> by their names in / and sometimes by their names in /usr.
This is mostly accurate. As you pointed out, we need to know which path
to refer to when we modify files, but when we access (read-only) files,
we are free to use either name by relying on the aliasing.
> But as I say I don't think the end state being worked towards here, is
> far from the best end state.
I can understand how you get there. My impression is that your view is a
minority opinion.
Speaking for myself without any Freexian or CTTE hat, I mainly want the
transition to be finished such that I can spend brain cells on more
useful (to me) matters.
> If I'm ever entitled to play the "I wrote dpkg" card, I think it's
> now. As the author of dpkg, which I intended to be highly reliable
> software (and, I like to think, I succeeded), I think this ia very
> poor system design.
You are entitled to play that card. Having you and Guillem agree on
these matters is quite a strong argument to me. The other aspect that I
see as rather strong is the cross-distribution consistency on this
aspect. Together, these pose a non-trivial conflict. I kinda hoped that
this question was over as the CTTE ruled on it twice already and few
things have changed since then. DEP-17 mostly writes down what Guillem
said earlier in more detail.
Let me also point out that Guillem is doing work towards better
supporting such aliasing in dpkg. For instance, he is reworking the dpkg
database to be able to track filetypes. That would allow dpkg to at
least recognize some of the problems such as directory vs symlink
conflicts.
> And, the approach being taken very seriously privileges Debian itself,
> and those well-staffed derivatives able to do the necessary transition
> auditing (albeit, indeed, with tooling from Debian). I am
> firmly ideologically opposed to such a tradeoff.
I have difficulties disagreeing with that. Getting this done is more
important to me though.
> The non-aliased approach
> ========================
> >From my conversation with Helmut, it seems that we are envisaging, as
> part of the aliased-usrmerge approach, that there will be tools to
> detect violations of the "refer only by canonical path" rule.
>
> But detecting violations of "these directories only ought to contain
> compat symlinks into /usr" rule is a *lot* simpler. It can be done,
> quite reliably, on end-user systems.
While that may be true, I don't think we do QA on end-user systems. What
we do QA on is binary packages. In that sense, we can more reliably
detect problems in packages, no?
> If we had done usrmerge the non-aliased way, then such a checking
> program would be able to detect a /-vs-/usr bug analogous to #911225.
> So I think a non-directory-aliases variant of this bug is more
> tractable than a directory-aliases variant.
Do you recognize that in the aliased world #911225 is not a bug at all? You
are comparing a bug to "it just works" here.
> I hope this helps clarify my thinking.
It very much did. Thanks again for taking the time to write this down.
Your mail and our earlier conversation certainly shifts my judgement on
this matter. Still, I see rather strong arguments for that other view as
well.
Helmut
Reply to: