Bug#1050001: Unwinding directory aliasing [and 3 more messages]
On Sat, 26 Aug 2023 at 11:27, Ian Jackson
<ijackson@chiark.greenend.org.uk> wrote:
>
> Helmut Grohne writes ("Bug#1050001: Unwinding directory aliasing"):
> > On Wed, Aug 23, 2023 at 05:04:36PM +0100, Ian Jackson wrote:
> > > And, the approach being taken very seriously privileges Debian itself,
> > > and those well-staffed derivatives able to do the necessary transition
> > > auditing (albeit, indeed, with tooling from Debian). I am
> > > firmly ideologically opposed to such a tradeoff.
> >
> > I have difficulties disagreeing with that. Getting this done is more
> > important to me though.
>
> I have hoisted this to the start of the mail. I think it is a hugely
> important point.
>
> Debian is not simply a technical project trying to thread its way in a
> complicated world. Debian is an ideological project. At its best,
> Debian is the infrastructrure that enables vast swathes of people to
> massively enhance their own technological autonomy. Many of our most
> controversial decisions served this goal, and stood the test of time.
Exactly, and usr-merge provides exactly that infrastructure, which is
one of the many reasons why it is necessary, as it should be beyond
obvious by now.
> If you want to think about it on more practical (or even, selfish)
> level, we want Debian to continue to be the preferred choice, when
> someone is choosing an upstream. We didn't get where we are today by
> following bad technical decisions made by others.
Very true, and a perfect example of such a "bad technical decision" is
the symlinks-farm layout that is proposed by a couple of people: it
doesn't solve any real problem, and just causes issues for developers
and users, and is purely designed to carefully tip-toe around one
singular hopelessly broken debian-specific tool, dpkg, which suffers
from long-standing bad design decisions that have never been fixed to
this day. It was attempted once, it failed, it was rolled back and
actual usr-merge was successfully deployed.
> > * Basically every other distro uses aliasing now. I expect that
> > a few upstreams have stopped paying attention to systems in your end
> > state. Therefore, they may not only hard code paths in /usr/bin, but
> > also the other way round assume that everything is to be found in /.
>
> This is indeed a plausible practical reason to do it the aliased way.
> >From my point of view, it amounts to saying "everyone else has made
> this mistake, so to be compatible, we must too".
>
> But I think that seriously underestimates our influence. Debian
> derivatives make up well over half of all Linux installations.
> They're the default basis for most CI images. If we decide this was a
> failed experiment, then indeed there will be some pain for a while,
> but fairly soon people will stop making this assumption.
In case you haven't noticed, it's not 1998 anymore. It's 2023, and
Debian is sliding fast into irrelevance, largely because it takes a
decade of fighting off trolls and saboteurs to achieve the most
innocuous of changesl, while most other projects can just implement
obviously correct and needed features such as this one and many others
in a couple of months and then move on to the next. In other words,
other distros innovate while we stagnate. Nobody would truly care if
somehow madness descended on this project and such a grievous act of
self-harm was actually perpetrated, at most some raised eyebrows and
some "they are at it again, aren't they" snarky remarks. Certainly
nobody would move back. For example, I can provide a cast-iron
guarantee that systemd will keep supporting only an usr-merged layout
and not work anywhere else starting from the next version (currently
in main), so there goes ~99.5% of Debian installations.
> > * A key motivation cited for doing the merged-/usr work is called
> > "hermetic /usr". [...] Setting up the aliasing symlinks is easier and
> > less prone to change over time than setting up the symlink farm that
> > you are proposing.
>
> I don't like the phrase "symlink farm" because it suggests that all,
> most, or even a substantial minority, of files have these symlinks.
> True, at the start, there will be at least a symlink allotment
> but I'm hoping that in the end it'll be a symlink flowerbed.
"I'm hoping" doesn't cut it. There is only one example of such an
attempt, in OpenSUSE, and it never materialized. It's been 10 years,
and still the handful of proponents of symlinks-farm have absolutely
nothing to show other than "I'm hoping". What are we even doing here?
> > And then we have a large body of people who simply
> > want this to be over and not having to thing about /usr-merge
> > consequences anymore.
>
> Well, of course it is very tempting to declare the matter as settled
> and hope that it will go away. I too want an end state where we will
> eventually be able to forget about all of this.
The matter is settled, and was settled long ago to boot.
> Or to put it another way, the delays to completion of this project
> have not been due to the political opposition,. They have been
> because the project encountered technical problems. Problems whose
> existence was predicted by subject matter experts but dismissed at the
> time as FUD. Problems which were apparently not regarded as real by the
> non-expert decisionmakers on the TC. Problems which still remain in
> large part unresolved, albeit in some caes "mitigated".
These so-called problems were and remain all FUD. We were told
installations would break, machines would stop working, the sky would
fall on our heads. It's been years, and no such thing has happened.
There's a couple of dpkg bugs that are being worked around, but that's
hardly surprising given the sorry state of that tool.
> > > Aliasing is EBW, and "Only use canonical names" is not good enough
> > > ==================================================================
> > >
> > > There is basically one underlying technical reason for preferring the
> > > un-aliased usrmerge approach: aliasing directories in this way leads
> > > to great complication in file management, especially in package
> > > management software and in individual packages.
> >
> > I'm not sure I follow this argument precisely.
>
> This argument is basically drawing together the common factor in the
> DEP-17 problem list.
>
> > these complications mostly affect ourselves and
> > our package management while end users are mostly unaffected.
>
> I think "package management" is the wrong term here. It's not just
> our tools and packages that are affected. All forms of operating
> system management are affected: anything that changes the software,
> and not just its configuration.
>
> Affected tooling includes not just our .debs, but also remote
> configuration management systems like Ansible, image construction
> (Dockerfiles), and 3rd-party software installation progrmas (including
> both 3rd-party .debs and 3rd-party script-based installation systems).
>
> And yes, actual *end users* (especially of something like Ubuntu) are
> largely unaffected because they don't do much operation system
> management. Regarding Ubuntu specifically: Ubuntu's approach to 3rd
> party softare during upgrades is (for very plausible reasons) quite
> sledgehammery.
Users don't do OS management? Is this some kind of joke? Have you
never heard of this little thing called 'the cloud'? It's kinda of a
big thing in the past decade in case you missed it. Orchestration and
OS management happen daily on a scale that non-Windows OSes have never
seen before (barring smartphones, but that's a world of its own). Of
course Debian is a rounding error there, but Ubuntu is most certainly
not, and guess what, Ubuntu has finished usr-merge a few years before
us. And so have other distros that represent the vast, vast majority
of cloud instances.
The reality, beside the usual evidence-free narrative that
symlinks-farm proponents have gotten us used to (where is the evidence
of Ansible/Docker/whatever issues? *crickets* ), is of course that the
only issue here is in dpkg and its deeply ingrained brokenness, which
makes it mostly a build-time packaging problem, that is being
successfully solved as such.
> > > Naming files by their canonical names will have to be done everywhere.
> ...
> > This seems overgeneralized to me. [...]
>
> Well, this is a key part of the problem. IMO we need to be able to
> state simple and clear rules, which when followed, will result in
> reliable construction of working software systems.
>
> We build our systems by building on layers of abstractions. Layers of
> abstraction allow us to narrow the scope of our consideration. In our
> systems, the filesystem is a pervasive abstraction. A filesystem with
> directory aliasing is a much more complicated and subtle abstraction.
>
> So we can either write broad and restrictive rules ("overgeneralised"
> as you put it), or subtle and complicated ones.
>
> I would be much more convinced that all of this isn't a problem, if
> there existed, and had been formally adopted (eg by existing in some
> manual somewhere) a short and clear set of rules about what is and
> isn't allowed - not just rules for us within Debian, but rules for
> everyone, everywhere, referring to and modifying files.
>
> I think one reason that hasn't been done is that it's hard. Another
> is perhaps that some of the rules required for successful and reliable
> operation contradict some of the ostensible goals of the aliasing.
The rule is there, and it is simple and straightforward: OS vendor
stuff goes in /usr. End of rule.
> > > Spotting and mitigating violations is hard
> > > ------------------------------------------
> > >
> > > We do not currently have good tooling that will spot violations of
> > > this rule. It's not clear precisley what the right behaviour of our
> > > tools would be; we need to alert *the right set of users* to the
> > > mistakes, and *with the right level of severity*. Many of our key
> > > tools don't have a good way to produce "critical warnings". The
> > > consequences of violations are unpredicatable and can depend on event
> > > ordering. But they can be very severe. So we are creating a source
> > > of bad heisenbugs.
> >
> > We already have the Debian Usr Merge Analysis Tool available at
> > https://salsa.debian.org/helmutg/dumat and its output at
> > https://subdivi.de/~helmut/dumat.yaml. As explained on d-d@l.d.o, I want
> > to turn those findings into automatic RC bugs. Does that alleviate your
> > concerns to some extent?
>
> That's certainly helpful for the transition now. Are we going to
> maintain this or something like it indefinitely ?
>
> But I don't think it addresses the point I intended. We need to be
> able to spot when a user installs a .deb, that they got from
> "somewhere", when a directory-aliasing thing is going to go wrong.
No, we do not, just like we do not need to "spot" when a user installs
a .deb that does anything else that is not supported, in the many ways
that dpkg has to break a system - like a broken maintainer script, to
name the obvious. Third party packages shipping to / are broken, and
can be ignored, and if they decide to move things around without care
it just means they are even more broken. But guess what: those cases
do not exist, it's yet another case of evidence-free narrative from
the symlinks-farm camp.
> > > Also, we only have direct control over the behaviour of our own
> > > packages, images, etc. that we (Debian) ship. [...]
> >
> > This is very true. On the flip side, shipping files in /bin or /lib is
> > usually done by essential packages or packages required for booting.
> > Neither of these topics are very popular in 3rd party software.
>
> But, a goal of the directory aliasing is to be compatible with other
> systems, so that 3rd party software is *allowed* to refer to, and,
> presumably, ship files in /bin (because "there's no difference now,
> right?").
Just like that very same 3rd party software is *allowed* to put 'rm
-rf /' in postinst, and dpkg will happily raze your filesystem to the
ground. Like your example, this one is also fictional.
> > > Violations of the "only use canonical names" rule are required
> ...
> > > Now, those references are almost all in "immutable" contexts, where it
> > > doesn't actually matter, since the file is in fact available by the
> > > non-canonical name. However, this introduces a new implied rule:
> > > it becomes a bug to take a filename you see in a place where the file
> > > is being *read*, and apply it in a context where the file is going to
> > > be *updated*.
> >
> > This is accurate and a nice way to describe it. Are there actually that
> > many tools that need to update files of a Debian installation?
>
> I listed some above. There are going to be a lot more.
No, you didn't list anything real. Just some evidence-free
speculation, but that doesn't count as 'listing'.
> We, here in Debian, are likely to have a much more restricted view of
> what really goes on out there in the world. After all when *we* want
> to modify a Debian system we usually do it by modifying .debs,
> modifying .dsc's within Debian. That is easy for us but a lot harder
> for our downstreams and users.
>
> Our downstreams (of all kinds) are are more likely to use other
> tooling (of all kinds).
>
> > This feels vague to me. Given the general adoption of the aliasing
> > approach in other distributions, I'd expect that we have at least
> > anecdotal evidence. Are you aware of any?
>
> No. But most of my closer friends, and most of the systems I maintain
> myself (especially the more complicated ones) have so far managed to
> put off having usrmerge, in the hope that the technical situation
> would become less buggy in the future. And, frankly, the political
> situation is so toxic that I expect many people who have trouble with
> usrmerge will avoid sharing it with anyone but their close friends.
Ah well, why do we even bother with popcon at all these days I wonder,
when we could just ask your mates down the pub! But worry not, we will
solve that "put off having usrmerge" situation soon enough for you and
your mates and any other straggler that may or may not exist.
> It took a significant externally funded research programme to document
> the DEP-17 list. Let us not say, once more, that categories of bug
> predicted by an expert are probably imaginary or rare.
They are not rare, they are non-existing. It's just narrative.
Otherwise, please provide some evidence, I'll wait - we've waited 10
years for any and just saw tumbleweeds, but I guess today might be our
lucky day after all.
> > > Looking towards the future
> > > --------------------------
> > >
> > > It seems to me that directory aliasing will continue to be a source of
> > > very annoying bugs indefinitely, well after the transition is fully
> > > complete. In another 20 years we'll still be debugging strange
> > > installation breakage that will turn out to be due to directory
> > > aliasing.
> >
> > I very much hope that this is not where we're heading. Quite contrary,
> > over the past 20 years, we've had packages move between / and /usr and
> > catching up with where stuff was located. I expect that we get rid of
> > this bug class and trade it for that other bug class that you are
> > describing. The big question then is the frequency of each class.
>
> We can get rid of the former bug class simply by moving everything to
> /usr, once. We'll *experience* those bugs now, but if we do it as
> part of a coordinated programme we can have tooling to spot it. When
> that transition is complete, those bugs won't arise any more.
That's DEP-17
> > Speaking for myself without any Freexian or CTTE hat, I mainly want the
> > transition to be finished such that I can spend brain cells on more
> > useful (to me) matters.
>
> As I say, I don't think the directory aliasing situation is ever going
> to be finished. We can revert it, or have it forever be weird.
More evidence-free narrative. Where is this "forever-weird situation"
in SUSE, Ubuntu, Arch, Fedora, ...? Nowhere to be seen.
> Simon McVittie writes ("Bug#1050001: Unwinding directory aliasing"):
> > This is not merged-/usr with the meaning used by the technical committee's
> > past resolutions,
>
> (Assuming you're not just arguing terminology:)
>
> I am indeed asking the TC to change its mind. We now have a great
> deal more information than we did. We now know for sure that
> Guillem's concerns were not just FUD. I think it unlikely that the TC
> in 2019 would have decided as they did, if they had had the list from
> DEP-17 in front of them.
On the contrary, we now know for sure it was all FUD. All new machines
have the new layout for 3 stable releases, all machines have the new
layout for one stable release, nothing broke. Nothing fell from the
sky. The earth didn't open up and swallow any user machine. It's been
years, surely by now symlinks-farm proponents should be able to
provide ONE real case as evidence?
> Ansgar writes ("Bug#1050001: Unwinding directory aliasing"):
> > Why should we invent again another, incompatible filesystem layout?
>
> I'm not inventing a new filesystem layout. What I am proposing is in
> fact the simple composition of tried and tested layout principles:
> (i) the traditional Unix filesystem layout + (ii) file motion
> (from / to /usr), which is a thing we know how to do and can do
> routinely.
It's not strictly new in the sense that the symlinks-farm approach you
are proposing is old, and a known failed strategy that was attempted
only once, did not work, and never again. But who cares about real
world evidence, right?
Reply to: