On Mon, Dec 04, 2023 at 01:13:43PM +0100, Helmut Grohne wrote:
> David Kalnischkies made me aware […]
Oh, did he? I think he wanted to tell you something else… 😉
As IRC seems to be really bad at transporting complicated things (who
had guessed?) and I need to sort my thoughts anyhow let me recount the
last few days in a lengthy mail… perhaps that helps me and whoever else
might be interested.
Disclaimer: /me is an independent APT developer for ~14 years aka
super biased if the topic is upgrades and APT.
Content warning: Gory details of APT and dpkg internals are described
in text. Reader discretion advised.
First of, I care only about APT (= meaning everything using libapt, be
it apt, aptitude, unattended-upgrades or some python-apt script), not
about someone who believes they could perform an upgrade from bookworm
to trixie by hand with dpkg¹. I mean, APT can calculated it, so there
obviously is a way, but its so tedious for a human being that nobody
does that. Sure, there are people who believe "dpkg -i *.deb" would
work, but a) that wouldn't be affected by this problem to begin with and
b) it doesn't work as you have to spell out pre-depends, the order in
which the debs are given is important and last but not least you have to
work around a few things in dpkg (e.g. #832972, #844300). So whoever
believes they can do it without APT are probably lying to themselves
or at the very least wasting a lot of time such experts could use far
better to improve APT and/or dpkg for the benefit of all…
Source: I stumbled over many bugs while trying to simplify APT down to
that dpkg call more than seven years ago, so that is not even a new
thing and so not even worthy of grabbing your pitchforks because
I supposedly impose new things you haven't adapted to yet…
The release notes are actually saying you have to use 'apt', so I am
affording a significant amount of leeway here talking about APT.
(bootstrapping and such stuff which gets away with not using APT isn't
upgrading anything, so this problem with Conflicts is of no concern)
So, after clearing that one up, lets focus on the issue at hand:
APT tells dpkg about all removes it will eventually schedule ahead of
time so technically all (well… some?) Conflicts your yaml includes could
be made to exhibit the problem, but APT usually always makes the removal
of a package explicit as well and in that case you/we/usr-merge is fine.
There is one exception we have to talk about: If a package is scheduled
to be removed in one dpkg call and in the next one unpacked. I figured
out today that I implemented that sorta by accident… actually, I was
working on crossgrades (= a package changing architecture in an upgrade,
which contrary to common usage of the word is at least for APT also the
case for all <-> any and as such can happen even on a single architecture
system) and accidentally its also catching temporary removals which
don't change architecture, but are also removing a package before
unpacking it (Its Debians fault for trusting me with this stuff…).
While the former might be interesting to look at as another source of
esoteric problems, lets focus on the temporary removals here:
Unversioned conflicts [usually] do not cause temporary removal. They
cause "permanent" removal as the packages aren't co-installable (yes,
I mentioned you could conflict on a provides from bookworm which is
removed in trixie. I don't think that actually happens in reality).
A versioned conflict is discouraged by policy and by lintian. Okay, we
need it here, so: It can cause it, but only if the conflict is mutual
but the packages still co-installable on trixie AND they were also
co-installable in bookworm (and for it to actually happen, they have
to be installed both from bookworm on the user system of course).
If "pkga conflicts pkgb (<< 2)" and "pkgb depends pkga (>= 2)" (like in
a package rename perhaps) APT can just "unpack pkgb pkga" and all is
fine. The problem you came with was instead "pkga conflicts pkgb (<< 2)"
and "pkgb conflicts pkga (<< 2)" (its the same if ONE of them is
a breaks, but not if both are) as APT can't just unpack them it has to
remove one of them before unpack both. That is what I called a temporary
removal as the package is only removed for a very short time.
But that short amount of time is already too long if the packages
involved are essential, which I suppose is the reason for §6.6 as if APT
(as it currently does by happy accident as described above) just tells
dpkg that it is allowed to deinstall one of them and unpacks both dpkg
can do its §6.6 dance to avoid the actual removal from disk. Yeah!
I implemented a generic improvement by accident instead of a bug!
Sadly, for usr-merge we need the removals to happen explicitly as dpkg
can't untangle the aliasing. The barrier idea achieves this by the way
of pre-depends as APT has to spell out pre-depends for dpkg aka
it has to configure the barrier package before it can unpack the ones
pre-depending on it, so APT schedules "remove pkga; unpack barrier;
configure barrier; unpack pkga pkgb". remove and unpack of pkga are not
next to each other, so this is "clearly" not a crossgrade and so the
removal remains explicit.
So, in more natural words the situation might be described as:
"pkga and pkgb need both be installed in bookworm. The maintainer moves
a file from /lib to /usr/lib while also moving the file from pkga to
pkgb in trixie and keeping the ability to co-install pkga and pkgb"
Ability here means that they don't have to be installed together.
I mention this explicitly as moving a file from -bin to a -data package
is usually not affected as the -bin will usually depend rather than
break the -data package which eliminates the need for a temporal
removal in general.
So, I have some hope that the actual amount of packages which
need this sort of barrier trickery is countable with two hands
(and I don't mean in binary). It happens rather seldom in normal
upgrades, but usr-merge changes Breaks to Conflicts, so I
suppose the number is not zero like it usually is.
> barrier package approach can be used here, but it requires each provider
> to have its own barrier package rather than one central barrier package.
For the trick to work the amount of barrier packages doesn't matter at
all and there are no real requirements except that the package it
conflicts with is not (pseudo)essential (or protected), but APT
complains loudly if that would be the case so people would notice…
Julian and I were suggesting a single one as that /might/ work better
with APTs ordering code. If it really does, we will know only on rollout
as no amount of testing can actually replicate the ingenuity of the
maintainers of 60000+ package with 2+ years each of coming up with
interwoven dependency-linking between them all.
My personal opinion was to not have it be usrmerge/usr-is-merged as
especially the former has dependencies and dependencies mean they could
theoretically form a loop. I think the more detached it is the better.
Prior art in the form of multiarch-support suggests it can work,
although I think that had only the Pre-Depends pointing at it (but a
massive amount of them) and not the Conflicts from it…
I mentioned in passing that we sorta could do both by having a barrier
package provide a bunch of barrier-for-x and have packages pre-depend on
those. Its another layer of indirection, but it should work™.
A very tiny reason I favor a single/very few barrier packages is that
they appear in the NEW packages listing of APT and that just looks silly
if its many… (yes, THAT is the problem here to be concerned with… 😉)
> For versioned conflicts (where the conflicted version generally is in
> bookworm and not in trixie), a single central barrier package might do,
> but we can also start with one and split it up later if providing
> multiple barriers from the same binary package initially. For instance,
> the files diverted by molly-guard would require a barrier package that
> also handles bfh-container and progress-linux-container on one side and
> systemd, finit, kexec-tools, runit and sysvinit on the other. Such
At the time of writing of this mail none of those packages exhibits the
required negative dependency setup in unstable. I suppose at least
molly-guard will be if a version >> 0.8.2~ enters unstable which
breaks/conflicts versioned with systemd-sysv (I only looked at the
dependencies, not if the file moving is happening, I just assume it is
or we would probably not speak about it). The rest are unversioned
in systemd-sysv 255~rc4-2 and hence not co-installable. I suppose you
are working with unreleased versions.
I haven't looked at a lot of other examples from the yaml, I would
predict most of them to be unversioned conflicts through and as such
to be unaffected and even many of the versioned conflicts to have
redeeming qualities disqualifying them for temporal removal trigger.
> If we have a choice between introducing a barrier package and […]
Yes. If its reasonably easy and contained dealing with it at the
problems source (= individual packages) should be preferred.
I have some trust in the barrier idea, but its a relatively high-level
trick with drawbacks which doesn't always work (e.g. for essentials).
It is also our usual approach even IF we discover that a package runs
into a bug in APT/dpkg – the package has to somehow work around it
as we usually can fix bugs only for stable+1 (well, given we do this
because dpkg doesn't do aliasing, I suppose I have proven my point).
Best regards
David Kalnischkies
¹ Yes, I know of dselect and cupt. They don't even support M-A. Now have
a guess who implemented that ~14 years ago in APT. I warned you that
I am biased, didn't I? cupt doesn't use selections as far as I know
and dselect uses them, but calls apt for the actual work if
dselect-upgrade is any indication… haven't used either so no promises.
Attachment:
signature.asc
Description: PGP signature