Re: testing for rootfs vs. /usr reproducibility regressions
On Thu, 19 Aug 2021 at 10:06:27 +0200, Helmut Grohne wrote:
> On Tue, Aug 17, 2021 at 05:19:06PM +0100, Simon McVittie wrote:
> > If we want to make buildd chroots merged-/usr any time soon, then I
> > think we need to say this class of bugs is RC for bookworm.
>
> For a moment, let us assume perfection of this plan: Reproducible builds
> identify all of the cases that need to be fixed. We bump them to RC
> bugs. We fix them all. Then we change unstable to be merged-/usr and
> we're done. Are we? We still need to support partial upgrades from
> bullseye to bookworm, so - as you pointed out earlier - all packages in
> bookworm will have to keep this support property. However, our
> defaulting to merged-/usr makes it impossible for reproducible builds to
> test it as producing an unmerged chroot is no longer possible.
> Effectively, the unmerged case will be untested and as a consequence
> will be broken.
Is this the scenario you're concerned about?
* We reach a point where every package in bookworm builds identically
(or with only non-/usr-related variation) in non-merged-/usr and
merged-/usr chroots, i.e. the bugs falling into classification
https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
have all been fixed
* We deploy an automatic transition mechanism that makes all bookworm
installations, including chroots, convert to merged-/usr (for example,
making usrmerge transitively Essential would be one possible
implementation, although perhaps not the best one)
* Now our buildd chroots are all merged-/usr, and additionally we can
no longer easily test for /usr-related reproducibility
* The foo maintainer uploads foo_1.0 that has a regression, introducing a
*new* /usr-related unreproducibility (for example it might end up
hard-coding /usr/bin/sh when built in a merged-/usr environment, rather
than the /bin/sh that is correct for traditional environments, even
though earlier versions of foo did not have that bug)
* The regression is not detected because we can't easily test for it any more
* A user carries out a partial upgrade in which foo_1.0 is upgraded before
whatever part of the system is responsible for carrying out the transition
mechanism
* Now foo doesn't work on that user's system
I agree that's a potential failure mode, although the window for regression
is not necessarily very long (foo would have to regress after the automatic
transition is already in place).
> For this plan to work, we will have to support unmerged chroots until
> the end of bookworm, which appears to contradict the premise we started
> with.
One way we could escape from this would be to observe that the earliest
point at which it is OK for package maintainers to rely on merged-/usr is
when the bookworm+1 development cycle opens, therefore it is OK to have
a mechanism for bookworm chroots/containers to be special-cased to opt out
from the merged-/usr transition, as long as those chroots/containers will
never be upgraded to bookworm+1 (because in practice we tend to discard
and re-bootstrap chroots/containers rather than upgrading them).
Conceptually, those chroots/containers are stuck in the "upgrade from
bullseye to bookworm almost but not quite complete" state, forever.
If desired, we could declare this to be formally unsupported, but
make reproducible-builds do it anyway, purely as a way to get test
coverage. This would be conceptually similar to the way people regularly
do archive rebuilds with a gcc that is not supported as the default yet,
in order to get the information we need before we can make it the default
- but in reverse, with the "old" configuration being the one that is
formally unsupported.
reproducible-builds could do something like this:
* for build1: do a merged-/usr chroot/container as (will become) usual
* for build2: bootstrap a chroot/container with --no-merged-usr, and set
whatever flag opts-out from the transition
or (matching what they do for buster/bullseye):
* bootstrap a chroot/container with --no-merged-usr
* for build1: use it as-is (opting out from the transition if necessary)
* for build2: enable the transition and let it happen, or install usrmerge
manually
Or, if the transition ends up being done during boot, either in the
initramfs or in normal user-space, then chroots/containers would
"naturally" not transition without manual action - similar to the way
chroots/containers don't pick up kernel or initramfs behaviour changes
like the changed default for /proc/sys/kernel/unprivileged_userns_clone
in bullseye, because their kernel is not their own.
smcv
Reply to: