Re: Supporting alternative zlib implementations
* Guillem Jover <guillem@debian.org> [2024-09-25 01:55]:
> Hi!
>
> On Wed, 2024-09-25 at 00:39:10 +0200, Fay Stegerman wrote:
> > * Guillem Jover <guillem@debian.org> [2024-09-24 17:45]:
> > > Personally, I think fully migrating from zlib to zlib-ng would sound
> > > great (even for trixie), but I guess we can take it slow if you do not
> > > feel confident or have concerns over this.
> >
> > As using an alternative zlib implementation could impact Reproducible Builds
> > [1], I would recommend taking that into consideration before deciding on this
> > kind of change.
>
> Ah, this is related to something I wanted to mention too and forgot.
>
> I don't think the specific case you mention is in itself a concern for
> Debian, because we only guarantee reproducibility given the same inputs,
> which includes the set of packages and their versions that were used
> when building the binaries. So if there was a switch, those would end up
> being recorded as well, and used when reproducing the outputs. And this
> could also happen with a newer version of zlib itself.
>
> The problem though is, that because the compressed stream is going to
> change, that can make certain test suites fail if we perform this
> switch, which I think would be the main fallout that we'd see from
> this and would need manual fixing, although I assume Fedora has probably
> handled most of these already. For example when I added explicit
> zlib-ng support to dpkg, I had to fix its test suite to parametrize
> sizes for test artifacts.
Whilst it indeed may not affect the reproducibility guarantees for Debian
packages themselves, it does affect being able to use a Debian system for
Reproducible Builds of other software for which the reference artefacts were
built with regular zlib and thus can no longer be reproduced on Debian if that
uses a different zlib implementation (so far I've only encountered the reverse,
which seems relatively rare -- for now).
For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention. Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].
AFAIK all rebuilders (including my own [3]) for Android APKs use Debian base
systems, so this could cause quite a bit of breakage for Reproducible Builds
within that ecosystem, which is something I would like to avoid (or at least
have a decent workaround for -- e.g. being able to easily choose between
multiple zlib implementations during runtime in my Python tooling would be
great).
As you point out, we've been lucky that zlib has remained backwards-compatible
for a long time (even though it doesn't provide any guarantees of that AFAIK).
Which also makes me wonder how much more likely zlib-ng might be to produce
different compressed streams between different versions or using different
hardware (configurations).
There might also be issues with reproducibility of Debian packages themselves if
e.g. zlib-ng output can differ on different hardware (e.g. number of cores) even
with an otherwise identical build environment. At the very least I think it
would be good to know how all this could be affected (and how likely things are
to remain as stable as zlib has been so far) before making a decision to switch.
> I think it would be pretty easy to at least see the extent of this
> fallout by performing a mass rebuild for packages build-depending
> on zlib1g-dev with a zlib-ng version.
- Fay
[1] https://tracker.debian.org/pkg/apksigcopier
[2] https://github.com/obfusk/reproducible-apk-tools
[3] https://github.com/obfusk/rbtlog
Reply to: