Re: Adding support for LZIP to dpkg, using that instead of xz, archive wide

[Guillem Jover <guillem@debian.org>]

On Sun, 2015-06-14 at 14:08:24 +0200, Thomas Goirand wrote:
> On 06/14/2015 05:46 AM, Guillem Jover wrote:
> > Well if you want reproducible output, then use the same tool version.
> 
> That's not possible: Jessie, Sid and Trusty don't have the same version,
> and we need to generate the orig.tar file in all of them. The
> contributors for the Debian OpenStack packaging are mostly using Ubuntu,
> and they need to keep a workflow with the orig.tar file in the Git
> repository.
>
> I did tell them to just get the file from the Debian archive, but it
> doesn't work. One of the reason it doesn't is because sometimes, they
> upload first to Ubuntu, and then I do in Debian, and we end up with
> different orig.tar.xz files, meaning it's hard for them to sync back
> with Debian. I would like this to not be an issue anymore.

From where I'm sitting it all pretty much looks like a self-inflicted
problem. Upstream does not believe in releasing source tarballs (so
each user has to generate them from git-archive, which should be
considered inherently non-reproducible across different versions), and
then for packaging you want to use a pristine-tar workflow w/o using
pristine-tar, nor by creating a single common upstream source tarball,
nor by communicating/coordinating the creation of the first upstream
source tarball and reusing that on the other distro.

> > That's the equivalent of expecting that using a different gcc version
> > will give you the same output.
> 
> I fail to see what gcc and a lossless compressor have in common.

A lossless compressor is not defined in terms of it needing to
generate the same compressed bistream, but in that it should produce
the same output when uncompressed.

gcc and a lossless compressor are similar in that they might produce
different "bitstreams" as long as the result is correct, for example
by performing optimizations that improve speed or reduce size.

> > As long as the bitstream is compatible with previous versions, I don't
> > see it as a problem
> 
> The problem, I just explained it: I can't use xz in a pristine-tar like
> workflow, because it wouldn't reproduce the same output. And I'd like to
> use something better than the 20 years old gzip.

See Felipe and Russ replies. And I don't think you've explained why you
cannot use pristine-tar, which was implemented precisely to overcome such
problems.

> > And how does that have anything to do with what gets packaged in Debian.
> > For Debian you only need to generate it once, why would you want to
> > generate it anew every time you build a new Debian revision instead
> > of just reusing the same tarball that is on the archive, if you don't
> > keep source tarball releases around?
> 
> See above. It's a pristine-tar like workflow. Your question is
> equivalent to: "why do people use pristine-tar?". The answer is: because
> it's convenient to just use git, without having to look into the Debian
> archive.

No, my question would actually be: why are you not using the tool that
was designed to do what you want to do?

Personally, which is besides a bit the point here, I think pristine-tar
is an interesting idea, but I find it pretty fragile and requiring an
ongoing amount of hacks to keep it going over time, so not something I'd
rely on for my data for long-term storage. For more details, see
pristine-tar O: bug #737871. I'm also part of the Cult of the Tarball,
and the Cult of no Autogenerated Cruft in a VCS.

> And by the way, xz wouldn't be usable with pristine-tar for the
> same reason.

If that is based on actual usage and pristine-tar is not working for
you, the correct solution is to fix it.

> > Adding lzip support for source packages *might* make some sense, as
> > I pointed out in the bug report. But doing so does have a very high cost:
> > 
> >   <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_compressors_for_.dsc_packages.3F>
> 
> I do understand the cost. But there's a valid reason. If you believe
> there's something better than lz, with the same properties, and that we
> had support for it, I'd happily adopt it. It is just that xz doesn't
> work right now, and most likely will break again in future versions of
> xz-utils.

It's not a matter of there being something better than lzip, I think
your expectations are unrealistic. Either lzip has a guarantee that it
will not change its bitstream ever again (does it?), which implies that
part of the compressor cannot ever be improved anymore, which makes it
quite unintersting as it's stale on arrival, or would require users to
tune parameters manually, removing simplicity, one of it's claimed
advantages over xz-utils; or it can change and then it loses the
property you seem to believe that you want from it.

The same applies to gzip, bzip2, tar, or even git versions (see the
above mentioned prsintine-tar O: bug), all those have changed and can
change in the future.

Regards,
Guillem