.deb format: let's use 0.939, zstd, drop bzip2

[Adam Borowski <kilobyte@angband.pl>]

Hi!
I've recently did some research on how can we improve the speed of unpacking
packages.  There's a lot of other stages that can be improved, but let's
talk about the .deb format.

First, the 0.939 format, as described in "man deb-old".  While still being
accepted by dpkg, it had been superseded before even the very first stable
release.  Why?  It has at least two upsides over 2.0:

* there's no 10¹⁰ bytes (~9.31GB) limit
  While no package this big is in the archive _yet_ (max being 1⎖652⎖244⎖360
  bytes), both storage sizes and software bloat grow pretty fast, thus we'll
  break this barrier in a few years.  And there's a world outside the
  official archive -- I bet someone already has been burned by this limit.
* it's faster by a small but non-negligible factor.  A task "unpack all
  packages in default XFCE GUI install" gets done by stock dpkg (after
  repacking everything as gzip) 3% faster.

Obviously, 3% is not worth fighting for, but as the size limit needs fixing
anyway...

Alas, while current dpkg handles 0.939 archives well, it supports only two
compressors: gzip and cat.  Neither of them is adequate these days.  Thus,
we'd need to enable others -- which means not being able to unpack new .debs
with old dpkg.  Barring ugly versioned pre-depends on dpkg, that'd require
waiting two release cycles.

So let's pick compressors to enable.  For compression ratio, xz still wins
(at least among popular compressors).  But there's a thing to say about
zstd: firefox.deb zstd -19 takes to unpack:
* 2.644s .xz, stock dpkg
* 2.532s .xz, my tool (libarchive based)
* 0.290s .zst, my tool
* 0.738s .gz, stock dpkg
* 0.729s .gz 0.939, stock dpkg
File sizes being 60628216 gz, 47959544 zstd, 44506304 xz.

XFCE install total: 723M xz, 773M zstd, 963M gzip.

Thus, even though we'd want to stick with xz for the official archive, speed
gains from zstd are so massive that it's tempting to add support for it,
at least for non-official uses, possibly also for common Build-Depends.
The usual objection, "we don't want to bloat the Essential set" doesn't hold
water because 1. libzstd is already a part of the Required set in Buster,
2. a non-default compressor can be dlopened.

Thoughts?

But, the dlopen idea shows a potential victim: bzip2.  Let's kill it.

Stats for Buster's packages:

.deb format:
2.0:    100%

control:
gz      11671
xz      45210

data:
gz      966
xz      55915

With not a single package in the archive still using bz2, removing support
would be reasonable.  It'd be okay to give a clear error message telling the
user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still
unpack historic .debs if need be.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ .globl _start↵.data↵rc: .ascii "/etc/init.d/rcS\0"↵.text↵_start
⣾⠁⢰⠒⠀⣿⡁ mov $57,%rax↵syscall↵cmp $0,%rax↵jne child↵parent:↵mov $61,%rax
⢿⡄⠘⠷⠚⠋⠀ mov $-1,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall↵jmp parent↵child:
⠈⠳⣄⠀⠀⠀⠀ mov $59,%rax↵mov $rc,%rdi↵xor %rsi,%rsi↵xor %rdx,%rdx↵syscall