Package: lintian
Version: 2.5.45
Severity: wishlist
Tags: patch
Dear Maintainer,
as not known to everybody, xz's higher compression levels have -
besides improving compression of big files - the side effect of taking
a lot of memory for the dictionary, even when unpacking. There is
however no sense in using a compression level that (roughly) takes
more DictSize than the size of the uncompressed file. [1] has a
discussion on this,
In other words,
| override_dh_builddeb:
| dh_builddeb -- -Zxz -z9
in the traceroute package triggered an OOM upon installation on an
embedded hardware with 128MiB RAM since ...
| $ ar x traceroute_1%3a2.0.20-2+b1_armel.deb data.tar.xz
| $ xz --list --verbose --verbose data.tar.xz
| (...)
| Compressed size: 47,9 KiB (49.056 B)
| Uncompressed size: 130,0 KiB (133.120 B)
| (...)
| Memory needed: 65 MiB
| (...)
... it caused an allocation of 65 Mibyte for nothing on an also
otherwise busy computer.
In my opinion lintian is the right place to place a warning about such
unncessary ressource usage.
The patch attached is just a proof of concept and not ready for
production yet, especially since data.tar.xz is unpacked (and later
removed) to the current working directory.
Let me know if you consider such a check a good idea, then I'll do the
final polishing and sane error handling. Also the alarm threshold will
probably need some reconsideration.
Example output:
W: traceroute: overeager-compression-for-data-tarball 65.0 MiB RAM required for 0.1 MiB uncompressed data
Aside, does the lab provide a good place for extraction, or should I
just use tempdir?
Christoph
[1] https://www.mirbsd.org/permalinks/wlog-10_e20130104-tg.htm
-- System Information:
Debian Release: stretch/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 4.4.13 (SMP w/4 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Init: unable to detect
Versions of packages lintian depends on:
ii binutils 2.26-12
ii bzip2 1.0.6-8
ii diffstat 1.61-1
ii file 1:5.28-1
ii gettext 0.19.8.1-1
ii hardening-includes 2.8+nmu2
ii intltool-debian 0.35.0+20060710.4
ii libapt-pkg-perl 0.1.29+b5
ii libarchive-zip-perl 1.57-1
ii libclass-accessor-perl 0.34-1
ii libclone-perl 0.38-1+b1
ii libdata-alias-perl 1.20-1+b1
ii libdpkg-perl 1.18.7
ii libemail-valid-perl 1.198-1
ii libfile-basedir-perl 0.07-1
ii libipc-run-perl 0.94-1
ii liblist-moreutils-perl 0.413-1+b1
ii libparse-debianchangelog-perl 1.2.0-8
ii libperl5.22 [libdigest-sha-perl] 5.22.2-1
ii libtext-levenshtein-perl 0.13-1
ii libtimedate-perl 2.3000-2
ii liburi-perl 1.71-1
ii libyaml-libyaml-perl 0.41-6+b1
ii man-db 2.7.5-1
ii patchutils 0.3.4-1
ii perl 5.22.2-1
ii t1utils 1.39-2
ii xz-utils 5.1.1alpha+20120614-2.1
Versions of packages lintian recommends:
ii dpkg 1.18.7
pn libperlio-gzip-perl <none>
ii perl 5.22.2-1
ii perl-modules-5.22 [libautodie-perl] 5.22.2-1
Versions of packages lintian suggests:
pn binutils-multiarch <none>
ii dpkg-dev 1.18.7
ii libhtml-parser-perl 3.72-1
ii libtext-template-perl 1.46-1
-- no debconf information
diff --git a/checks/deb-format.desc b/checks/deb-format.desc
index 85b9a7a..add7893 100644
--- a/checks/deb-format.desc
+++ b/checks/deb-format.desc
@@ -92,3 +92,13 @@ Info: The data portion of this binary package uses a non-compressed
.
Except if data is non-compressible, use gzip for
maximum compatibility and speed, and xz for maximum compression ratio.
+
+Tag: overeager-compression-for-data-tarball
+Severity: normal
+Certainty: certain
+Info: The data portion of this binary package was xz-compressed with
+ a compression level above reason. Creating and also unpacking it will
+ use a lot of RAM without any benefit.
+ .
+ Reduce the compression level to a value where the uncompressed size
+ is not bigger than the related dictionary size. See xz(1) for details.
diff --git a/checks/deb-format.pm b/checks/deb-format.pm
index e0b750a..841066d 100644
--- a/checks/deb-format.pm
+++ b/checks/deb-format.pm
@@ -164,6 +164,31 @@ sub run {
} elsif ($type eq 'udeb'
&& $data_member !~ m/^data\.tar\.[gx]z$/) {
tag 'udeb-uses-unsupported-compression-for-data-tarball';
+ } elsif ($data_member eq 'data.tar.xz') {
+ my $success = spawn($opts, ['ar', 'x', $deb, $data_member]);
+ if ($success) {
+ my $uncompressed; # in MiB
+ my $memory_needed; # in MiB
+ open(my $fd, '-|', 'xz', '--list', '--verbose', '--verbose', $data_member) or die;
+ while (my $line = <$fd>) {
+ chomp($line);
+ ($line =~ /^\s+Uncompressed size: .* \(([0-9]+) B\)/) and
+ ($uncompressed = $1 / 1048576);
+ ($line =~ /^\s+Memory needed:\s+([0-9]+) MiB/) and
+ ($memory_needed = $1);
+ }
+ close ($fd);
+ # warn if
+ # - more than 10 MiB is needed for decompression and
+ # - memory needed is >120% of uncompressed size
+ if ($uncompressed && $memory_needed &&
+ $memory_needed > 10 &&
+ $memory_needed > $uncompressed * 1.2) {
+ tag 'overeager-compression-for-data-tarball',
+ sprintf ('%.1f MiB RAM required for %.1f MiB uncompressed data', $memory_needed, $uncompressed);
+ }
+ unlink ($data_member);
+ }
} elsif ($data_member eq 'data.tar.lzma') {
tag 'uses-deprecated-compression-for-data-tarball', 'lzma';
# Ubuntu's archive allows lzma packages.
Attachment:
signature.asc
Description: Digital signature