[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#829100: marked as done (lintian: [patch] Warn about over-eagerly xz-compressed data.tar.xz)

Your message dated Tue, 16 Jan 2018 01:20:10 +0000
with message-id <E1ebFv8-0003oA-Fs@fasolo.debian.org>
and subject line Bug#829100: fixed in lintian 2.5.69
has caused the Debian Bug report #829100,
regarding lintian: [patch] Warn about over-eagerly xz-compressed data.tar.xz
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
829100: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=829100
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message --- 
Package: lintian
Version: 2.5.45
Severity: wishlist
Tags: patch

Dear Maintainer,

as not known to everybody, xz's higher compression levels have -
besides improving compression of big files - the side effect of taking
a lot of memory for the dictionary, even when unpacking. There is
however no sense in using a compression level that (roughly) takes
more DictSize than the size of the uncompressed file. [1] has a
discussion on this,

In other words,

| override_dh_builddeb:
|     dh_builddeb -- -Zxz -z9

in the traceroute package triggered an OOM upon installation on an
embedded hardware with 128MiB RAM since ...

| $ ar x traceroute_1%3a2.0.20-2+b1_armel.deb data.tar.xz
| $ xz --list --verbose --verbose data.tar.xz
| (...)
|   Compressed size:    47,9 KiB (49.056 B)
|   Uncompressed size:  130,0 KiB (133.120 B)
| (...)
|   Memory needed:      65 MiB
| (...)

... it caused an allocation of 65 Mibyte for nothing on an also
otherwise busy computer.

In my opinion lintian is the right place to place a warning about such
unncessary ressource usage.

The patch attached is just a proof of concept and not ready for
production yet, especially since data.tar.xz is unpacked (and later
removed) to the current working directory.

Let me know if you consider such a check a good idea, then I'll do the
final polishing and sane error handling. Also the alarm threshold will
probably need some reconsideration.

Example output:

W: traceroute: overeager-compression-for-data-tarball 65.0 MiB RAM required for 0.1 MiB uncompressed data

Aside, does the lab provide a good place for extraction, or should I
just use tempdir?

    Christoph

[1] https://www.mirbsd.org/permalinks/wlog-10_e20130104-tg.htm


-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.4.13 (SMP w/4 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Init: unable to detect

Versions of packages lintian depends on:
ii  binutils                          2.26-12
ii  bzip2                             1.0.6-8
ii  diffstat                          1.61-1
ii  file                              1:5.28-1
ii  gettext                           0.19.8.1-1
ii  hardening-includes                2.8+nmu2
ii  intltool-debian                   0.35.0+20060710.4
ii  libapt-pkg-perl                   0.1.29+b5
ii  libarchive-zip-perl               1.57-1
ii  libclass-accessor-perl            0.34-1
ii  libclone-perl                     0.38-1+b1
ii  libdata-alias-perl                1.20-1+b1
ii  libdpkg-perl                      1.18.7
ii  libemail-valid-perl               1.198-1
ii  libfile-basedir-perl              0.07-1
ii  libipc-run-perl                   0.94-1
ii  liblist-moreutils-perl            0.413-1+b1
ii  libparse-debianchangelog-perl     1.2.0-8
ii  libperl5.22 [libdigest-sha-perl]  5.22.2-1
ii  libtext-levenshtein-perl          0.13-1
ii  libtimedate-perl                  2.3000-2
ii  liburi-perl                       1.71-1
ii  libyaml-libyaml-perl              0.41-6+b1
ii  man-db                            2.7.5-1
ii  patchutils                        0.3.4-1
ii  perl                              5.22.2-1
ii  t1utils                           1.39-2
ii  xz-utils                          5.1.1alpha+20120614-2.1

Versions of packages lintian recommends:
ii  dpkg                                 1.18.7
pn  libperlio-gzip-perl                  <none>
ii  perl                                 5.22.2-1
ii  perl-modules-5.22 [libautodie-perl]  5.22.2-1

Versions of packages lintian suggests:
pn  binutils-multiarch     <none>
ii  dpkg-dev               1.18.7
ii  libhtml-parser-perl    3.72-1
ii  libtext-template-perl  1.46-1

-- no debconf information


diff --git a/checks/deb-format.desc b/checks/deb-format.desc
index 85b9a7a..add7893 100644
--- a/checks/deb-format.desc
+++ b/checks/deb-format.desc
@@ -92,3 +92,13 @@ Info: The data portion of this binary package uses a non-compressed
  .
  Except if data is non-compressible, use gzip for
  maximum compatibility and speed, and xz for maximum compression ratio.
+
+Tag: overeager-compression-for-data-tarball
+Severity: normal
+Certainty: certain
+Info: The data portion of this binary package was xz-compressed with
+ a compression level above reason. Creating and also unpacking it will
+ use a lot of RAM without any benefit.
+ .
+ Reduce the compression level to a value where the uncompressed size
+ is not bigger than the related dictionary size. See xz(1) for details.
diff --git a/checks/deb-format.pm b/checks/deb-format.pm
index e0b750a..841066d 100644
--- a/checks/deb-format.pm
+++ b/checks/deb-format.pm
@@ -164,6 +164,31 @@ sub run {
             } elsif ($type eq 'udeb'
                 && $data_member !~ m/^data\.tar\.[gx]z$/) {
                 tag 'udeb-uses-unsupported-compression-for-data-tarball';
+            } elsif ($data_member eq 'data.tar.xz') {
+                my $success = spawn($opts, ['ar', 'x', $deb, $data_member]);
+                if ($success) {
+                    my $uncompressed;   # in MiB
+                    my $memory_needed;  # in MiB
+                    open(my $fd, '-|', 'xz', '--list', '--verbose', '--verbose', $data_member) or die;
+                    while (my $line = <$fd>) {
+                        chomp($line);
+                        ($line =~ /^\s+Uncompressed size: .* \(([0-9]+) B\)/) and
+                            ($uncompressed = $1 / 1048576);
+                        ($line =~ /^\s+Memory needed:\s+([0-9]+) MiB/) and
+                            ($memory_needed = $1);
+                    }
+                    close ($fd);
+                    # warn if
+                    # - more than 10 MiB is needed for decompression and
+                    # - memory needed is >120% of uncompressed size
+                    if ($uncompressed && $memory_needed &&
+                        $memory_needed > 10 &&
+                        $memory_needed > $uncompressed * 1.2) {
+                        tag 'overeager-compression-for-data-tarball',
+                            sprintf ('%.1f MiB RAM required for %.1f MiB uncompressed data', $memory_needed, $uncompressed);
+                    }
+                    unlink ($data_member);
+                }
             } elsif ($data_member eq 'data.tar.lzma') {
                 tag 'uses-deprecated-compression-for-data-tarball', 'lzma';
                 # Ubuntu's archive allows lzma packages.

Attachment: signature.asc
Description: Digital signature


--- End Message ---
--- Begin Message --- 
Source: lintian
Source-Version: 2.5.69

We believe that the bug you reported is fixed in the latest version of
lintian, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 829100@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Chris Lamb <lamby@debian.org> (supplier of updated lintian package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Tue, 16 Jan 2018 00:41:30 +0000
Source: lintian
Binary: lintian
Architecture: source all
Version: 2.5.69
Distribution: unstable
Urgency: medium
Maintainer: Debian Lintian Maintainers <lintian-maint@debian.org>
Changed-By: Chris Lamb <lamby@debian.org>
Description:
 lintian    - Debian package checker
Closes: 829100 863384 883772 886930 886961 887083 887120 887124
Changes:
 lintian (2.5.69) unstable; urgency=medium
 .
   * Summary of tag changes:
     + Added:
       - insecure-copyright-format-uri
       - package-contains-file-in-etc-skel
       - package-contains-python-tests-in-global-namespace
       - python-package-missing-depends-on-python
       - xz-compression-level-too-high
 .
   * checks/cruft.pm:
     + [CL] Ignore TeX \section (etc.) titles when checking for GFDL
       license. Thanks, Norbert Preining for the report.  (Closes: #863384)
   * checks/fields.{pm,desc}:
     + [CL] Downgrade severity of wrong-section-according-to-package-name
       from "W:" to "I:".  (Closes: #883772)
     + [CL] Thanks to Niels Thykier, update the description of the
       orphaned-package-not-maintained-in-debian-infrastructure tag.
     + [CL] Include the offending uri in the output of the
       vcs-deprecated-in-debian-infrastructure tag.
   * checks/python.{pm,desc}:
     + [CL] Don't emit new-package-should-not-package-python2-module if
       the maintainer justifies its inclusion in the changelog entry.
     + [CL] Improve the description and reasoning for the
       new-package-should-not-package-python2-module tag.
     + [CL] Include the offending package name when warning about
       new-package-should-not-package-python2-module.
     + [CL] Warn about packages that ship Python modules but are missing
       dependencies on any Python interpreter.  (Closes: #887083)
     + [CL] Remark that new-package-should-not-package-python2-module's
       appearance on https://lintian.debian.org/ can be ignored.
       (Closes: #887124)
   * checks/rules.pm:
     + [CL] Allow rules-not-should-not-use data-based tags to capture
       variables and include them in the emitted tag.
   * checks/source-copyright.{desc,pm}:
     + [CL] Warn about insecure "Format:" URIs that reference debian.org.
       Based on a patch by Nicolas Braud-Santoni.  (Closes: #886930)
   * checks/standards-version.pm:
     + [CL] Include the date the Standards-Version was actually released in
       the output of the ancient-standards-version and the
       out-of-date-standards-version tags.
 .
   * data/debhelper/*:
     + [ADB] Refresh.
   * data/files/fnames:
     + [CL] Warn about packages that ship (eg.) test_foo.py files in the
       global Python module namespace.
     + [CL] Emit an error if packages ship files in /etc/skel. Thanks to
       Paul Wise for the suggestion.  (Closes: #887120)
   * data/files/fonts:
     + [ADB] Refresh.
   * data/fields/name_section_mappings:
     + [CL] Ensure that NSS (Name Services Switch) modules are placed in the
       "admin" section. Thanks to Mathieu Parent (sathieu) for the patch.
       (Closes: #886961)
   * data/rules/rules-should-not-use:
     + [CL] Include the assigned value in the Lintian output for the
       debian-rules-should-not-use-DH_EXTRA_ADDONS tag.
     + [CL] Detect overly-compressed xz packages.  (Closes: #829100)
   * data/spelling/corrections:
     + [PW] Add a number of corrections.
Checksums-Sha1:
 e5a76b8ff7528ecdaca8299519d894cb08e029c5 3516 lintian_2.5.69.dsc
 1cc1bf9d3cced913b810fc17920242a9e3352c8f 1465732 lintian_2.5.69.tar.xz
 96d3f2b4c754c3af7133d261d0a97a6f279a7de0 1087936 lintian_2.5.69_all.deb
 ec551c921ac5f208e955f5337bc0547b0f37e256 15880 lintian_2.5.69_amd64.buildinfo
Checksums-Sha256:
 da19fff297f24536807a121a5741160c6a7a0a2eec2afccd6825b7149568a692 3516 lintian_2.5.69.dsc
 9679bdf5f3943fb76894f0b05761f40c7146e81f7714251cd1fc617815993178 1465732 lintian_2.5.69.tar.xz
 f7566b1199d80b6aace9bc11d29312737aab5abb6eea4a2fd9ac95934c2fa4f2 1087936 lintian_2.5.69_all.deb
 bedb6343c865ad60f5b5c86aa84c16f0863a8a2102a87256dc92a3fbdaf7fa81 15880 lintian_2.5.69_amd64.buildinfo
Files:
 bc45e5f97f63a8d374d5fe553d2b1570 3516 devel optional lintian_2.5.69.dsc
 658a23251c0cf8400f40d2592a5c20bc 1465732 devel optional lintian_2.5.69.tar.xz
 ad3314f6ae274185556b92529b15cfe1 1087936 devel optional lintian_2.5.69_all.deb
 41a3e262baa819d19fb43db404bd4117 15880 devel optional lintian_2.5.69_amd64.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEwv5L0nHBObhsUz5GHpU+J9QxHlgFAlpdT7QACgkQHpU+J9Qx
HlidFw/+N2CFymLZeFQ4JiF6eEr2+fRhc84zsl17/dC74+HBMPv/JPbBySzXJEsT
W6DedBER/HWZB8bfLDvj0xUaliJbCUnojfuv0um0EEPko9AFbKA8N3Wlt/7Bnz9l
qxN/+WzWdV7gTf3cXJ6GZNzHA1ISuhReIqdHzrBn/uhiG2y8S4joixPKlYvAxqET
5z0sYYNPJhZmWoomBhKDynQ8MP878DZEq29BcF/Tk5M8aqv+rKONqOUgLhlMDanT
cBu27YRRgnWTnE8kG9juGgduL2ZeHAROSb+aDfkJPYYiMLn4AU1gvXA2ABiyp6pC
pRSIw2QmkKCNtREWXNRbdYwMnZozxhAZFSldMigBcc1nW3G2kDWjs7IgtdPOVLUk
m/KZeE4zOu5XzsQvOGL6E3H6qN7EZ3EvwoOgrcs0Rvts3X4u7kOpU2XYsLz1x9YD
gUioZ2lw9f1GkWsSmlSx8WUshwuRtRQxS6eKANzP2y4xh5lB4vZZy5QISmRpmkhr
W+2k5DiRVNz4boIMd3WHUhbcDaDN1PGqEUPkcpB3jfU/O3l+dDatZmRwHB9+tSUD
QDGnkujKx9zvxFzCgi63hZXRBYD2RItYDlHBdL+6jCVDUjn02QFopeLesgQOHhye
dH/vdF6rSwFhjoqFBH+IN2WeC31ki4eXjzF8JEsgaP3K3BbezBY=
=OlWi
-----END PGP SIGNATURE-----

--- End Message ---
Reply to: