Bug#715035: marked as done (lintian: Possible memory optimizations)
Your message dated Wed, 22 Jul 2015 21:37:43 +0000
with message-id <E1ZI1hv-0003jN-Pt@franck.debian.org>
and subject line Bug#715035: fixed in lintian 2.5.34
has caused the Debian Bug report #715035,
regarding lintian: Possible memory optimizations
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)
--
715035: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=715035
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: lintian: Possible memory optimizations
- From: Niels Thykier <niels@thykier.net>
- Date: Fri, 05 Jul 2013 18:38:39 +0200
- Message-id: <20130705163839.14679.11670.reportbug@mikazuki.thykier.net>
Package: lintian
Version: 2.5.13
Severity: minor
Hi,
I have spent a little time looking at the memory consumption of
Lintian. I have already optimized a few trivial cases, but I have
also seen futher potential that may be worth deploying.
Method: I have only looked at the memory consumed by L::Collect at the
end of the run (i.e. the memory data you get with -dddd from the
master branch). I suspect there is also a lot to be gained from not
slurping copyright files (etc.) without any regards to their size in
checks.
Observations so far:
- (sorted_)index is(/are) the primary memory consumer(s).
- fortunately, they share a large part of the memory so the output
of "-dddd" looks worse than it is.
- AFAICT I can tell, memory seems to be "leaked" by Perl allocating
large buffers for the strings[1]. With my test case source:linux,
I suspect we can save about 5MB if we can reduce the size of
these buffers to problem.
- changelog is also pretty expensive and could probably very often be
shared between different L::Collect instances related to the same
processable group. For lintian, it is about 2Mb for both the .deb
and the source. The common "non-sharing" case would be binNMUs or
a bug in the package[2].
- If we can dedup this on a disk level, we are also recovering
disk storage. I haven't checked if this is worth it even for
lintian.d.o. The savings will be a lot less on the disk though,
since Perl/Parse::DebianChangelog spends more memory on it than
its actual size (about factor 4 for lintian's changelog)[3].
I have attached the memory usage information of running lintian with
-dddd (-X man) on some of the linux binaries and their source package,
which can be used as reference.
~Niels
[1] This makes sense if the strings are to be changed later, but
generally this strings will only be read.
[2] E.g. installing the wrong changelog in one of the binary
packages. :)
[3] If we are doing this disk de-duplication, we can probably trivally
apply it to copyright files as well. But back to memory ...
N: Memory usage [source:linux/3.2.20-1]: 35.91 MB
N: -- base_dir: 120.00 B
N: -- binaries: 105.41 kB
N: -- binary_field: 1022.11 kB
N: -- binary_relation: 1513.17 kB
N: -- changelog: 1440.98 kB
N: -- debfiles: 116.00 B
N: -- diffstat: 116.00 B
N: -- field: 109.57 kB
N: -- file_info: 3.92 MB
N: -- index: 29.32 MB
N: -- is_non_free: 116.00 B
N: -- name: 64.00 B
N: -- native: 20.00 B
N: -- relation: 12.07 kB
N: -- relation_noarch: 5.85 kB
N: -- sorted_index: 28.74 MB
N: -- source_field: 1.71 kB
N: -- type: 64.00 B
N: -- unpacked: 116.00 B
N: Memory usage [binary:linux-source-3.2/3.2.20-1/all]: 1462.71 kB
N: -- base_dir: 136.00 B
N: -- changelog: 1440.99 kB
N: -- control: 144.00 B
N: -- control-index: 2.38 kB
N: -- field: 2.00 kB
N: -- file_info: 1249.00 B
N: -- index: 9.39 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 904.00 B
N: -- name: 72.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 4.98 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 8.42 kB
N: -- type: 64.00 B
N: Memory usage [binary:linux-support-3.2.0-2/3.2.20-1/all]: 1476.09 kB
N: -- base_dir: 140.00 B
N: -- changelog: 1440.99 kB
N: -- control: 148.00 B
N: -- control-index: 3.66 kB
N: -- field: 1473.00 B
N: -- file_info: 3.01 kB
N: -- index: 22.93 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 2.45 kB
N: -- name: 76.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1.59 kB
N: -- scripts: 625.00 B
N: -- sorted_control-index: 2.78 kB
N: -- sorted_index: 21.78 kB
N: -- type: 64.00 B
N: -- unpacked: 152.00 B
N: Memory usage [binary:linux-doc-3.2/3.2.20-1/all]: 7.65 MB
N: -- base_dir: 132.00 B
N: -- changelog: 1440.99 kB
N: -- control: 140.00 B
N: -- control-index: 2.38 kB
N: -- field: 1.57 kB
N: -- file_info: 1016.66 kB
N: -- index: 5.03 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 1150.69 kB
N: -- name: 68.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1476.00 B
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 4.95 MB
N: -- type: 64.00 B
N: -- unpacked: 144.00 B
N: Memory usage [binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64]: 9.28 MB
N: -- base_dir: 148.00 B
N: -- changelog: 1441.00 kB
N: -- control: 156.00 B
N: -- control-index: 5.65 kB
N: -- field: 2.21 kB
N: -- file_info: 893.27 kB
N: -- index: 2.78 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 532.94 kB
N: -- name: 80.00 B
N: -- native: 64.00 B
N: -- objdump_info: 4.35 MB
N: -- relation: 7.02 kB
N: -- scripts: 602.00 B
N: -- sorted_control-index: 4.63 kB
N: -- sorted_index: 2.74 MB
N: -- strings: 156.00 B
N: -- type: 64.00 B
N: -- unpacked: 156.00 B
N: Memory usage [binary:linux-manual-3.2/3.2.20-1/all]: 4.65 MB
N: -- base_dir: 136.00 B
N: -- changelog: 1440.99 kB
N: -- control: 144.00 B
N: -- control-index: 2.38 kB
N: -- field: 1.96 kB
N: -- file_info: 338.43 kB
N: -- index: 2.70 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 566.65 kB
N: -- name: 72.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 2.12 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 2.65 MB
N: -- type: 64.00 B
N: Memory usage [binary:linux-manual-3.2/3.2.35-2/all]: 4.75 MB
N: -- base_dir: 136.00 B
N: -- changelog: 1525.74 kB
N: -- control: 144.00 B
N: -- control-index: 2.38 kB
N: -- field: 2.06 kB
N: -- file_info: 339.34 kB
N: -- index: 2.70 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 568.13 kB
N: -- name: 72.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1.81 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 2.66 MB
N: -- type: 64.00 B
N: Memory usage [binary:linux-headers-3.2.0-4-amd64/3.2.35-2/amd64]: 6.86 MB
N: -- base_dir: 148.00 B
N: -- changelog: 1525.75 kB
N: -- control: 156.00 B
N: -- control-index: 3.02 kB
N: -- field: 1.77 kB
N: -- file_info: 691.72 kB
N: -- index: 4.63 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 761.51 kB
N: -- name: 80.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 2.83 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 2.17 kB
N: -- sorted_index: 4.55 MB
N: -- type: 64.00 B
N: Memory usage [binary:xen-linux-system-3.2.0-4-amd64/3.2.35-2/amd64]: 1.50 MB
N: -- base_dir: 152.00 B
N: -- changelog: 1525.75 kB
N: -- control: 160.00 B
N: -- control-index: 2.38 kB
N: -- field: 1409.00 B
N: -- file_info: 815.00 B
N: -- index: 6.22 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 454.00 B
N: -- name: 84.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 2.22 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 5.32 kB
N: -- type: 64.00 B
N: Memory usage [binary:linux-headers-3.2.0-4-rt-amd64/3.2.35-2/amd64]: 6.85 MB
N: -- base_dir: 152.00 B
N: -- changelog: 1525.75 kB
N: -- control: 160.00 B
N: -- control-index: 3.02 kB
N: -- field: 1.78 kB
N: -- file_info: 704.93 kB
N: -- index: 4.62 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 770.92 kB
N: -- name: 84.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 2.83 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 2.17 kB
N: -- sorted_index: 4.54 MB
N: -- type: 64.00 B
N: Memory usage [binary:linux-support-3.2.0-4/3.2.35-2/all]: 1.52 MB
N: -- base_dir: 140.00 B
N: -- changelog: 1525.74 kB
N: -- control: 148.00 B
N: -- control-index: 3.66 kB
N: -- field: 1.54 kB
N: -- file_info: 2.98 kB
N: -- index: 22.93 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 2.45 kB
N: -- name: 76.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1.50 kB
N: -- scripts: 625.00 B
N: -- sorted_control-index: 2.78 kB
N: -- sorted_index: 21.78 kB
N: -- type: 64.00 B
N: -- unpacked: 152.00 B
N: Memory usage [binary:linux-headers-3.2.0-4-all/3.2.35-2/amd64]: 1.50 MB
N: -- base_dir: 148.00 B
N: -- changelog: 1525.74 kB
N: -- control: 156.00 B
N: -- control-index: 2.38 kB
N: -- field: 1525.00 B
N: -- file_info: 800.00 B
N: -- index: 6.20 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 444.00 B
N: -- name: 80.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1.52 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 5.30 kB
N: -- type: 64.00 B
N: Memory usage [binary:linux-headers-3.2.0-4-all-amd64/3.2.35-2/amd64]: 1.50 MB
N: -- base_dir: 152.00 B
N: -- changelog: 1525.75 kB
N: -- control: 160.00 B
N: -- control-index: 2.38 kB
N: -- field: 1.53 kB
N: -- file_info: 818.00 B
N: -- index: 6.23 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 456.00 B
N: -- name: 84.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 2.32 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 5.32 kB
N: -- type: 64.00 B
N: Memory usage [binary:linux-headers-3.2.0-4-common/3.2.35-2/amd64]: 4.46 MB
N: -- base_dir: 152.00 B
N: -- changelog: 1525.75 kB
N: -- control: 160.00 B
N: -- control-index: 2.38 kB
N: -- field: 1.53 kB
N: -- file_info: 374.14 kB
N: -- index: 2.48 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 562.20 kB
N: -- name: 84.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1393.00 B
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 2.44 MB
N: -- type: 64.00 B
N: Memory usage [binary:linux-doc-3.2/3.2.35-2/all]: 7.74 MB
N: -- base_dir: 132.00 B
N: -- changelog: 1525.73 kB
N: -- control: 140.00 B
N: -- control-index: 2.38 kB
N: -- field: 1.67 kB
N: -- file_info: 1017.94 kB
N: -- index: 5.04 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 1152.63 kB
N: -- name: 68.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1393.00 B
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 4.96 MB
N: -- type: 64.00 B
N: -- unpacked: 144.00 B
N: Memory usage [binary:linux-libc-dev/3.2.35-2/amd64]: 2.13 MB
N: -- base_dir: 136.00 B
N: -- changelog: 1525.74 kB
N: -- control: 144.00 B
N: -- control-index: 2.38 kB
N: -- field: 1.79 kB
N: -- file_info: 61.68 kB
N: -- index: 541.39 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 102.33 kB
N: -- name: 68.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1.83 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 531.17 kB
N: -- type: 64.00 B
N: -- unpacked: 144.00 B
N: Memory usage [binary:linux-headers-3.2.0-4-common-rt/3.2.35-2/amd64]: 4.48 MB
N: -- base_dir: 152.00 B
N: -- changelog: 1525.75 kB
N: -- control: 160.00 B
N: -- control-index: 2.38 kB
N: -- field: 1.53 kB
N: -- file_info: 384.59 kB
N: -- index: 2.50 MB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 573.07 kB
N: -- name: 84.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 1393.00 B
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 2.46 MB
N: -- type: 64.00 B
N: Memory usage [binary:linux-source-3.2/3.2.35-2/all]: 1.51 MB
N: -- base_dir: 136.00 B
N: -- changelog: 1525.74 kB
N: -- control: 144.00 B
N: -- control-index: 2.38 kB
N: -- field: 2.11 kB
N: -- file_info: 1237.00 B
N: -- index: 9.39 kB
N: -- is_non_free: 52.00 B
N: -- java_info: 92.00 B
N: -- md5sums: 904.00 B
N: -- name: 72.00 B
N: -- native: 64.00 B
N: -- objdump_info: 92.00 B
N: -- relation: 5.32 kB
N: -- scripts: 92.00 B
N: -- sorted_control-index: 1.56 kB
N: -- sorted_index: 8.42 kB
N: -- type: 64.00 B
--- End Message ---
--- Begin Message ---
Source: lintian
Source-Version: 2.5.34
We believe that the bug you reported is fixed in the latest version of
lintian, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 715035@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Niels Thykier <niels@thykier.net> (supplier of updated lintian package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Wed, 22 Jul 2015 21:55:58 +0200
Source: lintian
Binary: lintian
Architecture: source all
Version: 2.5.34
Distribution: unstable
Urgency: medium
Maintainer: Debian Lintian Maintainers <lintian-maint@debian.org>
Changed-By: Niels Thykier <niels@thykier.net>
Description:
lintian - Debian package checker
Closes: 715035 792167 792237 792501 793068
Changes:
lintian (2.5.34) unstable; urgency=medium
.
* Summary of tag changes:
+ Added:
- unstripped-static-library
.
* checks/binaries.{desc,pm}:
+ [NT] Apply patch from Guillem Jover to improve the tag
description of the LFS tag. (Closes: #792167)
+ [NT] Detect unstripped static libraries (as an
experimental tag).
* checks/cruft.pm:
+ [NT] Avoid svn-conflict-file tag for ".r0" files, since
there cannot be a conflict involving revision 0.
Thanks to Andreas Beckmann for the report.
(Closes: #793068)
+ [BR] Detect pdf, ps, and eps lenna file.
* checks/files.{desc,pm}:
+ [BR] Fix detection of sources.list file. (Closes: #792237)
* checks/systemd.pm:
+ [NT] Apply patches from Felipe Sateler to ignore known
masked init.d scripts. (Closes: #792501)
.
* commands/info -> info.pm:
+ [NT] Renamed info to info.pm to remove some boilerplate
code.
+ [NT] Forbid late --user-dirs and --include-dir options.
.
* frontend/lintian:
+ [NT] Forbid late --user-dirs and --include-dir options.
+ [NT] Remove the following obsolete command line
options: --root, --all, --binary, --source, --udeb.
They are no longer useful.
.
* lib/Lintian/Collect{,/*}.pm:
+ [NT] Add a shared memory cache that can be used to
reduce memory comsumption a bit for sharable objects.
(Closes: #715035)
* lib/Lintian/Data.pm:
+ [NT] Remove references to LINTIAN_ROOT as Lintian::Data
no longer uses it.
* lib/Lintian/Path.pm:
+ [NT] Rework some implementation details to reduce memory
consumption slightly.
+ [NT] The undocumented "type" and "time" methods have been
removed from the API.
+ [NT] The "uid" and "gid" methods now return 0 rather than
undef when numeric ownership is not collected.
.
* reporting/templates:
+ [NT] Remove (uses of) the "invisible-anchor" css class as
invisible (display: none) elements cannot be used as
anchors.
Checksums-Sha1:
491fcb7d7bbd7f06e3e3979080fce97c24d770eb 2731 lintian_2.5.34.dsc
b09752870cd488407f2e89ad3b50586cf8e8bd78 1220984 lintian_2.5.34.tar.xz
896995e5227c750702ee96075ba36a3548211040 819008 lintian_2.5.34_all.deb
Checksums-Sha256:
f5aee181b48017eefe5382b272eefa840cbbba617133ae8053b48d250c8e8ed7 2731 lintian_2.5.34.dsc
590d7c8c4457e4c76e23daafdd5e518d00d56aab5efb6220e818f764462e2678 1220984 lintian_2.5.34.tar.xz
40712986199ffc4a9e3d493cbfa6d00f74195bbee762c231191aaf46d2c10290 819008 lintian_2.5.34_all.deb
Files:
b964bbcab97bb435b6b36263f34bc3c8 2731 devel optional lintian_2.5.34.dsc
75301e71130477d2eb6b9caf8ab56a4b 1220984 devel optional lintian_2.5.34.tar.xz
a1deabae89110bc2c31987cbcce44c42 819008 devel optional lintian_2.5.34_all.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBCAAGBQJVr/m2AAoJEAVLu599gGRCPbkQAKS4vvGSsSs5cRajjrSugQyo
Eqf1088LhFW91fulZCZA04CHFZB6YQrsEMk/119sueaf76STbUovifF9Vx/TOgr1
8C+Ohub84Q9QGyGZP2uk8HyPwaSokoWdHkTTBhQNmbSKE/l3S1JEjd5pH3yOjqMg
+hMTjfWQySNQcnRD5FXZr5HGLNZY6TomCOWcQSZZrhFuCY85m8Z33VT5f5anKfx9
1FmHQQpSS75nv0aH52FukZQYrbiUpJi1LEK1wCD+0dVveCaDeQRGqm1mBT/BTGVJ
hkiaFBd2e3PtCZ7AKiuumn/eQjnw5iSQAtI5/Dvl7K1B/cb6v1HlHTsEnY0xg+JK
l0NdX3dfS6myT/L1mvJVupG6qJ6pzBvCkvtTVuSJiyQHYiIe9l9/ptumpnughzZj
rDqBeErAGE9iERE862zzwtIDxVaf6Fs7TSdOKz778QeNY0sTC1r5dRu/Y2Wxx1Bw
6ky/0VB7RQc03HcsajxNaAnalDl9c4mvpgRvlnWP2NqTwgtNT/yhtjeUHidUH3m6
mdRcONUtQ30BXrvPv5s+yMyE3Bj+l0APqlSdA5Kr9cgo6K5ykoBJF28c7SBdmyZU
F6ImaaeRSHWg4IGjNOZHCt/Z68/gnQXI5l1lnym8nnBkuffcBMpVxK1A/vLtMOLr
YkKF4sUL7jsKwuiMe+Ea
=IsKl
-----END PGP SIGNATURE-----
--- End Message ---
Reply to: