[lintian] Performance data from lintian.d.o

[Niels Thykier <niels@thykier.net>]

Hi,

I have taken the liberty of collecting some simple wall-times of lintian
as it processes packages on lintian.d.o.  I would have attached the
result, but even xz compressed, it does not seem to be accepted by the list.

The top 15 offenders in runtime two days ago  were (NB: tasks exclude
"total"/aggregated values):

> Processable id					Task			Wall time (s)
> binary:libwebkit2gtk-4.0-37-dbg/2.6.1+dfsg1-2/i386	coll/unpacked		380.37953
> binary:libwebkit2gtk-4.0-37-dbg/2.6.1+dfsg1-2/amd64	coll/unpacked		378.34206
> binary:libwebkit2gtk-4.0-37-dbg/2.6.1+dfsg1-2/i386	coll/md5sums		338.527949
> binary:libwebkit2gtk-4.0-37-dbg/2.6.1+dfsg1-2/amd64	coll/md5sums		332.526015
> source:qtwebkit/2.3.4.dfsg-3				check/source-copyright	327.574843
> binary:libncarg-dev/6.2.0-3/amd64			check/manpages		321.68114
> binary:libwebkit2gtk-3.0-25-dbg/2.4.6-2/amd64		coll/unpacked		176.7548
> source:gnome-applets/3.8.1-1				check/cruft		142.666446
> source:webkit2gtk/2.6.1+dfsg1-2			check/cruft		126.058139
> binary:libwebkitgtk-3.0-0-dbg/2.4.6-2/amd64		coll/unpacked		117.748018
> binary:libwebkitgtk-1.0-0-dbg/2.4.6-2/amd64		coll/unpacked		108.593638
> source:gimp-help/2.6.1-1				coll/unpacked		104.796028
> source:gnulib/20140202+stable-2			check/source-copyright	102.252265
> binary:libwebkit2gtk-3.0-25-dbg/2.4.6-2/amd64		coll/md5sums		80.975489
> binary:libimage-exiftool-perl/9.74-1/all		check/manpages		80.789133


I am a bit surprised to see check/source-copyright on this list.  Unlike
the others (except checks/manpages[1]) it does not read from / write to
all files.

The logs are placed in /srv/lintian.debian.org/logs/lintian-perf.log and
are rotated like most other logs.

Other fun facts:
 - There are 111 524 tasks
 - The wall time of these tasks sums to 33 627s (~9.3 hours)
   - According to harness, lintian spent ~7 hours (24 604s) processing
     the runs.  This gives us 27% wall-time reduction due threading.
   - The runtime is spent as the following:
     - Check, sum: 8 563s, avg: 0.1s, count: 80 846
     - Coll, sum: 23 867s, avg: 1.4s, count: 17 224
     - Coll-auto-RM, sum: 1 196s, avg: 0.09s, count: 13 454[2]
   - The runs processed 780 package groups.
 - There are 8 136 tasks with a runtime over a second (7.3%).

Enjoy - and performance optimising patches welcome,
~Niels

[1] checks/manpages is known to be slow on packages with many manpages,
which is why it is not a surprise.

[2] NB: It is inflated due to garbage collection of existing collections
that are now deprecated (e.g. doc-files, init.d).