Hi,
Lintian 2.5.34 has run on lintian.d.o for 3 days (6 runs) now. The
results are very promising so far.
* It is at 49% reprocessed. At this rate, we can expect it to finish
after 13 runs (6.5 days).
- 2.5.33 took ~8 days to do the same.
- This puts at an average of ~350 more packages per run. Caveats
apply[0].
* Memory consumption is consistently below 2.6GB[1]
- Generally we seem to stay ~1.2GB
- We had a "bad" usage Friday for 4 hours at 2.6GB
(Most of this is presumably Perl not giving memory back)
- I suspect gcc based on where time was spent.
- If we can solve this, we can probably afford using another
unpacker job (we are currently running with 2).
Future work
===========
Last night's run included libreoffice and that revealed:
processed libreoffice/1:4.4.4-1 successfully (time: 4162.291s)
processed libreoffice/1:5.0.0~rc3-1 successfully (time: 6579.308s)
These numbers are ~70 minutes and 1h + 50 minutes respectively. A quick
look at the numbers for the latter. Its the usual suspects, though a
few things stand out:
* checks/cruft (source) takes 69% of the total group check time (~35
minutes).
* checks/files (libreoffice-dev-doc) takes 22.5% of the total group
check time (11 minutes).
* coll/file-info (libreoffice-dev-doc) takes 25% of the total unpack
time
* coll/md5sums (libreoffice-dev-doc) takes 25% of the total unpack time
- These two are likely scheduled at the same time, so with only 2
parallel jobs, these can trivially stall everything else for 15
minutes.
You can find the original performance logs at [2]. Libreoffice was in
run 6 (and gcc-5 was in run 3).
~Niels
[0] Caveats:
* Each run processes 1 to 16 groups of 512 sources + binaries. The
last group always overrun the 4 hour limit.
- has happened twice (gcc-5 + libreoffice) for 2.5.34. For
libreoffice, we had 5 "fast" groups followed by the 6th group
with libreoffice.
- I do not have the numbers for 2.5.33.
- In summary, fortunate scheduling can make a world of difference.
* Each run also processes newly uploaded packages. IOW, there will be
a significant noise ratio when "heavy" packages (gcc-5, libreoffice
etc.) is uploaded.
- My numbers do not track/account for that.
[1] https://munin.debian.org/debian.org/lindsay.debian.org/
Notably:
* https://munin.debian.org/debian.org/lindsay.debian.org/memory.html
* https://munin.debian.org/debian.org/lindsay.debian.org/cpu.html
[2] https://people.debian.org/~nthykier/lintian/performance-2.5.34/
Attachment:
signature.asc
Description: OpenPGP digital signature