Hi, Lintian 2.5.34 has run on lintian.d.o for 3 days (6 runs) now. The results are very promising so far. * It is at 49% reprocessed. At this rate, we can expect it to finish after 13 runs (6.5 days). - 2.5.33 took ~8 days to do the same. - This puts at an average of ~350 more packages per run. Caveats apply[0]. * Memory consumption is consistently below 2.6GB[1] - Generally we seem to stay ~1.2GB - We had a "bad" usage Friday for 4 hours at 2.6GB (Most of this is presumably Perl not giving memory back) - I suspect gcc based on where time was spent. - If we can solve this, we can probably afford using another unpacker job (we are currently running with 2). Future work =========== Last night's run included libreoffice and that revealed: processed libreoffice/1:4.4.4-1 successfully (time: 4162.291s) processed libreoffice/1:5.0.0~rc3-1 successfully (time: 6579.308s) These numbers are ~70 minutes and 1h + 50 minutes respectively. A quick look at the numbers for the latter. Its the usual suspects, though a few things stand out: * checks/cruft (source) takes 69% of the total group check time (~35 minutes). * checks/files (libreoffice-dev-doc) takes 22.5% of the total group check time (11 minutes). * coll/file-info (libreoffice-dev-doc) takes 25% of the total unpack time * coll/md5sums (libreoffice-dev-doc) takes 25% of the total unpack time - These two are likely scheduled at the same time, so with only 2 parallel jobs, these can trivially stall everything else for 15 minutes. You can find the original performance logs at [2]. Libreoffice was in run 6 (and gcc-5 was in run 3). ~Niels [0] Caveats: * Each run processes 1 to 16 groups of 512 sources + binaries. The last group always overrun the 4 hour limit. - has happened twice (gcc-5 + libreoffice) for 2.5.34. For libreoffice, we had 5 "fast" groups followed by the 6th group with libreoffice. - I do not have the numbers for 2.5.33. - In summary, fortunate scheduling can make a world of difference. * Each run also processes newly uploaded packages. IOW, there will be a significant noise ratio when "heavy" packages (gcc-5, libreoffice etc.) is uploaded. - My numbers do not track/account for that. [1] https://munin.debian.org/debian.org/lindsay.debian.org/ Notably: * https://munin.debian.org/debian.org/lindsay.debian.org/memory.html * https://munin.debian.org/debian.org/lindsay.debian.org/cpu.html [2] https://people.debian.org/~nthykier/lintian/performance-2.5.34/
Attachment:
signature.asc
Description: OpenPGP digital signature