Bug#695839: lintian: Long-running instances reserves 2.5+ GBs, but avg RES is < 1G MB
clone 695839 -1
retitle -1 lintian: regression in memory usage or memory leak
found -1 2.5.11
notfound -1 2.5.10.2
severity -1 serious
retitle 695839 [reporting/harness]: Schedule check packages in chucks
tags 695839 pending
thanks
On 2012-12-13 10:55, Niels Thykier wrote:
> [...]
>
> Currently Lintian on lintian.d.o is doing a full run and top says:
>
> [...] VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> [...] 2567m 818m 1528 R 81 40.6 432:36.97 lintian
>
> The run started on the 11th and I remember seeing it at 900ish VIRT/RES.
> Yesterday, it was at 2.5 G VIRT (don't remember the RES). Today, we are
> still at 2.5 G VIRT and RES has been 800-1000 MB for the past 30 minutes.
>
> We could be looking at a leak, because the VIRT is still growing a
> bit. Though, if so, only a subset of the packages triggers the leak.
> I am more inclined to believe it is long-living objects that we no
> longer have use, but still have a reference to.
>
> L::Tags keeps a table of statistics for each file processed that is
> not cleaned (afaict), so a slow grow is expected I guess. However,
> I have a feeling this is not the (only) cause of grow.
>
> During the write up of this email the stats has changed to:
>
> [...] VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> [...] 2699m 977m 1484 S 0 48.6 442:20.99 lintian
>
>
> If it is not a leak and just Perl not releasing memory back
> (regardless of reason), I think we should look at splitting
> full runs into smaller "short" lived processes.
>
> ~Niels
>
>
The bad news first; Lintian 2.5.11 was unable to finish the full run.
Around gnome-scan, Lintian started spewing errors - first a fork failed
and then a couple of:
"""[...] cannot open X: Cannot allocate memory at ... """
After that, it just said "collect info X of ... failed" a couple of
times for each of the remaining packages. The VIRT memory usage had
grown over 3GB, so I now strongly suspect that 2.5.11 introduces a
memory leak.
On the plus side, I have managed to finish up a set of patches to make
harness schedule groups in smaller chucks. I pulled them onto
lintian.d.o and restarted to run as a work-around for now.
~Niels
Reply to: