Bug#776658: lintian: Memory consumption of harness and html_reports

[Niels Thykier <niels@thykier.net>]

On 2015-01-30 18:30, Niels Thykier wrote:
> Package: lintian
> Version: 2.5.30+deb8u3
> Severity: important
> 
> The reporting framework consumes a rather substantial amount of
> memory.
> 
> [...]
> 
> The html_reports process itself consumes up to 2GB while processing
> templates.  It is possible that there is nothing we can do about that
> as there *is* a lot of data in play.  But even then, we can free it as
> soon as possible (so we do not keep it while running gnuplot at the
> end of the run).
> 
> Currently, when harness -i runs, the gnuplot process seems to die for
> "no apparent" reason.  I suspect it is OOM'ed though harness +
> html_reports "only" consumes 65-70%ish of the memory available and
> gnuplot seems fairly cheap memory-wise in comparison.
> [...]
>
> ~Niels
> 
> 

I managed to reproduce the original issue locally.  The problem is
indeed an OOM issue.  The root cause is that html_reports reserves a
substantial part of the available and with "overcommit memory" set to
false, the fork() call will fail due to insufficient memory.  In
summary, we (briefly) require the memory usage of html_reports x2 when
generating graphs.

This issue is fundamentally still present in html_reports, but the
changes so far have freed sufficient memory that lilburn^Wlindsay.d.o
have enough memory to pull off the fork.

Solutions may include the SQLite database Russ talked about or a
"pre-fork" the graph generation process.  Sadly, reading the lintian.log
is both the primary source of our memory consumption *and* a hard
dependency for generating the graphs.

~Niels