Bug#776658: lintian: Memory consumption of harness and html_reports
On Fri, 30 Jan 2015 17:38:40 -0800 Russ Allbery <rra@debian.org> wrote:
> Niels Thykier <niels@thykier.net> writes:
>
> > The html_reports process itself consumes up to 2GB while processing
> > templates. It is possible that there is nothing we can do about that
> > as there *is* a lot of data in play. But even then, we can free it as
> > soon as possible (so we do not keep it while running gnuplot at the
> > end of the run).
>
> I think the code currently takes a very naive approach and loads the
> entire state of the world into memory, and Perl's memory allocation is
> known to aggressively trade space for speed.
>
> If instead it stored the various things it cared about in a local SQLite
> database, it would be a bit slower, but it would consume much less
> memory. I bet the speed difference wouldn't be too bad. And this would
> have the possibly useful side effect of creating a SQLite database full of
> interesting statistics that one could run rich queries against.
>
> --
> Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>
>
>
Hi Russ (and others),
I have been considering the expand the scope of the reporting framework
to include testing and to de-tangle suite-related data in the reports
(i.e. get a separate report for each suite).
If I am to add that, I think a database might be the only realistic way
forward for that with all the bells and whistles we currently have.
However, I do not really have a lot of experience with Perl database
frameworks, so I could use some help here.
I suspect it would make sense to obsolete the harness state cache as
well (the YAML file), so most tools only need to deal with the database.
FTR, I would probably ask for a postgres database on lindsay.d.o. That
said, SQLite support would be great for local testing.
Thanks,
~Niels
Reply to: