Re: Refactoring the tracker

[Florian Weimer <fw@deneb.enyo.de>]

* Raphael Geissert:

> Florian Weimer wrote:
>> Another issue which has gained some significance lately is that the
>> package and CVE lists have grown quite a bit, leading to longer and
>> longer processing times on soler.  I've removed a few unused features
>> to speed things up a bit, but it seems that we're at the bare minimum
>> right now.
>
> I think it would be a great opportunity to use post-commit hooks to make the 
> scripts on soler only update the database instead of regenerating it all the 
> time.

The internal dependencies are bit convuleted, that's why I asked about
a dependency tracking framework such as PyCells.

However, if we have enough RAM available, all that shouldn't be
necessary.  I've implemented the fundamental join that determines
non-vulnerable package versions in pure Python, and it results in
Python process of about 100 MB.  This is also the data which is needed
to serve web pages.  soler has about 800 MB of free RAM, so we've got
plenty of room for growth.  (Parsing the data/*/list files and doing
the join takes a few seconds on my machine; soler will be a bit
slower, but I'm confient that it will still acceptable.  We can reduce
the parse time for the list files by simply splitting them by year,
anyway.)

> I personally found the OO approach to the generation of the web
> pages really painful.

I'm open to different suggestions.  My experience with web frameworks
is rather limited and does not include any current frameworks.