RFC: queue-viewer / debrelease.projectb speed-ups
Hi,
[tl;dr: it's much quicker, question at the end]
Currently, a "no-op" run of queue-viewer for stable and oldstable (i.e.
creating no new debdiffs and performing no new installability checks)
takes a little under 3.5 minutes end-to-end. While that's not the end of
the world for something that we run once every 30 minutes, any diffs or
installability checks that need performing add to the run time.
I've therefore been having a look at what can be done to improve things
and have been testing a change that maintains an internal table of the
binary packages associated with each source package being processed,
which is populated the first time the source/suite combination is seen
and then used in place of the repeated calls to the database to retrieve
binary package lists, versions and architecture information.
>From my testing this evening, a similar "no-op" queue-viewer run can now
take less than 30 seconds.
The only difference in the generated XML in each case (checked using
"xmllint --c14n | xmllint --format") was the update timestamp and in one
case the order of an architecture list.
We don't have a test-suite for any of the code beyond the current
archive state so it's possible I've missed a corner case somewhere, but
it certainly seems to work okay as things are now.
Now, the bit that needs comments... :-)
Currently, all of the changes I've made are in one simple method in
debrelease.projectb and then a set of changes (88+, 40-) in
debrelease.proposed to use the new method and the data returned from it
in place of existing ones. While that works, it feels cleaner to move
all of the work into the projectb class and have "proposed" remain as it
is, with the cache being an implementation detail of the database work.
The question is, should the projectb class be cloned so that we have
versions that use the cached data and the existing calls, or simply
merge all of the changes in to the existing class?
Other than queue-viewer, the projectb class is used by d, hint and
point-release-mail.
Regards,
Adam
Reply to: