Re: RFC: queue-viewer / debrelease.projectb speed-ups
On Sat, 2015-06-13 at 20:18 +0100, Adam D. Barratt wrote:
> Currently, a "no-op" run of queue-viewer for stable and oldstable (i.e.
> creating no new debdiffs and performing no new installability checks)
> takes a little under 3.5 minutes end-to-end. While that's not the end of
> the world for something that we run once every 30 minutes, any diffs or
> installability checks that need performing add to the run time.
>
> I've therefore been having a look at what can be done to improve things
> and have been testing a change that maintains an internal table of the
> binary packages associated with each source package being processed,
> which is populated the first time the source/suite combination is seen
> and then used in place of the repeated calls to the database to retrieve
> binary package lists, versions and architecture information.
Due to a push earlier this evening to deploy a change to testing.pl I
inadvertently pushed the changes I'd made to lib/debrelease/projectb.py
and pulled them live.
The world doesn't appear to have exploded, so I've decided to leave
things as they are for now. Please yell if you spot any regressions.
Unfortunately they've not had the speed-up effect that I hoped, as it
turns out that the change I was originally seeing wasn't just due to
there being fewer database lookups involved, but also the lookups being
effectively inlined in to queue-viewer itself, removing hundreds -
possibly thousands - of method invocations; the inner binary processing
loop makes four calls to methods in the projectb module for every binary
package it sees, at least.
Regards,
Adam
Reply to: