On 10.05.2012 17:42, Julian Gilbey wrote:
On Mon, Feb 13, 2012 at 01:23:49PM +0000, Adam D. Barratt wrote:On 06.02.2012 20:58, Adam D. Barratt wrote:>migration/testing.pl needs to be updated to make it react better when>called repeatedly, or to avoid the need to do so. Right now it's >sufficiently resource hungry to make a 16-core machine unhappy. > >An initial idea might be to statically generate package pages, >although>this would need to be benchmarked for feasibility. testing.pl could>then simply become a redirect to /package/blah or similar.Stupid question: is it running under mod_perl?
The server in question doesn't appear to have libapache2-mod-perl2 installed, so I'm going to say "no". ;-)
If so, surely there's some way to cache the database between invocations so it doesn't have to be regenerated on each call?
The per-package pages were already cached and it's that cache that we're currently using, via mod_rewrite, to serve the pages that are linked from services such as the PTS. There's no support for any of the other querystring parameters right now, because we'd end up with n-combinatorial versions of each file around.
In theory, yes, it could probably be rewritten to store the interesting data somewhere and cross-reference that, rather than re-reading the packages files all the time. As I mentioned in my mail opening this report though, whatever is available needs to be sufficiently sane not to make a machine of the size of ftp-master/release.d.o fall over when comment spammers spider several thousand of the pages in a short time and thus make DSA frown a good deal.
Regards, Adam