Hi All, On Mon, 2014-07-21 at 11:02 +0100, Daniel Lintott wrote: > On 21/07/14 04:11, Paul Wise wrote: > > Unfortunately the final word on that is that Debian needs to replace > > our current redirector with one based on the RSS feature and also to I honestly was expecting such answer! > > add a cache mechanism (they suggested a 1 hour cache time) so that we > > don't overload the RSS feature. Are we really consuming so much bandwidth for that feature? I assume this will happen each time a user or a daemon wants to check a particular package. I'm not convinced this is worth especially they ask for a cache of 1 hour, do we expect that per package we do a check more than twice per day (daily daemon + random user) > > Do you think it would be possible for you to add a caching mechanism > > and convert your version into a patch that we can apply to SVN? > > > > It should definitely be possible to add a caching mechanism to the the > new redirector, currently I have a couple of ideas on this but both have > drawbacks. > > 1. Use a Berkeley DB to store the retrieved data, similar to what is > currently done. > > Something like: > | project | file | update_time | > -------------------------------- > | | | | > > Problems I foresee: > * My intention would be to check at the time the script is requested if > the update_time > 1 hour ago... > - if yes... get the new RSS and update the DB > - if no.... use the information from the DB > > What happens if this happens for multiple requests at the same time? > > 2. Save the XML file to a cache folder I like this one > Then at request time check the time on that file and it's age. > > The only problem I can see this causing is disk space (I don't know how > much of an issue this is for Debian) > > The RSS file for the VPCS project is almost 52KB. Picking a figure out > of the air (as I've no idea how packages use the redirector) of 10000, > this is going to create 520MB of cached files. > > Obviously some projects may have a smaller RSS and others larger.... > which may skew that estimate. > > Any comments on these would be most appreciated. What about compressing the files? This can reduce the size dramatically. Can you please check for the file you used as example? Cheers, Abou Al Montacir
Attachment:
signature.asc
Description: This is a digitally signed message part