Re: Bug#752384: HEAnet sourceforge mirror is outdated
Hi Daniel,
many thanks for your work on this!
> It should definitely be possible to add a caching mechanism to the the
> new redirector, currently I have a couple of ideas on this but both have
> drawbacks.
>
> 1. Use a Berkeley DB to store the retrieved data, similar to what is
> currently done.
BDB may not be a particularly good choice either [1] -- there are other DBs
suggested in that thread that seem to have a better future.
https://lists.debian.org/debian-devel/2014/06/msg00328.html
> Problems I foresee:
> * My intention would be to check at the time the script is requested if
> the update_time > 1 hour ago...
> - if yes... get the new RSS and update the DB
> - if no.... use the information from the DB
>
> What happens if this happens for multiple requests at the same time?
If the update of the db is atomic (which is easy to arrange for most DBs),
then I'd be tempted to ignore that you might very occasionally request the
same RSS feed twice in quick succession. Others may disagree with me here...
but I'd worry about that later if it is actually a problem.
An alternative to worrying about locking would be to only update the db from
cron. This adds some latency to the scan which is annoying for the
maintainer sometimes but not really an issue from the QA perspective. If the
RSS feed offers a Last-Modified header for HEAD requests, then the cron job
can be done easily and often (perhaps that should be investigated anyway?).
> 2. Save the XML file to a cache folder
>
> Then at request time check the time on that file and it's age.
>
> The only problem I can see this causing is disk space (I don't know how
> much of an issue this is for Debian)
Rather than keeping all of them all the time, you could delete them as soon
as they are older than the refresh time; that could be done with find from
cron. If we were to split the QA cron jobs that check for outdated sources
across the day, it would be easy to keep that number down to 10-20% of the
total XML.
> The RSS file for the VPCS project is almost 52KB. Picking a figure out
> of the air (as I've no idea how packages use the redirector) of 10000,
> this is going to create 520MB of cached files.
Just to help with one of the two random numbers:
udd=> select count(*) from upstream where watch_file like '%sf.net/%';
count
-------
2409
(1 row)
hope that helps!
cheers
Stuart
--
Stuart Prescott http://www.nanonanonano.net/ stuart@nanonanonano.net
Debian Developer http://www.debian.org/ stuart@debian.org
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Reply to: