Re: Expires: headers for Packages.gz, Sources.gz
On Sat, Feb 10, 2001 at 01:12:15AM -0700, Jason Gunthorpe wrote:
> On Sat, 10 Feb 2001, Matt Zimmerman wrote:
> > Hmm. Well, even if the mirrors are botching the timestamps, they should at
> > least be doing it in a consistent way (off by a fixed amount), so estimating
> > expiration time relative to the modification time should still work.
>
> Not quite, because now the expires time will be off by several hours
> relative to the time the file is actually updated, so you are back to the
> level of uncertainty you have with the max_age=1day scheme.
Not quite, because max_age will still be relative to the object refresh time,
which will have less relation to the actual file modification time than the
munged modification time.
> > > Expires make it so things like stable (which has static package files)
> > > will always result in a IMS query, instead of 1 every day, as it is now.
>
> > It would only make sense to implement this for package files which change
> > on a daily basis.
>
> Which is starting to move rapidly into ugly hack, hard to keep working
> territory :>
>
> > > The default expiry mechanisms in a cache should handle this case well
> > > enough. DEBs do expire BTW, not their content, but the value in keeping
> > > them (we churn DEB files at the rate of ~150 meg/day in the archive)
> >
> > I don't think that Expires: is intended to have any bearing on how long a
> > cache will hold onto the object, only how long it can be considered to be
> > fresh.
>
> A good cache will prefer to churn files that have actually been expired over
> files that are still valid, then it will prefer files that have been used
> frequently, etc.
Which is good behavior. If someone has a .deb cached that is frequently
downloaded (say, a package that has been updated in stable since the time the
site's CDs were created), it _should_ be cached for as long as it is useful.
If it isn't used a lot, it will be preferred to be dropped over other, more
frequently accessed fresh content.
> Really what you need for maximum happyness is a APT-specific proxy. I had a
> nice idea on how to implement a really cool one (with predictive download
> even, ooh) at one point, but who has time..
Or better, an APT module for a modular caching engine. But who has even more
time...
> > > BTW, you can use apt-get to fetch those source files for you..
>
> > I am using apt-get to fetch the source archives. If it can also determine
> > which packages to fetch based on priority, do let me know how.
>
> Use your script on /var/state/apt/lists/*Sources, you don't need wget in the
> mix.
True, I suppose I could do that. I started off hoping to allow the whole bit
to run as non-root, but I need apt-get update to get apt-get source to work
anyhow.
--
- mdz
Reply to: