Re: Expires: headers for Packages.gz, Sources.gz
On Sat, 10 Feb 2001, Matt Zimmerman wrote:
> > > I don't see what effect timezone skew would have...Expires: contains the
> > > time zone information. If the object is expired after 24 hours, but the
> > > file hasn't
> > It does in HTTP, most of our mirrors are FTP based and have a nasty habit of
> > adding hours to the times, this totally wacks the whole idea of using
> > absolute Expires headers.
> Hmm. Well, even if the mirrors are botching the timestamps, they should at
> least be doing it in a consistent way (off by a fixed amount), so estimating
> expiration time relative to the modification time should still work.
Not quite, because now the expires time will be off by several hours
relative to the time the file is actually updated, so you are back to the
level of uncertainty you have with the max_age=1day scheme.
> Blecch. In that case, the package files would have to be touched daily
> regardless of whether or not they changed. A waste of bandwidth for ftp-based
> mirroring.
All HTTP download schemes, including APT, rely on unchange mtime ==
unchanged content, so you basically screw everyone who is using APT's
native caching mechanism.
> > Expires make it so things like stable (which has static package files) will
> > always result in a IMS query, instead of 1 every day, as it is now.
> It would only make sense to implement this for package files which
> change on a daily basis.
Which is starting to move rapidly into ugly hack, hard to keep working
territory :>
> > The default expiry mechanisms in a cache should handle this case well enough.
> > DEBs do expire BTW, not their content, but the value in keeping them (we
> > churn DEB files at the rate of ~150 meg/day in the archive)
>
> I don't think that Expires: is intended to have any bearing on how long a cache
> will hold onto the object, only how long it can be considered to be fresh.
A good cache will prefer to churn files that have actually been expired
over files that are still valid, then it will prefer files that have been
used frequently, etc.
Really what you need for maximum happyness is a APT-specific proxy. I had
a nice idea on how to implement a really cool one (with predictive
download even, ooh) at one point, but who has time..
> Quite right. I wish I could do this with Squid's refresh_pattern, but
> it has no way to express mtime + a constant factor (only a percentage of
> the age).
As I said, IMS is cheap, so you don't care about aging. You care about
preventing squid from expiring the Packages file. Let APT force squid to
IMS the thing, but return the file from cache, when it hasn't changed.
> > BTW, you can use apt-get to fetch those source files for you..
> I am using apt-get to fetch the source archives. If it can also determine
> which packages to fetch based on priority, do let me know how.
Use your script on /var/state/apt/lists/*Sources, you don't need wget in
the mix.
Jason
Reply to: