Re: mirroring question

[Ray <debpost@byrnie.otak.com>]

On Fri, Aug 09, 2002 at 04:03:05PM -0400, Loren Jordan wrote:
> The question I have is, how can I get a mirror copy over http only?  There 
> are a couple of apt-get'able archives I use parts of and would like to 
> mirror them as well for my own use.  The problem is, they only allow http 
> access.  I have tried wget in "mirror" mode but I have yet to be able to 
> keep it from wandering outside of the specific section I want to 
> mirror.  It also gets all the possible index versions (Apache index sorted 
> by each possible thing) and just makes thing rather cluttered.

[As if this thread didn't already have enough shameless plugs]

I've got a strange partial mirror tool I wrote which uses libapt. It can
use any method supported by libapt (http, ftp, copy, ...) and uses a
sources.list file to specify what archives to mirror.

Since it was originally intended just to let me 'fetch' packages without
pulling down dependencies, I named it apt-fetch.

apt-fetch fetch [package list] 
  - same as "apt-get install -d" except it does no dependency checking.

apt-fetch sync
  - syncs up the mirror. Uses a dpkg 'status' file to determine which
    packages should be "held" (not downloaded). This could probably be
    mixed with one of those apt-proxy type tools to select packages to
    mirror based on which packages have been requested.

There's a simple shell script which does things like set APT_CONFIG, clean
out dead links and call apt-fetch with various options. apt-fetch itself
is written in c++ and isn't exactly the prettiest batch of code I've ever
written.

The tool has been used to keep a campus partial mirror going for close to
two years now. It has some limitations (only one architecture at a time,
doesn't grab the Release files, etc), but is otherwise pretty nice.  It
downloads everything into a single directory and then makes hard links to
create the actual mirror structure, which may be a little inelegant, but
it works and allows me to clean out old packages with "apt-get autoclean".
If you just want to maintain a small mirror and generate your own Packages
files, then it's actually a good solution.

I haven't touched the code in over a year, but if anyone thinks this might
be a useful tool and wants to test it or take it over, let me know.

- Ray (duster on #debian)