dart; source mirroring tool
Ive been working on a source mirroring program, i call dart, for Debian
ARchiving Tool, which is available at http://people.debian.org/~bug1/dart/
Its not release quality yet (e.g. the mirror dir is hardcoded), but it is
getting to the useable stage, if anyone is particularly keen try it out,
but checkout the BUGS file first.
There are a few other programs that can be used to mirror source, my
motivations for a new program are mainly becasue i want a tool written in
C, so i can work on it, eventually i want to make it easy to use one of
the autobuilders with it, to lower the bandwidth requirments for mirroring
debian
It uses pthreads and libcurl to do download with one thread (and
connection) per site.
To keep track of which version of a packages belong to which distro in the
package pool, im experimenting with a <dist>.index.dsc.gz file which is
just a header and the path to every .dsc file in the dist, currently in
just downloading the whole Sources.gz file and generating the
.index.dsc.gz file from that. But maybe an index file like this would have
wider usefullness...
How do we currently keep track of which version of a package belongs to
either testing or unstable, how is the Sources.gz generated ?
AFAIK the Sources.gz gile is generated from either apt-ftparchive, or
dpkg-scanpackages, both of which search a given directory for _ALL_ .dsc
files.
The only way i can see how this can work is if unstable and testing exist
in seperate directories somewhere, not in the merged package pool that is
mirrored. The override file doesnt have version information, so that
doesnt help, how else can it be done ?
If an <dist>.index.dsc.gz (which would be about 45kB) and
<dist>-<date>.dsc.tar.gz (400 kB or so, i forget) were used then it would
be possible to only download new .dsc files rather than a new 600kB
Sources.gz when only a few files have changed.
The Sources.gz does have extra info, the Section: and Priority: fields
which is in the overrides file, and Directory: field which is trivial.
Glenn
Reply to: