Re: snapshot.debian.net

[Fumitoshi UKAI <ukai@debian.or.jp>]

At Tue, 26 Apr 2005 22:00:47 +0200,
Jonas Smedegaard wrote:

> On 26-04-2005 18:33, Fumitoshi UKAI wrote:
> 
> > So, one issues to create replica or backup site in Europe (or other
> > region) is how to copy initial set of snapshot.debian.net. 
> > I'm afraid rsync whole archive once would be impossible because it 
> > would require huge memory resources (keep too many filename and
> > link information in memory).
> 
> The problem of rsync needing all directory info in memory at once was
> one of the reasons for the invention of its successor, rdiff.
> 
> Have a look at the Debian packages "rdiff" and "rdiff-backup". I haven't
> played with "rdiff" but believe it works on single files, whereas
> "rdiff-backup" although primarily used for maintaining multiple
> snapshots of an archive is also usable for single shots, and handles
> memory nicely.

If you suggest rdiff-backup to mirror snapshot.debian.net, then
I think it will also take a lot of memory. rdiff-backup(1) says
"--no-hard-links ... If many hard-linked files  are  present,  this 
option can drastically decrease memory usage." and snapshot.debian.net
uses hardlink excessively.

If you suggest rdiff-backup to manage archive in snapshot.debian.net, 
most file contents wouldn't be changed but different filenames would be 
used for different file contents.  I'm not sure how well rdiff-backup 
handles such case in efficient way.

Regards,
Fumitoshi UKAI