Re: Debian mirror scripts

[Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de>]

otto.wyss@orpatec.ch (Otto Wyss) writes:

> Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de> wrote:
>
>> otto.wyss@orpatec.ch (Otto Wyss) writes:
>> 
>> > Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de> wrote:
>> >
>> >> > Sure? Anyway DpartialMirror "http://dpartialmirror.sourceforge.net/";
>> >> > can.
>> >> >
>> > I guess mirrorer doesn't care for bandwith saving as DpartialMirror,
>> > correct me if I'm wrong.
>> 
>> Currently it will always redownload the Packages/Sources files as gzip
>> on every update to fix a bug in the apt methods. But I already
>> suggested only updating those that don't match the Release file. And,
>> unless you have an rsync method for apt, it won't rsync files.
>> 
> Why there isn't there already a rsync method for apt is probably a
> mystery nobody ever will solve.

It is not wanted due to rsync causing excessive server load.

If Debian would provide zsync files a zsync module could be provided
in a matter of days.

>> While rsyncing the Packages files sounds like a good idea to save
>> traffic it actualy is a bit insignificant compared to the daily
>> traffic of new sources and debs.
>> 
> Do you mean there are up to 100 new packages each day? I get between 50
> - 150 packages updated each day for just i386. Or do you mean there are
> 100 new versions? DpartialMirror handles new versions of packages
> (sources and deps) in a way it save about 1% even when the packages are
> normal gzip'ed. It would save around 10% - 50% with rsyncable.

New versions. The size of the Packages files is comparatively tiny
compared to all the debs. Even the 1% saving for rsyncing debs is
hardly worth it due to the extra traffic for the checksums and the
server load it causes.

zsync has the option of looking into gziped files and rsync them as if
they would be ungziped (while still just downloading chunks of the
gziped file). Its a bit more complex algorithm but works even better
than rsyncable files and rsync.

> Is there any plan to add this feature to mirrored?
>
> O. Wyss

I doubt it. Everyone can add support for it to apt and reprepo
(mirrorer) can utilize it. I think that is the best way.

MfG
        Goswin