Re: Sudden increase in size of Debian?

["Jacob Anawalt" <jacob@cachevalley.com>]

Kevin McKinley said:
> On Wed, 17 Sep 2003 00:21:55 -0600
> Jacob Anawalt <jacob@cachevalley.com> wrote:
>
>> Now you have me interested. Do you already have a script to mirror only
>> stable and unstable with rsync? I think I would try only mirroring
>> stable with such a script, but I thought it would take having a program
>> parse things like the Packages files for each release and
>> main/contrib/non-free sub folder and <arch> that you were after.
>
> Yes, I do. It's attached.
>

That attachment, debian.rsync, seems to just be anonftpsync (rsync script)
only with your rules. Maybe there were suppose to be two attachments? If
not, I think you're mirroring all releases except experimental of i386 and
maybe some more stuff as well. I still prefer the include/exclude rules I
posted to this thread.

Ok, I'm only slightly less lazy in this area today. Here's a half-baked
idea of what I thought it would require.

1) Run rsync for just the dists/* files and links that you want, but
specifically getting all the Packages files for the targets you want but
at least dists/woody/main/binary-i386/Packages* if you're after i386.

2) Call a script that does this - only make yours better ;)

Modify the package reading script to read the Packages file for all
release, target and arch you are interested in, writing the output into
rsync_packages.txt. Use whatever language you like. Here's some Perl in
your eye.

#! /usr/bin/perl
# Create a --include-file= source for rsync

open(PACKAGES,"<debian/dists/woody/main/binary-i386/Packages") or die $
!;
open(RSYNCLIST,">rsync_packages.txt");

while ($line = <PACKAGES>) {
        next if ($line !~ /^Filename: (.*)$/);
        print RSYNCLIST $1,"\n";
}

close(PACKAGES);
close(RSYNCLIST);

3) rsync again, only this time you include from the file you wrote and
exclude everything else in pool, and dists/ because you already did that.
ie: --include-file=rsync_packages.txt --exclude dists/ --exclude=pool/



There's lots of improvement but I thought I'd share the half-baked idea
before I get around to implementing it since no one else pointed out a
script like this when I asked.

[snip]
>
>> Maybe if/when non-free is moved out of the mirrors that will also help
>> the mirror size. I don't feel like writing a Packages file parsing/file
>
> I don't think that will help much; I don't think non-free is that
> significant. (But I was mirroring only main and contrib.)
>

Agreed

-- 
Jacob
Trying out SquirrelMail