Kevin McKinley wrote:
Now you have me interested. Do you already have a script to mirror only stable and unstable with rsync? I think I would try only mirroring stable with such a script, but I thought it would take having a program parse things like the Packages files for each release and main/contrib/non-free sub folder and <arch> that you were after.On Tue, 16 Sep 2003 00:46:41 +0100 Colin Watson <cjwatson@debian.org> wrote:It's possible that this is caused by the substantial divergence between testing and unstable due to glibc 2.3.2 over the last month or so. 6.5Gb kind of shocks me, but it's a possibility.The binary package archive for i386 did indeed increase that much; I can't keep all three branches until this works itself out. My work-around for now is to mirror only stable and unstable. Hopefully that will get things back down under 15 Gb. (When I started this back in February all three branches would fit in ~12 Gb.)
Not that parsing would require that much of a Perl or Python script, but feeding the list of files to rsync seemed a little crazy. I don't know if rsync has a max for command line input, or if it would be a bad idea to connect to rsync each file or sub batch. I have read that rsync can use a lot of memory because it keeps a list of all the files in memory until it is done.
If I'm mirroring stable, doesn't it only change when a revision is done, so I don't really need to mirror it that often?
Do you use the packages on the mirror for anything other than as a local http/ftp target for apt-get? If not, what reasons do you have for mirroring over using something like apt-cacher? I'm asking to learn from your reasons, and to help me decide when I'm going to stop wasting (?) bandwidth with my mirror and shut it down. If I was able to share those files with the public and become part of the ftp/http mirror round robin that would be a cool way of giving back and the mirror would seem more usefull, but at the moment it's not on the table for discussion.
Maybe if/when non-free is moved out of the mirrors that will also help the mirror size. I don't feel like writing a Packages file parsing/file size counting script at the moment but doing a rough comparision just by the size of main's Packages file (9.7M) to non-free's Packages file (252K) in unstable, I don't think it's going to drop _that_ much off.
Jacob