Re: using rsync to keep squeeze amd64 image freshened.

["J.A. Bezemer" <j.a.bezemer@opensourcepartners.nl>]

On Fri, 17 Dec 2010, shirish  wrote:

> Hi all,
>       I tried to get my query of how to use rsync by mailing
> debian-user. You can see the whole thread starting from
> http://lists.debian.org/debian-user/2010/12/msg00651.html
>
> What I want to do is to keep the weekly amd64 squeeze build , on
> http:// its http://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-dvd/debian-testing-amd64-DVD-1.iso
>
> The above is what I want to achieve. But I'm facing problems as I
> don't know how to use rsync.
>
> Looking from http://www.debian.org/CD/mirroring/rsync-mirrors and
> http://www.debian.org/CD/mirroring/#exclude
>
> the command seems to come something like this :-
>
> rsync --times --links --hard-links --partial -archive -verbose
> -compress --block-size=8192 --exclude=source/
> --exclude='*businesscard*.iso' --exclude='*netinst*.iso'
> --exclude=alpha/ --exclude=arm/ --exclude=hppa/ --exclude=hurd/
> --exclude=ia64/ --exclude=m68k/ --exclude=mips/ --exclude=mipsel/
> --exclude=powerpc/ --exclude=s390/ --exclude=sh/ --exclude=sparc/
> --exclude=5.0.7/ --exclude=5.0.7-live/ --exclude=i386/
> --include='*-dvd.iso' ftp.de.debian.org::debian-cd/ .
>
> While ftp.de.debian.org has only 5.0.7 images (not something I'm looking for)
>
> The Taiwan mirror seems to have everything . I had to look at quite a
> few mirrors as most of them have Debian Lenny (5.0.x series)  but not
> Debian squeeze (6.0.x series).
>
> ftp://ftp.tw.debian.org/debian-cdimage/weekly-builds/amd64/iso-dvd/debian-testing-amd64-DVD-1.iso
>
> This is the one I want to rsync. Only DVD-1 does anybody know how can
> I achieve that ?

Rsync is no "weird" tool. You can use it just like cp:

  cp /somepath/filename .

  rsync /somepath/filename .

Just the same. Try it. Don't get alarmed by the multitude of options in 
the examples...

Now the nice thing of rsync is that it can copy from a remote server, 
that's just:

  rsync servername::somepath/filename .

To find out what a server is offering, say ftp.tw.debian.org, use 
"rsync -vn". That's mostly like "ls -al":

  rsync -vn ftp.tw.debian.org::

So it's offering debian-cd. Let's get a quick listing:

  rsync -vn ftp.tw.debian.org::debian-cd

There's no weekly-builds dir there, so this is the wrong entrance.

  rsync -vn ftp.tw.debian.org::pub
  rsync -vn ftp.tw.debian.org::debian

Well, it seems they're just not offering the weekly-builds via rsync. This 
happens often.

The main host, cdimage.debian.org, does have them available:

  rsync -vn cdimage.debian.org::
  rsync -vn cdimage.debian.org::debian-cd     -> nope
  rsync -vn cdimage.debian.org::cdimage       -> aha!
  rsync -vn cdimage.debian.org::cdimage/weekly-builds/     -> mind the /
  rsync -vn cdimage.debian.org::cdimage/weekly-builds/amd64/
  rsync -vn cdimage.debian.org::cdimage/weekly-builds/amd64/iso-dvd/

There it is!

Now you would just do:

  rsync cdimage.debian.org::cdimage/weekly-builds/amd64/iso-dvd/debian-testing-amd64-DVD-1.iso .

but this is where the options become useful:

  rsync --times --partial --verbose --block-size=8192 cdimage.debian.org::cdimage/weekly-builds/amd64/iso-dvd/debian-testing-amd64-DVD-1.iso .

and personally I like --progress when running interactively.
Don't use --compress, everything in the DVD image is already compressed.
And the other options you quote are mostly for full-archive copying, so 
not useful in this case.


So, continuing from the other thread on debian-user:

Rsync doesn't display an update while it's checking your local file, but 
after that, if you use --progress, it will say how the download is going.

Rsync will download to a temporary filename starting with a dot (.name). 
It will rename the file to the correct name once it's finished and 
correct. _If_ you press Ctrl-C while rsync is downloading, _and_ you used 
--partial, both the original file and the temporary "half-file" are kept 
(though mind the filenames!). To continue: "cat half-download >> 
originalfile", mind the filenames again, and just restart the same rsync 
command.

Note in case you wondered: rsync will speed up the download by using 
blocks of the file it already has, but this works _only_ if there is a 
file with the exact same name in the directory you're downloading to (i.e. 
you will overwrite that file). If you want to keep the old version of the 
file you're updating, make a copy to a different filename before you start 
rsync.


Hopefully this is enough to get you going.

Best regards,

Anne Bezemer