Re: retrieving Packages/Sources files from mirrors

[Johannes Schauer <j.schauer@email.de>]

Hi,

Quoting Povl Ole Haarlev Olsen (2014-08-03 16:59:02)
> Couldn't you pretend to be a push-mirror yourself?
> 
> Your upstream mirror would notify you after it's done syncing, but instead 
> of you syncing everything from the upstream mirror, you would just get the 
> Packages/Sources and do whatever it is you do with them.
> 
> If you get a new notification from your upstream mirror while you're still 
> getting the Packages/Sources from the previous run, you know the mirror 
> was updated while you were fetching files and the files therefore might 
> not be from the same push. In that case you could get the files from 
> snapshot instead.

this would be a great way to do this and in fact, Carsten Otto already offered
to do this for me with having ftp2.de.debian.org as my upstream mirror.

Unfortunately the machine that creates the content of bootstrap.debian.net is
behind a restrictive university NAT. So the only way to use the push-mirror
method would be to use a third server which is connected to the net more
directly. Since this seems to be quite a fragile setup, it seems that a
poll-based solution would be easier maintainable.

I now wrote a script which retrieves the Release file, downloads the
Packages/Sources files and checks them all for integrity. If anything fails
(for example because a mirror push happened in the meantime) then it tries a
second time. Downloading everything does not take longer than six hours. I
think this is more robust and requires less coordination.

Thanks a lot for coming up with that solution, though :)

I might implement things that way once bootstrap.debian.net runs on the Debian
infrastructure :)

cheers, josch