Re: jigdo and unofficial Debian images
Hi tpc,
sorry to hear about your trouble with jigdo.
On Thu, Jun 10, 2004 at 03:48:20PM -0700, tpc@csua.berkeley.edu wrote:
> The most common problem with jigdo I've found is that what is specified
> in the .jigdo file to be downloaded to complete the image is not an
> accurate reflection of what actually exists on the corresponding mirror.
This is a general problem with Debian mirrors: They are a "moving target" -
while CD images are only created weekly, the mirror contents change every
day. However, jigdo has a mechanism to counter this. From the debian-cd
README:
About jigdo "fallback servers":
jigdo works by downloading individual packages and other files from a
normal Debian mirror, and using them to regenerate a CD/DVD image.
However, the content of Debian mirrors changes over time, files are
added and removed. But jigdo must have access to all files needed for
the image it has to regenerate, even those that have been removed from
the normal Debian mirrors.
A fallback server contains a backup of the Debian FTP space for the
moment the .jigdo files were generated. This backup is made available
under a certain URL which is written to the .jigdo files. jigdo will
*only* revert to the fallback server after an unsuccessful attempt to
retrieve a file from the normal user-selected Debian mirror, so the
bandwidth requirements are modest.
> For example, when I used the .jigdo from:
>
> ftp://ftp.fsn.hu/pub/CDROM-Images/debian-unofficial/sid/jigdo/
>
> which was linked from:
>
> http://www.debian.org/CD/jigdo-cd/#which
>
> to get the Debian Sid ISO image, no matter which corresponding mirror I
> specified I got the same error message: "Aaargh - <n> files could not be
> downloaded. This should not happen!".
OK - this might mean that those particular .jigdo files are broken. Hmmm...
but the sid-i386-1.jigdo file looks OK.
> I searched the web to figure out a solution, and when I tried to rsync my
> temporary iso with an actual iso, I got multiple error messages because
> the rsync syntax for the server is different from the syntax posted at:
>
> http://www.debian.org/CD/jigdo-cd/#faq
Yes, those notes are very terse and only try to guide you towards finding
the right rsync arguments.
> rsync --verbose --progress rsync://ftp.fsn.hu/pub/CDROM-Images/debian-unofficial/sid/sid-i386-1.iso
> debian-20040601-i386-binary-1.iso
>
> I got:
>
> You can access the archive using 'rsync' program like rsync -av ftp.fsn.hu::[Module]/path/to/the/files
>
> with no explanation of what [Module] is in reference to. [...]
As the FAQ entry says, you can get directory listings by ending requests
with a slash, e.g. try just the command "rsync rsync://ftp.fsn.hu/". This
allows you to find your way through the directories on the server. The
right command is
rsync rsync://ftp.fsn.hu/cdimages/debian-unofficial/sid/sid-i386-1.iso \
debian-20040601-i386-binary-1.iso
> http://www.mail-archive.com/debian-cd@lists.debian.org/msg06700.html
>
> where you tell a user to try and get the files from the Debian snapshot
> mirror, so I compared one of the missing files that could not be
> downloaded:
>
> ftp://ftp.fsn.hu/debian/pool/main/m/mutt/mutt_1.5.6-1_i386.deb...No such file `mutt_1.5.6-1_i386.deb'.
Thanks for mentioning an example file that fails. I just checked the fsn.hu
.jigdo file and everything seems OK - a backup copy of the file is held by
the server as
<ftp://ftp.fsn.hu/pub/debian-superseded/2Vz6uhSEq3fld7RfFss-dg>, and the
.jigdo file correctly specifies <ftp://ftp.fsn.hu/pub/debian-superseded/>
as the backup location.
The way jigdo-lite _should_ have behaved was to say at one point: "n files
not found in previous pass, trying alternative download locations", and
then it should have started downloading files from the debian-superseded
directory. If you still feel like playing around with jigdo-lite, you could
retry and send me the last couple of screens of its output.
> with what exists:
>
> http://snapshot.debian.net/archive/2004/06/10/debian/pool/main/m/mutt/mutt_1.5.6-20040523+2_i386.deb
>
> I guess I could edit the .jigdo file to reflect what actually exists, but
> how am I to generate the md5:
>
> 2Vz6uhSEq3fld7RfFss-dg=Debian:pool/main/m/mutt/mutt_1.5.6-1_i386.deb
You seem to have misunderstood the mapping that takes place: The .template
file contains the md5 - data with exactly that md5 is needed to build the
image. The .jigdo file provides a mapping which says where to download data
with that md5. So you only need to change the part after the "=", say to
this:
2Vz6uhSEq3fld7RfFss-dg=http://snapshot.debian.net/archive/2004/06/10/debian/pool/main/m/mutt/mutt_1.5.6-20040523+2_i386.deb
But there's an easier way: You can make jigdo-lite use the snapshot server
by adding a line like this in the [Servers] section at the bottom of the
.jigdo file:
Debian=http://snapshot.debian.net/archive/2004/06/10/
> After failing to get the Debian Sid ISO image with jigdo, I was able to
> download the Debian Sarge ISO image with jigdo, but this only works with
> an official .jigdo and mirror. I humbly request that jigdo compare
> what's in the .jigdo with what exists on the mirror and let you know in
> advance that some files cannot be downloaded so as to prevent others from
> experiencing what I had to go through.
...except doing it like that wouldn't work reliably, either. If jigdo did
an extra pass over all the URLs it has to download, it'd spend hours and
hours not downloading, but just checking whether files are there. During
this time, anything can happen, including a mirror update of the server...
There are two ways to counter the problem. The first solution is to create
backup/fallback mirrors which are guaranteed to hold all the files. This
solution is in place and working on fsn.hu AFAICT.
The second thing, currently unimplemented, might be something like this:
Each jigdo image download reports back success/failure to some web server
when it has finished. Before someone starts downloading an image, they can
view some statistics about the image they're about to download. For
example, if for the last 24 hours, no one was able to download the image,
it would be safe to say that the file is broken.
Cheers,
Richard
--
__ _
|_) /| Richard Atterer
| \/¯| http://atterer.net
¯ '` ¯
Reply to: