[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: JTE (Jigdo Template Export) v1.0

On Mon, Jun 07, 2004 at 02:02:24PM +0100, Steve McIntyre wrote:
> 1. Modify jigdo so it knows about the internals of ISO images and can
>    efficiently scan them (bad, not very generic for jigdo)
> 2. Write a helper tool to dump extra information for jigdo to use
>    alongside the ISO image (helper tool written, but modifying jigdo
>    to use this looks HARD)
> 3. Patch mkisofs to write .jigdo and .template files alongside the
>    ISO image
> I've now done #3, and the patch for mkisofs is at

That's really cool!!! Many thanks for doing this!

Something like #2 was my favourite: Write a special .iso image, but omit
the file data from it, just embed information about the names of files 
which were omitted. Later, tell jigdo-file to process the results and write 
out proper .jigdo/.template files.

I guess it's my fault for not offering to add support to jigdo-file for
this, sorry Steve. IMHO, it would have prevented some code from being
duplicated in jte and jigdo-file, permitted the use of jigdo-file's cache
for checksums (=> even faster!), and resulted in a smaller patch to mkisofs
which has better chances of being accepted by mkisofs upstream. And 
(unsurprisingly) it doesn't look /that/ hard to me. ;)

> To use this code, specify the location of the output .jigdo, .template
> and .jte files alongside the ISO image. The .jte file is an
> intermediate helper file that I'll probably lose for the next
> release.

What does the .jte file contain now, and why is it needed?

> How fast is it?
> ===============
> On my *laptop* (600MHz P3, slow laptop disk) I can make a template file
> in parallel with the ISO image from a typical 500MB data set in about 2
> minutes. By simply not creating the ISO (-o /dev/null), this time halves
> again. The data set I'm using here is a copy of the woody i386 r2 update
> CD, as it's a handy image I had lying around.

Yum. :)

> More features:
> 2. Add support for -jigdo-exclude option(s), so that we can exclude
>    (from the jigdo) README.* etc and other files that go on Debian CDs
>    but often change on the mirrors. Reasonably easy to do, and I'm
>    playing with this now.
> 3. Add pattern-matching in the .jigdo file (e.g. /mirror/debian ->
>    Debian:). Again, should be easy.

We'd get both 2. and 3. for free with the "post-processing by jigdo-file" 
approach, hmm...

> 5. MUCH harder: re-reading and re-encoding .iso images that have been
>    modified since they were first written. This is necessary for
>    the boot code used on several architectures in debian-cd. I see how
>    to do it - basically diff the image on disk to the one we would
>    recreate from the .template file and write a new template file to
>    match that. It's going to take some work...

But scanning the changed image would also need a lot of time... I think a
different approach looks more promising: Write a special "dd-jigdo" program
which behaves and works like the original "dd" in every respect, except
that it works directly on the .template. A simple version of such a program 
could get away without modifying the .jigdo file, and the .template format 
does not make it necessary to recompress all the data in the template - 
just a part of it needs to be recompressed.

Alternatively: While you're busy hacking on mkisofs, why not add the
functionality there? The use of dd by debian-cd only illustrates that
mkisofs lacks a feature here!

I think dd is the only tool used by debian-cd, to write a boot sector for
some arches - or is there something else?



  __   _
  |_) /|  Richard Atterer     |  GnuPG key:
  | \/¯|  http://atterer.net  |  0x888354F7
  ¯ '` ¯

Reply to: