Re: JTE (Jigdo Template Export) v1.0

[Richard Atterer <richard@list04.atterer.net>]

On Mon, Jun 07, 2004 at 04:49:00PM +0100, Steve McIntyre wrote:
> On Mon, Jun 07, 2004 at 05:00:04PM +0200, Richard Atterer wrote:
> >Something like #2 was my favourite: Write a special .iso image, but omit
> >the file data from it, just embed information about the names of files
> >which were omitted. Later, tell jigdo-file to process the results and
> >write out proper .jigdo/.template files.
> 
> Yup, that's what I was planning to do. But (as I said a while ago) I
> don't grok C++ at all and mktemplate.cc is _scary_, especially the
> hand-unrolled loop in the middle of scanImage().

I agree it's complicated, but that central loop needn't be touched at all. 
The code for jte would replace the entire scanImage() method with something
much, much simpler. Essentially, something like this:

  create Zobstream for gzipping of template data;
  MkTemplate::Desc desc;
  while (! end of .jte data) {
    if (next chunk of .jte data is a matched file) {
      desc.matchedFile();
      matchExecCommands();
    } else {
      desc.unmatchedData();
      write unmatched data to Zobstream;
    }
  }

This shouldn't be more than 200 lines of code when spelled out.

> :-) To be honest, the code I have is fairly small and self-contained,
> and might even be easier to port if necessary - it's all written in
> ANSI C. Also, on the caching front - this new code shouldn't slow
> things down too much anyway, as we're already reading all the file
> data through mkisofs. Most modern machines will MD5 data blocks faster
> than they can read them.

Yes, but once you can use the cache, mkisofs needn't even read the file
data anymore. It could just skip over the file, output the filename to the
.jte and leave the rest to jigdo-file. I can imagine that image creation
would be a matter of 30 seconds that way...

> It's also not a bad idea to have two different implementations of some of
> this code now - it makes it more accessible for other people to hack on
> it. It might also have shown up any bugs where your code didn't match the
> spec.

I agree.

> They won't be hard, and I'm on a roll now. These will be written
> tonight... :-)

OK! :-)

> >> 5. MUCH harder: re-reading and re-encoding .iso images that have been
> >>    modified since they were first written.
[...]
> 
> Absolutely - I'm hoping to just be able to modify a small chunk of the
> compressed template data, and of course then we don't need to touch
> the contents of the .jigdo. But we _will_ have to update the template
> MD5, of course.

Right, I forgot about that.

> >I think dd is the only tool used by debian-cd, to write a boot sector
> >for some arches - or is there something else?
> 
> It varies by platform, I'm afraid. Several arches have a post-iso process
> to bless / make a CD bootable. Alpha uses isomarkboot, for example. I'd
> expect most of the changes to be contained at the beginning of the image,
> but we can't guarantee that. Depending on how intelligent the BIOS /
> firmware is on the arch in question, we may have to scan most of the
> image to make sure we don't miss anything. :-(

I see - too bad. Well... what about some LD_PRELOAD trick? The preloaded
library could create a protocol of which parts of the image were
overwritten.

Cheers,

  Richard

-- 
  __   _
  |_) /|  Richard Atterer     |  GnuPG key:
  | \/¯|  http://atterer.net  |  0x888354F7
  ¯ '` ¯