On Mon, Jun 07, 2004 at 05:00:04PM +0200, Richard Atterer wrote: >On Mon, Jun 07, 2004 at 02:02:24PM +0100, Steve McIntyre wrote: >> 1. Modify jigdo so it knows about the internals of ISO images and can >> efficiently scan them (bad, not very generic for jigdo) >> >> 2. Write a helper tool to dump extra information for jigdo to use >> alongside the ISO image (helper tool written, but modifying jigdo >> to use this looks HARD) >> >> 3. Patch mkisofs to write .jigdo and .template files alongside the >> ISO image >> >> I've now done #3, and the patch for mkisofs is at > >That's really cool!!! Many thanks for doing this! > >Something like #2 was my favourite: Write a special .iso image, but omit >the file data from it, just embed information about the names of files >which were omitted. Later, tell jigdo-file to process the results and write >out proper .jigdo/.template files. Yup, that's what I was planning to do. But (as I said a while ago) I don't grok C++ at all and mktemplate.cc is _scary_, especially the hand-unrolled loop in the middle of scanImage(). >I guess it's my fault for not offering to add support to jigdo-file for >this, sorry Steve. IMHO, it would have prevented some code from being >duplicated in jte and jigdo-file, permitted the use of jigdo-file's cache >for checksums (=> even faster!), and resulted in a smaller patch to mkisofs >which has better chances of being accepted by mkisofs upstream. And >(unsurprisingly) it doesn't look /that/ hard to me. ;) :-) To be honest, the code I have is fairly small and self-contained, and might even be easier to port if necessary - it's all written in ANSI C. Also, on the caching front - this new code shouldn't slow things down too much anyway, as we're already reading all the file data through mkisofs. Most modern machines will MD5 data blocks faster than they can read them. It's also not a bad idea to have two different implementations of some of this code now - it makes it more accessible for other people to hack on it. It might also have shown up any bugs where your code didn't match the spec. >> To use this code, specify the location of the output .jigdo, .template >> and .jte files alongside the ISO image. The .jte file is an >> intermediate helper file that I'll probably lose for the next >> release. > >What does the .jte file contain now, and why is it needed? The .jte file _is_ the output for #2 above, as I wrote a while ago. I was hacking on it further, then decided to move more intelligence into the mkisofs patch itself. It's essentially my own simple version of a .template file. >> More features: >> >> 2. Add support for -jigdo-exclude option(s), so that we can exclude >> (from the jigdo) README.* etc and other files that go on Debian CDs >> but often change on the mirrors. Reasonably easy to do, and I'm >> playing with this now. >> >> 3. Add pattern-matching in the .jigdo file (e.g. /mirror/debian -> >> Debian:). Again, should be easy. > >We'd get both 2. and 3. for free with the "post-processing by jigdo-file" >approach, hmm... They won't be hard, and I'm on a roll now. These will be written tonight... :-) >> 5. MUCH harder: re-reading and re-encoding .iso images that have been >> modified since they were first written. This is necessary for >> the boot code used on several architectures in debian-cd. I see how >> to do it - basically diff the image on disk to the one we would >> recreate from the .template file and write a new template file to >> match that. It's going to take some work... > >But scanning the changed image would also need a lot of time... I think a >different approach looks more promising: Write a special "dd-jigdo" program >which behaves and works like the original "dd" in every respect, except >that it works directly on the .template. A simple version of such a program >could get away without modifying the .jigdo file, and the .template format >does not make it necessary to recompress all the data in the template - >just a part of it needs to be recompressed. Absolutely - I'm hoping to just be able to modify a small chunk of the compressed template data, and of course then we don't need to touch the contents of the .jigdo. But we _will_ have to update the template MD5, of course. >Alternatively: While you're busy hacking on mkisofs, why not add the >functionality there? The use of dd by debian-cd only illustrates that >mkisofs lacks a feature here! > >I think dd is the only tool used by debian-cd, to write a boot sector for >some arches - or is there something else? It varies by platform, I'm afraid. Several arches have a post-iso process to bless / make a CD bootable. Alpha uses isomarkboot, for example. I'd expect most of the changes to be contained at the beginning of the image, but we can't guarantee that. Depending on how intelligent the BIOS / firmware is on the arch in question, we may have to scan most of the image to make sure we don't miss anything. :-( I think maybe it's time to check out what these boot progs do, and see if we can do it directly within mkisofs too. More work... :-) -- Steve McIntyre, Cambridge, UK. steve@einval.com Into the distance, a ribbon of black Stretched to the point of no turning back
Attachment:
signature.asc
Description: Digital signature