Re: JTE (Jigdo Template Export) v1.0
On Wed, Jun 09, 2004 at 05:34:03PM +0100, Steve McIntyre wrote:
> Unfortunately, I don't see us (quite) getting that far. To generate the
> md5 of the full image file (which is kind of useful), we need to read all
> of the data through anyway. You can't simply lump together multiple md5
> chunks.
Ah - indeed, you're right. :-/
Hmmm. If the ability to create images this fast turns out to be a
"must-have" feature one day, the template format could be changed: Either
the image md5sum field could be left at zero to indicate "no md5sum
available", or we could switch over to using what I call a "64 bit rsync
sum" - a (cryptographically weak) checksum which allows "lumping together",
already used internally by jigdo.
> >Richard: I remember that you were about to put out a new version of
> >jigdo that could help in this, how is this going?
>
> That's a bug to do with large file support in libstdc++ IIRC? I've
> noticed the problem myself with large jigdo images. I'm just testing JTE
> v1.1 right now to make sure I don't have similar issues.
There *is* currently a problem with big files for gcc 3.x, x<4. However,
the issue Manty was referring to is something else:
Currently, the algorithm which searches for matching files inside the image
sometimes has to discard prospective matches in order to avoid becoming too
slow (ie not O(n²) instead of O(n) time). I have an idea for a way to
improve the accuracy, which should result in a smaller template size
because more files are found in the image.
> Absolutely. It'd be great to be able to get _lots_ of jigdo files created
> for all the different options including multiple variants of CD and
> different DVDs. There are still some more optimisations that should be
> possible in the image-building stages; at the moment we end up md5summing
> the entirety of the data on each disk several times, and that's a little
> bit wasteful.
If you're talking about the md5sums.txt files on the CDs, note that you can
take advantage of jigdo-file's cache when creating them:
jigdo-file md5sum --cache=x.db --hex FILES...
Cheers,
Richard
--
__ _
|_) /| Richard Atterer | GnuPG key:
| \/¯| http://atterer.net | 0x888354F7
¯ '` ¯
Reply to: