Re: [debian-knoppix] Improved cloop driver
On Thu, 1 Jul 2004 10:41 pm, Klaus Knopper wrote:
> Hi Michael,
Hi Klaus,
> This is a disadvantage, in my opinion, because some CDR-Media tends to
> get unreliable or at least produces more CRC rechecksum at the end of
> the medium. It is better to place indexes in front of the file, which is
> what the current cloop version does.
Is this also the case with dvd's? I can imagine knoppix growing to a dvd size
(e.g. 4GB) this will require between 8-12GB of virtual memory to store the
compressed image prior to writing it on the disk - not to mention it would be
very slow while the system keeps swapping.
The index is not very big in size (64bits/chunk is about 250k/Gb), and if the
CRC issue becomes a problem, maybe its possible to store a number of copies
of the index at the end?
> Plus, I'm not especially happy with frequent changes of the internal
> cloop file format, of course. ... :-/
Understandable.... The current cloop format allows for versioning which would
help.
> > 2) The sizes of the compressed blocks is interleaved between the blocks.
> > This allows the index to be reconstructed in the event that the index is
> > missing (i.e. the file is truncated for example). This makes the
> > compressed format much more robust.
>
> And adds overhead, and is only necessary if the index is NOT in the
> header. ;-)
This modification is not really necessary because the index is at the end -
its just there to provide forensic integrity - to be able to recover from
corrupted files etc. This may well be outside the knoppix scope of interest.
> > The reason for these changes is because this compressed format is used in
> > a digital forensic package called pyflag (http://pyflag.sourceforge.net).
> > In this context, compressed images can be taken of entire hard disks
> > (multi-gigabytes) and then the modified cloop driver can be used to mount
> > them in ro mode. The new compressed format is called "sgzip", but it may
> > be best if the cloop format was changed in future to incorporate the
> > changes.
>
> A better approach would be, IMHO, to first generate an empty index in
> the header (this is possible if you know in advance how many blocks must
> be compressed), then create the image, and then seeking back to the
> header, writing the index with the computed offsets. This would only
> require to keep the index pointers in memory, not the entire compressed
> image, and would need no changes in the compressed file format.
> Disadvantage: Images cannot be written "on the fly" anymore because you
> need a seek()able storage device in order to rewrite the header.
This solution is not ideal since we lose the ability to pipe data from the
compressor due to the need to seek in the output file (we could still pipe
into the compressor though, so its not that big a deal really). Also you will
need to know in advance how big the uncompressed image is so you can make a
large enough empty index at the start - this may not be easy to know if you
just do a mkisofs for example, and pipe the result to the compressor.
Since the index is really not very large, maybe it would be possible to store
it both at the start and at the end of the file. If the compressor is writing
to a stream, it leaves space for the index at the start, and put the real
index at the end. If the compressor can seek, it can go back and put the
index at the start as well. The kernel module can look in both places for the
index, the front first, and then the end. This will not require a change in
the file format since old cloop files will already have the index at the
start, so the new driver can easily use old files.
New files will usually have the index at the start if they have been created
with a file output rather than a stream, and so would be compatible with old
drivers too. Also the cloop format ignores extra data stuck on the end (since
its not indexed) and would not care if there is an extra meg at the end.
What do you think?
Michael.
>
> Regards
> -Klaus Knopper
_______________________________________________
debian-knoppix mailing list
debian-knoppix@linuxtag.org
http://mailman.linuxtag.org/mailman/listinfo/debian-knoppix
Reply to: