Re: [debian-knoppix] New create_compressed_fs utility
On Thu, Dec 19, 2002 at 02:37:32PM +0100, Valentijn Sessink wrote:
> Hello folks,
>
> I did a total rewrite of create_compressed_fs. Find it at
> http://projects.openoffice.nl/downloads/compressloop/
Great, finally a clean solution. Just a little drawback that it can't
handle input and output streams/pipes anymore.
> Usage: ./compressloop [options] infile outfile
> Options:
> -b blocksize set block size (multiple of 512 bytes)
> -c compression set compression (0-9, 0 is fastest, 9 is best)
> -i imagesize set image size if it cannot be determined
> -I forced-imagesize force image size, even if determined
> -v verbose (the more v's, the more verbosity)
>
> compressloop does not use malloc() calls, but writes directly to the output
> file. This saves memory and is much faster (up to almost 50%) if you have a
> computer that does *not* have 650M memory lying around...
...or swap. True, but what about pipes... ;-)
> You can now specify compression level as well, so it's possible to specify a
> larger but faster image if you want so.
Have you tried it? I'm pretty sure that the compressed file actually gets
larger if you try a higher compression with a small blocksize.
But it's a nice feature. Maybe, you can also probvide a benchmark script
that tries to determine the "optimal" compression level for a given
blocksize?
> Please note that due to the lseek() calls, it is no longer possible to use
> stdout as output file. Those who want to use images to pipe them to a CD
> burner will still need the old create_compressed_fs utility. This is due to
> the fact that IMHO the image is not a stream - create_compressed_fs builds
> an image, then streams the image to stdout. But that doesn't make it a
> stream.
>
> You can use stdin as input, but please note that compressloop needs the
> correct input size to build an image. Explanation is that a compressed loop
> file system is built as follows:
>
> [header][index of blocks][blocks...............................]
>
> The index is not known until the blocks have all been compressed. Also, the
> *size* of the image is not known until the number of blocks is known.
>
> compressloop works around this by making up an image size if it cannot
> determine one. It specifies 2,000,000 bytes for this - the index size will
> be ( 2,000,000 / blocksize * 4 ) bytes large. If there are less blocks,
> that's not a problem, the index will simply end prematurely.
> However, the index fills up valuable space, so whenever you know the correct
> input size - or have a rough estimate - then use the "-i" option to specify
> the block size.
I will have to check if the 2.4 kernel is content with vmalloc() instead
of kmalloc() memory for the block index (it should never get swapped
out). The loading of the cloop file will fail if your block index is
larger than about 130kb due to kmalloc() limitations.
> If the number you specify is too low, compressloop will quit.
>
> EXAMPLE
> compressloop -v -c 0 -b 65536 uncompressedinput.iso compressedoutput.iso
>
> A better method for all of the indexing hassle would probably be a rewrite
> of cloop where:
>
> - a version number is included in the header
> - the index is at the end of the file, followed by
> - a number that indicates the exact position of the start of the index.
I'm not really fond of the idea of putting the blockindex at the very
end of the file, because CD media is sometimes unreliable in the higher
block numbers, I have to admit. Plus, you noticed that the header is
actually a shellscript, so you can execute a cloop file directly? I
think that stuff like version information or indexes really belong to
the beginning of a file, not at the end, so you still can access them if
later parts of the file are corrupted.
> Good luck with compressloop. Questions?
Did you include your copyright and changelog? Can I include it
in the official Debian cloop-utils package?
> Praise?
Very good work! Nice code! Thanks a lot!
> Money?
Uhm... Would you volunteer to make it backwards-compatible with
create_compressed_fs, and (re-)add a piping feature, for a negotiable
price of course? :-)
Regards
-Klaus Knopper
--
Klaus Knopper Technical Solutions & Finances
knopper@linuxtag.org http://www.linuxtag.org/
Phone +49-(0)631-3109371 Fax +49-(0)631-3109372
LinuxTag 2003 - Europes largest Linux Expo Where .com meets .org
_______________________________________________
debian-knoppix mailing list
debian-knoppix@linuxtag.org
http://mailman.linuxtag.org/mailman/listinfo/debian-knoppix
Reply to: