Re: [debian-knoppix] New create_compressed_fs utility
At Thu, Dec 19, 2002 at 02:37:26PM +0100, Klaus Knopper wrote:
> On Thu, Dec 19, 2002 at 02:37:32PM +0100, Valentijn Sessink wrote:
> > I did a total rewrite of create_compressed_fs. Find it at
> > http://projects.openoffice.nl/downloads/compressloop/
> Great, finally a clean solution. Just a little drawback that it can't
> handle input and output streams/pipes anymore.
Yep... Does anyone know if it's possible to have a "memory stream", i.e.
instead of using malloc() to allocate memory, have a sort of "mopen()" call
that handles the allocation of new blocks? (I'm not much of a C programmer
as you see).
> > -i imagesize set image size if it cannot be determined
> > -I forced-imagesize force image size, even if determined
Eduard: I'll explain this better ("set input image size in bytes", would that
do?)
> > compressloop does not use malloc() calls, but writes directly to the output
> > file. This saves memory and is much faster (up to almost 50%) if you have a
> > computer that does *not* have 650M memory lying around...
> ...or swap. True, but what about pipes... ;-)
Ah, use "mktemp". I'll write a script for you, is that OK? ;-)
> > You can now specify compression level as well, so it's possible to specify a
> > larger but faster image if you want so.
> Have you tried it? I'm pretty sure that the compressed file actually gets
> larger if you try a higher compression with a small blocksize.
I was thinking of using a lower compression rate - this will speed things
up. I did not benchmark things.
> But it's a nice feature. Maybe, you can also probvide a benchmark script
> that tries to determine the "optimal" compression level for a given
> blocksize?
Oh dear... Maybe next week, I'll try something.
> > be ( 2,000,000 / blocksize * 4 ) bytes large. If there are less blocks,
> > that's not a problem, the index will simply end prematurely.
> > However, the index fills up valuable space, so whenever you know the correct
> > input size - or have a rough estimate - then use the "-i" option to specify
> > the block size.
> I will have to check if the 2.4 kernel is content with vmalloc() instead
> of kmalloc() memory for the block index (it should never get swapped
> out). The loading of the cloop file will fail if your block index is
> larger than about 130kb due to kmalloc() limitations.
Hmm, I didn't know this. You might want to check my code if my checks for
this are correct. I doubt it.
> > A better method for all of the indexing hassle would probably be a rewrite
> > of cloop where:
> > - a version number is included in the header
> > - the index is at the end of the file, followed by
> > - a number that indicates the exact position of the start of the index.
> I'm not really fond of the idea of putting the blockindex at the very
> end of the file, because CD media is sometimes unreliable in the higher
> block numbers, I have to admit. Plus, you noticed that the header is
Hmmm. Well, maybe you're right. In which case I don't think it makes much
sense to rewrite compressloop to do piping - piping to stdout essentially
means duplicating code, and could be done much faster with a shell script
and mktemp.
> actually a shellscript, so you can execute a cloop file directly? I
> think that stuff like version information or indexes really belong to
> the beginning of a file, not at the end, so you still can access them if
> later parts of the file are corrupted.
> Did you include your copyright and changelog? Can I include it
It's a rewrite, so the original GPL still applies. I mentioned my name - so
there's the copyright ;-) , but not the whole changelog. I'll look at my CVS
tree and add some info. In the next version, that is :)
> in the official Debian cloop-utils package?
Oh dammit, I was going to make loads of money with it but this GPL license
forbids me to hide the sources! ;-)
Yes, off course you can.
> > Praise?
> Very good work! Nice code! Thanks a lot!
:-)
> > Money?
> Uhm... Would you volunteer to make it backwards-compatible with
> create_compressed_fs, and (re-)add a piping feature, for a negotiable
> price of course? :-)
Ah, money. Uhm... I will think about it. You're going to pay me in Knoppix
Client Access Licenses, right? Cool!
I'll give it a thought - although I'm a bit reluctant to rewrite parts of
the code to re-establish the old behaviour. If (not when) these are mutually
exclusive, I thing a shell script around compressloop is a better idea.
However, if a few lines of code make the stdout-thing go - I don't have
anything against the very idea of streaming.
V.
p.s. no need to Cc: me, I read the list.
--
http://www.openoffice.nl/ Open Office - Linux Office Solutions
Valentijn Sessink valentyn+sessink@nospam.openoffice.nl
_______________________________________________
debian-knoppix mailing list
debian-knoppix@linuxtag.org
http://mailman.linuxtag.org/mailman/listinfo/debian-knoppix
Reply to: