Re: mkisofs memory hungry?
Joerg Schilling wrote:
mkisofs shows its age and simplicity... Written in the days of CDs, it
doesn't use any data organization beyond brute force, and therefore the
only way to handle large metadata is to beat the problem to death with
larger hardware. mkisofs doesn't "need" to keep all metadata in memory,
it's just written to do so.
å¼ é?¡æ¦ <email@example.com> wrote:
Hello. I choose one machine in the office to be the DVD backup server.
This machine has 96MB memory, in most cases the debian system running on
it keep 64MB memory free. In this office we don't have other machine
that is supposed to keep up in midnight for backup.
the DVD backup server is supposed to collect 4GB files (many of them are
pretty small) and make an iso image out if it, burn it to DVD burner. At
the first time mkisofs runs (this evening), I was surprised this script
runs several hours with HDD extremely busy, and still the output iso
image stays at 0-bit. top(1) says it takes 64% memory, while 93MB
physical memory and 160MB swap space are occupied (168MB swap space in
I thought mkisofs is somewhat pipeline process, files comes in, iso
images goes out. I thought the memory needed is not directly related to
the amount of data to make ISO image, so I feel this behaviour is
mkisofs needs to keep all metadata of all files in allocated memory.
Mkisofs seems to behave OK.
It seems a good candidate for putting the metadata in something like a
B+tree temporary database, which offers both quite fast random access
and allows forward and backward searching in order. In other words, the
best of random and sequential data access.
Joerg: I'm not criticizing mkisofs, it was written for a smaller
problem, and you took it over from the original author. However, I do
disagree with your defense of it, it really uses far more memory than is
justified for what it does, and the generation of the TOC from metadata
could be done from a simple database just as well, as long as sequential
ordered access was supported. The current implementation is functional
but inellegant. There's a great word in German for something like that,
but I can't remember what it is.
Zhang: if you are building a list of files to back up, sorting that list
may improve the pattern of memory use in the metadata, and therefore do
less paging of the data. You don't have enough memory, but if you can
add even a little your problem may go away; I moved an old machine from
96 to 160MB and saw a huge difference in performance.
bill davidsen <firstname.lastname@example.org>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979