Joerg Schilling wrote:
mkisofs shows its age and simplicity... Written in the days of CDs, it doesn't use any data organization beyond brute force, and therefore the only way to handle large metadata is to beat the problem to death with larger hardware. mkisofs doesn't "need" to keep all metadata in memory, it's just written to do so.å¼ é?¡æ¦ <zhangweiwu@realss.com> wrote:Hello. I choose one machine in the office to be the DVD backup server. This machine has 96MB memory, in most cases the debian system running on it keep 64MB memory free. In this office we don't have other machine that is supposed to keep up in midnight for backup. the DVD backup server is supposed to collect 4GB files (many of them are pretty small) and make an iso image out if it, burn it to DVD burner. At the first time mkisofs runs (this evening), I was surprised this script runs several hours with HDD extremely busy, and still the output iso image stays at 0-bit. top(1) says it takes 64% memory, while 93MB physical memory and 160MB swap space are occupied (168MB swap space in total). I thought mkisofs is somewhat pipeline process, files comes in, iso images goes out. I thought the memory needed is not directly related to the amount of data to make ISO image, so I feel this behaviour is strange.mkisofs needs to keep all metadata of all files in allocated memory. Mkisofs seems to behave OK.
It seems a good candidate for putting the metadata in something like a B+tree temporary database, which offers both quite fast random access and allows forward and backward searching in order. In other words, the best of random and sequential data access.
Joerg: I'm not criticizing mkisofs, it was written for a smaller problem, and you took it over from the original author. However, I do disagree with your defense of it, it really uses far more memory than is justified for what it does, and the generation of the TOC from metadata could be done from a simple database just as well, as long as sequential ordered access was supported. The current implementation is functional but inellegant. There's a great word in German for something like that, but I can't remember what it is.
Zhang: if you are building a list of files to back up, sorting that list may improve the pattern of memory use in the metadata, and therefore do less paging of the data. You don't have enough memory, but if you can add even a little your problem may go away; I moved an old machine from 96 to 160MB and saw a huge difference in performance.
-- bill davidsen <davidsen@tmr.com> CTO TMR Associates, Inc Doing interesting things with small computers since 1979