Re: dpkg memory usage
Ian Jackson <ian@chiark.greenend.org.uk> writes:
> It's mainly because the access method you're using is (I surmise)
> reinvoking dpkg each time. That involves loading the more robust data
> structures in /var/lib/dpkg/info into a fast-to-access in-core format.
It's mostly when I invoke it on the command line that I notice this.
(When I ocassionally do a "dpkg -i", "dpkg -e" or "dpkg -S".) I use
rpm on some of my systems, and it always surprises me how quickly the
commands return because I'm used to dpkg.
> I also intend to change the format of the /var/lib/dpkg/info/*.list
> database to make it faster to load, and I may change
> /var/lib/dpkg/status too. (The resulting structures will still be
> editable with emacs.)
>From an academic standpoint, I'm interested in knowing what you have
in mind here. I think a single text file would be noticably faster
than a bunch of *.list files, but I don't know how much time is spent
on I/O and how much is spent on building data structures in memory.
(It would save the time of scanning the directory, opening and closing
all the files.)
> Using a dbm file or something is fine if you just want read-only
> access. However, they're no good for updating, because such systems
> do not have sensible behaviour on filesystem failures like disk full -
> they can't be updated atomically. You end up having to read
> everything in and rewrite the whole database after every update.
Ok, I'll concede this on the atomic and error handling points. I
didn't know that dpkg was designed that robustly.
I guess a libdpkg would fix the startup time issues and if I want
"dpkg -S" to work faster, I can always write a perl script that does
it, caching the information in a DBM hash that it rebuilds on demand.
> Steve Dunham writes ("Re: dpkg memory usage"):
> > And the text version is still prone to severe corruption. Mine was
> > scrambled the other day when I upgraded the modutils package running a
> > 2.1.x kernel - the machine locked up, and when I rebooted and tried to
> > install more packages, dpkg mixed up a bunch of scripts and .list
> > files.
> I think this was probably a simple kernel bug. dpkg cannot defend
> against your kernel scrambling its filesystem data structures. It
> does ask the kernel to confirm that changes have been committed to
> disk before it continues.
The crash was a kernel bug. After the system came back up there were
files left in /var/lib/dpkg/updates. When I tried to run dpkg after
that, it complained about two of those files being corrupt, so I
deleted those two and left the rest.
My guess was that the corruption was related to those files being left
in the updates directory, does this make sense? What would happen to
dpkg if stuff was left in there? (The filenames were just numbers,
from 1 to 12 I believe, I deleted 10 and 11.) That's why I suggested
that dpkg mixed them up, although it could be a matter of the
directory not being correct.
Anyways, I did a "ls -lart" of the info directory, renamed some stuff
and reinstalled the effected packages, and it seems to be recovered.
(If I had a local copy of the Debian archive, I could have rebuilt all
of the info directory using the status file.)
Don't take any of this the wrong way. dpkg is a great program; you
did a damn fine job with it. The startup speed issue is only a minor
one. The only real issue I have with Debian packages is interactivity
and this isn't the fault of dpkg.
Steve
dunham@cps.msu.edu
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Reply to: