Re: Slow package database

[Brad Sawatzky <brad+debian@swatter.net>]

On Fri, 30 Mar 2007, Adam Borowski wrote:

> It's not that.  It's /var/lib/dpkg/available.
> 
> Try this:
> 
> for x in /var/lib/apt/lists/*Packages;do dpkg --merge-avail "$x";done
> (or "dselect update")
> Install+purge of an empty package:
> 	9-ish seconds on one box (my home desktop, years of use)
> 	6-ish seconds on another (a slow server)
> 
> dpkg --clear-avail
> Install+purge of an empty package:
> 	0.182s on the slow server
> 	3ish seconds on the desktop
> 
> The difference between those 0.182s and 3s is due to removed and even purged
> packages staying in the available file.

Hmm.  I've been wondering why upgrades seemed to take so long on my shiny
new desktop...

Here's a couple more data points for a 2GB/AMD64 box running 2.6.19.2 with
and ext3 filesystem.  My test package was the sid/etch ethereal 'dummy'
package.

Original baseline:
  % time (apt-get -y install ethereal && apt-get -y remove ethereal)
    [ . . . ]
    real  0m10.729s
    user  0m6.692s
    sys 0m1.224s
  I threw out the first couple iterations after which these timings were
  consistent to +- 0.1s across 5 subsequent samples.

Now, I blow away my (22 MB) /var/lib/dpkg/available file:
  % dpkg --clear-avail

  % time (apt-get -y install ethereal && apt-get -y remove ethereal)
    [ . . . ]
    real  0m6.492s
    user  0m4.124s
    sys 0m0.800s
  I threw out the first couple iterations after which these timings were
  consistent to +- 0.1s across 5 subsequent samples.

--> Result: 40% faster with an empty 'available' file.

On a whim I enabled dir_index on my ext3 FS, _and_ rebuilt info/:
  % cd /var/lib/dpkg/
  % cp -a info info.tmp
  % mv info info.orig
  % mv info.tmp info
I gather that should both defrag the info files (FWIW), and rebuild the
director using the hashed b-trees.  (I don't see how the hashed b-trees
could impact this particular test, but what the hell...)

Then I redid the timing test:
%time (apt-get -y install ethereal && apt-get -y remove ethereal)
  [ . . . ]
  real  0m6.847s
  user  0m4.100s
  sys 0m0.776s

--> Result: No difference at all this time...

Then I replaced the 'available' file with an archived version and ran
the test again:
%time (apt-get -y install ethereal && apt-get -y remove ethereal)
  [ . . . ]
  real  0m10.869s
  user  0m6.592s
  sys 0m1.228s

--> Result: right back to the original (slow) baseline times

Looks like '/var/lib/dpkg/available' is the main culprit here.  (I wish I
could test the same setup on a different filesystem like JFS though...)

-- Brad