Bug#832326: very slow processing of Contents files
Control: tag -1 moreinfo
On Sun, Jul 24, 2016 at 12:43:23PM +0200, Eduard Bloch wrote:
> Package: apt
> Version: 1.3~pre2
> Severity: minor
>
> Hello,
>
> since Contents file handling has been added recently, the processing of
> them seems to be very slow. It takes about two minutes (guessed, not
> measured) where all other stuff is done within the first ~10 seconds.
>
> <first analysis>
> I think, the basic problem here is the massive size of the data in the
> Index files - they are already big and compression ratio is very high.
> Uncompressed versions of both amd64 and i386 add up to about one
> gigabyte! OTOH when I zcat them both, it takes just about 5 seconds!
> So I guess the problem is the amount of data that needs to be rotated
> while patching the code.
> I measured a bit how ed performs and it takes about 11 seconds for
> Contents-amd64.gz (and about 166k of patch lines in a combined patch).
> Patch was made before from the series of related pdiff files, of course.
Not sure what is happening at your side, but APT should normally store
Contents files using LZ4 compression, not gzip; unless you force it to
do otherwise.
We specifically switched to LZ4 to solve this issue.
Does your system not use .lz4 compressed Contents files?
> APT::Compressor::lz4::Binary "false";
My system says:
APT::Compressor::lz4::Binary "lz4";
but maybe just because I have an lz4 binary, it should not make a
difference.
Your configuration looks correct otherwise, though, so I'm not sure
what your problem is.
--
Debian Developer - deb.li/jak | jak-linux.org - free software dev
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to (`inline'). Thank you.
Reply to: