Bug#564137: cachefile srcpkgcache.bin is discarded without good reason
2010/2/2 Ben Hutchings <ben@decadent.org.uk>:
> OK, thanks for the repeated explanation. You sent it to the wrong bug
> report previously and I think I must not have read it fully.
Mhh. The BTS really hates me at times… way to many mails
are lost in both directions in the last few days… and sometime
it seems i "strike back" and confuse the bts with bugcontrol
typos and omitting bugnumbers i intended to post to… :/
(old mail is for history reasons attached as fullquote)
> I wrote a program to dump the sizes and times in these cache files, and
> in the latest appearance of this bug they all matched the status of the
> text files except for /var/lib/dpkg/status.
APT should include also a new Debug option to track exactly that:
Debug::pkgCacheGen
Another note, as this is might confusing at first:
An apt-get operation like remove, purge, install or (dist-)upgrade will
NOT update the cache files after dpkg has completed his operation.
It will notice on the next run that the pkgcache.bin is invalid and
regenerate it ~ maybe apt should trigger the regeneration directly
after a successful dpkg-run to have a complete valid file for normal users…
Best regards / Mit freundlichen Grüßen,
David "DonKult" Kalnischkies
2010/1/10 David Kalnischkies <kalnischkies+debian@gmail.com>:
> Hello Ben Hutchings,
>
>> Each invocation of apt-cache was taking about 3 seconds (with a warm
>> disk cache) and it was invoked once for each of ~400 binary package
>> names. I think that must be a bug in APT, and indeed it runs much
>> faster following an 'apt-get update'
>
> Yeah, the disk cache is "warm", but the APT cache is somehow cold as it
> resists in /var/cache/apt and is only writable for root - so APT needs to
> parse all files in /var/lib/apt/lists and the status files for each run...
> "apt-get update" helps in this regard as it rebuilds the cache(s) (as root)
> so the next APT runs have an up-to-date cache. Every other APT command
> executed as root would have achieved the same (apt-cache gencaches).
> We can't relax the write-access to the cache as APT depends on this
> information and an evil user modifying this cache files would open all
> sort of problems...
>
> But the real question is: Why is the APT cache no longer up-to-date?
> For APT a cache is no longer valid than the mtime of the cached file
> has changed - as this happens more often for the status files than
> the lists-file APT builds two caches:
> srcpkgcache.bin - Cache of the /var/lib/apt/lists files
> pkgcache.bin - Cache of srcpkgcache.bin + status files
> The status files are pretty small and therefore fast to parse.
> The real timesucker is the parsing of the lists files, so i guess,
> (as long as your machine isn't really slow) APT thinks the
> srcpkgcache.bin file is out-of-date - as APT can't write the cache
> to the file it is only build in memory - 400 times in a row in your case...
>
> I noticed these "no longer up-to-date" from time to time but
> always thought the cause is my APT setup... I will take a deeper
> look at it than it hits me the next time - and as the next APT will
> include a debug option for it you are all invited to do the same. :)
> (The option will be called Debug::pkgCacheGen)
>
>>> I agree we can optimize a bit the code (but there's probably a reason
>>> is done one-by-one)
> While on it: As reportbug is a python app i would suggest to have
> a look at APTs python interface "python-apt" - using it could
> obsolete the whole calling and output-parsing (overhead) ...
> (and maybe also look at reportbug-ng which seems to use this already).
>
>
> Best regards / Mit freundlichen Grüßen,
>
> David "DonKult" Kalnischkies
>
Reply to: