Bug#764204: apt-cache calls fcntl() on 65536 FDs
2015-05-25 21:25 GMT+02:00 Sebastian Boehm <sebastian@sometimesfood.org>:
> Hi,
>
> On 25 May 2015 at 16:58, Julian Andres Klode <jak@debian.org> wrote:
>> Those seem to be uncached or on compressed indices, and thus not
>> relevant for this bug.
>
> fair enough. Still, even with cached and uncompressed indices a single
> apt-cache show takes about 1.6 seconds on jessie and about 1.2 seconds
> on wheezy.
My slowest machine has 0.9 seconds on the first run, and 0.09 on all
subsequent ones, once the indices have been read into the kernel's
cache.
>
>>> How about using the O_CLOEXEC flag when opening files?
>>
>> We do, AFAIK.
>
> Wouldn't that render the fcntl calls superfluous? Also I can't find
> any relevant occurences of O_CLOEXEC in the codebase (apart from one
> in pkgDPkgPM::StartPtyMagic).
Right, sorry, we do not really use O_CLOEXEC, we use FD_CLOEXEC on
descriptors we open, by calling SetCloseExec().
Anyway, apart from the race with that, there might be some open fds we
have inherited and code using libapt-pkg might open other fds, so it's
a good idea to set FD_CLOEXEC anyway.
>
>> I'm wondering a bit because show only does about 38 fcntl() calls for
>> me, it does not seem to ForkExec() here at all (and I don't know why
>> it should do that at all).
>
> Interesting. You haven't set your Apt::ARCHITECTURES to a single arch
> by any chance, have you?
No, it's multi-arch.
>
>> I just tested the closing call, and it took about 4 ms (or 10-12 in
>> jessie) for the program to start up, call fcntl() on the 65533 file
>> descriptors, and exit. I cannot believe that this is the major
>> performance difference you are experiencing between wheezy and jessie.
>>
>> Anyway, I fixed this in
>> http://anonscm.debian.org/cgit/apt/apt.git/commit/?id=870a2b6d683e58c0584bbd3614a76cf25a055928
>> to use /proc/self/fd where available. This brings us down to 0.72 ms
>> in my optimal test case.
>
> While this version is still a bit slower than the one in wheezy, it is
> significantly faster than the one currently in jessie:
>
> Most recent git version:
> ~ # time apt-cache show mg > /dev/null
> apt-cache policy mg > /dev/null 0.01s user 0.00s system 51% cpu 0.015 total
>
> apt 1.0.9.8 (jessie):
> ~ # time apt-cache show mg > /dev/null
> apt-cache show mg > /dev/null 0.01s user 0.04s system 92% cpu 0.061 total
>
> On the docker instance on my laptop with uncompressed indices I'm down
> from 2.2s to 1.4s. All in all a noticeable improvement.
It might have been significantly faster in the previous commit because
the sysconf() was moved out of the loop (released in 1.0.9.10), but
using /proc/self/fd sped it up even further.
--
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
Reply to: