On Mon, May 25, 2015 at 04:58:38PM +0200, Julian Andres Klode wrote: > 2015-05-25 16:22 GMT+02:00 Sebastian Boehm <sebastian@sometimesfood.org>: > > On 25 May 2015 at 15:32, Julian Andres Klode <jak@debian.org> wrote: > >> I don't really see the performance issue here, though. I don't know > >> why the submitter thinks that there is a performance issue here. The > >> closing takes a few milliseconds. An apt-cache show runs in 0.040s > >> here, and has never taken longer for me. > > > > on my fastest machine, a single apt-cache show takes 0.06s on Jessie. > > However, the same apt-cache show takes less than 0.006s on Wheezy. > > > > For a single run on a fast machine, this isn't too bad, but on a slow > > machine this really hurts a lot. When running docker on my laptop, a > > single apt-cache show takes about 2.7 seconds on jessie, whereas it > > took only 1.4 seconds on wheezy. > > > > Jessie: > > > > root@b49835d0cc64:~# time apt-cache show mg > /dev/null > > real 0m2.708s > > user 0m1.830s > > sys 0m0.850s > > > > Wheezy: > > > > root@df40952198bb:~# time apt-cache show mg > /dev/null > > real 0m1.441s > > user 0m1.260s > > sys 0m0.190s > > Those seem to be uncached or on compressed indices, and thus not > relevant for this bug. Also, Wheezy has more packages and therefore more data than Jessie. That takes more time to parse and work with of course. Some improvements will be made (or were made already) for Stretch in experimental, but I wouldn't be surprised too much if by the time Stretch releases the improvements are eaten up by new stuff… Pro-Tip: Lookup how big /var/cache/apt/pkgcache.bin is in bytes. Add a few megabytes and round it up. Set the value you calculated this way in a config file as value for APT::Cache-Start. Read more about the value itself and how to set it in 'man apt.conf'. (The default is ~20MB which is very close to a minimal setup. Its not a silver bullet, but if we talk about millisecond improvements…) Running some apt command as root (or at least apt-cache gencaches) before doing a lot of operations as 'normal' user can be a huge timesafe btw. 2 seconds sounds like a (at least partially) bad cache as even my 6 years old armel system (a sheevaplug) does it in less than half a second with a good one… > > Even the 0.06s are quite noticeable when using tools such as Chef, > > which produce one apt-cache call per package check. On my fastest > > machine, checking the status for ~70 packages used to take > > approximately 400ms on wheezy, but takes more than four seconds with > > jessie. Obviously the situation is much worse with docker and on > > slower systems. > > > >> So I'd prefer to just close this, as it's not worth the extra effort > >> to read out /proc/self/fd. > > > > How about using the O_CLOEXEC flag when opening files? > > We do, AFAIK. It's hard to believe, but apt code predates the O_CLOEXEC by ~ a decade. It's also not so much what apt code does, but what code does who uses libapt. Interesting is e.g. our FileFd. The fds it opens aren't opened with this flag and I don't think we could just do it as this might break users. A first step would be to let it grow an option to set this flag… (Not that we would use it everywhere, but its usage is increasing steadily compared to more 'manual' ways of opening files as transparent (de)compression is kinda handy to have, even if the code who made that happen is a bit insane^Wcomplex…) > I'm wondering a bit because show only does about 38 fcntl() calls for > me, it does not seem to ForkExec() here at all (and I don't know why > it should do that at all). Asking for the (foreign) architectures dpkg supports is using ForkExec and is probably happening in close to every command. Haven't checked through (apt does this since wheezy through, so this can't be a regression in jessie). [It explains Michael Hale's "solution" through as this autodetection isn't run if the list is set manually. Also, as Ubuntu is activating M-A by default parsing only one arch is faster than parsing two… Parsing none with a good cache would be a lot better of course…] > Anyway, I fixed this in > http://anonscm.debian.org/cgit/apt/apt.git/commit/?id=870a2b6d683e58c0584bbd3614a76cf25a055928 > to use /proc/self/fd where available. This brings us down to 0.72 ms > in my optimal test case. (Frankly I would be happy if we could drop this code entirely, but that is going to be a lot of busy work for questionable benefit so I am not going to cry "here". I would tag it newcomer, but I don't thing the experience working on this would be very nice, so I don't.) Best regards David Kalnischkies
Attachment:
signature.asc
Description: Digital signature