--- Begin Message ---
Package: python-apt
Version: 1.0.1
Severity: wishlist
Dear Maintainers,
Unattended-upgrades needs to find all upgradable and autoremovable
packages, but the filtering itself happens in Python code making it
slow - and a very significant part of u-u's CPU time starting with u-u
1.4 (~60% when there is nothing to install):
$ python3 -c "import timeit; print(timeit.timeit(stmt='{pkg.name for
pkg in cache if pkg.is_upgradable}', setup='import apt;cache =
apt.Cache();gc.enable()',number=10)/10)"
0.703306365699973
$ sudo time ./unattended-upgrade --verbose
Initial blacklisted packages:
Initial whitelisted packages:
Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security,
o=UbuntuESM,a=bionic
No packages found that can be upgraded unattended and no pending auto-removals
2.21user 0.09system 0:02.33elapsed 98%CPU (0avgtext+0avgdata 109296maxresident)k
0inputs+21072outputs (0major+28423minor)pagefaults 0swaps
I tried using FilteredCache, but as it filters in Python, too, it did
not speed up filtering and actually it made the code slower.
Please extend the API to provide filtering in native code (or just
much faster code) at least for .is_auto_removable and .is_upgradable.
Thanks,
Balint
--
Balint Reczey
Ubuntu & Debian Developer
--- End Message ---
--- Begin Message ---
On Fri, Jul 06, 2018 at 11:23:33AM +0200, Balint Reczey wrote:
> Package: python-apt
> Version: 1.0.1
> Severity: wishlist
>
> Dear Maintainers,
>
> Unattended-upgrades needs to find all upgradable and autoremovable
> packages, but the filtering itself happens in Python code making it
> slow - and a very significant part of u-u's CPU time starting with u-u
> 1.4 (~60% when there is nothing to install):
>
> $ python3 -c "import timeit; print(timeit.timeit(stmt='{pkg.name for
> pkg in cache if pkg.is_upgradable}', setup='import apt;cache =
> apt.Cache();gc.enable()',number=10)/10)"
> 0.703306365699973
>
> $ sudo time ./unattended-upgrade --verbose
> Initial blacklisted packages:
> Initial whitelisted packages:
> Starting unattended upgrades script
> Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security,
> o=UbuntuESM,a=bionic
> No packages found that can be upgraded unattended and no pending auto-removals
> 2.21user 0.09system 0:02.33elapsed 98%CPU (0avgtext+0avgdata 109296maxresident)k
> 0inputs+21072outputs (0major+28423minor)pagefaults 0swaps
>
> I tried using FilteredCache, but as it filters in Python, too, it did
> not speed up filtering and actually it made the code slower.
>
> Please extend the API to provide filtering in native code (or just
> much faster code) at least for .is_auto_removable and .is_upgradable.
I'm not going to extend the high-level API. The low-level API is
substantially faster, applications needing performance at that level
should look at that. For the specific benchmark above, it's about 35
times as fast:
$ python3 <(echo "import timeit; print(timeit.timeit(stmt='{pkg.get_fullname(True) for pkg in cache._cache.packages if pkg.current_ver is not None and cache._depcache.is_upgradable(pkg)}', setup='import apt;cache = apt.Cache();gc.enable()',number=10)/10)")
0.021239974099444225
$ python3 <(echo "import timeit; print(timeit.timeit(stmt='{pkg.name for pkg in cache if pkg.is_upgradable}', setup='import apt;cache = apt.Cache();gc.enable()',number=10)/10)")
0.7345808134006802
with my current python-apt branch, we're down to 0.3362563787959516s,
after optimising the iteration a bit.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
--- End Message ---