Bug#760473: apt method http uses 100% cpu with high transfer rate
Package: apt
Version: 1.0.7
Severity: normal
Dear Maintainer,
The current implementation of the http method puts a much higher load
on the CPU than a similar call done in wget or curl.
As an example I downloaded a large file from the deb mirror (0ad-data)
which is then cached by local squid. Which in turn then provides higher transfer
rates in case I download it again and causing this bug to appear:
cli:
% apt-get download 0ad-data
Get:1 http://ftp.de.debian.org/debian/ jessie/main 0ad-data all 0.0.16-1 [530 MB]
46% [1 0ad-data 247 MB/530 MB 46%] 24.6 MB/s 11s
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8516 weller 20 0 5516 4200 3944 R 99.4 0.0 0:04.60 http
For comparison the same download with wget:
cli:
% wget http://ftp.de.debian.org/debian/pool/main/0/0ad-data/0ad-data_0.0.16-1_all.deb
(58.5 MB/s) - '0ad-data_0.0.16-1_all.deb' saved [530253138/530253138]
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12578 weller 20 0 5992 4104 3712 R 18.2 0.0 0:00.57 wget
As you can see from these lines the download is capped by the http
method to almost 40% of the transfer rate due to the single core CPU usage.
My guess is that this problem stems from the circular buffer design used
in the http method which might've provided a speed boost back in the day
but hinders it now.
/proc/cpuinfo:
processor : 31
model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
cpu MHz : 2328.042
-- Package-specific info:
-- (/etc/apt/preferences present, but not submitted) --
-- (/etc/apt/sources.list present, but not submitted) --
-- System Information:
Debian Release: jessie/sid
APT prefers testing
APT policy: (550, 'testing'), (500, 'stable'), (210, 'unstable'), (1, 'experimental')
Architecture: i386 (x86_64)
Kernel: Linux 3.15.3-64+ (SMP w/32 CPU cores; PREEMPT)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Versions of packages apt depends on:
ii debian-archive-keyring 2012.4
ii gnupg 1.4.18-2
ii libapt-pkg4.12 1.0.7
ii libc6 2.19-9bfw1
ii libgcc1 1:4.9.1-4bfw1
ii libstdc++6 4.9.1-4bfw1
apt recommends no packages.
Versions of packages apt suggests:
pn apt-doc <none>
ii aptitude 0.6.11-1bfw1
ii dpkg-dev 1.17.13
ii python-apt 0.9.3.9
-- Configuration Files:
/etc/logrotate.d/apt changed [not included]
-- no debconf information
Reply to: