Bug#918592: apt update incredibly slow due to writing lists files using zstd
On Mon, Jan 07, 2019 at 06:40:02PM +0100, Michael Stapelberg wrote:
> Package: apt
> Version: 1.8.0~alpha3
> Severity: normal
>
> For the past few weeks, I haven’t been able to update my Debian system. I
> finally took some time to look into what’s going wrong.
We'd have to figure out when this started, I guess.
>
> The symptom is that “apt update” just hung forever after fetching all files from
> my configured mirrors.
>
> A closer look revealed 100% CPU usage of 1 core in the
> /usr/lib/apt/methods/store process, which was writing to files in
> /var/lib/apt/lists/partial.
>
> Attaching gdb revealed that store was spending its time in a zstd function:
>
> #0 0x00007f376cc0bf78 in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
> #1 0x00007f376cbcc275 in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
> #2 0x00007f376cbcdedf in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
> #3 0x00007f376cbce461 in ZSTD_compressContinue () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
> #4 0x00007f376cbcfe9e in ?? () from /usr/lib/x86_64-linux-gnu/libzstd.so.1
> #5 0x00007f376d9874af in ?? () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
> #6 0x00007f376d97ffd1 in FileFd::Write(void const*, unsigned long long) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
> #7 0x000055afaedf1073 in StoreMethod::Fetch(pkgAcqMethod::FetchItem*) () at include/apt-pkg/fileutl.h:168
> #8 0x00007f376d939f83 in pkgAcqMethod::Run(bool) () from /usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0
> #9 0x000055afaedf0759 in main () at /usr/include/c++/8/bits/basic_string.h:252
> #10 0x00007f376d32e09b in __libc_start_main (main=0x55afaedf06e0 <main>, argc=1, argv=0x7ffe0782f238, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe0782f228) at ../csu/libc-start.c:308
> #11 0x000055afaedf083a in _start () at ../methods/store.cc:147
>
> Sure enough, raising the cost of the zstd compression in my apt config fixed the
> issue:
>
> % cat /etc/apt/apt.conf.d/99disable-zstd
> APT::Compressor::zstd::Cost "999";
>
> I have a few remarks:
>
> 1. Why are the lists re-compressed at all?
> (Also note that I set Acquire::GzipIndexes "yes", so I have already signaled
> that I would prefer to accept the lists as they are.)
No. You have indicated that you want them stored compressed. Which picks the
cheapest compressor, that's usually lz4.
Yes, it used to mean to not uncompress files and take them directly, but that
changed a few years ago, in 1.2 or so, as lz4 basically has not much overhead,
so it's much faster than keeping xz indexes around.
>
> 2. Why is store only using 1 of 16 CPU cores for compression?
store is using 1 process per file (up to a limit that's a multiple of your
cores), parallel compression of single files is meh.
>
> 3. Should we disable zstd for the time being, given that it turns a 1-minute apt
> update into a many-minute apt update, to the point where I had assumed it was
> hanging?
It's not used by default. It's unclear to me why this happens to you. Do you have
another config file which sets a lower zstd cost perhaps?
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
Reply to: