[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#1067440: marked as done (Compression makes searching packages very slow)

Your message dated Fri, 22 Mar 2024 10:34:36 +0000
with message-id <E1rncEO-00FsMG-OS@fasolo.debian.org>
and subject line Bug#1067440: fixed in apt 2.7.14
has caused the Debian Bug report #1067440,
regarding Compression makes searching packages very slow
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
1067440: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067440
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message --- 
Package: apt
Version: 2.7.12

I noticed that searching for packages is very slow if the package lists are compressed. To reproduce, remove `/var/lib/apt/lists`, enable

    Acquire::GzipIndexes "true"; Acquire::CompressionTypes::Order:: "gz";

, run `apt update`. This enables LZ4 compression on my systems, but I don't think the exact method matters. You can then run `apt search librust`, which takes about 19 seconds in a Debian 12 container (docker.io/debian:12 has compression already set up), compared to 0.4 seconds without compression.

Also tested on Ubuntu 22.04 and 24.04, so the exact APT version shouldn't matter too much.

I tried to look into it, and `strace -e trace=openat apt-cache search librust` shows it reopen and re-read one of the package lists:

openat(AT_FDCWD, "/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_universe_binary-amd64_Packages.lz4", O_RDONLY) = 16
librust-addr2line+default-dev - Cross-platform symbolication library - feature "default"
openat(AT_FDCWD, "/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_universe_binary-amd64_Packages.lz4", O_RDONLY) = 16
librust-addr2line+object-dev - Cross-platform symbolication library - feature "object"
openat(AT_FDCWD, "/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_universe_binary-amd64_Packages.lz4", O_RDONLY) = 16
librust-addr2line+rustc-demangle-dev - Cross-platform symbolication library - feature "rustc-demangle"
openat(AT_FDCWD, "/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_jammy_universe_binary-amd64_Packages.lz4", O_RDONLY) = 16
librust-addr2line+std-dev - Cross-platform symbolication library - feature "std"

(you can use -e trace=openat,read to confirm that it's actually reading the file)

I believe it's quadratic in the number of search results, and this is related to the pseudo-indexing mechanism used by APT (see `pkgRecords::Lookup` in apt-pkg). Each lookup call will have to decompress the file in order to seek to the destination.

Unfortunately, I suspect this isn't exactly an easy fix, given the current design.

--- End Message ---
--- Begin Message --- 
Source: apt
Source-Version: 2.7.14
Done: Julian Andres Klode <jak@debian.org>

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1067440@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 22 Mar 2024 11:11:44 +0100
Source: apt
Architecture: source
Version: 2.7.14
Distribution: unstable
Urgency: medium
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Closes: 1065513 1065517 1065831 1067440
Changes:
 apt (2.7.14) unstable; urgency=medium
 .
   [ Steve Langasek ]
   * Support building without gnutls
   * Add an artificial Conflicts: against libnettle8
     to force upgrades in launchpad buildd chroots
 .
   [ David Kalnischkies ]
   * Handle EINTR in the static FileFd::Write overload
   * Remove non-existent Debug::BuildDeps from apt.conf(5)
   * Parse unsupported != relation in dependencies
 .
   [ Frans Spiesschaert ]
   * Dutch program translation update (Closes: #1065513)
   * Dutch manpages translation update (Closes: #1065517)
 .
   [ Wesley Schwengle ]
   * Update documentation for apt-get upgrade with pkg arg (Closes: #1065831)
   * Update documentation for apt upgrade with pkg arg (Closes: #1065831)
   * Include Dutch translation for apt/apt-get upgrade documenation update
   * Update Graphviz URL to https://graphviz.org/
   * Update VCG tool URI to new location
   * s#http://bugs.debian.org/src/#https://bugs.debian.org/src#
 .
   [ Julian Andres Klode ]
   * pkgTagFile::Jump: Use lookback buffer to rejump to current position
     (Closes: #1067440)
   * debrecords: Do not reparse if given same location (Closes: #1067440)
   * Revert "debrecords: Do not reparse if given same location"
Checksums-Sha1:
 9d586a5ba08a6570b9387a79359437e3dd5268a9 2977 apt_2.7.14.dsc
 9444824cb80675b51e84200a8b8ecc1aa2f432f9 2353208 apt_2.7.14.tar.xz
 171c4eec9b5a4ebf8befd82471edb2f1e49703ee 7911 apt_2.7.14_source.buildinfo
Checksums-Sha256:
 99d27439b093bb20a378ae9114db378a29115159a9bace930153316d49930db7 2977 apt_2.7.14.dsc
 7a01783f05b7d90f36aea7871cc06757499ae8ade933497aaeb6c508274e7358 2353208 apt_2.7.14.tar.xz
 66c5fec4a4df6cb13013b8e3ff9d0cec8f7ae8b0bfa736caed9b13f28c908c73 7911 apt_2.7.14_source.buildinfo
Files:
 294343830f0aa8779c42b59c119696c6 2977 admin required apt_2.7.14.dsc
 90a0084d85973b56212ac7b3b95e3f1e 2353208 admin required apt_2.7.14.tar.xz
 69c3aa426a9dfb284e78c0f76a857bca 7911 admin required apt_2.7.14_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJDBAEBCgAtFiEET7WIqEwt3nmnTHeHb6RY3R2wP3EFAmX9WnIPHGpha0BkZWJp
YW4ub3JnAAoJEG+kWN0dsD9xhOcP/2EhL/Mhv5hm3epTekS3EjbPpqutNU+AAPwC
3ke2N82kD8d2aPMVV3aNfOrdNaAPlOFT24i9/3n0I0TuK9n4KXwwIpZ7zEsSOVdv
E2hFWuJUuuk7ydXgwhVJsUPQj9zx0WGK4bDxm73XnHYgBOPbCQFTz/pODezUZsSQ
ZY98Z5Iax4s6A7OF7tAFBtL6jsKTb9P0EMGjeW6TJYvSsq8GIX+AQo5IOGdR0l31
DKa8HiBLyXM5yGEg8XzCCE5Ys+mL5LG9Qhq8jPArfAyus7uc36juZM2wDAQEfMQh
jRmK6ig0KWvdUc1GNTnix1JYVFhW+5zed9FKMO0TJEarHAbYEYEyZDvONUz0pjKz
L4u6D+YLa4rUfECtuMWrgeadup1dOvZb6U/+NtEKYYDKifPucJ8Lr7TqO914DaoR
Ep/7hq5mb6z5uOpd+FD9v/eNAalaTZv/juRqHhMsYcpCWpoNIPQwjO84TkAmieSb
39T2tsM3yljnRnVnR5VfB83oSuZbPVSXw+CO1KIoGnQrVJSrx2hBoJsP2g8+s0K9
AREcqFlJjIaLLdw0XqS85fK1C0xdOgIhRfxTPvM/f54c6Ll4wmMTn6L6W4sJl9tY
5rl9EAU8OQ5dUT3ML1vCLaupFet1z/ZD3t4Ym/Ql11E6WKVQY4sroA0kV16JiVX0
XkLP+x+d
=97lD
-----END PGP SIGNATURE-----

Attachment: pgpJBt4l2Izi0.pgp
Description: PGP signature


--- End Message ---
Reply to: