Bug#986899: [pre-approval] unblock: apt/2.2.3
Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: unblock
X-Debbugs-Cc: jak@debian.org
Please unblock package apt
[ Reason ]
Fix downloading packages from repositories without a Size field; those
fail if the unsized package is the largest one on the server that's in
the pipeline.
Add warnings for such repositories, to actually surface such
repositories.
We also fix a unit test to not trigger a test failure and hence FTBFS.
This only got triggered on Ubuntu's LTO toolchain so far, but is an
actual bug - it's unclear why we haven't seen it before.
[ Impact ]
Repositories without Size fields, such as those generated by pulp,
will have failing downloads.
Without the warning, users will have no clear deprecation, and the error
in 2.3.y that will land in bookworm will be hard on them.
The test case fix should not have any impact on bullseye; well it
_should_ not have worked before. It's mostly there for other
downstreams, but I can't rule out the possibility of it triggering at
some point after a toolchain update or by luck or whatever :D
[ Tests ]
We have added automatic integration tests for the unsized package
stuff; and the unit test is well a unit test itself.
[ Risks ]
CI that checks for APT warnings will fail on broken repositories, as
they'll get the warning :)
The maximum pipeline size now being calculated correctly for unsized
packages should not cause any issue, as that could have returned 0
(unknown) before already; though in practice, most times, you don't end
up with packages with unknown size.
If you don't have a repo without a Size field, there should be no risk,
as none of the code paths should be triggered.
[ Checklist ]
[x] all changes are documented in the d/changelog
[x] I reviewed all changes and I approve them
[x] attach debdiff against the package in testing
[ Other info ]
unblock apt/2.2.3
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
diff -Nru apt-2.2.2/apt-pkg/acquire-item.cc apt-2.2.3/apt-pkg/acquire-item.cc
--- apt-2.2.2/apt-pkg/acquire-item.cc 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/apt-pkg/acquire-item.cc 2021-04-13 17:53:32.000000000 +0200
@@ -3469,6 +3469,12 @@
Version.VerStr(), Version.ParentPkg().FullName(false).c_str());
return;
}
+ if (FileSize == 0 && not _config->FindB("Acquire::AllowUnsizedPackages", false))
+ {
+ _error->Warning("Repository is broken: %s (= %s) has no Size information",
+ Version.ParentPkg().FullName(false).c_str(),
+ Version.VerStr());
+ }
// Check if we already downloaded the file
struct stat Buf;
diff -Nru apt-2.2.2/CMakeLists.txt apt-2.2.3/CMakeLists.txt
--- apt-2.2.2/CMakeLists.txt 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/CMakeLists.txt 2021-04-13 17:53:32.000000000 +0200
@@ -200,7 +200,7 @@
# Configure some variables like package, version and architecture.
set(PACKAGE ${PROJECT_NAME})
set(PACKAGE_MAIL "APT Development Team <deity@lists.debian.org>")
-set(PACKAGE_VERSION "2.2.2")
+set(PACKAGE_VERSION "2.2.3")
string(REGEX MATCH "^[0-9.]+" PROJECT_VERSION ${PACKAGE_VERSION})
if (NOT DEFINED DPKG_DATADIR)
diff -Nru apt-2.2.2/debian/changelog apt-2.2.3/debian/changelog
--- apt-2.2.2/debian/changelog 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/debian/changelog 2021-04-13 17:53:32.000000000 +0200
@@ -1,3 +1,16 @@
+apt (2.2.3) unstable; urgency=medium
+
+ * tests: Check for and discard expected warning from MaybeAddAuth. For some
+ reason, this was only noticed with LTO enabled, but should be a general
+ issue.
+ * Fix downloads of unsized files that are largest in pipeline (LP: #1921626)
+ * Warn on packages without a Size field. Such repositories are broken and
+ need to be fixed, as we do not test apt against them, see the bug above
+ for more details. Set Acquire::AllowUnsizedPackages to disable the
+ warning.
+
+ -- Julian Andres Klode <jak@debian.org> Tue, 13 Apr 2021 17:53:32 +0200
+
apt (2.2.2) unstable; urgency=medium
[ David Kalnischkies ]
diff -Nru apt-2.2.2/doc/apt-verbatim.ent apt-2.2.3/doc/apt-verbatim.ent
--- apt-2.2.2/doc/apt-verbatim.ent 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/doc/apt-verbatim.ent 2021-04-13 17:53:32.000000000 +0200
@@ -274,7 +274,7 @@
">
<!-- this will be updated by 'prepare-release' -->
-<!ENTITY apt-product-version "2.2.2">
+<!ENTITY apt-product-version "2.2.3">
<!-- (Code)names for various things used all over the place -->
<!ENTITY debian-oldstable-codename "buster">
diff -Nru apt-2.2.2/doc/examples/configure-index apt-2.2.3/doc/examples/configure-index
--- apt-2.2.2/doc/examples/configure-index 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/doc/examples/configure-index 2021-04-13 17:53:32.000000000 +0200
@@ -257,6 +257,7 @@
AllowInsecureRepositories "<BOOL>";
AllowWeakRepositories "<BOOL>";
AllowDowngradeToInsecureRepositories "<BOOL>";
+ AllowUnsizedPackages "<BOOL>";
// allow repositories to change information potentially breaking user config like pinning
AllowReleaseInfoChange "<BOOL>"
{
diff -Nru apt-2.2.2/doc/po/apt-doc.pot apt-2.2.3/doc/po/apt-doc.pot
--- apt-2.2.2/doc/po/apt-doc.pot 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/doc/po/apt-doc.pot 2021-04-13 17:53:32.000000000 +0200
@@ -5,9 +5,9 @@
#, fuzzy
msgid ""
msgstr ""
-"Project-Id-Version: apt-doc 2.2.2\n"
+"Project-Id-Version: apt-doc 2.2.3\n"
"Report-Msgid-Bugs-To: APT Development Team <deity@lists.debian.org>\n"
-"POT-Creation-Date: 2021-03-12 09:16+0100\n"
+"POT-Creation-Date: 2021-04-13 15:38+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
diff -Nru apt-2.2.2/methods/basehttp.cc apt-2.2.3/methods/basehttp.cc
--- apt-2.2.2/methods/basehttp.cc 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/methods/basehttp.cc 2021-04-13 17:53:32.000000000 +0200
@@ -882,7 +882,11 @@
{
unsigned long long MaxSizeInQueue = 0;
for (FetchItem *I = Queue; I != 0 && I != QueueBack; I = I->Next)
+ {
+ if (I->MaximumSize == 0)
+ return 0;
MaxSizeInQueue = std::max(MaxSizeInQueue, I->MaximumSize);
+ }
return MaxSizeInQueue;
}
/*}}}*/
diff -Nru apt-2.2.2/po/apt-all.pot apt-2.2.3/po/apt-all.pot
--- apt-2.2.2/po/apt-all.pot 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/po/apt-all.pot 2021-04-13 17:53:32.000000000 +0200
@@ -5,9 +5,9 @@
#, fuzzy
msgid ""
msgstr ""
-"Project-Id-Version: apt 2.2.2\n"
+"Project-Id-Version: apt 2.2.3\n"
"Report-Msgid-Bugs-To: APT Development Team <deity@lists.debian.org>\n"
-"POT-Creation-Date: 2021-03-12 09:16+0100\n"
+"POT-Creation-Date: 2021-04-13 15:38+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
diff -Nru apt-2.2.2/test/integration/framework apt-2.2.3/test/integration/framework
--- apt-2.2.2/test/integration/framework 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/test/integration/framework 2021-04-13 17:53:32.000000000 +0200
@@ -486,6 +486,10 @@
echo 'APT::Machine-ID "912e43bd1c1d4ba481f9f8ccab25f9ee";' > rootdir/etc/apt/apt.conf.d/machine-id
+ # XXX: We really ought to make the packages in the test suite all have Size fields.
+ # But this needs a lot more changes, so let's keep it simple for stable updates.
+ echo 'Acquire::AllowUnsizedPackages "true";' >> rootdir/etc/apt/apt.conf.d/temporary-allow-unsized-packages
+
configcompression '.' 'gz' #'bz2' 'lzma' 'xz'
confighashes 'SHA256' # these are tests, not security best-practices
diff -Nru apt-2.2.2/test/integration/test-ubuntu-bug-1921626-unsized-packages apt-2.2.3/test/integration/test-ubuntu-bug-1921626-unsized-packages
--- apt-2.2.2/test/integration/test-ubuntu-bug-1921626-unsized-packages 1970-01-01 01:00:00.000000000 +0100
+++ apt-2.2.3/test/integration/test-ubuntu-bug-1921626-unsized-packages 2021-04-13 17:53:32.000000000 +0200
@@ -0,0 +1,46 @@
+#!/bin/sh
+set -e
+
+TESTDIR="$(readlink -f "$(dirname "$0")")"
+. "$TESTDIR/framework"
+
+setupenvironment
+configarchitecture "i386"
+confighashes 'SHA512'
+
+mkdir tree
+head -c $((5*1024)) /dev/urandom > tree/data-file
+
+buildsimplenativepackage 'a' 'all' '1.0' 'stable'
+buildsimplenativepackage 'b' 'all' '1.0' 'stable' '' '' '' '' "$PWD/tree"
+buildsimplenativepackage 'c' 'all' '1.0' 'stable'
+
+setupaptarchive
+changetowebserver
+
+# Disable sandbox
+echo 'APT::Sandbox::User "root";' > rootdir/etc/apt/apt.conf.d/no-acquire-sandbox
+# XXX: Remove temporary hack
+rm rootdir/etc/apt/apt.conf.d/temporary-allow-unsized-packages
+
+testsuccess apt update
+
+for file in rootdir/var/lib/apt/lists/*Packages; do
+ awk '!(/^Size/ && ++cnt==2)' $file > $file.new
+ mv $file.new $file
+done
+
+
+size_a=$(wc -c aptarchive/pool/a_1.0_all.deb | awk '{print $1}')
+size_b=$(wc -c aptarchive/pool/b_1.0_all.deb | awk '{print $1}')
+size_c=$(wc -c aptarchive/pool/c_1.0_all.deb | awk '{print $1}')
+
+testsuccessequal "Get:1 http://localhost:${APTHTTPPORT} stable/main all a all 1.0 [$size_a B]
+Get:2 http://localhost:${APTHTTPPORT} stable/main all b all 1.0 [$size_b B]
+Get:3 http://localhost:${APTHTTPPORT} stable/main all c all 1.0 [$size_c B]" apt download a b c -o Acquire::AllowUnsizedPackages=true
+
+rm *.deb
+testwarningequal "Get:1 http://localhost:${APTHTTPPORT} stable/main all a all 1.0 [$size_a B]
+Get:2 http://localhost:${APTHTTPPORT} stable/main all b all 1.0 [$size_b B]
+Get:3 http://localhost:${APTHTTPPORT} stable/main all c all 1.0 [$size_c B]
+W: Repository is broken: b:i386 (= 1.0) has no Size information" apt download a b c
diff -Nru apt-2.2.2/test/libapt/authconf_test.cc apt-2.2.3/test/libapt/authconf_test.cc
--- apt-2.2.2/test/libapt/authconf_test.cc 2021-03-12 09:15:59.000000000 +0100
+++ apt-2.2.3/test/libapt/authconf_test.cc 2021-04-13 17:53:32.000000000 +0200
@@ -1,5 +1,6 @@
#include <config.h>
+#include <apt-pkg/error.h>
#include <apt-pkg/fileutl.h>
#include <apt-pkg/netrc.h>
#include <apt-pkg/strutl.h>
@@ -235,11 +236,15 @@
EXPECT_EQ("foo1", U.User);
EXPECT_EQ("bar", U.Password);
+ _error->PushToStack();
EXPECT_TRUE(fd.Seek(0));
U = URI("http://https.example/foo";);
EXPECT_TRUE(MaybeAddAuth(fd, U));
EXPECT_TRUE(U.User.empty());
EXPECT_TRUE(U.Password.empty());
+ EXPECT_FALSE(_error->empty());
+ EXPECT_TRUE(U.Password.empty());
+ _error->RevertToStack();
EXPECT_TRUE(fd.Seek(0));
U = URI("http://http.example/foo";);
Reply to: