Bug#886803: apt: makes unattended-upgrades FTBFS due to changed acquire behaviour

[Balint Reczey <balint.reczey@canonical.com>]

Package: apt
Version: 1.6~alpha6
Severity: important
Control: affects -1 unattended-upgrades

Dear Maintainers,

Tests of unattended-upgrades started failing with the latest apt update.
https://ci.debian.net/data/autopkgtest/unstable/amd64/u/unattended-upgrades/20180105_155142/log.gz
:
...
Running ./test_remove_unused.py with python3
EE
======================================================================
ERROR: test_remove_unused_dependencies (__main__.TestRemoveUnused)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_remove_unused.py", line 100, in test_remove_unused_dependencies
    options, rootdir="./root.unused-deps")
  File "/tmp/autopkgtest-lxc.r972p9ul/downtmp/build.b49/src/test/unattended_upgrade.py",
line 1462, in main
    sys.exit(1)
SystemExit: 1

======================================================================
ERROR: test_remove_unused_dependencies_new_unused_only
(__main__.TestRemoveUnused)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_remove_unused.py", line 122, in
test_remove_unused_dependencies_new_unused_only
    options, rootdir="./root.unused-deps")
  File "/tmp/autopkgtest-lxc.r972p9ul/downtmp/build.b49/src/test/unattended_upgrade.py",
line 1462, in main
    sys.exit(1)
SystemExit: 1

----------------------------------------------------------------------
Ran 2 tests in 0.776s

FAILED (errors=2)
An error occurred: '404  Not Found [IP: 91.189.88.162 80]'
The URI 'http://archive.ubuntu.com/ubuntu/test-package_2.0_all.deb'
failed to download, aborting
An error occurred: '404  Not Found [IP: 91.189.88.161 80]'
The URI 'http://archive.ubuntu.com/ubuntu/test-package_2.0_all.deb'
failed to download, aborting
Makefile:6: recipe for target 'check' failed
make: *** [check] Error 1
...

Unattended upgrades uses a pre-prepared apt state in tests in which
already downloaded packages are installed. Those packages don't exist
in the archive and apt did not try to download them, but starting with
the following commit in apt.git apt started looking for them online,
too:

commit 355e1aceac1dd05c4c7daf3420b09bd860fd169d (HEAD, refs/bisect/bad)
Author: David Kalnischkies <david@kalnischkies.de>
Date:   Fri Oct 27 19:09:45 2017 +0200

    implement fallback to alternative URIs for all items

    For deb files we always supported falling back from one server to the
    other if one failed to download the deb, but that was hardwired in the
    handling of this specific item. Moving this alongside the retry
    infrastructure we can implement it for all items and allow methods to
    use this as well by providing additional URIs in a redirect.

I plan making the tests use local repositories of packages instead of
relying on apt's behavior, but looking for an already downloaded .deb
file online may not be desired from apt either.

Cheers,
Balint

-- 
Balint Reczey
Ubuntu & Debian Developer