Bug#439031: Reproducing #439031

To: 439031@bugs.debian.org
Subject: Bug#439031: Reproducing #439031
From: Devendra Gera <dgera@deviantart.com>
Date: Wed, 9 Oct 2013 13:17:13 +0200
Message-id: <[🔎] CAGdWyGxxFEUrB_JKBcmvJF7kwxG2vJhcCR=E9zF-5RH6=vfT4A@mail.gmail.com>
Reply-to: Devendra Gera <dgera@deviantart.com>, 439031@bugs.debian.org
Hi,

Apologies for the long report, but I've tried to be rather precise.

I've run into what I believe is this bug, although there's a
possibility of my case being slight different (more on that later).

To test and reproduce the issue, I've set up a couple of simple
repositories on a (xubuntu, but it shouldn't matter) laptop, which are
served over HTTP via the python SimpleHTTPServer module. This is the
directory structure:

gera@gera-tp:/tmp/repos$ tree
.
├── debian-squeeze
│   └── squeeze
│       ├── InRelease
│       ├── Packages
│       ├── Packages.bz2
│       ├── Packages.gz
│       ├── Release
│       ├── Release.gpg
│       ├── Sources
│       ├── Sources.bz2
│       └── Sources.gz
└── debian-wheezy
    └── wheezy
        ├── InRelease
        ├── Packages
        ├── Packages.bz2
        ├── Packages.gz
        ├── Release
        ├── Release.gpg
        ├── Sources
        ├── Sources.bz2
        └── Sources.gz

4 directories, 18 files
gera@gera-tp:/tmp/repos$

This is served over HTTP via:

gera@gera-tp:/tmp/repos$ sudo python -mSimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...

And over HTTPS via:

gera@gera-tp:/tmp/repos$ sudo socat
OPENSSL-LISTEN:443,reuseaddr,fork,certificate=/etc/ssl/certs/ssl-cert-snakeoil.pem,key=/etc/ssl/private/ssl-cersnakeoil.key,verify=0
TCP4:localhost:80

[ Note that I'm using the snakeoil certs that come with xubuntu and
have the CN set to 'xubuntu' ]

A freshly installed, updated VM is set to consume from these
repositories. To narrow down on a minimal test case, I've removed all
other repositories. The 'apt', 'apt-transport-https' and 'apt-utils'
packages seem up-to-date:

root@vm-dgera:~# apt-cache policy apt apt-transport-https apt-utils
apt:
  Installed: 0.9.7.9
  Candidate: 0.9.7.9
  Version table:
 *** 0.9.7.9 0
        100 /var/lib/dpkg/status
apt-transport-https:
  Installed: 0.9.7.9
  Candidate: 0.9.7.9
  Version table:
 *** 0.9.7.9 0
        100 /var/lib/dpkg/status
apt-utils:
  Installed: 0.9.7.9
  Candidate: 0.9.7.9
  Version table:
 *** 0.9.7.9 0
        100 /var/lib/dpkg/status

The VM is set up to trust the snakeoil certificate and has an entry in
/etc/hosts for the name 'xubuntu', to match the CN in the cert.

root@vm-dgera:~# wget -nv 'https://xubuntu/'
2013-10-09 03:56:27 URL:https://xubuntu/ [276/276] -> "index.html" [1]
root@vm-dgera:~# wget -nv 'https://192.168.122.1/'
The certificate's owner does not match hostname `192.168.122.1'
root@vm-dgera:~#


Now, here are a couple of scenarios:

SCENARIO I.
---------------------

The repositories are set up via https. That is,

root@vm-dgera:~# apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 900 https://xubuntu/debian-wheezy/ wheezy/ Packages
     release o=deviantART,a=wheezy,n=wheezy,l=deviantART wheezy,c=
     origin xubuntu
 500 https://xubuntu/debian-squeeze/ squeeze/ Packages
     release o=deviantART,a=squeeze,n=squeeze,l=deviantART squeeze,c=
     origin xubuntu
Pinned packages:
root@vm-dgera:~#


An apt-get update works fine:

root@vm-dgera:~# apt-get -o "Acquire::Languages=none"  update
Get:1 https://xubuntu squeeze/ Release.gpg [287 B]
Hit https://xubuntu squeeze/ Release.gpg [287 B]
Get:2 https://xubuntu wheezy/ Release.gpg [287 B]
Hit https://xubuntu wheezy/ Release.gpg [287 B]
Get:3 https://xubuntu squeeze/ Release [1,544 B]
Hit https://xubuntu squeeze/ Release [1,544 B]
Get:4 https://xubuntu wheezy/ Release [1,540 B]
Hit https://xubuntu wheezy/ Release [1,540 B]
Get:5 https://xubuntu squeeze/ Packages [31.2 kB]
Hit https://xubuntu squeeze/ Packages [31.2 kB]
Get:6 https://xubuntu wheezy/ Packages [38.0 kB]
Hit https://xubuntu wheezy/ Packages [38.0 kB]
Reading package lists... Done
root@vm-dgera:~#

It still works fine, if I delete all the repository files from
/var/lib/apt/lists/:

root@vm-dgera:~# rm /var/lib/apt/lists/xubuntu_debian-*
root@vm-dgera:~# apt-get -o "Acquire::Languages=none"  update
Get:1 https://xubuntu squeeze/ Release.gpg [287 B]
Get:2 https://xubuntu wheezy/ Release.gpg [287 B]
Get:3 https://xubuntu squeeze/ Release [1,544 B]
Get:4 https://xubuntu wheezy/ Release [1,540 B]
Get:5 https://xubuntu squeeze/ Packages [31.2 kB]
Get:6 https://xubuntu wheezy/ Packages [38.0 kB]
Fetched 72.8 kB in 0s (103 kB/s)
Reading package lists... Done
root@vm-dgera:~#

HOWEVER - if I delete only a single repository's files (squeeze here,
but it doesn't matter), it fails:

root@vm-dgera:~# rm /var/lib/apt/lists/xubuntu_debian-squeeze_squeeze_*
root@vm-dgera:~# ls /var/lib/apt/lists/
lock  partial  xubuntu_debian-wheezy_wheezy_Packages
xubuntu_debian-wheezy_wheezy_Release
xubuntu_debian-wheezy_wheezy_Release.gpg
root@vm-dgera:~# apt-get -o "Acquire::Languages=none"  update
Get:1 https://xubuntu squeeze/ Release.gpg [287 B]
Get:2 https://xubuntu wheezy/ Release.gpg [287 B]
Hit https://xubuntu wheezy/ Release.gpg [287 B]
Get:3 https://xubuntu squeeze/ Release [1,544 B]
Hit https://xubuntu squeeze/ Release [1,544 B]
Ign https://xubuntu squeeze/ Release
Get:4 https://xubuntu wheezy/ Release [1,540 B]
Hit https://xubuntu wheezy/ Release [1,540 B]
Get:5 https://xubuntu squeeze/ Packages [31.2 kB]
Hit https://xubuntu squeeze/ Packages [31.2 kB]
Get:6 https://xubuntu wheezy/ Packages [38.0 kB]
Hit https://xubuntu wheezy/ Packages [38.0 kB]
Fetched 287 B in 0s (425 B/s)
Reading package lists... Done
W: GPG error: https://xubuntu squeeze/ Release: Unknown error executing gpgv
root@vm-dgera:~#

It keeps failing till I remove all repo entries (from the host -
although that's always the case here since all repos come from a
single host) from /var/lib/apt/lists/


SCENARIO II
---------------------

If, however, I change the repo URLs to http instead of https:

root@vm-dgera:~# apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 900 http://xubuntu/debian-wheezy/ wheezy/ Packages
     release o=deviantART,a=wheezy,n=wheezy,l=deviantART wheezy,c=
     origin xubuntu
 500 http://xubuntu/debian-squeeze/ squeeze/ Packages
     release o=deviantART,a=squeeze,n=squeeze,l=deviantART squeeze,c=
     origin xubuntu
Pinned packages:
root@vm-dgera:~#


'apt-get update' works fine initially:

root@vm-dgera:~# apt-get -o "Acquire::Languages=none"  update
Get:1 http://xubuntu squeeze/ Release.gpg [287 B]
Get:2 http://xubuntu wheezy/ Release.gpg [287 B]
Get:3 http://xubuntu squeeze/ Release [1,544 B]
Get:4 http://xubuntu wheezy/ Release [1,540 B]
Get:5 http://xubuntu squeeze/ Packages [31.2 kB]
Get:6 http://xubuntu wheezy/ Packages [38.0 kB]
Fetched 72.8 kB in 30s (2,418 B/s)
Reading package lists... Done
root@vm-dgera:~#

Also when I remove everything from /var/lib/apt/lists/:

root@vm-dgera:~# rm /var/lib/apt/lists/xubuntu_debian-*
root@vm-dgera:~# apt-get -o "Acquire::Languages=none"  update
Get:1 http://xubuntu squeeze/ Release.gpg [287 B]
Get:2 http://xubuntu wheezy/ Release.gpg [287 B]
Get:3 http://xubuntu squeeze/ Release [1,544 B]
Get:4 http://xubuntu wheezy/ Release [1,540 B]
Get:5 http://xubuntu squeeze/ Packages [31.2 kB]
Get:6 http://xubuntu wheezy/ Packages [38.0 kB]
Fetched 72.8 kB in 30s (2,421 B/s)
Reading package lists... Done
root@vm-dgera:~#

And *ALSO* when I remove the files related to a single repository:

root@vm-dgera:~# rm /var/lib/apt/lists/xubuntu_debian-squeeze_squeeze_*
root@vm-dgera:~# apt-get -o "Acquire::Languages=none"  update
Get:1 http://xubuntu squeeze/ Release.gpg [287 B]
Get:2 http://xubuntu wheezy/ Release.gpg [287 B]
Get:3 http://xubuntu squeeze/ Release [1,544 B]
Get:4 http://xubuntu wheezy/ Release [1,540 B]
Get:5 http://xubuntu squeeze/ Packages [31.2 kB]
Get:6 http://xubuntu wheezy/ Packages [38.0 kB]
Fetched 72.8 kB in 30s (2,417 B/s)
Reading package lists... Done
root@vm-dgera:~#


--------

The only change between scenarios I and II is the http/https change.

In order to track this down, I moved the original /usr/bin/gpgv to
/usr/bin/gpgv.orig and dropped in the following script in its place:

root@vm-dgera:~# cat /usr/bin/gpgv
#!/bin/bash

mkdir -p /tmp/gpgv_files
for i in "$@"; do
    if [ -f "$i" ]; then
        cp "$i" /tmp/gpgv_files/
    else
        echo "'$i' does not exist" >> /tmp/gpgv.log
    fi
done
exec /usr/bin/gpgv.orig "$@"

root@vm-dgera:~#


Apart from some noise (it checks and reports all arguments), this
makes it clear that
'/var/lib/apt/lists/xubuntu_debian-squeeze_squeeze_Release' does not
exist when gpgv is asked to check its signature.

I'd be happy to run any other required tests and provide logs.

Cheers,
--gera.
Reply to:
Follow-Ups:
- Bug#439031: Reproducing #439031
  - From: David Kalnischkies <kalnischkies+debian@gmail.com>
Prev by Date: Re: Please give-back python-apt 0.8.9.1 on armel
Next by Date: Bug#617643: marked as done (apt: https method breaks with ignored range requests)
Previous by thread: Processed: Re: Bug#723815: marked as done (python3-apt: aptsources.sourceslist.uniq(s) throws an exception in python3)
Next by thread: Bug#439031: Reproducing #439031
Index(es):
- Date
- Thread