[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#364272: apt: recent difficxulty in updateing, "Remote end closed connection" and MD5 sum errors

#include <hallo.h>
* Eduard Bloch [Sat, Jun 24 2006, 12:01:53PM]:

> > Oops.  Sorry for being unclear.  Could you run it again, this time
> > using tcpdump's -w option instead of capturing its output?  This way,
> > it's far easier to use tools to process the data.
> 
> Ok, stored under the same URL.

And now I have a log where the update process reproducible fails on
Sources.bz2 files (and only such, no others seem to be affected). That
is exactly what I have seen on Ubuntu a while ago, randomly appearing
and disappearing.

You see there that something is seriously broken in apt's control flow.
First it try to decompress something non-existent (empty files?). But,
despite of shocking error message, the files are in fact silently
fetched/unpacked and the contents is correct. What is going on?

I also noticed that after such a breakage there is some junk left in
/var/lib/apt/lists/partial/. Reruning updates removes the garbage and
there are no errors anymore. But when I remove the garbage manually, apt
also produces failed bzip2 runs. Sorry, WTF?



zombie:/tmp# apt-get update
Get:1 http://ftp.uni-kl.de unstable Release.gpg [189B]
Get:2 http://ftp.uni-kl.de experimental Release.gpg [189B]
Get:3 http://ftp.uni-kl.de stable Release.gpg [189B]
Ign http://ftp.uni-kl.de unstable Release.gpg
Hit http://ftp.uni-kl.de unstable Release
Hit http://ftp.uni-kl.de experimental Release 
Hit http://ftp.uni-kl.de stable Release       
Ign http://ftp.uni-kl.de unstable Release     
Hit http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/contrib Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/non-free Packages/DiffIndex
Get:4 http://ftp.uni-kl.de unstable/main Sources/DiffIndex [12.5kB]
Get:5 http://ftp.uni-kl.de unstable/contrib Sources/DiffIndex [12.5kB]
Get:6 http://ftp.uni-kl.de unstable/non-free Sources/DiffIndex [12.5kB]
Get:7 http://ftp.uni-kl.de unstable/main Sources/DiffIndex [12.5kB]
Get:8 http://ftp.uni-kl.de unstable/contrib Sources/DiffIndex [12.5kB]
Get:9 http://ftp.uni-kl.de unstable/non-free Sources/DiffIndex [12.5kB]
Hit http://ftp.uni-kl.de experimental/main Packages/DiffIndex
Get:10 http://ftp.uni-kl.de experimental/main Sources/DiffIndex [12.5kB]
Ign http://ftp.uni-kl.de stable/main Sources/DiffIndex
Ign http://ftp.uni-kl.de stable/contrib Sources/DiffIndex
Ign http://ftp.uni-kl.de stable/non-free Sources/DiffIndex
Ign http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Ign http://ftp.uni-kl.de unstable/unikl Packages/DiffIndex
Ign http://ftp.uni-kl.de stable/main Sources
Ign http://ftp.uni-kl.de stable/contrib Sources
Ign http://ftp.uni-kl.de stable/non-free Sources
Ign http://ftp.uni-kl.de unstable/main Packages
Ign http://ftp.uni-kl.de unstable/unikl Packages
Hit http://ftp.uni-kl.de stable/main Sources
Hit http://ftp.uni-kl.de stable/contrib Sources
Hit http://ftp.uni-kl.de stable/non-free Sources
Hit http://ftp.uni-kl.de unstable/main Packages
Hit http://ftp.uni-kl.de unstable/unikl Packages
Fetched 88.1kB in 2s (33.2kB/s)
Reading package lists... Done
zombie:/tmp# rm /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_*
rm: cannot remove `/var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_*': No such file or directory
zombie:/tmp# rm /var/lib/apt/lists/*Source*
zombie:/tmp# apt-get update
Get:1 http://ftp.uni-kl.de unstable Release.gpg [189B]
Get:2 http://ftp.uni-kl.de experimental Release.gpg [189B]
Get:3 http://ftp.uni-kl.de stable Release.gpg [189B]
Ign http://ftp.uni-kl.de unstable Release.gpg
Hit http://ftp.uni-kl.de unstable Release
Hit http://ftp.uni-kl.de experimental Release 
Hit http://ftp.uni-kl.de stable Release       
Ign http://ftp.uni-kl.de unstable Release     
Hit http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/contrib Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/non-free Packages/DiffIndex
Get:4 http://ftp.uni-kl.de unstable/main Sources [1211kB]
Get:5 http://ftp.uni-kl.de unstable/contrib Sources [21.5kB]
Get:6 http://ftp.uni-kl.de unstable/non-free Sources [27.7kB]
Get:7 http://ftp.uni-kl.de unstable/main Sources [1211kB]
Get:8 http://ftp.uni-kl.de unstable/contrib Sources [21.5kB]
Get:9 http://ftp.uni-kl.de unstable/non-free Sources [27.7kB]
Hit http://ftp.uni-kl.de experimental/main Packages/DiffIndex
Ign http://ftp.uni-kl.de experimental/main Sources 
Ign http://ftp.uni-kl.de stable/main Sources       
Ign http://ftp.uni-kl.de stable/contrib Sources    
Ign http://ftp.uni-kl.de stable/non-free Sources   
Ign http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Ign http://ftp.uni-kl.de unstable/unikl Packages/DiffIndex
Get:10 http://ftp.uni-kl.de experimental/main Sources [34.1kB]
Get:11 http://ftp.uni-kl.de stable/main Sources [1322kB]               
Get:12 http://ftp.uni-kl.de stable/contrib Sources [24.8kB]            
Get:13 http://ftp.uni-kl.de stable/non-free Sources [23.5kB]                
Ign http://ftp.uni-kl.de unstable/main Packages                              
99% [13 Sources gzip 81125] [7 Sources bzip2 0] [Waiting for headers]      
bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Resource temporarily unavailable
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Err http://ftp.uni-kl.de unstable/main Sources                       
  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_main_source_Sources - open (2 No such file or directory)
99% [13 Sources gzip 81125] [8 Sources bzip2 0] [Waiting for headers]
bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Resource temporarily unavailable
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Err http://ftp.uni-kl.de unstable/contrib Sources                    
  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_contrib_source_Sources - open (2 No such file or directory)
99% [13 Sources gzip 81125] [9 Sources bzip2 0] [Waiting for headers]
bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Resource temporarily unavailable
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Err http://ftp.uni-kl.de unstable/non-free Sources                   
  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_non-free_source_Sources - open (2 No such file or directory)
Ign http://ftp.uni-kl.de unstable/unikl Packages                     
Hit http://ftp.uni-kl.de unstable/main Packages
Hit http://ftp.uni-kl.de unstable/unikl Packages
Fetched 3925kB in 3s (1226kB/s)
Failed to fetch http://ftp.uni-kl.de/debian/dists/unstable/main/source/Sources.bz2  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_main_source_Sources - open (2 No such file or directory)
Failed to fetch http://ftp.uni-kl.de/debian/dists/unstable/contrib/source/Sources.bz2  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_contrib_source_Sources - open (2 No such file or directory)
Failed to fetch http://ftp.uni-kl.de/debian/dists/unstable/non-free/source/Sources.bz2  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_non-free_source_Sources - open (2 No such file or directory)
Reading package lists... Done
E: Some index files failed to download, they have been ignored, or old ones used instead.
zombie:/tmp# ls -la /var/lib/apt/lists/*Source*
-rw-r--r-- 1 root root  117120 Jul  5 21:43 /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_experimental_main_source_Sources
-rw-r--r-- 1 root root   90341 Apr 18 03:34 /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_stable_contrib_source_Sources
-rw-r--r-- 1 root root 4787381 Apr 18 14:05 /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_stable_main_source_Sources
-rw-r--r-- 1 root root   81125 Jun  6  2005 /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_stable_non-free_source_Sources
-rw-r--r-- 1 root root   88337 Jul  6 21:42 /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_unstable_contrib_source_Sources
-rw-r--r-- 1 root root 6043597 Jul  6 21:42 /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_unstable_main_source_Sources
-rw-r--r-- 1 root root  114856 Jul  6 21:42 /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_unstable_non-free_source_Sources
zombie:/tmp# md5sum /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_stable_main_source_Sources
51e757f3b13635ecc6abea5e7527363e  /var/lib/apt/lists/ftp.uni-kl.de_debian_dists_stable_main_source_Sources
zombie:/tmp# grep 51e757f3b13635ecc6abea5e7527363e /var/lib/apt/lists/*Release*
/var/lib/apt/lists/ftp.uni-kl.de_debian_dists_stable_Release: 51e757f3b13635ecc6abea5e7527363e  4787381 main/source/Sources
zombie:/tmp# apt-get update
Get:1 http://ftp.uni-kl.de unstable Release.gpg [189B]
Get:2 http://ftp.uni-kl.de experimental Release.gpg [189B]
Get:3 http://ftp.uni-kl.de stable Release.gpg [189B]
Ign http://ftp.uni-kl.de unstable Release.gpg
Hit http://ftp.uni-kl.de unstable Release
Hit http://ftp.uni-kl.de experimental Release 
Hit http://ftp.uni-kl.de stable Release       
Ign http://ftp.uni-kl.de unstable Release     
Hit http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/contrib Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/non-free Packages/DiffIndex
Get:4 http://ftp.uni-kl.de unstable/main Sources/DiffIndex [12.5kB]
Get:5 http://ftp.uni-kl.de unstable/contrib Sources/DiffIndex [12.5kB]
Get:6 http://ftp.uni-kl.de unstable/non-free Sources/DiffIndex [12.5kB]
Get:7 http://ftp.uni-kl.de unstable/main Sources/DiffIndex [12.5kB]
Get:8 http://ftp.uni-kl.de unstable/contrib Sources/DiffIndex [12.5kB]
Get:9 http://ftp.uni-kl.de unstable/non-free Sources/DiffIndex [12.5kB]
Hit http://ftp.uni-kl.de experimental/main Packages/DiffIndex
Get:10 http://ftp.uni-kl.de experimental/main Sources/DiffIndex [12.5kB]
Ign http://ftp.uni-kl.de stable/main Sources/DiffIndex
Ign http://ftp.uni-kl.de stable/contrib Sources/DiffIndex
Ign http://ftp.uni-kl.de stable/non-free Sources/DiffIndex
Ign http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Ign http://ftp.uni-kl.de unstable/unikl Packages/DiffIndex
Ign http://ftp.uni-kl.de stable/main Sources
Ign http://ftp.uni-kl.de stable/contrib Sources
Ign http://ftp.uni-kl.de stable/non-free Sources
Ign http://ftp.uni-kl.de unstable/main Packages
Ign http://ftp.uni-kl.de unstable/unikl Packages
Hit http://ftp.uni-kl.de stable/main Sources
Hit http://ftp.uni-kl.de stable/contrib Sources
Hit http://ftp.uni-kl.de stable/non-free Sources
Hit http://ftp.uni-kl.de unstable/main Packages
Hit http://ftp.uni-kl.de unstable/unikl Packages
Fetched 88.1kB in 2s (31.0kB/s)
Reading package lists... Done
zombie:/tmp# ls -la /var/lib/apt/lists/partial/
total 32
drwxr-xr-x 2 ed   ed       1 Jul  7 21:01 .
drwxr-xr-x 3 root root 16384 Jul  7 21:01 ..
zombie:/tmp# rm /var/lib/apt/lists/*Source*
zombie:/tmp# apt-get update
Get:1 http://ftp.uni-kl.de unstable Release.gpg [189B]
Get:2 http://ftp.uni-kl.de experimental Release.gpg [189B]
Get:3 http://ftp.uni-kl.de stable Release.gpg [189B]
Ign http://ftp.uni-kl.de unstable Release.gpg
Hit http://ftp.uni-kl.de unstable Release
Hit http://ftp.uni-kl.de experimental Release 
Hit http://ftp.uni-kl.de stable Release       
Ign http://ftp.uni-kl.de unstable Release     
Hit http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/contrib Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/non-free Packages/DiffIndex
Get:4 http://ftp.uni-kl.de unstable/main Sources [1211kB]
Get:5 http://ftp.uni-kl.de unstable/contrib Sources [21.5kB]
Get:6 http://ftp.uni-kl.de unstable/non-free Sources [27.7kB]
Get:7 http://ftp.uni-kl.de unstable/main Sources [1211kB]
Get:8 http://ftp.uni-kl.de unstable/contrib Sources [21.5kB]
Get:9 http://ftp.uni-kl.de unstable/non-free Sources [27.7kB]
Hit http://ftp.uni-kl.de experimental/main Packages/DiffIndex
Ign http://ftp.uni-kl.de experimental/main Sources
Ign http://ftp.uni-kl.de stable/main Sources      
Ign http://ftp.uni-kl.de stable/contrib Sources    
Ign http://ftp.uni-kl.de stable/non-free Sources   
Ign http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Ign http://ftp.uni-kl.de unstable/unikl Packages/DiffIndex
Get:10 http://ftp.uni-kl.de experimental/main Sources [34.1kB]
Get:11 http://ftp.uni-kl.de stable/main Sources [1322kB]               
Get:12 http://ftp.uni-kl.de stable/contrib Sources [24.8kB]                
Get:13 http://ftp.uni-kl.de stable/non-free Sources [23.5kB]               
Ign http://ftp.uni-kl.de unstable/main Packages                            
Ign http://ftp.uni-kl.de unstable/unikl Packages                             
Hit http://ftp.uni-kl.de unstable/main Packages                              
99% [7 Sources bzip2 0] [Waiting for headers]                        
bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Resource temporarily unavailable
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Err http://ftp.uni-kl.de unstable/main Sources
  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_main_source_Sources - open (2 No such file or directory)
99% [8 Sources bzip2 0] [Waiting for headers]
bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Resource temporarily unavailable
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Hit http://ftp.uni-kl.de unstable/unikl Packages
Err http://ftp.uni-kl.de unstable/contrib Sources
  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_contrib_source_Sources - open (2 No such file or directory)
99% [9 Sources bzip2 0]
bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Resource temporarily unavailable
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Err http://ftp.uni-kl.de unstable/non-free Sources
  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_non-free_source_Sources - open (2 No such file or directory)
Fetched 3925kB in 2s (1408kB/s)
Failed to fetch http://ftp.uni-kl.de/debian/dists/unstable/main/source/Sources.bz2  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_main_source_Sources - open (2 No such file or directory)
Failed to fetch http://ftp.uni-kl.de/debian/dists/unstable/contrib/source/Sources.bz2  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_contrib_source_Sources - open (2 No such file or directory)
Failed to fetch http://ftp.uni-kl.de/debian/dists/unstable/non-free/source/Sources.bz2  Could not open file /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_non-free_source_Sources - open (2 No such file or directory)
Reading package lists... Done
E: Some index files failed to download, they have been ignored, or old ones used instead.
zombie:/tmp# ls -la /var/lib/apt/lists/partial/
total 36
drwxr-xr-x 2 ed   ed      24 Jul  7 21:05 .
drwxr-xr-x 3 root root 16384 Jul  7 21:05 ..
-rw-r--r-- 1 root root     0 Jul  7 21:05 ftp.uni-kl.de_debian_dists_unstable_contrib_source_Sources.decomp
-rw-r--r-- 1 root root     0 Jul  7 21:05 ftp.uni-kl.de_debian_dists_unstable_main_source_Sources.decomp
-rw-r--r-- 1 root root     0 Jul  7 21:05 ftp.uni-kl.de_debian_dists_unstable_non-free_source_Sources.decomp
zombie:/tmp# rm /var/lib/apt/lists/partial/ftp.uni-kl.de_debian_dists_unstable_*
zombie:/tmp# apt-get update
Get:1 http://ftp.uni-kl.de unstable Release.gpg [189B]
Get:2 http://ftp.uni-kl.de experimental Release.gpg [189B]
Get:3 http://ftp.uni-kl.de stable Release.gpg [189B]
Ign http://ftp.uni-kl.de unstable Release.gpg
Hit http://ftp.uni-kl.de unstable Release
Hit http://ftp.uni-kl.de experimental Release 
Hit http://ftp.uni-kl.de stable Release       
Ign http://ftp.uni-kl.de unstable Release     
Hit http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/contrib Packages/DiffIndex
Hit http://ftp.uni-kl.de unstable/non-free Packages/DiffIndex
Get:4 http://ftp.uni-kl.de unstable/main Sources/DiffIndex [12.5kB]
Get:5 http://ftp.uni-kl.de unstable/contrib Sources/DiffIndex [12.5kB]
Get:6 http://ftp.uni-kl.de unstable/non-free Sources/DiffIndex [12.5kB]
Get:7 http://ftp.uni-kl.de unstable/main Sources/DiffIndex [12.5kB]
Get:8 http://ftp.uni-kl.de unstable/contrib Sources/DiffIndex [12.5kB]
Get:9 http://ftp.uni-kl.de unstable/non-free Sources/DiffIndex [12.5kB]
Hit http://ftp.uni-kl.de experimental/main Packages/DiffIndex
Get:10 http://ftp.uni-kl.de experimental/main Sources/DiffIndex [12.5kB]
Ign http://ftp.uni-kl.de stable/main Sources/DiffIndex
Ign http://ftp.uni-kl.de stable/contrib Sources/DiffIndex
Ign http://ftp.uni-kl.de stable/non-free Sources/DiffIndex
Ign http://ftp.uni-kl.de unstable/main Packages/DiffIndex
Ign http://ftp.uni-kl.de unstable/unikl Packages/DiffIndex
Ign http://ftp.uni-kl.de stable/main Sources
Ign http://ftp.uni-kl.de stable/contrib Sources
Ign http://ftp.uni-kl.de stable/non-free Sources
Ign http://ftp.uni-kl.de unstable/main Packages
Ign http://ftp.uni-kl.de unstable/unikl Packages
Hit http://ftp.uni-kl.de stable/main Sources
Hit http://ftp.uni-kl.de stable/contrib Sources
Hit http://ftp.uni-kl.de stable/non-free Sources
Hit http://ftp.uni-kl.de unstable/main Packages
Hit http://ftp.uni-kl.de unstable/unikl Packages
Fetched 88.1kB in 2s (35.4kB/s)
Reading package lists... Done
Reply to: