Bug#290694: marked as done (apt: http method garbles data with 'chunked' transfer-encoding)
Your message dated Wed, 18 Jan 2006 16:02:12 -0800
with message-id <E1EzNFk-0000iF-N2@spohr.debian.org>
and subject line Bug#290694: fixed in apt 0.6.43.2
has caused the attached Bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I am
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Debian bug tracking system administrator
(administrator, Debian Bugs database)
--------------------------------------
Received: (at submit) by bugs.debian.org; 16 Jan 2005 00:21:28 +0000
>From loethi-debian@meck.oche.de Sat Jan 15 16:21:28 2005
Return-path: <loethi-debian@meck.oche.de>
Received: from (meck.oche.de) [212.21.70.4]
by spohr.debian.org with smtp (Exim 3.35 1 (Debian))
id 1CpyAY-0000zH-00; Sat, 15 Jan 2005 16:21:27 -0800
Received: (qmail 20102 invoked from network); 16 Jan 2005 00:20:43 -0000
Received: from unknown (HELO nevermind.krynn) (192.168.144.210)
by haven.krynn with SMTP; 16 Jan 2005 00:20:43 -0000
Received: from lothar by nevermind.krynn with local (Exim 4.34)
id 1Cpy9q-00031D-FQ; Sun, 16 Jan 2005 01:20:42 +0100
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: Lothar Wassmann <loethi-debian@meck.oche.de>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: apt: http method garbles data with 'chunked' transfer-encoding
X-Mailer: reportbug 3.2
Date: Sun, 16 Jan 2005 01:20:41 +0100
Message-Id: <E1Cpy9q-00031D-FQ@nevermind.krynn>
Delivered-To: submit@bugs.debian.org
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2005_01_02
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-6.6 required=4.0 tests=BAYES_00,HAS_PACKAGE,
WEIRD_PORT autolearn=no version=2.60-bugs.debian.org_2005_01_02
X-Spam-Level:
Package: apt
Version: 0.5.27
Severity: important
Hi,
retrieving packages with apt-get via an HTTP proxy (e.g. apt-proxy)
that uses "Transfer-Encoding: chunked" leads to random data corruption
in the retrieved package files and an error message:
|nevermind:~# apt-get install libx11-6
|Failed to fetch http://nevermind.krynn:9999/debian/pool/main/x/xfree86/libx11-6_4.3.0.dfsg.1-10_i386.deb MD5Sum mismatch
Using ethereal to capture the network traffic and comparing the
garbled package file with a good one I found that data corruption
always occurs when a chunk of data starts with a <CR> (0x0d).
"strace -f apt-get install ..." revealed that /usr/lib/apt/methods/http is the culprit:
>> pid 26438 is /usr/lib/apt/methods/http
>> fd 3 is the socket of the server connection
>> fd 4 is "/var/cache/apt/archives/partial/libx11-6_4.3.0.dfsg.1-10_i386.deb"
[...]
[pid 26438] read(3, "ff0\r\n\r\364\6{C\275\351\336L\357p\357d\357\266\336\35"..., 11906) = 1448
^^^^^^ start of DATA
^^^^ CRLF delimiter
^^^
Data Length Token of the Chunked Transfer-Encoding
[pid 26438] read(3, 0x8063656, 10458) = -1 EAGAIN (Resource temporarily unavailable)
[pid 26438] select(5, [0 3], [4], NULL, {120, 0}) = 1 (out [4], left {120, 0})
[pid 26438] write(4, "\364\6{C\275\351\336L\357p\357d\357\266\336\35\275\317"..., 1442) = 1442
[...]
^
This block should begin with '\r' and the size should be 1443!
(1448 byte returned by read() minus 3 byte chunk length code minus 2 byte CRLF)
A look at the source code of the http method revealed that the
CircleBuf::WriteTillEl() method is broken.
Assuming we entered the routine to read the data length token at the
start of a data chunk this is what happens:
| for (unsigned long I = OutP; I < InP; I++)
| {
| if (Buf[I%Size] != '\n')
| continue;
>> we get here with 'I' pointing to the '\n' after the token,
>> having already skipped the preceding '\r'
| ++I;
>> we skip the delimiter, leaving 'I' pointing to the first DATA
>> byte of the chunk!
| if (I < InP && Buf[I%Size] == '\r')
| ++I;
>> IF the first DATA byte happens to be '\r' it will also be skipped!
>> Exactly what I have seen in the strace log
>> Since the block size is known and the DATA block is delimited
>> with '\r\n' The '\r' of the final delimiter will show up in the
>> data block, making the block appear to have been rotated by one
>> byte.
|
| if (Single == false)
| {
| if (Buf[I%Size] != '\n')
>> Not relevant here, BUT since 'I' might be equal to InP
>> already we might be checking an invalid byte from the buffer!
| continue;
| ++I;
| if (I < InP && Buf[I%Size] == '\r')
| ++I;
| }
At the first glance of the routine the sequence of comparisons '\n',
'\r' looked suspicious to me (the delimiter sequence is '\r\n', so the
comparisons ought to be in the same order!).
So I decided to swap them (directly editing the binary using hexl-mode
of emacs ;) so that the loop terminates after it has found a '\r\n'
sequence as it should.
With this change I have successfully installed a dozen packages that
have failed with MD5Sum errors before.
Lothar Wassmann
-- Package-specific info:
-- apt-config dump --
APT "";
APT::Architecture "i386";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Clean-Installed "off";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::userstatus "status.user";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::dpkg "/usr/bin/dpkg";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
Acquire "";
Acquire::http "";
Acquire::http::Pipeline-Depth "0";
-- (no /etc/apt/preferences present) --
-- /etc/apt/sources.list --
# This sources.list was generated by apt-sources v2.20
# Copyright(c) 2001 by Danny Rodriguez
#
deb http://nevermind.krynn:9999/debian testing main contrib non-free
deb http://nevermind.krynn:9999/non-US testing/non-US main contrib non-free
deb http://nevermind.krynn:9999/security testing/updates main contrib
-- System Information:
Debian Release: 3.1
APT prefers testing
APT policy: (500, 'testing')
Architecture: i386 (i686)
Kernel: Linux 2.6.8.1
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Versions of packages apt depends on:
ii libc6 2.3.2.ds1-20 GNU C Library: Shared libraries an
ii libgcc1 1:3.4.3-6 GCC support library
ii libstdc++5 1:3.3.5-5 The GNU Standard C++ Library v3
-- no debconf information
---------------------------------------
Received: (at 290694-close) by bugs.debian.org; 19 Jan 2006 00:10:31 +0000
>From katie@ftp-master.debian.org Wed Jan 18 16:10:31 2006
Return-path: <katie@ftp-master.debian.org>
Received: from katie by spohr.debian.org with local (Exim 4.50)
id 1EzNFk-0000iF-N2; Wed, 18 Jan 2006 16:02:12 -0800
From: Michael Vogt <mvo@debian.org>
To: 290694-close@bugs.debian.org
X-Katie: $Revision: 1.65 $
Subject: Bug#290694: fixed in apt 0.6.43.2
Message-Id: <E1EzNFk-0000iF-N2@spohr.debian.org>
Sender: Archive Administrator <katie@ftp-master.debian.org>
Date: Wed, 18 Jan 2006 16:02:12 -0800
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2005_01_02
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Level:
X-Spam-Status: No, hits=-6.0 required=4.0 tests=BAYES_00,HAS_BUG_NUMBER
autolearn=no version=2.60-bugs.debian.org_2005_01_02
X-CrossAssassin-Score: 5
Source: apt
Source-Version: 0.6.43.2
We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive:
apt-doc_0.6.43.2_all.deb
to pool/main/a/apt/apt-doc_0.6.43.2_all.deb
apt-utils_0.6.43.2_i386.deb
to pool/main/a/apt/apt-utils_0.6.43.2_i386.deb
apt_0.6.43.2.dsc
to pool/main/a/apt/apt_0.6.43.2.dsc
apt_0.6.43.2.tar.gz
to pool/main/a/apt/apt_0.6.43.2.tar.gz
apt_0.6.43.2_i386.deb
to pool/main/a/apt/apt_0.6.43.2_i386.deb
libapt-pkg-dev_0.6.43.2_i386.deb
to pool/main/a/apt/libapt-pkg-dev_0.6.43.2_i386.deb
libapt-pkg-doc_0.6.43.2_all.deb
to pool/main/a/apt/libapt-pkg-doc_0.6.43.2_all.deb
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 290694@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Michael Vogt <mvo@debian.org> (supplier of updated apt package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Format: 1.7
Date: Thu, 19 Jan 2006 00:06:33 +0100
Source: apt
Binary: apt-utils libapt-pkg-doc libapt-pkg-dev apt-doc apt
Architecture: source all i386
Version: 0.6.43.2
Distribution: unstable
Urgency: low
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Michael Vogt <mvo@debian.org>
Description:
apt - Advanced front-end for dpkg
apt-doc - Documentation for APT
apt-utils - APT utility programs
libapt-pkg-dev - Development files for APT's libapt-pkg and libapt-inst
libapt-pkg-doc - Documentation for APT development
Closes: 280844 290694 344642 346369 346450 347258 347349 347435 347729 347970 348158 348348 348574
Changes:
apt (0.6.43.2) unstable; urgency=low
.
* Merge bubulle@debian.org--2005/apt--main--0 up to patch-166:
- en_GB.po, de.po: fix spaces errors in "Ign " translations Closes: #347258
- makefile: make update-po a pre-requisite of clean target so
that POT and PO files are always up-to-date
- sv.po: Completed to 511t. Closes: #346450
- sk.po: Completed to 511t. Closes: #346369
- fr.po: Completed to 511t
- *.po: Updated from sources (511 strings)
- el.po: Completed to 511 strings Closes: #344642
- da.po: Completed to 511 strings Closes: #348574
- es.po: Updated to 510t1f Closes: #348158
- gl.po: Completed to 511 strings Closes: #347729
- it.po: Yet another update Closes: #347435
* added debian-archive-keyring to the Recommends (closes: #347970)
* fixed message in apt-key to install debian-archive-keyring
* typos fixed in apt-cache.8 (closes: #348348, #347349)
* add patch to fix http download corruption problem (thanks to
Petr Vandrovec, closes: #280844, #290694)
Files:
7d4268f9c499131b00a68761fc1de89f 789 admin important apt_0.6.43.2.dsc
278495ad6c4170efb952e51c2abc2f76 1489394 admin important apt_0.6.43.2.tar.gz
7cf2202b2c7ee8d58f55c4de75fd5798 86688 doc optional apt-doc_0.6.43.2_all.deb
1e97e61039da99bb4573cdce95fee97d 110094 doc optional libapt-pkg-doc_0.6.43.2_all.deb
98cb5abcd88b6872dd599ff7b832b9c4 1260966 admin important apt_0.6.43.2_i386.deb
15c9456a802723303160f45c8a2e27c9 80312 libdevel optional libapt-pkg-dev_0.6.43.2_i386.deb
aff14302921020807f22d3a8f0880313 195674 admin important apt-utils_0.6.43.2_i386.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFDztQjliSD4VZixzQRAngXAJ4pJYfCyzRVLi5qgLQ8z3Tl2EyJTACfS1dj
rbDk4dwXeJGwYdvZaj4qLi8=
=JEWN
-----END PGP SIGNATURE-----
Reply to: