Bug#383487: parsing packages files: Reads from pipe results in corrupt data

[Jeroen van Wolffelaar <jeroen@wolffelaar.nl>]

Package: python-apt
Version: 0.6.19
Severity: serious

Please find attached a simple python script that can parse and print a
package file, via a pipe using cat or directly with 'open()'. With open,
the result will be identical to the original, as verifiable via 'diff'.
However, using os.popen with cat will result in inserted newlines, which
indicates that stanza's with packages are not read in correctly. In some
bigger files, such as current i386 packages files, you'll also see
partial reads of the long descriptions. So far, I can reproduce this
with all Packages files in unstable/main, or basicly any packages file
that has sufficient many entries. I cannot at first sight find and
pattern, it's seemingly random, but it's reproducible to the exact same
places when dealing with the same packages file.

I've reproduced (actually, first encountered) this bug on the
0.6.19.dsa.1 backport as installed on merkel.debian.org (otherwise ia64
stable). The places where stanza's were broken up were identical there.

On merkel this is with python 2.3, on my own system I could reproduce it
with both python2.3 and python2.4.

--Jeroen

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.8-3-k7
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)

Versions of packages python-apt depends on:
ii  apt [libapt-pkg-libc6.3-6-3.1 0.6.45     Advanced front-end for dpkg
ii  apt-utils [libapt-inst-libc6. 0.6.45     APT utility programs
ii  libc6                         2.3.6-19   GNU C Library: Shared libraries
ii  libgcc1                       1:4.1.1-10 GCC support library
ii  libstdc++6                    4.1.1-10   The GNU Standard C++ Library v3
ii  python                        2.4.3-11   An interactive high-level object-o
ii  python-central                0.5.5      register and build utility for Pyt

python-apt recommends no packages.

-- no debconf information

-- 
Jeroen van Wolffelaar
Jeroen@wolffelaar.nl (also for Jabber & MSN; ICQ: 33944357)
http://Jeroen.A-Eskwadraat.nl

#!/usr/bin/python2.3

import os, sys, apt_pkg

# Doesn't trigger bug:
#file = open(sys.argv[1])

# Does trigger bug:
file = os.popen("cat "+sys.argv[1], 'r')

parse = apt_pkg.ParseTagFile(file)
while parse.Step() == 1:
    for k in parse.Section.keys():
        print k + ": " + parse.Section.get(k)

    print ""