On Sun, Oct 10, 2004 at 04:42:11AM +0800, Dan Jacobson wrote: > Here we see the effect reading from local wwwoffle on an offline system: > > # aptitude install ftm > Need to get 27.2kB of archives. After unpacking 73.7kB will be used. > Get:1 http://debian.linux.org.tw sid/main ftm 0.0.7 [27.2kB] > Err http://debian.linux.org.tw sid/main ftm 0.0.7 > Error reading from server Remote end closed connection > # aptitude install ftm > Need to get 27.2kB of archives. After unpacking 73.7kB will be used. > Get:1 http://debian.linux.org.tw sid/main ftm 0.0.7 [27.2kB] > Fetched 27.2kB in 0s (2411kB/s) > > One cannot reproduce it usually without rebooting, as internal buffers > filling affect timing issues, etc. I can reproduce the problem, and I think I know why it happens. I wrote a proxy that converts every data stream going over it to Transfer-Encoding: Chunked. It's available at: <URL:http://greek0@gmx.net/~greek0/div/chunked-proxy.py> It works quite well AFAICT, but it relies on python's httplib, which turned out to have some problems as well, at least if the servers send strange data (\r in the headers without \r\n), as ftp.at.debian.org does. So if you try it out and get garbled data please make sure that it really is apt's fault. The problem seems to be timing sensitive, since singlestepping in gdb, as well as adding adding sleep(10) calls to the proxy prevent it from happening. The error message and the corrupted data comes from HttpMethod::ServerDie, which is called by HttpMethod::Go when it fails to read data from the server (ie. the server closed the connection). ServerDie then flushes all the Data that we have already read (usually either all the Data the server sent us excluding headers, or just the last part of the file that we haven't written out yet). It does this even if we are using a Transfer-Encoding, so the resulting file is corrupted, since the Encoding is not stripped. Additionally we usually get a "Remote host closed connection" error. I don't really know what's the right point to fix the problem. IMHO ServerDie might be too late. HttpMethod::Go might be right, but I don't know how we should detect a failing connection for chunked encoding then. Cheers, Greek0
Attachment:
signature.asc
Description: Digital signature