Re: Bug#64766: apt: The apt program does not obey the HTTP/1.1 RFC
On Fri, 26 May 2000 amb@gedanken.demon.co.uk wrote:
> The http method that comes with apt claims to be HTTP/1.1 compliant, but
> checking the behaviour against the RFC indicates non-compliances that may cause
> it to fail with HTTP/1.0 servers.
You have some confusion about what aspects of the spec require what
behaviors..
> | has been signaled, the client MUST NOT send any more requests on that
> | connection.
>
> I think that apt does this.
Actually it doesn't do this. I suppose it should, but that is a very minor
point. Any server that has problems with APT has them before the first
request completes, so the 6th request has not been sent yet (APT keeps a 5
requeust pipeline)
This should probably be fixed, but it will not result in any improved
real-world compatibility.
> apt does not send "Connection: close" headers because it wants to use persistent
> connections where possible.
It sends Connection: keep-alive, except when talking to a proxy. (as the
spec says, read the proxy section)
> | If either the client or the server sends the close token in the
> | Connection header, that request becomes the last one for the
> | connection.
>
> ## Non-compliance ##
Barring my statement above about sending more requests this is exactly how
APT behaves. It does however wait for the server to close the connection,
the RFC says nothing at all about this last point.
> apt assumes that HTTP/1.0 servers will have persistent connections.
Not quite, APT pipelines from the start, see below for section 19.6.2
which allows this. It assumes that any server unwilling to support
persistant connections will do one of two things when presented with a
pipelined request:
1) Immediately die upon a pipelined query
2) Process only the first query and flush the rest of the queries from
the socket buffer [linger]
The RFC reall makes the same assumptions about the server.
> | Clients which assume persistent connections and pipeline immediately
> | after connection establishment SHOULD be prepared to retry their
> | connection if the first pipelined attempt fails. If a client does
> | such a retry, it MUST NOT pipeline before it knows the connection is
> | persistent. Clients MUST also be prepared to resend their requests if
> | the server closes the connection before sending all of the
> | corresponding responses.
>
> ## Non-compliance ##
>
> apt will not retry if the there is a failure. There also seem to be conditions
> with HTTP/1.0 servers where apt apt expects persistent connections, but failure
> to read the second set of data is seen as a server failure.
Nope, it does retry if there are very specific kinds of failures common
to defective servers and it does disable pipelining upon retry. See the
Server->Pipeline setting. This covers the SHOULD and MUST NOT clause.
[Older APT versions didn't do this, but it is only a SHOULD clause]
One could argue that this case should be broadend to cover the case when
the server dies in the middle of a transmission, but I do not think that
is the intention of the RFC.
It also does the second MUST clause (obviosly) if any if the remaining
pipelined requests are dropped it reconnects and retries.
> | Some clients and servers might wish to be compatible with some
> | previous implementations of persistent connections in HTTP/1.0
> | clients and servers. Persistent connections in HTTP/1.0 are
> | explicitly negotiated as they are not the default behavior.
>
> ## Non-compliance ##
>
> apt assumes that persistent connections are the default to HTTP/1.0 servers.
> There is no way that apt will consider a connection to be non-persistent.
APT follows Section 19.6.2 because it pipelines from the start. APT
inserts the 'Connection: keep-alive' header which is the HTTP/1.0
extension negotiation header. This covers the 'Explicit Negotiation' bit
of your quote. Thus is foes follow the recommendations (again barring the
bit above about ignoring close headers)
BTW, you are confusing Pipelining with Persistance, you need to sort that
out to make correct sense of the RFC.
APT does not have any idea what the server is until it is too late [that
is what you get for pipelining from the start]. If the server panics on
this initial pipelined request and immediately aborts then things go
according to the RFC [reconnect, retry with pipelining off]. If the server
panics and sends half a data response then APT aborts [this situtation is
not covered by the RFC].
There are some very defective HTTP/1.0 implementations that do this second
thing, and I think they are in violation of the HTTP/1.0 RFC. Certianly
the HTTP/1.1 RFC does not anticipate these servers exist and provides no
explicit recommendations for dealing with them!
IIRC the only way a server can have a short response (aside from big
server screw ups, like SEGV's) is if the server does not do a lingering
close, or read all pipelined requests into an internal buffer. This
causes the TCP stack to abort the unread data upon close and results in a
short file. This is a bug in the server, not the clients.
The only resonable way to deal with these servers is to not pipeline from
the start at all (even though Section 19.6.2 says you can). It is the
pipelined request that breaks the server, not anything to do with
persistance. You can make APT do this by setting the pipline depth to 0.
I have only seen two servers which are this badly broken, Boa and
wwwoffle, new versions of Boa are largely fixed.
Jason
Reply to: