Bug#28954: marked as done (libc6: DoS fix which breaks rpc.nfsd)
Your message dated Sun, 14 Feb 1999 11:58:13 -0800
with message-id <v04104425b2ecdb1b206b@[206.163.71.146]>
and subject line Fixed in 2.0.7.19981211-4
has caused the attached bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I'm
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Ian Jackson
(administrator, Debian bugs database)
Received: (at submit) by bugs.debian.org; 4 Nov 1998 20:27:40 +0000
Received: (qmail 21159 invoked from network); 4 Nov 1998 20:26:52 -0000
Received: from bigfoot.eecs.umich.edu (steveh@141.213.4.21)
by master.debian.org with SMTP; 4 Nov 1998 20:26:52 -0000
Received: (from steveh@localhost)
by bigfoot.eecs.umich.edu (8.9.1a/8.9.1/Debian/GNU) id PAA07016;
Wed, 4 Nov 1998 15:26:37 -0500
Date: Wed, 4 Nov 1998 15:26:37 -0500
Message-Id: <199811042026.PAA07016@bigfoot.eecs.umich.edu>
From: <steveh@eecs.umich.edu>
Subject: libc6: DoS fix which breaks rpc.nfsd
To: submit@bugs.debian.org
X-Mailer: bug 3.1.5
Package: libc6
Version: 2.0.7u-2
This is FYI; I'm just passing on a note from the nfsd developer,
Olaf Kirch. Is there an older version of libc6 available which
doesn't have the call to svc_getreqset() and replace the readrcp()
function in glibc/2.0.7/sunrpc/svc_tcp.c around line 331?
>From okir@monad.swb.de Wed Nov 4 15:20:50 1998
Date: Thu, 29 Oct 1998 01:26:18 +0100
From: Olaf Kirch <okir@monad.swb.de>
To: drepper@cygnus.com
Cc: alan@lxorguk.ukuu.org.uk, Steve Hsieh <steveh@eecs.umich.edu>
Subject: Bug in glibc Sunrpc TCP code
Hi Uli,
there are some recent changes in glibc's handling of TCP requests
that are supposed to stave off the DoS attack discussed on bugtraq
two months ago. This patch has some problems. The worst of it is that
it makes the Linux unfsd die at random intervals.
What the patch basically does is to enable RPC request scheduling
within the readtcp routine that is supposed to fetch data from a
TCP stream and store it in a buffer used by the xdr_rec routines.
Problem number one, which also seems to be the one killing nfsd, is
when you have the following pattern:
connection 1: incomplete record arrives.
xdr_rec calls readtcp, which stalls.
connection 2: incomplete record arrives.
readtcp notices, and calls svc_getreqset.
We now enter the xdr_rec routines for
connection 2, and xdr_rec calls readtcp, which
stalls.
connection 1: remainder of data arrives. readtcp calls
svc_getreqset, which in turn calls the xdr_rec
routines. These routines find that the receive
buffer is empty, and fetch data from the stream
via readtcp. So far, so good. The record is
complete, we return to handle the complete
request.
connection 2: remainder of data arrives.
We return from readtcp, and business goes on
as usual.
connection 1: another request arrives. We now break out
of the select loop in readtcp and return.
The calling xdr_rec function now assumes nothing
has happened inbetween, when if fact all of its
internal state has changed.
The last sentence is deliberately vague, because I have not been able
to pinpoint the exact location of where things get hosed, but for
NFS the symptom is that a decoded WRITE request contains mostly 0xff
bytes. At the very least, what happens is that the xdr_rec code is
now out of sync wrt record markers. The fbtbc (fragment bytes to be
comsumed) counter will have random values.
The second problem with this approach is that it gives rise to another,
much more effective DoS attack that simply hanging a server. Imagine
a client opening TCP connections to the server in lockstep:
open connection 1, send some data
open connection 2, send some data
tear down connection 1
open connection 3, send some data
tear down connection 2
open connection 4, send some data
...
This will make the RPC server nest invocations of readtcp and
svc_getreqset deeper and deeper until it runs out of stack, or
memory, or both. The attack can even be performed across slow links if
you create a large number of connections in parallel, and is quite
bandwidth-efficient because you don't have to send more than a single
byte to make the code descend into readtcp().
I've got a patch (that we're also using in Caldera's OpenLinux) that fixes
the TCP problem in a different way, by collecting the entire request in
a buffer before letting the request decoder routines have their way. This
does require that you give the RPC library an advance upper limit on the
requests you're willing to handle via TCP, though. If you're interested,
let me know.
Cheers
Olaf
--
Olaf Kirch | --- o --- Nous sommes du soleil we love when we play
okir@monad.swb.de | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax
okir@caldera.de +-------------------- Why Not?! -----------------------
UNIX, n.: Spanish manufacturer of fire extinguishers.
Reply to: