Bug#592187: [stable] Bug#576838: virtio network crashes again
Hi all,
> > I was finally able to identify the patch series that introduced the fix
> > (they were introduced to -stable in 2.6.33.2):
> >
> > cb63112 net: add __must_check to sk_add_backlog
> > a12a9a2 net: backlog functions rename
> > 51c5db4 x25: use limited socket backlog
> > c531ab2 tipc: use limited socket backlog
> > 37d60aa sctp: use limited socket backlog
> > 9b3d968 llc: use limited socket backlog
> > 230401e udp: use limited socket backlog
> > 20a92ec tcp: use limited socket backlog
> > ab9dd05 net: add limit for socket backlog
> >
> > After applying these to 2.6.32.17, I wasn't able to trigger the failure
> > anymore.
>
> What "failure"?
>From my other mail, for public reference:
> With 2.6.32.17 as a KVM guest using virtio_net, large nfs reads and
> writes cause the network to crash. Only rmmod virtio_net/modprobe
> virtio_net fixes it. I found that this bug was fixed in 2.6.33.2, and
> git bisect pointed me to the following patch series, which, when
applied
> to 2.6.32.17, fixes the problem:
I have to add that this also happens to bare-bone systems on real
hardware - we just had a machine crash during it's nightly nfs backup
with a slew of page allocation failures.
> > 230401e didn't apply cleanly with git cherry-pick on top of 2.6.32.17,
> > so there might be some additional work needed.
> >
> > @Greg: would it be possible to have these fixes in the next 2.6.32? See
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592187#69 for details:
> > they fix a guest network crash during heavy nfs-io using virtio.
>
> These are a lot of patches, looking like they are adding a new feature.
> I would need to get the ack of the network maintainer before I can add
> them.
>
> David?
fyi, the comment of the original patch series applied to 2.6.33. It's
not a new feature per se, but a fix to a general problem.
Author: Zhu Yi <yi.zhu@intel.com>
Date: Thu Mar 4 18:01:40 2010 +0000
net: add limit for socket backlog
[ Upstream commit 8eae939f1400326b06d0c9afe53d2a484a326871 ]
We got system OOM while running some UDP netperf testing on the loopback
device. The case is multiple senders sent stream UDP packets to a single
receiver via loopback on local host. Of course, the receiver is not able
to handle all the packets in time. But we surprisingly found that these
packets were not discarded due to the receiver's sk->sk_rcvbuf limit.
Instead, they are kept queuing to sk->sk_backlog and finally ate up all
the memory. We believe this is a secure hole that a none privileged user
can crash the system.
The root cause for this problem is, when the receiver is doing
__release_sock() (i.e. after userspace recv, kernel udp_recvmsg ->
skb_free_datagram_locked -> release_sock), it moves skbs from backlog to
sk_receive_queue with the softirq enabled. In the above case, multiple
busy senders will almost make it an endless loop. The skbs in the
backlog end up eat all the system memory.
The issue is not only for UDP. Any protocols using socket backlog is
potentially affected. The patch adds limit for socket backlog so that
the backlog size cannot be expanded endlessly.
Reported-by: Alex Shi <alex.shi@intel.com>
Cc: David Miller <davem@davemloft.net>
Cc: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: "Pekka Savola (ipv6)" <pekkas@netcore.fi>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Vlad Yasevich <vladislav.yasevich@hp.com>
Cc: Sridhar Samudrala <sri@us.ibm.com>
Cc: Jon Maloy <jon.maloy@ericsson.com>
Cc: Allan Stephens <allan.stephens@windriver.com>
Cc: Andrew Hendry <andrew.hendry@gmail.com>
Signed-off-by: Zhu Yi <yi.zhu@intel.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Acked-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
> thanks,
>
> greg k-h
>
Regards,
Lukas
Reply to: