--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: virtio_net: page allocation failure on memory pressure, disables network interface
- From: Apollon Oikonomopoulos <apoikos@gmail.com>
- Date: Wed, 2 Jun 2010 16:20:53 +0300
- Message-id: <20100602132053.GA14185@noc.grnet.gr>
Package: linux-2.6
Version: 2.6.26-22lenny1
Severity: normal
Hi,
I have noticed the following behaviour on a couple of our hosts running
2.6.26-2-amd64 inside a KVM instance with virtio network interfaces.
Upon memory pressure, the following error appears:
Jun 1 11:10:38 worker1 kernel: [533872.896187] swapper: page allocation failure. order:0, mode:0x20
Jun 1 11:10:38 worker1 kernel: [533872.896187] Pid: 0, comm: swapper Not tainted 2.6.26-2-amd64 #1
Jun 1 11:10:38 worker1 kernel: [533872.896187]
Jun 1 11:10:38 worker1 kernel: [533872.896187] Call Trace:
Jun 1 11:10:38 worker1 kernel: [533872.896187] <IRQ> [<ffffffff80276bc5>] __alloc_pages_internal+0x3a6/0x3bf
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffff802956d0>] kmem_getpages+0x96/0x15f
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffff80295d60>] fallback_alloc+0x16b/0x1e1
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffff8029593d>] kmem_cache_alloc_node+0x105/0x138
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffff803b65f9>] __alloc_skb+0x3c/0x12d
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffff803b758f>] __netdev_alloc_skb+0x29/0x43
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffffa0170187>] :virtio_net:try_fill_recv+0x32/0xf1
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffffa0170c01>] :virtio_net:virtnet_poll+0x214/0x2c3
Jun 1 11:10:38 worker1 kernel: [533872.896187] [<ffffffff803bd117>] net_rx_action+0xab/0x1da
Jun 1 11:10:40 worker1 kernel: [533872.896187] [<ffffffff8023936d>] __do_softirq+0x5c/0xd1
Jun 1 11:10:40 worker1 kernel: [533872.896187] [<ffffffff8021c4ec>] ack_apic_level+0x53/0xd8
Jun 1 11:10:40 worker1 kernel: [533872.896187] [<ffffffff8020d2dc>] call_softirq+0x1c/0x28
Jun 1 11:10:40 worker1 kernel: [533872.896187] [<ffffffff8020f3e8>] do_softirq+0x3c/0x81
Jun 1 11:10:40 worker1 kernel: [533872.896187] [<ffffffff802392cb>] irq_exit+0x3f/0x85
Jun 1 11:10:40 worker1 kernel: [533872.896187] [<ffffffff8020f648>] do_IRQ+0xb9/0xd9
Jun 1 11:10:40 worker1 kernel: [533872.896187] [<ffffffff8020b0ae>] default_idle+0x0/0x46
Jun 1 11:10:40 worker1 kernel: [533872.897055] [<ffffffff8020c47d>] ret_from_intr+0x0/0x19
Jun 1 11:10:40 worker1 kernel: [533872.897055] <EOI> [<ffffffff8021eb64>] native_safe_halt+0x2/0x3
Jun 1 11:10:40 worker1 kernel: [533872.897055] [<ffffffff8021eb64>] native_safe_halt+0x2/0x3
Jun 1 11:10:40 worker1 kernel: [533872.897055] [<ffffffff8020b0d8>] default_idle+0x2a/0x46
Jun 1 11:10:40 worker1 kernel: [533872.897055] [<ffffffff8020ad04>] cpu_idle+0x8e/0xb8
Jun 1 11:10:40 worker1 kernel: [533872.897055]
Jun 1 11:10:40 worker1 kernel: [533872.897055] Mem-info:
Jun 1 11:10:40 worker1 kernel: [533872.897055] Node 0 DMA per-cpu:
Jun 1 11:10:40 worker1 kernel: [533872.897055] CPU 0: hi: 0, btch: 1 usd: 0
Jun 1 11:10:40 worker1 kernel: [533872.897055] CPU 1: hi: 0, btch: 1 usd: 0
Jun 1 11:10:40 worker1 kernel: [533872.897055] Node 0 DMA32 per-cpu:
Jun 1 11:10:40 worker1 kernel: [533872.897055] CPU 0: hi: 186, btch: 31 usd: 185
Jun 1 11:10:40 worker1 kernel: [533872.897055] CPU 1: hi: 186, btch: 31 usd: 179
Jun 1 11:10:40 worker1 kernel: [533872.897055] Active:195926 inactive:49822 dirty:14192 writeback:8653 unstable:0
Jun 1 11:10:40 worker1 kernel: [533872.897055] free:1342 slab:6630 mapped:1353 pagetables:713 bounce:0
Jun 1 11:10:40 worker1 kernel: [533872.899103] Node 0 DMA free:3988kB min:40kB low:48kB high:60kB active:5716kB inactive:1520kB present:10792kB pages_scanned:0 all_unreclaimable? no
Jun 1 11:10:40 worker1 kernel: [533872.899103] lowmem_reserve[]: 0 994 994 994
Jun 1 11:10:40 worker1 kernel: [533872.899103] Node 0 DMA32 free:1380kB min:4012kB low:5012kB high:6016kB active:777988kB inactive:197768kB present:1018016kB pages_scanned:143 all_unreclaimable? no
Jun 1 11:10:40 worker1 kernel: [533872.899103] lowmem_reserve[]: 0 0 0 0
Jun 1 11:10:40 worker1 kernel: [533872.899103] Node 0 DMA: 1*4kB 0*8kB 1*16kB 2*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3988kB
Jun 1 11:10:40 worker1 kernel: [533872.928240] Node 0 DMA32: 0*4kB 1*8kB 0*16kB 1*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1320kB
Jun 1 11:10:40 worker1 kernel: [533872.928240] 92409 total pagecache pages
Jun 1 11:10:40 worker1 kernel: [533872.928240] Swap cache: add 491980, delete 449040, find 122585/148674
Jun 1 11:10:40 worker1 kernel: [533872.928240] Free swap = 216884kB
Jun 1 11:10:40 worker1 kernel: [533872.928240] Total swap = 514072kB
Jun 1 11:10:40 worker1 kernel: [533872.928240] 262128 pages of RAM
Jun 1 11:10:40 worker1 kernel: [533872.928240] 4927 reserved pages
Jun 1 11:10:40 worker1 kernel: [533872.928240] 27390 pages shared
Jun 1 11:10:40 worker1 kernel: [533872.928240] 42940 pages swap cached
After a couple of these backtraces (always by swapper), the virtio interface
freezes, i.e. stops receiving/transmitting traffic, until it is re-configured
using ifdown/ifup.
This *might* be the same bug as
https://bugzilla.redhat.com/show_bug.cgi?id=520119
which was fixed upstream with this patch against 2.6.31:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=3161e453e4
Unfortunately, the 2.6.26 virtio code is significantly different and this patch
does not apply as-is, however it seems to me that 2.6.26 doesn't appear to be
checking for oom while allocating buffers as well.
Thanks,
Apollon
-- Package-specific info:
** Version:
Linux version 2.6.26-2-amd64 (Debian 2.6.26-22lenny1) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Wed May 12 18:03:14 UTC 2010
** Command line:
root=/dev/mapper/local-root ro
** Not tainted
** Model information
sys_vendor:
product_name:
product_version:
chassis_vendor:
chassis_version:
bios_vendor: QEMU
bios_version: QEMU
** PCI devices:
not available
** USB devices:
-- System Information:
Debian Release: 5.0.4
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.26-2-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages linux-image-2.6.26-2-amd64 depends on:
ii debconf [debconf-2.0] 1.5.24 Debian configuration management sy
ii initramfs-tools [linux-initra 0.92o tools for generating an initramfs
ii module-init-tools 3.4-1 tools for managing Linux kernel mo
linux-image-2.6.26-2-amd64 recommends no packages.
Versions of packages linux-image-2.6.26-2-amd64 suggests:
ii grub 0.97-47lenny2 GRand Unified Bootloader (Legacy v
pn linux-doc-2.6.26 <none> (no description available)
Versions of packages linux-image-2.6.26-2-amd64 is related to:
pn firmware-bnx2 <none> (no description available)
pn firmware-bnx2x <none> (no description available)
pn firmware-ipw2x00 <none> (no description available)
pn firmware-ivtv <none> (no description available)
pn firmware-iwlwifi <none> (no description available)
pn firmware-linux <none> (no description available)
pn firmware-linux-nonfree <none> (no description available)
pn firmware-qlogic <none> (no description available)
pn firmware-ralink <none> (no description available)
-- debconf information excluded
-- System Information:
Debian Release: 5.0.4
APT prefers stable
APT policy: (500, 'stable'), (80, 'testing'), (70, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32-4-amd64 (SMP w/2 CPU cores)
Locale: LANG=el_GR.UTF-8, LC_CTYPE=el_GR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages linux-image-2.6.26-2-amd64 depends on:
ii debconf [debconf-2.0] 1.5.24 Debian configuration management sy
ii initramfs-tools [linux-initra 0.92o tools for generating an initramfs
ii module-init-tools 3.4-1 tools for managing Linux kernel mo
linux-image-2.6.26-2-amd64 recommends no packages.
Versions of packages linux-image-2.6.26-2-amd64 suggests:
ii grub-pc [grub] 1.96+20080724-16 GRand Unified Bootloader, version
ii linux-doc-2.6.26 2.6.26-22lenny1 Linux kernel specific documentatio
Versions of packages linux-image-2.6.26-2-amd64 is related to:
pn firmware-bnx2 <none> (no description available)
pn firmware-bnx2x <none> (no description available)
pn firmware-ipw2x00 <none> (no description available)
pn firmware-ivtv <none> (no description available)
pn firmware-iwlwifi <none> (no description available)
pn firmware-linux <none> (no description available)
ii firmware-linux-nonfree 0.22 Binary firmware for various driver
pn firmware-qlogic <none> (no description available)
pn firmware-ralink <none> (no description available)
-- debconf information excluded
--- End Message ---