Bug#692957: linux-image-3.2.0-4-amd64: NFS server causes high load on 3.2 kernel
Package: src:linux
Version: 3.2.32-1
Severity: important
I originally niticed the issue on Ubuntu 12.04.1, but could reproduce it on
Debian as well.
I upgraded a squeeze system to testing and found that NFS access
creates a high load on the NFS server. A single client machine with
a single write causes 40% system load, where the original load on Squeeze
was 7-10% (which still is high, but might be causes by running in a test VM).
My Ubuntu production server got completely killed by several clients
accessing.
Steps to reproduce:
Server:
- install vanilla Squeeze server system without Graphical UI and standard
settings otherwise
- install nfs-kernel-server
- add to exports:
/export client(rw,no_root_squash,async,subtree_check)
- exportfs -a
Client:
- same installation, install nfs-common
- mount with
mount -t nfs server:/export /mnt
- go to /mnt and write a file with
dd if=/dev/zero of /mnt/t1 bs=1M count=10000
- measure the load on the server while writing
- upgrade the server to Wheezy
- do the same writing test and see the load go up
If I boot Wheezy to the still installed 2.6 kernel, the load goes back to
normal.
I have played around with different nfs export and mount options. The only
change that I found was using -o proto=uds when mounting. This reduces the
load to 15% which is still higher than the original load, but still much lower
that 40%.
Ubuntu Bug #879334 and it's duplicates report the same issues.
-- Package-specific info:
** Version:
Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-12) ) #1 SMP Debian 3.2.32-1
** Command line:
BOOT_IMAGE=/boot/vmlinuz-3.2.0-4-amd64 root=UUID=0f95e4ef-e796-41b1-9e14-8878428a61a4 ro quiet
** Tainted: O (4096)
* Out-of-tree module has been loaded.
** Kernel log:
[ 1.312867] Switching to clocksource tsc
[ 1.461471] e1000 0000:00:03.0: eth0: (PCI:33MHz:32-bit) 08:00:27:f2:11:56
[ 1.461481] e1000 0000:00:03.0: eth0: Intel(R) PRO/1000 Network Connection
[ 1.461562] ohci_hcd 0000:00:06.0: setting latency timer to 64
[ 1.461574] ohci_hcd 0000:00:06.0: OHCI Host Controller
[ 1.461590] ohci_hcd 0000:00:06.0: new USB bus registered, assigned bus number 1
[ 1.461649] ohci_hcd 0000:00:06.0: irq 22, io mem 0xf0804000
[ 1.517407] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[ 1.517410] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 1.517412] usb usb1: Product: OHCI Host Controller
[ 1.517414] usb usb1: Manufacturer: Linux 3.2.0-4-amd64 ohci_hcd
[ 1.517415] usb usb1: SerialNumber: 0000:00:06.0
[ 1.517499] hub 1-0:1.0: USB hub found
[ 1.517511] hub 1-0:1.0: 8 ports detected
[ 1.517926] ahci 0000:00:0d.0: version 3.0
[ 1.518083] ahci: SSS flag set, parallel bus scan disabled
[ 1.518247] ahci 0000:00:0d.0: AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl SATA mode
[ 1.518251] ahci 0000:00:0d.0: flags: 64bit ncq stag only ccc
[ 1.518273] ahci 0000:00:0d.0: setting latency timer to 64
[ 1.519072] scsi0 : ahci
[ 1.519140] ata1: SATA max UDMA/133 abar m8192@0xf0806000 port 0xf0806100 irq 21
[ 1.519212] ata_piix 0000:00:01.1: version 2.13
[ 1.519276] ata_piix 0000:00:01.1: setting latency timer to 64
[ 1.520413] scsi1 : ata_piix
[ 1.520588] scsi2 : ata_piix
[ 1.520627] ata2: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0xd000 irq 14
[ 1.520630] ata3: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0xd008 irq 15
[ 1.676448] ata3.00: ATAPI: VBOX CD-ROM, 1.0, max UDMA/133
[ 1.676901] ata3.00: configured for UDMA/33
[ 1.840110] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 1.840185] ata1.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133
[ 1.840188] ata1.00: 32118784 sectors, multi 128: LBA48 NCQ (depth 31/32)
[ 1.840284] ata1.00: configured for UDMA/133
[ 1.840360] scsi 0:0:0:0: Direct-Access ATA VBOX HARDDISK 1.0 PQ: 0 ANSI: 5
[ 1.840852] scsi 2:0:0:0: CD-ROM VBOX CD-ROM 1.0 PQ: 0 ANSI: 5
[ 1.846811] sd 0:0:0:0: [sda] 32118784 512-byte logical blocks: (16.4 GB/15.3 GiB)
[ 1.846844] sd 0:0:0:0: [sda] Write Protect is off
[ 1.846846] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 1.846860] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.847914] sda: sda1 sda2 < sda5 >
[ 1.848330] sd 0:0:0:0: [sda] Attached SCSI disk
[ 1.848710] sr0: scsi3-mmc drive: 32x/32x xa/form2 tray
[ 1.848712] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 1.848949] sr 2:0:0:0: Attached scsi CD-ROM sr0
[ 1.853273] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 1.853324] sr 2:0:0:0: Attached scsi generic sg1 type 5
[ 1.924105] usb 1-1: new full-speed USB device number 2 using ohci_hcd
[ 1.961261] PM: Starting manual resume from disk
[ 1.961264] PM: Hibernation image partition 8:5 present
[ 1.961265] PM: Looking for hibernation image.
[ 1.961490] PM: Image not found (code -22)
[ 1.961492] PM: Hibernation image not present or could not be loaded.
[ 1.981826] kjournald starting. Commit interval 5 seconds
[ 1.981841] EXT3-fs (sda1): mounted filesystem with ordered data mode
[ 2.171387] usb 1-1: New USB device found, idVendor=80ee, idProduct=0021
[ 2.171390] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=0
[ 2.171392] usb 1-1: Product: USB Tablet
[ 2.171393] usb 1-1: Manufacturer: VirtualBox
[ 3.353470] udevd[273]: starting version 175
[ 3.633138] input: PC Speaker as /devices/platform/pcspkr/input/input1
[ 3.718429] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[ 3.719541] ACPI: Power Button [PWRF]
[ 3.719618] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input3
[ 3.719698] ACPI: Sleep Button [SLPF]
[ 3.849046] piix4_smbus 0000:00:07.0: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[ 3.964876] Error: Driver 'pcspkr' is already registered, aborting...
[ 3.972125] input: VirtualBox USB Tablet as /devices/pci0000:00/0000:00:06.0/usb1/1-1/1-1:1.0/input/input4
[ 3.972278] generic-usb 0003:80EE:0021.0001: input,hidraw0: USB HID v1.10 Mouse [VirtualBox USB Tablet] on usb-0000:00:06.0-1/input0
[ 3.972289] usbcore: registered new interface driver usbhid
[ 3.972291] usbhid: USB HID core driver
[ 3.986582] snd_intel8x0 0000:00:05.0: setting latency timer to 64
[ 4.005789] ACPI: AC Adapter [AC] (on-line)
[ 4.120551] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input5
[ 4.132142] parport_pc 00:04: reported by Plug and Play ACPI
[ 4.169039] input: Unspecified device as /devices/virtual/input/input6
[ 4.169571] vboxguest: major 0, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
[ 4.169574] vboxguest: Successfully loaded version 4.1.18_Debian (interface 0x00010004)
[ 4.340088] intel8x0_measure_ac97_clock: measured 54160 usecs (9432 samples)
[ 4.340090] intel8x0: measured clock 174150 rejected
[ 4.700096] intel8x0_measure_ac97_clock: measured 55939 usecs (7200 samples)
[ 4.700099] intel8x0: measured clock 128711 rejected
[ 5.060126] intel8x0_measure_ac97_clock: measured 55964 usecs (7200 samples)
[ 5.060128] intel8x0: measured clock 128654 rejected
[ 5.060130] intel8x0: clocking to 48000
[ 5.761383] Adding 702460k swap on /dev/sda5. Priority:-1 extents:1 across:702460k
[ 6.213599] EXT3-fs (sda1): using internal journal
[ 6.347686] loop: module loaded
[ 7.432791] RPC: Registered named UNIX socket transport module.
[ 7.432794] RPC: Registered udp transport module.
[ 7.432795] RPC: Registered tcp transport module.
[ 7.432796] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 7.525896] FS-Cache: Loaded
[ 7.567347] FS-Cache: Netfs 'nfs' registered for caching
[ 7.616799] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 7.806216] ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 7.833988] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 7.841704] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 8.473525] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 8.504888] NFSD: starting 90-second grace period
[ 18.616241] eth0: no IPv6 routers present
** Model information
sys_vendor: innotek GmbH
product_name: VirtualBox
product_version: 1.2
chassis_vendor: Oracle Corporation
chassis_version:
bios_vendor: innotek GmbH
bios_version: VirtualBox
board_vendor: Oracle Corporation
board_name: VirtualBox
board_version: 1.2
** Loaded modules:
nfsd
nfs
nfs_acl
auth_rpcgss
fscache
lockd
sunrpc
loop
vboxguest(O)
parport_pc
parport
psmouse
joydev
processor
ac
snd_intel8x0
snd_ac97_codec
usbhid
snd_pcm
snd_page_alloc
hid
snd_timer
snd
i2c_piix4
i2c_core
ac97_bus
soundcore
thermal_sys
button
serio_raw
power_supply
evdev
pcspkr
ext3
mbcache
jbd
sg
sr_mod
sd_mod
crc_t10dif
cdrom
ata_generic
ata_piix
ahci
libahci
ohci_hcd
ehci_hcd
libata
e1000
usbcore
usb_common
scsi_mod
** Network interface configuration:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp
** Network status:
*** IP interfaces and addresses:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:f2:11:56 brd ff:ff:ff:ff:ff:ff
inet 192.168.178.32/24 brd 192.168.178.255 scope global eth0
inet6 fe80::a00:27ff:fef2:1156/64 scope link
valid_lft forever preferred_lft forever
*** Device statistics:
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
lo: 260 4 0 0 0 0 0 0 260 4 0 0 0 0 0 0
eth0: 666331 869 0 0 0 0 0 6 26382 281 0 0 0 0 0 0
*** Protocol statistics:
Ip:
860 total packets received
5 with invalid addresses
0 forwarded
0 incoming packets discarded
855 incoming packets delivered
273 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
5 active connections openings
0 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
435 segments received
252 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
22 packets received
0 packets to unknown port received.
0 packet receive errors
22 packets sent
UdpLite:
TcpExt:
1 TCP sockets finished time wait in fast timer
1 delayed acks sent
356 packet headers predicted
15 acknowledgments not containing data payload received
IpExt:
InMcastPkts: 6
InBcastPkts: 392
InOctets: 653827
OutOctets: 21532
InMcastOctets: 216
InBcastOctets: 35378
** PCI devices:
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
00:01.1 IDE interface [0101]: Intel Corporation 82371AB/EB/MB PIIX4 IDE [8086:7111] (rev 01) (prog-if 8a [Master SecP PriP])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
Region 4: I/O ports at d000 [size=16]
Kernel driver in use: ata_piix
00:02.0 VGA compatible controller [0300]: InnoTek Systemberatung GmbH VirtualBox Graphics Adapter [80ee:beef] (prog-if 00 [VGA controller])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at e0000000 (32-bit, prefetchable) [size=16M]
Expansion ROM at <unassigned> [disabled]
00:03.0 Ethernet controller [0200]: Intel Corporation 82543GC Gigabit Ethernet Controller (Copper) [8086:1004] (rev 02)
Subsystem: Intel Corporation PRO/1000 T Server Adapter [8086:1004]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (63750ns min)
Interrupt: pin A routed to IRQ 19
Region 0: Memory at f0000000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at d010 [size=8]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [e4] PCI-X non-bridge device
Command: DPERE- ERO+ RBC=512 OST=1
Status: Dev=ff:1f.0 64bit- 133MHz- SCD- USC- DC=simple DMMRBC=2048 DMOST=1 DMCRS=8 RSCEM- 266MHz- 533MHz-
Kernel driver in use: e1000
00:04.0 System peripheral [0880]: InnoTek Systemberatung GmbH VirtualBox Guest Service [80ee:cafe]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 20
Region 0: I/O ports at d020 [size=32]
Region 1: Memory at f0400000 (32-bit, non-prefetchable) [size=4M]
Region 2: Memory at f0800000 (32-bit, prefetchable) [size=16K]
00:05.0 Multimedia audio controller [0401]: Intel Corporation 82801AA AC'97 Audio Controller [8086:2415] (rev 01)
Subsystem: Intel Corporation Device [8086:0000]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Interrupt: pin A routed to IRQ 21
Region 0: I/O ports at d100 [size=256]
Region 1: I/O ports at d200 [size=64]
Kernel driver in use: snd_intel8x0
00:06.0 USB controller [0c03]: Apple Inc. KeyLargo/Intrepid USB [106b:003f] (prog-if 10 [OHCI])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Interrupt: pin A routed to IRQ 22
Region 0: Memory at f0804000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci_hcd
00:07.0 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 08)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 9
00:0d.0 SATA controller [0106]: Intel Corporation 82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode] [8086:2829] (rev 02) (prog-if 01 [AHCI 1.0])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Interrupt: pin A routed to IRQ 21
Region 0: I/O ports at d240 [size=8]
Region 1: I/O ports at <ignored>
Region 2: I/O ports at d250 [size=8]
Region 3: I/O ports at <ignored>
Region 4: I/O ports at d260 [size=16]
Region 5: Memory at f0806000 (32-bit, non-prefetchable) [size=8K]
Capabilities: [70] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000002
Kernel driver in use: ahci
** USB devices:
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 80ee:0021 VirtualBox USB Tablet
-- System Information:
Debian Release: wheezy/sid
APT prefers testing-updates
APT policy: (500, 'testing-updates'), (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 3.2.0-4-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages linux-image-3.2.0-4-amd64 depends on:
ii debconf [debconf-2.0] 1.5.46
ii initramfs-tools [linux-initramfs-tool] 0.109
ii kmod 9-2
ii linux-base 3.5
ii module-init-tools 9-2
Versions of packages linux-image-3.2.0-4-amd64 recommends:
ii firmware-linux-free 3.1
Versions of packages linux-image-3.2.0-4-amd64 suggests:
pn debian-kernel-handbook <none>
ii grub-pc 1.99-23
pn linux-doc-3.2 <none>
Versions of packages linux-image-3.2.0-4-amd64 is related to:
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux <none>
pn firmware-linux-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-ralink <none>
pn firmware-realtek <none>
pn xen-hypervisor <none>
-- debconf information:
linux-image-3.2.0-4-amd64/postinst/depmod-error-initrd-3.2.0-4-amd64: false
linux-image-3.2.0-4-amd64/prerm/removing-running-kernel-3.2.0-4-amd64: true
linux-image-3.2.0-4-amd64/postinst/ignoring-ramdisk:
linux-image-3.2.0-4-amd64/postinst/missing-firmware-3.2.0-4-amd64:
Reply to: