Bug#592780: linux-image-2.6.32-5-amd64: pv-ops DomU kernel doesn't forward network traffic
Package: linux-2.6
Version: 2.6.32-18
Severity: normal
At work we have 4 Dom0s, all of them with different hardware. When 2.6.32 with pv-ops appeared on Debian testing (squeeze) we started testing it.
Everything went fine until we tried to configure an OpenVPN server on a DomU. We found out that forwarding traffic was not working. We tried hard to debug the problem (tcpdump, wireshark, etc) and packets reached the DomU but they never left.
After testing the DomU on every version of the Xen hypervisor (from Debian, 3.2, 3.4 and 4) and on every DomO we have we discovered almost by chance that with the -xen flavour of the kernel package the problem disappeared. AFAIK the -xen kernel has the forward-ported patches from 2.6.26, so most probably this has something to do with pv-ops.
If I can be of any help just ask, I'd be glad to.
Thanks.
-- Package-specific info:
** Version:
Linux version 2.6.32-5-amd64 (Debian 2.6.32-18) (ben@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-2) ) #1 SMP Sat Jul 24 01:47:24 UTC 2010
** Command line:
root=/dev/xvda2 ro root=/dev/xvda2 ro ip=:127.0.255.255::::eth0:dhcp
** Not tainted
** Kernel log:
[ 0.050975] type=2000 audit(1280879563.802:1): initialized
[ 0.053240] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.054473] VFS: Disk quotas dquot_6.5.2
[ 0.054521] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.054600] msgmni has been set to 1997
[ 0.054838] alg: No test for stdrng (krng)
[ 0.054891] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 0.054899] io scheduler noop registered
[ 0.054903] io scheduler anticipatory registered
[ 0.054907] io scheduler deadline registered
[ 0.054935] io scheduler cfq registered (default)
[ 0.056325] Linux agpgart interface v0.103
[ 0.056352] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.056508] input: Macintosh mouse button emulation as /devices/virtual/input/input0
[ 0.056552] PNP: No PS/2 controller found. Probing ports directly.
[ 0.057376] i8042.c: No controller found.
[ 0.057434] mice: PS/2 mouse device common for all mice
[ 0.057472] rtc_cmos: probe of rtc_cmos failed with error -16
[ 0.057495] cpuidle: using governor ladder
[ 0.057499] cpuidle: using governor menu
[ 0.057506] No iBFT detected.
[ 0.057730] TCP cubic registered
[ 0.057826] NET: Registered protocol family 10
[ 0.058194] lo: Disabled Privacy Extensions
[ 0.058384] Mobile IPv6
[ 0.060165] NET: Registered protocol family 17
[ 0.157944] PM: Resume from disk failed.
[ 0.157957] registered taskstats version 1
[ 0.157975] XENBUS: Device with no driver: device/vbd/51714
[ 0.157979] XENBUS: Device with no driver: device/vbd/51713
[ 0.157984] XENBUS: Device with no driver: device/vbd/51715
[ 0.157988] XENBUS: Device with no driver: device/vbd/51716
[ 0.157992] XENBUS: Device with no driver: device/vbd/51717
[ 0.157997] XENBUS: Device with no driver: device/vbd/51718
[ 0.158001] XENBUS: Device with no driver: device/vbd/51719
[ 0.158005] XENBUS: Device with no driver: device/vbd/51720
[ 0.158009] XENBUS: Device with no driver: device/vif/0
[ 0.158013] XENBUS: Device with no driver: device/console/0
[ 0.158027] /build/buildd-linux-2.6_2.6.32-18-amd64-qDNI1z/linux-2.6-2.6.32/debian/build/source_amd64_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[ 0.158047] Initalizing network drop monitor service
[ 0.158100] Freeing unused kernel memory: 588k freed
[ 0.158279] Write protecting the kernel read-only data: 4220k
[ 0.171348] udev: starting version 158
[ 0.202482] Initialising Xen virtual ethernet driver.
[ 0.241506] alloc irq_desc for 16 on node 0
[ 0.241510] alloc kstat_irqs on node 0
[ 0.250760] alloc irq_desc for 17 on node 0
[ 0.250763] alloc kstat_irqs on node 0
[ 0.260666] alloc irq_desc for 18 on node 0
[ 0.260669] alloc kstat_irqs on node 0
[ 0.270383] alloc irq_desc for 19 on node 0
[ 0.270386] alloc kstat_irqs on node 0
[ 0.282152] blkfront: xvda2: barriers enabled
[ 0.287507] blkfront: xvda1: barriers enabled
[ 0.289848] blkfront: xvda3: barriers enabled
[ 0.290843] blkfront: xvda4: barriers enabled
[ 0.291697] blkfront: xvda5: barriers enabled
[ 0.292553] blkfront: xvda6: barriers enabled
[ 0.293401] blkfront: xvda7: barriers enabled
[ 0.294160] blkfront: xvda8: barriers enabled
[ 0.294662] alloc irq_desc for 20 on node 0
[ 0.294665] alloc kstat_irqs on node 0
[ 1.357317] PM: Starting manual resume from disk
[ 1.357329] PM: Resume from partition 202:1
[ 1.357332] PM: Checking hibernation image.
[ 1.358119] PM: Error -22 checking image file
[ 1.358123] PM: Resume from disk failed.
[ 1.381852] kjournald starting. Commit interval 5 seconds
[ 1.381873] EXT3-fs: mounted filesystem with ordered data mode.
[ 2.505218] udev: starting version 158
[ 2.806308] input: PC Speaker as /devices/platform/pcspkr/input/input1
[ 3.147747] Error: Driver 'pcspkr' is already registered, aborting...
[ 3.272690] Adding 1048568k swap on /dev/xvda1. Priority:-1 extents:1 across:1048568k SS
[ 3.557375] EXT3 FS on xvda2, internal journal
[ 3.982768] EXT4-fs (xvda3): barriers disabled
[ 3.997998] EXT4-fs (xvda3): mounted filesystem with ordered data mode
[ 4.011050] EXT4-fs (xvda4): barriers disabled
[ 4.025188] EXT4-fs (xvda4): mounted filesystem with ordered data mode
[ 4.106832] EXT4-fs (xvda5): barriers disabled
[ 4.126327] EXT4-fs (xvda5): mounted filesystem with ordered data mode
[ 4.252909] EXT4-fs (xvda6): barriers disabled
[ 4.297945] EXT4-fs (xvda6): mounted filesystem with ordered data mode
[ 4.320913] EXT4-fs (xvda7): barriers disabled
[ 4.341559] EXT4-fs (xvda7): mounted filesystem with ordered data mode
[ 4.358895] EXT4-fs (xvda8): barriers disabled
[ 4.373572] EXT4-fs (xvda8): mounted filesystem with ordered data mode
[ 6.603694] RPC: Registered udp transport module.
[ 6.603706] RPC: Registered tcp transport module.
[ 6.603713] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 6.886321] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 6.998078] svc: failed to register lockdv1 RPC service (errno 97).
[ 6.999734] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 7.001824] NFSD: starting 90-second grace period
[ 8.397825] lp: driver loaded but no devices found
[ 8.411937] ppdev: user-space parallel port driver
[ 16.040035] eth0: no IPv6 routers present
[ 441.489557] hrtimer: interrupt took 324145830 ns
[16343.725153] nfsd: nfsv4 idmapping failing: has idmapd not been started?
[243871.194273] nfsd: nfsv4 idmapping failing: has idmapd not been started?
[746193.623447] udev: starting version 160
** Model information
not available
** Loaded modules:
Module Size Used by
parport_pc 18855 0
ppdev 5030 0
lp 7462 0
parport 27954 3 parport_pc,ppdev,lp
nfsd 254118 11
lockd 57603 1 nfsd
nfs_acl 2031 1 nfsd
auth_rpcgss 33460 1 nfsd
sunrpc 161317 14 nfsd,lockd,nfs_acl,auth_rpcgss
exportfs 3170 1 nfsd
ext4 285611 6
jbd2 66919 1 ext4
crc16 1319 1 ext4
snd_pcm 60471 0
snd_timer 15582 1 snd_pcm
snd 46446 2 snd_pcm,snd_timer
soundcore 4598 1 snd
snd_page_alloc 6249 1 snd_pcm
pcspkr 1699 0
evdev 7352 0
ext3 106518 1
jbd 37085 1 ext3
mbcache 5050 2 ext4,ext3
xen_netfront 15180 0
xen_blkfront 8102 8
** Network interface configuration:
auto lo
iface lo inet loopback
auto eth0
iface eth0:0 inet dhcp
iface eth0 inet static
address 192.168.0.220
netmask 255.255.255.0
broadcast 192.168.0.255
gateway 192.168.0.1
auto eth0:1
iface eth0:1 inet static
address 192.168.11.220
netmask 255.255.255.0
broadcast 192.168.11.255
** Network status:
*** IP interfaces and addresses:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:16:3e:b1:9b:6e brd ff:ff:ff:ff:ff:ff
inet 192.168.0.220/24 brd 192.168.0.255 scope global eth0
inet 192.168.11.220/24 brd 192.168.11.255 scope global eth0:1
inet6 fe80::216:3eff:feb1:9b6e/64 scope link
valid_lft forever preferred_lft forever
*** Device statistics:
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
lo:46253828 72490 0 0 0 0 0 0 46253828 72490 0 0 0 0 0 0
eth0:185383130466 568751722 0 0 0 0 0 0 1207832827970 231385864 0 0 0 0 0 0
*** Protocol statistics:
Ip:
558120756 total packets received
436 with invalid addresses
0 forwarded
0 incoming packets discarded
556051309 incoming packets delivered
231365330 requests sent out
Icmp:
10919 ICMP messages received
212 input ICMP message failed.
ICMP input histogram:
destination unreachable: 8551
echo requests: 2365
echo replies: 3
10946 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 8578
echo request: 3
echo replies: 2365
IcmpMsg:
InType0: 3
InType3: 8551
InType8: 2365
OutType0: 2365
OutType3: 8578
OutType8: 3
Tcp:
13197 active connections openings
96404 passive connection openings
112 failed connection attempts
21488 connection resets received
34 connections established
555733139 segments received
230891771 segments send out
273165 segments retransmited
0 bad segments received.
80716 resets sent
Udp:
230879 packets received
65 packets to unknown port received.
45962 packet receive errors
189452 packets sent
RcvbufErrors: 45962
UdpLite:
TcpExt:
2 resets received for embryonic SYN_RECV sockets
44 packets pruned from receive queue because of socket buffer overrun
8986 TCP sockets finished time wait in fast timer
19 time wait sockets recycled by time stamp
243859 delayed acks sent
5915 delayed acks further delayed because of locked socket
Quick ack mode was activated 65806 times
53388988 packets directly queued to recvmsg prequeue.
3407050928 bytes directly in process context from backlog
2960662096 bytes directly received in process context from prequeue
4395491 packets dropped from prequeue
65574100 packet headers predicted
81897165 packets header predicted and directly queued to user
39308772 acknowledgments not containing data payload received
328733394 predicted acknowledgments
6 times recovered from packet loss due to fast retransmit
97961 times recovered from packet loss by selective acknowledgements
1 bad SACK blocks received
Detected reordering 3 times using FACK
Detected reordering 1 times using time stamp
1 congestion windows fully recovered without slow start
7 congestion windows partially recovered using Hoe heuristic
1691 congestion windows recovered without slow start by DSACK
651 congestion windows recovered without slow start after partial ack
86403 TCP data loss events
TCPLostRetransmit: 293
5 timeouts after reno fast retransmit
8735 timeouts after SACK recovery
63 timeouts in loss state
145515 fast retransmits
19016 forward retransmits
35863 retransmits in slow start
51500 other TCP timeouts
1922 SACK retransmits failed
676 times receiver scheduled too late for direct processing
888 packets collapsed in receive queue due to low socket buffer
6791 DSACKs sent for old packets
15131 DSACKs received
720 connections reset due to unexpected data
170 connections reset due to early user close
343 connections aborted due to timeout
TCPDSACKIgnoredOld: 12079
TCPDSACKIgnoredNoUndo: 487
TCPSpuriousRTOs: 540
TCPSackShifted: 1639890
TCPSackMerged: 352611
TCPSackShiftFallback: 173774
IpExt:
InBcastPkts: 185207
OutBcastPkts: 53861
InOctets: -223694447
OutOctets: 2046398014
InBcastOctets: 32990469
OutBcastOctets: 11801947
*** Device features:
eth0: 0x50003
lo: 0x13865
** PCI devices:
not available
** USB devices:
not available
-- System Information:
Debian Release: squeeze/sid
APT prefers testing
APT policy: (100, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32-5-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages linux-image-2.6.32-5-amd64 depends on:
ii debconf [debconf-2.0] 1.5.33 Debian configuration management sy
ii initramfs-tools [linux-initra 0.97.2 tools for generating an initramfs
ii linux-base 2.6.32-18 Linux image base package
ii module-init-tools 3.12-1 tools for managing Linux kernel mo
Versions of packages linux-image-2.6.32-5-amd64 recommends:
ii firmware-linux-free 2.6.32-18 Binary firmware for various driver
Versions of packages linux-image-2.6.32-5-amd64 suggests:
ii grub-legacy [grub] 0.97-61 GRand Unified Bootloader (Legacy v
pn linux-doc-2.6.32 <none> (no description available)
Versions of packages linux-image-2.6.32-5-amd64 is related to:
pn firmware-bnx2 <none> (no description available)
pn firmware-bnx2x <none> (no description available)
pn firmware-ipw2x00 <none> (no description available)
pn firmware-ivtv <none> (no description available)
pn firmware-iwlwifi <none> (no description available)
pn firmware-linux <none> (no description available)
pn firmware-linux-nonfree <none> (no description available)
pn firmware-qlogic <none> (no description available)
pn firmware-ralink <none> (no description available)
pn xen-hypervisor <none> (no description available)
-- debconf information:
shared/kernel-image/really-run-bootloader: true
linux-image-2.6.32-5-amd64/postinst/bootloader-test-error-2.6.32-5-amd64:
linux-image-2.6.32-5-amd64/postinst/bootloader-error-2.6.32-5-amd64:
linux-image-2.6.32-5-amd64/prerm/would-invalidate-boot-loader-2.6.32-5-amd64: true
linux-image-2.6.32-5-amd64/postinst/ignoring-do-bootloader-2.6.32-5-amd64:
linux-image-2.6.32-5-amd64/postinst/depmod-error-initrd-2.6.32-5-amd64: false
linux-image-2.6.32-5-amd64/prerm/removing-running-kernel-2.6.32-5-amd64: true
linux-image-2.6.32-5-amd64/postinst/missing-firmware-2.6.32-5-amd64:
Reply to: