--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: linux-image-2.6.26-2-openvz-686: server is hangs on ?heavy? disk io on ocfs2
- From: Szabolcs Gyuris <gyuris.szabolcs@integrity.hu>
- Date: Thu, 25 Jun 2009 09:52:32 +0200
- Message-id: <20090625075232.5332.63830.reportbug@hoster-dmz-1.integrity.hu>
Package: linux-image-2.6.26-2-openvz-686
Version: 2.6.26-15lenny3
Severity: critical
Justification: breaks the whole system
I use openvz on ocfs2 fs which is on drbd on sw mirror raid.
Once or twice a day the server froze.
I got a lot of:
Jun 24 08:30:03 hoster-dmz-1 kernel: [116049.348676] (10868,0):dlmunlock_common:
128 ERROR: lockres F000000000000000028c8f0c0517267: Someone is calling dlmunlock
while waiting for an ast!<3>(10868,0):dlmunlock:685 ERROR: dlm status = DLM_BAD
PARAM
Jun 24 08:30:03 hoster-dmz-1 kernel: [116049.348676] (10868,0):ocfs2_cancel_conv
ert:3001 ERROR: DLM error -22 while calling ocfs2_dlm_unlock on resource F000000
000000000028c8f0c0517267
Jun 24 08:30:03 hoster-dmz-1 kernel: [116049.348676] (10868,0):ocfs2_flock_handl
e_signal:1505 ERROR: status = -22
Jun 24 08:30:03 hoster-dmz-1 kernel: [116049.348676] (10868,0):ocfs2_do_flock:79 ERROR: status = -22
but it seem to be non-fatal.
before the server hangs:
Jun 25 00:21:48 hoster-dmz-1 kernel: [222288.239258] (4197,3):dlmunlock_common:128 ERROR: lockres F00000000000000002e601561c073e7: Someone is calling dlmunlock while waiting for an ast!<3>(4197,3):dlmunlock:685 ERROR: dlm status = DLM_BADPARAM
Jun 25 00:21:48 hoster-dmz-1 kernel: [222288.239258] (4197,3):ocfs2_cancel_convert:3001 ERROR: DLM error -22 while calling ocfs2_dlm_unlock on resource F00000000000000002e601561c073e7
Jun 25 00:21:48 hoster-dmz-1 kernel: [222288.239258] (4197,3):ocfs2_flock_handle_signal:1505 ERROR: status = -22
Jun 25 00:21:48 hoster-dmz-1 kernel: [222288.239258] (4197,3):ocfs2_do_flock:79 ERROR: status = -22
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] (4197,2):dlmunlock_common:128 ERROR: lockres F00000000000000002e5ffb61c07a97: Someone is calling dlmunlock
while waiting for an ast!<3>(4197,2):dlmunlock:685 ERROR: dlm status = DLM_BADPARAM
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] (4197,2):ocfs2_cancel_convert:3001 ERROR: DLM error -22 while calling ocfs2_dlm_unlock on resource F00000000000000002e5ffb61c07a97
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] (4197,2):ocfs2_flock_handle_signal:1505 ERROR: status = -22
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] (4197,2):ocfs2_do_flock:79 ERROR: status = -22
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] ------------[ cut here ]------------
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] kernel BUG at fs/ocfs2/dlmglue.c:678!
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] invalid opcode: 0000 [#1] SMP
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] Modules linked in: ocfs2 ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs nfs lockd nfs_acl sunrpc vzethdev vznetdev simfs vzrst vzcpt tun vzdquota vzmon vzdev xt_tcpudp xt_length ipt_ttl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables x_tables ipv6 sha1_generic drbd cn loop psmouse i2c_i801 rng_core serio_raw i2c_core pcspkr iTCO_wdt button shpchp i3000_edac pci_hotplug edac_core evdev dcdbas ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod raid1 md_mod ide_cd_mod cdrom ata_generic libata dock sd_mod piix usb_storage floppy ide_pci_generic ide_core mptsas mptscsih mptbase scsi_transport_sas ehci_hcd scsi_mod uhci_hcd usbcore tg3 thermal processor fan thermal_sys
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131]
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] Pid: 2044, comm: dlm_thread Not tainted (2.6.26-2-openvz-686 #1 036test001)
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] EIP: 0060:[<f92982cc>] EFLAGS: 00010046 CPU: 2
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] EIP is at ocfs2_locking_ast+0x210/0x44a [ocfs2]
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] EAX: 00000241 EBX: da123394 ECX: 00000286 EDX: 00000002
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] ESI: 00000002 EDI: da12339c EBP: f7008000 ESP: f716ff50
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] Process dlm_thread (pid: 2044, veid: 0, ti=f716e000 task=c38ed830 task.ti=f716e000)
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] Stack: 00000286 d68e2c40 f8e4001e f5944e00 da072980 f8e4002f f91e2662 f5944e00
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] d68e2c40 da072980 da072980 f91d3ad6 f5944e00 000003e8 00000063 00000010
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] 00000000 00000001 e49d73d8 c399eb80 e49d73e0 c399eb98 c399eba0 00000000
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] Call Trace:
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<f8e4001e>] o2dlm_lock_ast_wrapper+0x0/0x12 [ocfs2_stack_o2cb]
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<f8e4002f>] o2dlm_lock_ast_wrapper+0x11/0x12 [ocfs2_stack_o2cb]
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<f91e2662>] dlm_do_local_ast+0x6c/0x71 [ocfs2_dlm]
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<f91d3ad6>] dlm_thread+0xb9e/0xfa8 [ocfs2_dlm]
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<c013a2ac>] autoremove_wake_function+0x0/0x2d
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<f91d2f38>] dlm_thread+0x0/0xfa8 [ocfs2_dlm]
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<c013a1e4>] kthread+0x7c/0xa8
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<c013a168>] kthread+0x0/0xa8
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] [<c0109597>] kernel_thread_helper+0x7/0x10
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] =======================
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] Code: 1f 2c f9 64 a1 04 80 3c c0 64 8b 15 00 80 3c c0 50 ff b2 10 01 00 00 68 cc 64 2c f9 e8 e9 22 e9 c6 83 c4 14 8b 43 20 a8 02 75 04 <0f> 0b eb fe a8 01 75 04 0f 0b eb fe 83 7b 44 00 75 15 8b 43 04
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043131] EIP: [<f92982cc>] ocfs2_locking_ast+0x210/0x44a [ocfs2] SS:ESP 0068:f716ff50
Jun 25 00:22:03 hoster-dmz-1 kernel: [222310.043664] ---[ end trace 5a018967b5848431 ]---
-- Package-specific info:
** Version:
Linux version 2.6.26-2-openvz-686 (Debian 2.6.26-15lenny3) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Thu May 28 17:08:31 UTC 2009
** Command line:
auto BOOT_IMAGE=LinuxOLD ro root=fd00
** Not tainted
** Kernel log:
** Loaded modules:
Module Size Used by
nfs 215432 0
lockd 56456 1 nfs
nfs_acl 3680 1 nfs
sunrpc 166816 5 nfs,lockd,nfs_acl
vzethdev 9184 0
vznetdev 14532 10
simfs 4756 5
vzrst 116532 0
vzcpt 97924 0
tun 10148 2 vzrst,vzcpt
vzdquota 32732 5 [permanent]
vzmon 23112 9 vzethdev,vznetdev,vzrst,vzcpt
vzdev 3624 4 vzethdev,vznetdev,vzdquota,vzmon
xt_tcpudp 3584 0
xt_length 2528 0
ipt_ttl 2368 0
xt_tcpmss 2752 0
xt_TCPMSS 4448 0
iptable_mangle 4640 0
iptable_filter 4288 6
xt_multiport 3584 6
xt_limit 2948 0
xt_dscp 3136 0
ipt_REJECT 3552 0
ip_tables 11056 2 iptable_mangle,iptable_filter
x_tables 14404 10 xt_tcpudp,xt_length,ipt_ttl,xt_tcpmss,xt_TCPMSS,xt_multiport,xt_limit,xt_dscp,ipt_REJECT,ip_tables
sha1_generic 3072 0
cn 7552 0
ipv6 242176 107 vzrst,vzcpt,vzmon
loop 13548 0
pcspkr 3200 0
i2c_i801 8688 0
rng_core 4708 0
i2c_core 20596 1 i2c_i801
iTCO_wdt 10276 0
shpchp 26360 0
i3000_edac 5192 0
button 6864 0
pci_hotplug 24260 1 shpchp
edac_core 36912 3 i3000_edac
evdev 8736 0
dcdbas 7076 0
ext3 106184 5
jbd 40244 1 ext3
mbcache 7876 1 ext3
dm_mirror 15872 0
dm_log 9252 1 dm_mirror
dm_snapshot 15108 0
dm_mod 46984 12 dm_mirror,dm_log,dm_snapshot
raid1 18688 2
md_mod 67836 3 raid1
ide_cd_mod 28420 0
cdrom 30976 1 ide_cd_mod
ata_generic 5444 0
libata 141344 1 ata_generic
dock 9072 1 libata
usb_storage 77792 0
sd_mod 22968 7
piix 7336 0 [permanent]
ide_pci_generic 4676 0 [permanent]
uhci_hcd 19504 0
ehci_hcd 29388 0
floppy 48516 0
ide_core 97000 3 ide_cd_mod,piix,ide_pci_generic
usbcore 119312 4 usb_storage,uhci_hcd,ehci_hcd
mptsas 23976 7
tg3 85860 0
mptscsih 22624 1 mptsas
mptbase 52580 2 mptsas,mptscsih
scsi_transport_sas 26240 1 mptsas
scsi_mod 130220 6 libata,usb_storage,sd_mod,mptsas,mptscsih,scsi_transport_sas
thermal 15996 0
processor 33536 1 thermal
fan 4932 0
thermal_sys 11624 3 thermal,processor,fan
** PCI devices:
not available
** Sound cards:
-- System Information:
Debian Release: 5.0.1
APT prefers stable
APT policy: (500, 'stable')
Architecture: i386 (i686)
Kernel: Linux 2.6.26-2-openvz-686 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages linux-image-2.6.26-2-openvz-686 depends on:
ii debconf [debconf-2.0] 1.5.24 Debian configuration management sy
ii initramfs-tools [linux-initra 0.92o tools for generating an initramfs
ii module-init-tools 3.4-1 tools for managing Linux kernel mo
ii vzctl 3.0.22-14 server virtualization solution - c
Versions of packages linux-image-2.6.26-2-openvz-686 recommends:
ii libc6-i686 2.7-18 GNU C Library: Shared libraries [i
Versions of packages linux-image-2.6.26-2-openvz-686 suggests:
ii lilo 1:22.8-7 LInux LOader - The Classic OS load
pn linux-doc-2.6.26 <none> (no description available)
-- debconf information:
shared/kernel-image/really-run-bootloader: true
linux-image-2.6.26-2-openvz-686/preinst/lilo-has-ramdisk:
linux-image-2.6.26-2-openvz-686/preinst/overwriting-modules-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/preinst/abort-overwrite-2.6.26-2-openvz-686:
linux-image-2.6.26-2-openvz-686/preinst/failed-to-move-modules-2.6.26-2-openvz-686:
linux-image-2.6.26-2-openvz-686/preinst/initrd-2.6.26-2-openvz-686:
linux-image-2.6.26-2-openvz-686/prerm/removing-running-kernel-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/postinst/bootloader-test-error-2.6.26-2-openvz-686:
linux-image-2.6.26-2-openvz-686/prerm/would-invalidate-boot-loader-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/postinst/old-system-map-link-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/preinst/elilo-initrd-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/postinst/old-initrd-link-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/postinst/create-kimage-link-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/postinst/depmod-error-2.6.26-2-openvz-686: false
linux-image-2.6.26-2-openvz-686/postinst/old-dir-initrd-link-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/preinst/abort-install-2.6.26-2-openvz-686:
linux-image-2.6.26-2-openvz-686/postinst/kimage-is-a-directory:
linux-image-2.6.26-2-openvz-686/preinst/lilo-initrd-2.6.26-2-openvz-686: true
linux-image-2.6.26-2-openvz-686/postinst/bootloader-error-2.6.26-2-openvz-686:
linux-image-2.6.26-2-openvz-686/postinst/depmod-error-initrd-2.6.26-2-openvz-686: false
linux-image-2.6.26-2-openvz-686/preinst/bootloader-initrd-2.6.26-2-openvz-686: true
--- End Message ---