linux-image-2.6.32-bpo.5-amd64 kernel panic when running on Xen
Hello,
I'm running Debian Lenny with 2.6.32 kernel from backports on a Xen
and got a kernel panic few times with it. The server (Dom0) have
4-core 2.5Ghz Xeon with 16GB mem and it is running CentOS kernel with
custom build of Xen hypervisor version 3.4. I cant manage Dom0 - it is
provided to me by some cloud hosting service.
On that server there are two instances (DomU) running. A small
instance with 1/5 of CPU and 1GB mem which is running Lenny plus
CentOS 5.2 kernel (or maybe 5.3) provided by hoster. And a big
instance with 4/5 of CPU, 14GB mem, Debian Lenny inside with a 2.6.32
kernel from backports.
Sometimes my big instance panics. It happens only if I running 2.6.32
kernel from backports. No problem with it if I switch to the old
CentOS kernel provided by hoster.
I have already asked hoster for support with my kernel panic and they
answered that it seems there are a bug in my kernel and I should ask
the 'kernel vendor' for the support or at least write a bug report.
The kernel panic already happens two times. Both times at was at
arch/x86/mm/tlb.c:61
There are two kernel messages output attached to this mail. Hope
they'll be useful.
If I'm writing this to the wrong place please point me to the proper
mailing list :)
The exact version on kernel package:
$ apt-cache policy linux-image-2.6-amd64 linux-image-2.6.32-bpo.5-amd64
linux-image-2.6-amd64:
Installed: 2.6.32+27~bpo50+1
Candidate: 2.6.32+27~bpo50+1
Version table:
2.6.32+29 0
201 http://ftp.us.debian.org testing/main Packages
*** 2.6.32+27~bpo50+1 0
202 http://backports.debian.org lenny-backports/main Packages
100 /var/lib/dpkg/status
2.6.26+17+lenny1 0
500 http://ftp.us.debian.org lenny/main Packages
linux-image-2.6.32-bpo.5-amd64:
Installed: 2.6.32-30~bpo50+1
Candidate: 2.6.32-30~bpo50+1
Version table:
*** 2.6.32-30~bpo50+1 0
202 http://backports.debian.org lenny-backports/main Packages
100 /var/lib/dpkg/status
Thank you.
Xen Minimal OS!
start_info: 0x2584000(VA)
nr_pages: 0x37e700
shared_inf: 0x93635000(MA)
pt_base: 0x2587000(VA)
nr_pt_frames: 0x17
mfn_list: 0x990000(VA)
mod_start: 0x0(VA)
mod_len: 0
flags: 0x0
cmd_line: (hd0)/boot/grub/grub.conf
stack: 0x94f860-0x96f860
MM: Init
_text: 0x0(VA)
_etext: 0x5fcad(VA)
_erodata: 0x78000(VA)
_edata: 0x80b00(VA)
stack start: 0x94f860(VA)
_end: 0x98fe68(VA)
start_pfn: 25a1
max_pfn: 37e700
Mapping memory range 0x2800000 - 0x37e700000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 418e000(418e000)-37e700000(37e700000)
MM: done
Demand map pfns at 37e701000-237e701000.
Heap resides at 237e702000-437e702000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x37e701000.
Initialising scheduler
Thread "Idle": pointer: 0x237e702010, stack: 0x5710000
Initialising xenbus
Thread "xenstore": pointer: 0x237e7027c0, stack: 0x5720000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x237e702f70, stack: 0x5730000
"main" "(hd0)/boot/grub/grub.conf"
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********
backend at /local/domain/0/backend/vbd/4/2049
Failed to read /local/domain/0/backend/vbd/4/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/4/2049/feature-flush-cache.
135528448 sectors of 512 bytes
**************************
vbd 2050 is hd1
******************* BLKFRONT for device/vbd/2050 **********
backend at /local/domain/0/backend/vbd/4/2050
Failed to read /local/domain/0/backend/vbd/4/2050/feature-barrier.
Failed to read /local/domain/0/backend/vbd/4/2050/feature-flush-cache.
58654720 sectors of 512 bytes
**************************
[H[J
GNU GRUB version 0.97 (14654464K lower / 0K upper memory)
[m[4;2H+-------------------------------------------------------------------------+[5;2H|[5;76H|[6;2H|[6;76H|[7;2H|[7;76H|[8;2H|[8;76H|[9;2H|[9;76H|[10;2H|[10;76H|[11;2H|[11;76H|[12;2H|[12;76H|[13;2H|[13;76H|[14;2H|[14;76H|[15;2H|[15;76H|[16;2H|[16;76H|[17;2H+-------------------------------------------------------------------------+[m
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.[5;78H [m[7m[5;3H Debian GNU/Linux, kernel 2.6.32-bpo.5-amd64 [5;75H[m[m[6;3H [6;75H[m[m[7;3H [7;75H[m[m[8;3H [8;75H[m[m[9;3H [9;75H[m[m[10;3H [10;75H[m[m[11;3H [11;75H[m[m[12;3H [12;75H[m[m[13;3H [13;75H[m[m[14;3H [14;75H[m[m[15;3H [15;75H[m[m[16;3H [16;75H[m[16;78H [5;75H[23;4H The highlighted entry will be booted automatically in 1 seconds. [5;75H[H[J Booting 'Debian GNU/Linux, kernel 2.6.32-bpo.5-amd64'
root (hd0)
Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.32-bpo.5-amd64 ipv6.disable=1 console=hvc0 root=/dev/
xvda1 ro
initrd /boot/initrd.img-2.6.32-bpo.5-amd64
close blk: backend at /local/domain/0/backend/vbd/4/2049
close blk: backend at /local/domain/0/backend/vbd/4/2050
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.32-bpo.5-amd64 (Debian 2.6.32-30~bpo50+1) (norbert@tretkowski.de) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP Mon Jan 17 18:24:45 UTC 2011
[ 0.000000] Command line: ipv6.disable=1 console=hvc0 root=/dev/xvda1 ro
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] ACPI in unprivileged domain disabled
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
[ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 000000037e700000 (usable)
[ 0.000000] DMI not present or invalid.
[ 0.000000] last_pfn = 0x37e700 max_arch_pfn = 0x400000000
[ 0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000
[ 0.000000] init_memory_mapping: 0000000000000000-0000000100000000
[ 0.000000] init_memory_mapping: 0000000100000000-000000037e700000
[ 0.000000] RAMDISK: 016a7000 - 02f03000
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at 0000000000000000-000000037e700000
[ 0.000000] Bootmem setup node 0 0000000000000000-000000037e700000
[ 0.000000] NODE_DATA [0000000000008000 - 000000000000ffff]
[ 0.000000] bootmap [0000000000010000 - 000000000007fcdf] pages 70
[ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 037e700000]
[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
[ 0.000000] #1 [0004afa000 - 0004b23000] XEN PAGETABLES ==> [0004afa000 - 0004b23000]
[ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
[ 0.000000] #3 [0001000000 - 0001686414] TEXT DATA BSS ==> [0001000000 - 0001686414]
[ 0.000000] #4 [00016a7000 - 0002f03000] RAMDISK ==> [00016a7000 - 0002f03000]
[ 0.000000] #5 [0002f03000 - 0004afa000] XEN START INFO ==> [0002f03000 - 0004afa000]
[ 0.000000] #6 [0000100000 - 00008d9000] PGTABLE ==> [0000100000 - 00008d9000]
[ 0.000000] #7 [0004b23000 - 0005f21000] PGTABLE ==> [0004b23000 - 0005f21000]
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000000 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] Normal 0x00100000 -> 0x0037e700
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000000 -> 0x000000a0
[ 0.000000] 0: 0x00000100 -> 0x0037e700
[ 0.000000] SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
[ 0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[ 0.000000] No local APIC present
[ 0.000000] APIC: disable apic facility
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[ 0.000000] PCI: Warning: Cannot find a gap in the 32bit address range
[ 0.000000] PCI: Unassigned devices with 32bit resource registers may break!
[ 0.000000] Allocating PCI resources starting at 37e800000 (gap: 37e800000:400000)
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 3.4.2 (preserve-AD)
[ 0.000000] NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 30 pages/cpu @ffff8800123b3000 s90328 r8192 d24360 u122880
[ 0.000000] pcpu-alloc: s90328 r8192 d24360 u122880 alloc=30*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[ 0.000000] Xen: using vcpu_info placement
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 3611419
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: ipv6.disable=1 console=hvc0 root=/dev/xvda1 ro
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Initializing CPU#0
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.000000] Placing 64MB software IO TLB between ffff880012434000 - ffff880016434000
[ 0.000000] software IO TLB at phys 0x12434000 - 0x16434000
[ 0.000000] Memory: 14297184k/14654464k available (3079k kernel code, 384k absent, 356896k reserved, 1876k data, 592k init)
[ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] NR_IRQS:4352 nr_irqs:304
[ 0.000000] Console: colour dummy device 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] console [hvc0] enabled
[ 0.000000] installing Xen timer for CPU 0
[ 0.000000] Detected 2493.774 MHz processor.
[ 0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4987.54 BogoMIPS (lpj=9975096)
[ 0.004000] Security Framework initialized
[ 0.004000] SELinux: Disabled at boot.
[ 0.004000] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[ 0.012717] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[ 0.016972] Mount-cache hash table entries: 256
[ 0.017159] Initializing cgroup subsys ns
[ 0.017167] Initializing cgroup subsys cpuacct
[ 0.017174] Initializing cgroup subsys devices
[ 0.017180] Initializing cgroup subsys freezer
[ 0.017185] Initializing cgroup subsys net_cls
[ 0.017229] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.017236] CPU: L2 cache: 6144K
[ 0.017242] CPU 0/0x2 -> Node 0
[ 0.017247] CPU: Physical Processor ID: 0
[ 0.017251] CPU: Processor Core ID: 2
[ 0.017257] Performance Events: unsupported p6 CPU model 23 no PMU driver, software events only.
[ 0.017279] SMP alternatives: switching to UP code
[ 0.020277] installing Xen timer for CPU 1
[ 0.020317] SMP alternatives: switching to SMP code
[ 0.004000] Initializing CPU#1
[ 0.004000] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.004000] CPU: L2 cache: 6144K
[ 0.004000] CPU 1/0x2 -> Node 0
[ 0.004000] CPU: Physical Processor ID: 0
[ 0.004000] CPU: Processor Core ID: 2
[ 0.024157] installing Xen timer for CPU 2
[ 0.004000] Initializing CPU#2
[ 0.004000] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.004000] CPU: L2 cache: 6144K
[ 0.004000] CPU 2/0x2 -> Node 0
[ 0.004000] CPU: Physical Processor ID: 0
[ 0.004000] CPU: Processor Core ID: 2
[ 0.024380] installing Xen timer for CPU 3
[ 0.004000] Initializing CPU#3
[ 0.004000] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.004000] CPU: L2 cache: 6144K
[ 0.004000] CPU 3/0x2 -> Node 0
[ 0.004000] CPU: Physical Processor ID: 0
[ 0.004000] CPU: Processor Core ID: 2
[ 0.024549] Brought up 4 CPUs
[ 0.024949] devtmpfs: initialized
[ 0.029020] Grant table initialized
[ 0.029020] regulator: core version 0.5
[ 0.029020] NET: Registered protocol family 16
[ 0.029020] PCI: Fatal: No config space access function found
[ 0.034450] bio: create slab <bio-0> at 0
[ 0.034450] ACPI: Interpreter disabled.
[ 0.034450] xen_balloon: Initialising balloon driver.
[ 0.034450] vgaarb: loaded
[ 0.034450] PCI: System does not support PCI
[ 0.034450] PCI: System does not support PCI
[ 0.034450] Switching to clocksource xen
[ 0.037605] pnp: PnP ACPI: disabled
[ 0.038078] NET: Registered protocol family 2
[ 0.039039] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
[ 0.041659] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[ 0.042042] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 0.042042] TCP: Hash tables configured (established 524288 bind 65536)
[ 0.042042] TCP reno registered
[ 0.042042] NET: Registered protocol family 1
[ 0.042042] Unpacking initramfs...
[ 0.078936] Freeing initrd memory: 24944k freed
[ 0.087871] platform rtc_cmos: registered platform RTC device (no PNP device found)
[ 0.088292] audit: initializing netlink socket (disabled)
[ 0.088316] type=2000 audit(1297263056.007:1): initialized
[ 0.091960] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.093793] VFS: Disk quotas dquot_6.5.2
[ 0.093852] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.093965] msgmni has been set to 27972
[ 0.094526] alg: No test for stdrng (krng)
[ 0.094599] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 0.094616] io scheduler noop registered
[ 0.094621] io scheduler anticipatory registered
[ 0.094626] io scheduler deadline registered
[ 0.094664] io scheduler cfq registered (default)
[ 0.097085] Linux agpgart interface v0.103
[ 0.097117] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.097305] input: Macintosh mouse button emulation as /devices/virtual/input/input0
[ 0.097361] PNP: No PS/2 controller found. Probing ports directly.
[ 0.098184] i8042.c: No controller found.
[ 0.098251] mice: PS/2 mouse device common for all mice
[ 0.098313] rtc_cmos: probe of rtc_cmos failed with error -16
[ 0.098339] cpuidle: using governor ladder
[ 0.098344] cpuidle: using governor menu
[ 0.098351] No iBFT detected.
[ 0.098618] TCP cubic registered
[ 0.098623] IPv6: Loaded, but administratively disabled, reboot required to enable
[ 0.098630] Mobile IPv6
[ 0.098634] mip6_init: can't add xfrm type(destopt)
[ 0.098639] NET: Registered protocol family 17
[ 0.098727] registered taskstats version 1
[ 0.098752] XENBUS: Device with no driver: device/vbd/2049
[ 0.098757] XENBUS: Device with no driver: device/vbd/2050
[ 0.098762] XENBUS: Device with no driver: device/vif/0
[ 0.098766] XENBUS: Device with no driver: device/console/0
[ 0.098782] /tmp/buildd/linux-2.6-2.6.32/debian/build/source_amd64_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[ 0.098801] Initalizing network drop monitor service
[ 0.098881] Freeing unused kernel memory: 592k freed
[ 0.099080] Write protecting the kernel read-only data: 4216k
Loading, please wait...
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... [ 0.147270] udevd version 125 started
[ 0.232254] Initialising Xen virtual ethernet driver.
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... kinit: name_to_dev_t(/dev/xvda2) = dev(202,2)
kinit: trying to resume from /dev/xvda2
[ 0.429632] PM: Starting manual resume from disk
kinit: No resume image, doing normal boot...
done.
[ 0.455044] kjournald starting. Commit interval 5 seconds
[ 0.455059] EXT3-fs: mounted filesystem with ordered data mode.
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.86 booting
Starting the hotplug events dispatcher: udevd[ 1.727634] udevd version 125 started
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Setting the system clock.
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Unable to set System Clock to: Wed Feb 9 14:50:58 UTC 2011 [33m(warning).[39;49m
Activating swap...[ 2.157758] Adding 29327352k swap on /dev/xvda2. Priority:-1 extents:1 across:29327352k SS
done.
Checking root file system...fsck 1.41.3 (12-Oct-2008)
/dev/xvda1: clean, 1360063/2117632 files, 8221535/16941056 blocks
done.
[ 2.492385] EXT3 FS on xvda1, internal journal
Setting the system clock.
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Unable to set System Clock to: Wed Feb 9 14:50:58 UTC 2011 [33m(warning).[39;49m
Cleaning up ifupdown....
Loading kernel modules...done.
Checking file systems...fsck 1.41.3 (12-Oct-2008)
done.
Setting kernel variables (/etc/sysctl.conf)...done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Setting up networking....
Configuring network interfaces...Internet Systems Consortium DHCP Client V3.1.1
Copyright 2004-2008 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/
Listening on LPF/eth0/00:16:3e:01:cd:bc
Sending on LPF/eth0/00:16:3e:01:cd:bc
Sending on Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 11
DHCPOFFER from 68.67.141.131
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPACK from 68.67.141.131
Invalid domain list.
Invalid domain list.
bound to 68.67.141.188 -- renewal in 12122 seconds.
Starting portmap daemon....
Starting NFS common utilities: statd.
[ 17.962142] RPC: Registered udp transport module.
[ 17.962154] RPC: Registered tcp transport module.
[ 17.962159] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 18.044714] Slow work thread pool: Starting up
[ 18.044848] Slow work thread pool: Ready
[ 18.044923] FS-Cache: Loaded
[ 18.104024] FS-Cache: Netfs 'nfs' registered for caching
done.
Starting portmap daemon...Already running..
Starting NFS common utilities: statd.
INIT: Entering runlevel: 2
Starting system logging: syslog-ng.
Starting system message bus: dbus.
Starting OpenBSD Secure Shell server: sshd.
Starting MySQL database server: mysqld . ..
Checking for corrupt, not cleanly closed and upgrade needing tables..
Starting atop system monitor: atop.
Starting Java program runner: nailgun-common.
Starting NFS common utilities: statd.
Starting NTP server: ntpd.
Starting Postfix Mail Transport Agent: postfix[ 25.827957] warning: `ntpd' uses 32-bit capabilities (legacy support in use)
.
sadc not enabled in /etc/default/sysstat, not starting. [33m(warning).[39;49m
Starting Zabbix agent: zabbix_agentd.
Starting puppet agent.
Debian GNU/Linux 5.0 xxx.xxx.xxx.xxx hvc0
xxx.xxx.xxx.xxx login: [ 65.156488] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3120.088537] INFO: task kjournald:340 blocked for more than 120 seconds.
[ 3120.088553] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.088562] kjournald D 0000000000000002 0 340 2 0x00000000
[ 3120.088574] ffff88037c8c0000 0000000000000246 ffff8803799c8000 0000000000000200
[ 3120.088597] ffff88037b3c5500 000000000000f9e0 ffff880379865fd8 0000000000015780
[ 3120.088609] 0000000000015780 ffff880379c67100 ffff880379c673f8 0000000179864000
[ 3120.088621] Call Trace:
[ 3120.088632] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.088639] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.088648] [<ffffffff812fa908>] ? io_schedule+0x73/0xb7
[ 3120.088655] [<ffffffff8110dd71>] ? sync_buffer+0x3b/0x40
[ 3120.088662] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[ 3120.088669] [<ffffffff812fae15>] ? __wait_on_bit+0x41/0x70
[ 3120.088676] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.088684] [<ffffffff812faeaf>] ? out_of_line_wait_on_bit+0x6b/0x77
[ 3120.088693] [<ffffffff81064d44>] ? wake_bit_function+0x0/0x23
[ 3120.088706] [<ffffffffa00151cd>] ? journal_commit_transaction+0x508/0xe2b [jbd]
[ 3120.088715] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.088725] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.088732] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[ 3120.088740] [<ffffffffa0018413>] ? kjournald+0xdf/0x226 [jbd]
[ 3120.088748] [<ffffffff8100e5d4>] ? xen_smp_send_reschedule+0x0/0x7
[ 3120.088755] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.088763] [<ffffffffa0018334>] ? kjournald+0x0/0x226 [jbd]
[ 3120.088770] [<ffffffff81064a49>] ? kthread+0x79/0x81
[ 3120.088778] [<ffffffff81011baa>] ? child_rip+0xa/0x20
[ 3120.088785] [<ffffffff81010d61>] ? int_ret_from_sys_call+0x7/0x1b
[ 3120.088792] [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[ 3120.088800] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[ 3120.088808] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.088815] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
[ 3120.088822] INFO: task flush-202:1:876 blocked for more than 120 seconds.
[ 3120.088827] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.088834] flush-202:1 D 0000000000000000 0 876 2 0x00000000
[ 3120.088843] ffffffff8145d1f0 0000000000000246 0000000000000000 ffff88037b176300
[ 3120.088855] 0000000000000000 000000000000f9e0 ffff88037bbe7fd8 0000000000015780
[ 3120.088867] 0000000000015780 ffff880379c654c0 ffff880379c657b8 0000000081440220
[ 3120.088879] Call Trace:
[ 3120.088885] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.088892] [<ffffffff812fa908>] ? io_schedule+0x73/0xb7
[ 3120.088898] [<ffffffff8110dd71>] ? sync_buffer+0x3b/0x40
[ 3120.088905] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[ 3120.088912] [<ffffffff812fad18>] ? __wait_on_bit_lock+0x3f/0x84
[ 3120.088919] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.088927] [<ffffffff812fadc8>] ? out_of_line_wait_on_bit_lock+0x6b/0x77
[ 3120.088934] [<ffffffff81064d44>] ? wake_bit_function+0x0/0x23
[ 3120.088941] [<ffffffff8110f103>] ? __block_write_full_page+0x159/0x2ac
[ 3120.088948] [<ffffffff8110df00>] ? end_buffer_async_write+0x0/0x13b
[ 3120.088956] [<ffffffff810ba2ca>] ? __writepage+0xa/0x25
[ 3120.088963] [<ffffffff810ba951>] ? write_cache_pages+0x20b/0x327
[ 3120.088969] [<ffffffff810ba2c0>] ? __writepage+0x0/0x25
[ 3120.088976] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.088983] [<ffffffff8100c043>] ? xen_mc_flush+0x159/0x185
[ 3120.088990] [<ffffffff8100b476>] ? xen_end_context_switch+0x9/0x12
[ 3120.088998] [<ffffffff81107b0a>] ? writeback_single_inode+0xe7/0x2da
[ 3120.089006] [<ffffffff81108810>] ? writeback_inodes_wb+0x424/0x4ff
[ 3120.089013] [<ffffffff81108a17>] ? wb_writeback+0x12c/0x1ab
[ 3120.089022] [<ffffffff8105a720>] ? try_to_del_timer_sync+0x63/0x6c
[ 3120.089029] [<ffffffff81108c8d>] ? wb_do_writeback+0x14f/0x165
[ 3120.089036] [<ffffffff81108cd4>] ? bdi_writeback_task+0x31/0xaa
[ 3120.089045] [<ffffffff810c8c0e>] ? bdi_start_fn+0x0/0xd2
[ 3120.089051] [<ffffffff810c8c7e>] ? bdi_start_fn+0x70/0xd2
[ 3120.089057] [<ffffffff810c8c0e>] ? bdi_start_fn+0x0/0xd2
[ 3120.089064] [<ffffffff81064a49>] ? kthread+0x79/0x81
[ 3120.089070] [<ffffffff81011baa>] ? child_rip+0xa/0x20
[ 3120.089076] [<ffffffff81010d61>] ? int_ret_from_sys_call+0x7/0x1b
[ 3120.089083] [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[ 3120.089090] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[ 3120.089097] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.089104] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
[ 3120.089116] INFO: task mysqld:1183 blocked for more than 120 seconds.
[ 3120.089122] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] mysqld D 0000000000000000 0 1183 1119 0x00000000
[ 3120.092507] ffffffff8145d1f0 0000000000000282 0000000000000000 000000010000000e
[ 3120.092507] ffff88037c6fdb38 000000000000f9e0 ffff88037a86bfd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff880379c61c40 ffff880379c61f38 00000000810bbd06
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffffa0017f64>] ? log_wait_commit+0xbf/0x112 [jbd]
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.092507] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[ 3120.092507] [<ffffffffa00203f8>] ? ext3_sync_file+0x94/0xc8 [ext3]
[ 3120.092507] [<ffffffff8110b880>] ? vfs_fsync_range+0x73/0x9e
[ 3120.092507] [<ffffffff8110b92a>] ? do_fsync+0x28/0x39
[ 3120.092507] [<ffffffff8110b959>] ? sys_fsync+0xb/0x10
[ 3120.092507] [<ffffffff81010b63>] ? sysret_check+0x17/0x5a
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 3120.092507] INFO: task zabbix_agentd:1400 blocked for more than 120 seconds.
[ 3120.092507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] zabbix_agentd D 0000000000000000 0 1400 1392 0x00000000
[ 3120.092507] ffff88037c860e20 0000000000000282 0000000000000000 ffffffff810b988b
[ 3120.092507] ffff8803647494e8 000000000000f9e0 ffff88037a70dfd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff88037a6f8000 ffff88037a6f82f8 00000001810fe486
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffff810b988b>] ? get_page_from_freelist+0x56b/0x760
[ 3120.092507] [<ffffffff812fb049>] ? __mutex_lock_common+0x122/0x192
[ 3120.092507] [<ffffffff812fb171>] ? mutex_lock+0x1a/0x31
[ 3120.092507] [<ffffffff810b5021>] ? generic_file_aio_write+0x44/0x9f
[ 3120.092507] [<ffffffff810ee626>] ? do_sync_write+0xce/0x113
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff810cc731>] ? handle_mm_fault+0x35c/0x80f
[ 3120.092507] [<ffffffff810eef9e>] ? vfs_write+0xa9/0x102
[ 3120.092507] [<ffffffff810ef0b3>] ? sys_write+0x45/0x6e
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 3120.092507] INFO: task zabbix_agentd:1401 blocked for more than 120 seconds.
[ 3120.092507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] zabbix_agentd D ffff8800123e0c40 0 1401 1392 0x00000000
[ 3120.092507] ffff88037c860e20 0000000000000286 ffff88037a7979e8 ffff88037a7979e4
[ 3120.092507] ffff8800123e6780 000000000000f9e0 ffff88037a797fd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff88037a6f8710 ffff88037a6f8a08 00000001123e6780
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.092507] [<ffffffff812fa908>] ? io_schedule+0x73/0xb7
[ 3120.092507] [<ffffffff8110dd71>] ? sync_buffer+0x3b/0x40
[ 3120.092507] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[ 3120.092507] [<ffffffff812fad18>] ? __wait_on_bit_lock+0x3f/0x84
[ 3120.092507] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[ 3120.092507] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.092507] [<ffffffff812fadc8>] ? out_of_line_wait_on_bit_lock+0x6b/0x77
[ 3120.092507] [<ffffffff81064d44>] ? wake_bit_function+0x0/0x23
[ 3120.092507] [<ffffffff8110e163>] ? sync_dirty_buffer+0x29/0x93
[ 3120.092507] [<ffffffffa0013e00>] ? journal_dirty_data+0xd1/0x1b0 [jbd]
[ 3120.092507] [<ffffffffa0022f1f>] ? ext3_journal_dirty_data+0xf/0x34 [ext3]
[ 3120.092507] [<ffffffffa00213f9>] ? walk_page_buffers+0x65/0x8b [ext3]
[ 3120.092507] [<ffffffffa0022f44>] ? journal_dirty_data_fn+0x0/0x13 [ext3]
[ 3120.092507] [<ffffffffa0024a42>] ? ext3_ordered_write_end+0x73/0x10f [ext3]
[ 3120.092507] [<ffffffffa002fb1f>] ? ext3_xattr_get+0x1ef/0x271 [ext3]
[ 3120.092507] [<ffffffff810b4b0d>] ? generic_file_buffered_write+0x18d/0x278
[ 3120.092507] [<ffffffff810b4fa9>] ? __generic_file_aio_write+0x25f/0x293
[ 3120.092507] [<ffffffff810f6cab>] ? __link_path_walk+0x6f0/0x6f5
[ 3120.092507] [<ffffffff810b5036>] ? generic_file_aio_write+0x59/0x9f
[ 3120.092507] [<ffffffff810ee626>] ? do_sync_write+0xce/0x113
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff810cc731>] ? handle_mm_fault+0x35c/0x80f
[ 3120.092507] [<ffffffff810eef9e>] ? vfs_write+0xa9/0x102
[ 3120.092507] [<ffffffff810ef0b3>] ? sys_write+0x45/0x6e
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 3120.092507] INFO: task zabbix_agentd:1402 blocked for more than 120 seconds.
[ 3120.092507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] zabbix_agentd D 0000000000000000 0 1402 1392 0x00000000
[ 3120.092507] ffffffff8145d1f0 0000000000000282 0000000000000000 ffffffff810b988b
[ 3120.092507] ffff880379ba9c00 000000000000f9e0 ffff88037a7b7fd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff88037a6f8e20 ffff88037a6f9118 00000000810fe486
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffff810b988b>] ? get_page_from_freelist+0x56b/0x760
[ 3120.092507] [<ffffffff812fb049>] ? __mutex_lock_common+0x122/0x192
[ 3120.092507] [<ffffffff812fb171>] ? mutex_lock+0x1a/0x31
[ 3120.092507] [<ffffffff810b5021>] ? generic_file_aio_write+0x44/0x9f
[ 3120.092507] [<ffffffff810ee626>] ? do_sync_write+0xce/0x113
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff810cc731>] ? handle_mm_fault+0x35c/0x80f
[ 3120.092507] [<ffffffff810eef9e>] ? vfs_write+0xa9/0x102
[ 3120.092507] [<ffffffff810ef0b3>] ? sys_write+0x45/0x6e
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 3120.092507] INFO: task zabbix_agentd:1403 blocked for more than 120 seconds.
[ 3120.092507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] zabbix_agentd D ffff8800123c8780 0 1403 1392 0x00000000
[ 3120.092507] ffff88037a6f8e20 0000000000000286 ffffffff8100e22f ffffffff810b988b
[ 3120.092507] ffff8803646e58e8 000000000000f9e0 ffff88037a7d9fd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff88037a6f9530 ffff88037a6f9828 00000000810fe486
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.092507] [<ffffffff810b988b>] ? get_page_from_freelist+0x56b/0x760
[ 3120.092507] [<ffffffff812fb049>] ? __mutex_lock_common+0x122/0x192
[ 3120.092507] [<ffffffff812fb171>] ? mutex_lock+0x1a/0x31
[ 3120.092507] [<ffffffff810b5021>] ? generic_file_aio_write+0x44/0x9f
[ 3120.092507] [<ffffffff810ee626>] ? do_sync_write+0xce/0x113
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff810cc731>] ? handle_mm_fault+0x35c/0x80f
[ 3120.092507] [<ffffffff810eef9e>] ? vfs_write+0xa9/0x102
[ 3120.092507] [<ffffffff810ef0b3>] ? sys_write+0x45/0x6e
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 3120.092507] INFO: task zabbix_agentd:1404 blocked for more than 120 seconds.
[ 3120.092507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] zabbix_agentd D ffff8800123c8780 0 1404 1392 0x00000000
[ 3120.092507] ffff88037a6f9530 0000000000000282 ffffffff8100e22f ffffffff810b988b
[ 3120.092507] 0000000000000002 000000000000f9e0 ffff88037a7f9fd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff88037a6f9c40 ffff88037a6f9f38 00000000810fe486
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.092507] [<ffffffff810b988b>] ? get_page_from_freelist+0x56b/0x760
[ 3120.092507] [<ffffffff812fb049>] ? __mutex_lock_common+0x122/0x192
[ 3120.092507] [<ffffffff812fb171>] ? mutex_lock+0x1a/0x31
[ 3120.092507] [<ffffffff810b5021>] ? generic_file_aio_write+0x44/0x9f
[ 3120.092507] [<ffffffff810ee626>] ? do_sync_write+0xce/0x113
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff810cc731>] ? handle_mm_fault+0x35c/0x80f
[ 3120.092507] [<ffffffff810eef9e>] ? vfs_write+0xa9/0x102
[ 3120.092507] [<ffffffff810ef0b3>] ? sys_write+0x45/0x6e
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 3120.092507] INFO: task zabbix_agentd:1405 blocked for more than 120 seconds.
[ 3120.092507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] zabbix_agentd D 0000000000000000 0 1405 1392 0x00000000
[ 3120.092507] ffff88037c861c40 0000000000000286 0000000000000000 ffffffff810b988b
[ 3120.092507] 0000000000000642 000000000000f9e0 ffff88037b61bfd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff88037a6fa350 ffff88037a6fa648 00000003810fe486
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffff810b988b>] ? get_page_from_freelist+0x56b/0x760
[ 3120.092507] [<ffffffff812fb049>] ? __mutex_lock_common+0x122/0x192
[ 3120.092507] [<ffffffff812fb171>] ? mutex_lock+0x1a/0x31
[ 3120.092507] [<ffffffff810b5021>] ? generic_file_aio_write+0x44/0x9f
[ 3120.092507] [<ffffffff810ee626>] ? do_sync_write+0xce/0x113
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff810cc731>] ? handle_mm_fault+0x35c/0x80f
[ 3120.092507] [<ffffffff810eef9e>] ? vfs_write+0xa9/0x102
[ 3120.092507] [<ffffffff810ef0b3>] ? sys_write+0x45/0x6e
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 3120.092507] INFO: task aggregate_db.pl:21038 blocked for more than 120 seconds.
[ 3120.092507] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3120.092507] aggregate_db. D 0000000000000000 0 21038 20696 0x00000000
[ 3120.092507] ffff88037c861c40 0000000000000286 0000000000000000 ffff88000000ad00
[ 3120.092507] ffff88034624a7f8 000000000000f9e0 ffff8803460cbfd8 0000000000015780
[ 3120.092507] 0000000000015780 ffff880379a8dbd0 ffff880379a8dec8 00000003810b9de7
[ 3120.092507] Call Trace:
[ 3120.092507] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.092507] [<ffffffff812fa908>] ? io_schedule+0x73/0xb7
[ 3120.092507] [<ffffffff8110dd71>] ? sync_buffer+0x3b/0x40
[ 3120.092507] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[ 3120.092507] [<ffffffff812fad18>] ? __wait_on_bit_lock+0x3f/0x84
[ 3120.092507] [<ffffffff8110dd36>] ? sync_buffer+0x0/0x40
[ 3120.092507] [<ffffffff812fadc8>] ? out_of_line_wait_on_bit_lock+0x6b/0x77
[ 3120.092507] [<ffffffff81064d44>] ? wake_bit_function+0x0/0x23
[ 3120.092507] [<ffffffff8110e163>] ? sync_dirty_buffer+0x29/0x93
[ 3120.092507] [<ffffffffa0013e00>] ? journal_dirty_data+0xd1/0x1b0 [jbd]
[ 3120.092507] [<ffffffffa0022f1f>] ? ext3_journal_dirty_data+0xf/0x34 [ext3]
[ 3120.092507] [<ffffffffa00213f9>] ? walk_page_buffers+0x65/0x8b [ext3]
[ 3120.092507] [<ffffffffa0022f44>] ? journal_dirty_data_fn+0x0/0x13 [ext3]
[ 3120.092507] [<ffffffffa0024a42>] ? ext3_ordered_write_end+0x73/0x10f [ext3]
[ 3120.092507] [<ffffffff810b4b0d>] ? generic_file_buffered_write+0x18d/0x278
[ 3120.092507] [<ffffffff810b4fa9>] ? __generic_file_aio_write+0x25f/0x293
[ 3120.092507] [<ffffffff8118dd6c>] ? cpumask_any_but+0x28/0x34
[ 3120.092507] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 3120.092507] [<ffffffff8100c18d>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
[ 3120.092507] [<ffffffff810b5036>] ? generic_file_aio_write+0x59/0x9f
[ 3120.092507] [<ffffffff810ee626>] ? do_sync_write+0xce/0x113
[ 3120.092507] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[ 3120.092507] [<ffffffff810ccb7f>] ? handle_mm_fault+0x7aa/0x80f
[ 3120.092507] [<ffffffff810eef9e>] ? vfs_write+0xa9/0x102
[ 3120.092507] [<ffffffff810ef0b3>] ? sys_write+0x45/0x6e
[ 3120.092507] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[ 7419.160340] hrtimer: interrupt took 23101590 ns
[858121.456403] ------------[ cut here ]------------
[858121.456554] kernel BUG at /tmp/buildd/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/mm/tlb.c:61!
[858121.456756] invalid opcode: 0000 [#1] SMP
[858121.456853] last sysfs file: /sys/devices/vbd-2049/block/xvda1/queue/scheduler
[858121.456861] CPU 2
[858121.456866] Modules linked in: xt_recent xt_tcpudp iptable_filter ip_tables x_tables nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext3 jbd mbcache xen_netfront xen_blkfront
[858121.456901] Pid: 12066, comm: khelper Not tainted 2.6.32-bpo.5-amd64 #1
[858121.456906] RIP: e030:[<ffffffff81036626>] [<ffffffff81036626>] leave_mm+0x10/0x44
[858121.456921] RSP: e02b:ffff8800123f2e48 EFLAGS: 00010046
[858121.456927] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88037c861530
[858121.456933] RDX: ffff8800123f2e78 RSI: 0000000000000000 RDI: 0000000000000002
[858121.456940] RBP: ffff8800d85e8380 R08: dead000000100100 R09: ffffffff814d2740
[858121.456947] R10: ffffffff8144bec0 R11: 0000000000000001 R12: ffff8800123f2e78
[858121.456953] R13: 000000000000000e R14: 0000000000000000 R15: 0000000000000000
[858121.456964] FS: 00007f8ef2b726e0(0000) GS:ffff8800123ef000(0000) knlGS:0000000000000000
[858121.456971] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[858121.456976] CR2: 00000000006b97f8 CR3: 000000037a6c8000 CR4: 0000000000002660
[858121.456983] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[858121.456990] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[858121.456997] Process khelper (pid: 12066, threadinfo ffff88030a1da000, task ffff8800c6d73f90)
[858121.457006] Stack:
[858121.457009] 0000000000000000 ffffffff8100d420 0001b0d6079460f2 0000000000000001
[858121.457009] <0> ffff8802e4dcfb38 ffffffff81074dd3 ffff8800123f2e78 ffff8800123f2e78
[858121.457009] <0> ffffffff8144c3c0 ffff88037e01c540 0000000000012b10 ffffffff8100e9b0
[858121.457009] Call Trace:
[858121.457009] <IRQ>
[858121.457009] [<ffffffff8100d420>] ? drop_other_mm_ref+0x24/0x55
[858121.457009] [<ffffffff81074dd3>] ? generic_smp_call_function_single_interrupt+0xcc/0x100
[858121.457009] [<ffffffff8100e9b0>] ? xen_call_function_single_interrupt+0xe/0x23
[858121.457009] [<ffffffff81094628>] ? handle_IRQ_event+0x58/0x126
[858121.641465] [<ffffffff81095d7b>] ? handle_percpu_irq+0x39/0x6a
[858121.641465] [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[858121.641465] [<ffffffff811eed0e>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[858121.641465] [<ffffffff811eedd3>] ? xen_evtchn_do_upcall+0x2e/0x41
[858121.641465] [<ffffffff81011cfe>] ? xen_do_hypervisor_callback+0x1e/0x30
[858121.641465] <EOI>
[858121.641465] [<ffffffff810f32c3>] ? flush_old_exec+0x399/0x548
[858121.641465] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[858121.641465] [<ffffffff8112630d>] ? load_elf_binary+0x39d/0x1954
[858121.641465] [<ffffffff810caa38>] ? vm_normal_page+0x11/0x61
[858121.641465] [<ffffffff810cb43c>] ? follow_page+0x2ad/0x303
[858121.641465] [<ffffffff810ccf98>] ? __get_user_pages+0x3b4/0x451
[858121.641465] [<ffffffff810f3e7b>] ? get_arg_page+0x61/0x110
[858121.641465] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[858121.641465] [<ffffffff810f2b5a>] ? search_binary_handler+0xb4/0x245
[858121.641465] [<ffffffff810f4357>] ? do_execve+0x1e4/0x2c3
[858121.641465] [<ffffffff8100f500>] ? sys_execve+0x35/0x4c
[858121.641465] [<ffffffff81011c28>] ? kernel_execve+0x68/0xd0
[858121.641465] [<ffffffff81152bbe>] ? cap_cred_commit+0x0/0x1
[858121.641465] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[858121.641465] [<ffffffff810600c2>] ? ____call_usermodehelper+0x114/0x11e
[858121.641465] [<ffffffff81011baa>] ? child_rip+0xa/0x20
[858121.641465] [<ffffffff81010d61>] ? int_ret_from_sys_call+0x7/0x1b
[858121.641465] [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[858121.641465] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[858121.641465] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[858121.641465] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[858121.641465] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[858121.641465] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
[858121.641465] Code: 08 ea 5c 81 00 00 00 00 66 ff 85 10 ea 5c 81 58 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 83 ec 08 65 8b 04 25 48 57 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 04 25 40 57 01 00 48 8d 90 b8 02 00 00 f0
[858121.641465] RIP [<ffffffff81036626>] leave_mm+0x10/0x44
[858121.641465] RSP <ffff8800123f2e48>
[858121.641465] ---[ end trace 2dd934112e2d638a ]---
[858121.641465] Kernel panic - not syncing: Fatal exception in interrupt
[858121.641465] Pid: 12066, comm: khelper Tainted: G D 2.6.32-bpo.5-amd64 #1
[858121.641465] Call Trace:
[858121.641465] <IRQ> [<ffffffff812f9d03>] ? panic+0x86/0x143
[858121.641465] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[858121.641465] [<ffffffff811b719c>] ? dummycon_dummy+0x0/0x3
[858121.641465] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[858121.641465] [<ffffffff8100e242>] ? check_events+0x12/0x20
[858121.641465] [<ffffffff811b719c>] ? dummycon_dummy+0x0/0x3
[858121.641465] [<ffffffff812fca65>] ? oops_end+0xa7/0xb4
[858121.641465] [<ffffffff810125f0>] ? do_invalid_op+0x8b/0x95
[858121.641465] [<ffffffff81036626>] ? leave_mm+0x10/0x44
[858121.641465] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[858121.641465] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[858121.641465] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[858121.641465] [<ffffffff8100e160>] ? xen_vcpuop_set_next_event+0x0/0x60
[858121.641465] [<ffffffff8100e1ac>] ? xen_vcpuop_set_next_event+0x4c/0x60
[858121.641465] [<ffffffff8101193b>] ? invalid_op+0x1b/0x20
[858121.641465] [<ffffffff81036626>] ? leave_mm+0x10/0x44
[858121.641465] [<ffffffff8100d420>] ? drop_other_mm_ref+0x24/0x55
[858121.641465] [<ffffffff81074dd3>] ? generic_smp_call_function_single_interrupt+0xcc/0x100
[858121.641465] [<ffffffff8100e9b0>] ? xen_call_function_single_interrupt+0xe/0x23
[858121.641465] [<ffffffff81094628>] ? handle_IRQ_event+0x58/0x126
[858121.641465] [<ffffffff81095d7b>] ? handle_percpu_irq+0x39/0x6a
[858121.641465] [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[858121.641465] [<ffffffff811eed0e>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[858121.641465] [<ffffffff811eedd3>] ? xen_evtchn_do_upcall+0x2e/0x41
[858121.641465] [<ffffffff81011cfe>] ? xen_do_hypervisor_callback+0x1e/0x30
[858121.641465] <EOI> [<ffffffff810f32c3>] ? flush_old_exec+0x399/0x548
[858121.641465] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[858121.641465] [<ffffffff8112630d>] ? load_elf_binary+0x39d/0x1954
[858121.641465] [<ffffffff810caa38>] ? vm_normal_page+0x11/0x61
[858121.641465] [<ffffffff810cb43c>] ? follow_page+0x2ad/0x303
[858121.641465] [<ffffffff810ccf98>] ? __get_user_pages+0x3b4/0x451
[858121.641465] [<ffffffff810f3e7b>] ? get_arg_page+0x61/0x110
[858121.641465] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[858121.641465] [<ffffffff810f2b5a>] ? search_binary_handler+0xb4/0x245
[858121.641465] [<ffffffff810f4357>] ? do_execve+0x1e4/0x2c3
[858121.641465] [<ffffffff8100f500>] ? sys_execve+0x35/0x4c
[858121.641465] [<ffffffff81011c28>] ? kernel_execve+0x68/0xd0
[858121.641465] [<ffffffff81152bbe>] ? cap_cred_commit+0x0/0x1
[858121.641465] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[858121.641465] [<ffffffff810600c2>] ? ____call_usermodehelper+0x114/0x11e
[858121.641465] [<ffffffff81011baa>] ? child_rip+0xa/0x20
[858121.641465] [<ffffffff81010d61>] ? int_ret_from_sys_call+0x7/0x1b
[858121.641465] [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[858121.641465] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[858121.641465] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[858121.641465] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[858121.641465] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[858121.641465] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Xen Minimal OS!
start_info: 0x2584000(VA)
nr_pages: 0x37e700
shared_inf: 0xce1e6000(MA)
pt_base: 0x2587000(VA)
nr_pt_frames: 0x17
mfn_list: 0x990000(VA)
mod_start: 0x0(VA)
mod_len: 0
flags: 0x0
cmd_line: (hd0)/boot/grub/grub.conf
stack: 0x94f860-0x96f860
MM: Init
_text: 0x0(VA)
_etext: 0x5fcad(VA)
_erodata: 0x78000(VA)
_edata: 0x80b00(VA)
stack start: 0x94f860(VA)
_end: 0x98fe68(VA)
start_pfn: 25a1
max_pfn: 37e700
Mapping memory range 0x2800000 - 0x37e700000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 418e000(418e000)-37e700000(37e700000)
MM: done
Demand map pfns at 37e701000-237e701000.
Heap resides at 237e702000-437e702000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x37e701000.
Initialising scheduler
Thread "Idle": pointer: 0x237e702010, stack: 0x5710000
Initialising xenbus
Thread "xenstore": pointer: 0x237e7027c0, stack: 0x5720000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x237e702f70, stack: 0x5730000
"main" "(hd0)/boot/grub/grub.conf"
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********
backend at /local/domain/0/backend/vbd/6/2049
Failed to read /local/domain/0/backend/vbd/6/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/6/2049/feature-flush-cache.
135528448 sectors of 512 bytes
**************************
vbd 2050 is hd1
******************* BLKFRONT for device/vbd/2050 **********
backend at /local/domain/0/backend/vbd/6/2050
Failed to read /local/domain/0/backend/vbd/6/2050/feature-barrier.
Failed to read /local/domain/0/backend/vbd/6/2050/feature-flush-cache.
58654720 sectors of 512 bytes
**************************
[H[J
GNU GRUB version 0.97 (14654464K lower / 0K upper memory)
[m[4;2H+-------------------------------------------------------------------------+[5;2H|[5;76H|[6;2H|[6;76H|[7;2H|[7;76H|[8;2H|[8;76H|[9;2H|[9;76H|[10;2H|[10;76H|[11;2H|[11;76H|[12;2H|[12;76H|[13;2H|[13;76H|[14;2H|[14;76H|[15;2H|[15;76H|[16;2H|[16;76H|[17;2H+-------------------------------------------------------------------------+[m
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.[5;78H [m[7m[5;3H Debian GNU/Linux, kernel 2.6.32-bpo.5-amd64 [5;75H[m[m[6;3H [6;75H[m[m[7;3H [7;75H[m[m[8;3H [8;75H[m[m[9;3H [9;75H[m[m[10;3H [10;75H[m[m[11;3H [11;75H[m[m[12;3H [12;75H[m[m[13;3H [13;75H[m[m[14;3H [14;75H[m[m[15;3H [15;75H[m[m[16;3H [16;75H[m[16;78H [5;75H[23;4H The highlighted entry will be booted automatically in 1 seconds. [5;75H[H[J Booting 'Debian GNU/Linux, kernel 2.6.32-bpo.5-amd64'
root (hd0)
Filesystem type is ext2fs, using whole disk
kernel /boot/vmlinuz-2.6.32-bpo.5-amd64 ipv6.disable=1 console=hvc0 root=/dev/
xvda1 ro
initrd /boot/initrd.img-2.6.32-bpo.5-amd64
close blk: backend at /local/domain/0/backend/vbd/6/2049
close blk: backend at /local/domain/0/backend/vbd/6/2050
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.32-bpo.5-amd64 (Debian 2.6.32-30~bpo50+1) (norbert@tretkowski.de) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP Mon Jan 17 18:24:45 UTC 2011
[ 0.000000] Command line: ipv6.disable=1 console=hvc0 root=/dev/xvda1 ro
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] ACPI in unprivileged domain disabled
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
[ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 000000037e700000 (usable)
[ 0.000000] DMI not present or invalid.
[ 0.000000] last_pfn = 0x37e700 max_arch_pfn = 0x400000000
[ 0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000
[ 0.000000] init_memory_mapping: 0000000000000000-0000000100000000
[ 0.000000] init_memory_mapping: 0000000100000000-000000037e700000
[ 0.000000] RAMDISK: 016a7000 - 02f03000
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at 0000000000000000-000000037e700000
[ 0.000000] Bootmem setup node 0 0000000000000000-000000037e700000
[ 0.000000] NODE_DATA [0000000000008000 - 000000000000ffff]
[ 0.000000] bootmap [0000000000010000 - 000000000007fcdf] pages 70
[ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 037e700000]
[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
[ 0.000000] #1 [0004afa000 - 0004b23000] XEN PAGETABLES ==> [0004afa000 - 0004b23000]
[ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
[ 0.000000] #3 [0001000000 - 0001686414] TEXT DATA BSS ==> [0001000000 - 0001686414]
[ 0.000000] #4 [00016a7000 - 0002f03000] RAMDISK ==> [00016a7000 - 0002f03000]
[ 0.000000] #5 [0002f03000 - 0004afa000] XEN START INFO ==> [0002f03000 - 0004afa000]
[ 0.000000] #6 [0000100000 - 00008d9000] PGTABLE ==> [0000100000 - 00008d9000]
[ 0.000000] #7 [0004b23000 - 0005f21000] PGTABLE ==> [0004b23000 - 0005f21000]
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000000 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] Normal 0x00100000 -> 0x0037e700
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000000 -> 0x000000a0
[ 0.000000] 0: 0x00000100 -> 0x0037e700
[ 0.000000] SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
[ 0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[ 0.000000] No local APIC present
[ 0.000000] APIC: disable apic facility
[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[ 0.000000] PCI: Warning: Cannot find a gap in the 32bit address range
[ 0.000000] PCI: Unassigned devices with 32bit resource registers may break!
[ 0.000000] Allocating PCI resources starting at 37e800000 (gap: 37e800000:400000)
[ 0.000000] Booting paravirtualized kernel on Xen
[ 0.000000] Xen version: 3.4.2 (preserve-AD)
[ 0.000000] NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 30 pages/cpu @ffff8800123b3000 s90328 r8192 d24360 u122880
[ 0.000000] pcpu-alloc: s90328 r8192 d24360 u122880 alloc=30*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[ 0.000000] Xen: using vcpu_info placement
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 3611419
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: ipv6.disable=1 console=hvc0 root=/dev/xvda1 ro
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Initializing CPU#0
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.000000] Placing 64MB software IO TLB between ffff880012434000 - ffff880016434000
[ 0.000000] software IO TLB at phys 0x12434000 - 0x16434000
[ 0.000000] Memory: 14297184k/14654464k available (3079k kernel code, 384k absent, 356896k reserved, 1876k data, 592k init)
[ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] NR_IRQS:4352 nr_irqs:304
[ 0.000000] Console: colour dummy device 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] console [hvc0] enabled
[ 0.000000] installing Xen timer for CPU 0
[ 0.000000] Detected 2493.776 MHz processor.
[ 0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4987.55 BogoMIPS (lpj=9975104)
[ 0.004000] Security Framework initialized
[ 0.004000] SELinux: Disabled at boot.
[ 0.004000] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[ 0.013640] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[ 0.017905] Mount-cache hash table entries: 256
[ 0.018090] Initializing cgroup subsys ns
[ 0.018099] Initializing cgroup subsys cpuacct
[ 0.018106] Initializing cgroup subsys devices
[ 0.018112] Initializing cgroup subsys freezer
[ 0.018117] Initializing cgroup subsys net_cls
[ 0.018161] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.018167] CPU: L2 cache: 6144K
[ 0.018173] CPU 0/0x1 -> Node 0
[ 0.018178] CPU: Physical Processor ID: 0
[ 0.018182] CPU: Processor Core ID: 1
[ 0.018188] Performance Events: unsupported p6 CPU model 23 no PMU driver, software events only.
[ 0.018210] SMP alternatives: switching to UP code
[ 0.020279] installing Xen timer for CPU 1
[ 0.020321] SMP alternatives: switching to SMP code
[ 0.004000] Initializing CPU#1
[ 0.004000] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.004000] CPU: L2 cache: 6144K
[ 0.004000] CPU 1/0x1 -> Node 0
[ 0.004000] CPU: Physical Processor ID: 0
[ 0.004000] CPU: Processor Core ID: 1
[ 0.024156] installing Xen timer for CPU 2
[ 0.004000] Initializing CPU#2
[ 0.004000] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.004000] CPU: L2 cache: 6144K
[ 0.004000] CPU 2/0x1 -> Node 0
[ 0.004000] CPU: Physical Processor ID: 0
[ 0.004000] CPU: Processor Core ID: 1
[ 0.024387] installing Xen timer for CPU 3
[ 0.004000] Initializing CPU#3
[ 0.004000] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.004000] CPU: L2 cache: 6144K
[ 0.004000] CPU 3/0x1 -> Node 0
[ 0.004000] CPU: Physical Processor ID: 0
[ 0.004000] CPU: Processor Core ID: 1
[ 0.024557] Brought up 4 CPUs
[ 0.026604] devtmpfs: initialized
[ 0.030468] Grant table initialized
[ 0.030468] regulator: core version 0.5
[ 0.030468] NET: Registered protocol family 16
[ 0.030468] PCI: Fatal: No config space access function found
[ 0.032126] bio: create slab <bio-0> at 0
[ 0.032126] ACPI: Interpreter disabled.
[ 0.032126] xen_balloon: Initialising balloon driver.
[ 0.032126] vgaarb: loaded
[ 0.036085] PCI: System does not support PCI
[ 0.036093] PCI: System does not support PCI
[ 0.036211] Switching to clocksource xen
[ 0.037696] pnp: PnP ACPI: disabled
[ 0.038206] NET: Registered protocol family 2
[ 0.039168] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
[ 0.041994] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[ 0.042172] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 0.042172] TCP: Hash tables configured (established 524288 bind 65536)
[ 0.042172] TCP reno registered
[ 0.042172] NET: Registered protocol family 1
[ 0.042172] Unpacking initramfs...
[ 0.079100] Freeing initrd memory: 24944k freed
[ 0.088323] platform rtc_cmos: registered platform RTC device (no PNP device found)
[ 0.088733] audit: initializing netlink socket (disabled)
[ 0.088756] type=2000 audit(1298123181.989:1): initialized
[ 0.092478] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 0.094150] VFS: Disk quotas dquot_6.5.2
[ 0.094210] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.094320] msgmni has been set to 27972
[ 0.094819] alg: No test for stdrng (krng)
[ 0.094885] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 0.094900] io scheduler noop registered
[ 0.094905] io scheduler anticipatory registered
[ 0.094910] io scheduler deadline registered
[ 0.094950] io scheduler cfq registered (default)
[ 0.097312] Linux agpgart interface v0.103
[ 0.097348] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.097559] input: Macintosh mouse button emulation as /devices/virtual/input/input0
[ 0.097625] PNP: No PS/2 controller found. Probing ports directly.
[ 0.098445] i8042.c: No controller found.
[ 0.098512] mice: PS/2 mouse device common for all mice
[ 0.098565] rtc_cmos: probe of rtc_cmos failed with error -16
[ 0.098591] cpuidle: using governor ladder
[ 0.098595] cpuidle: using governor menu
[ 0.098603] No iBFT detected.
[ 0.098865] TCP cubic registered
[ 0.098870] IPv6: Loaded, but administratively disabled, reboot required to enable
[ 0.098877] Mobile IPv6
[ 0.098881] mip6_init: can't add xfrm type(destopt)
[ 0.098887] NET: Registered protocol family 17
[ 0.098977] registered taskstats version 1
[ 0.099002] XENBUS: Device with no driver: device/console/0
[ 0.099007] XENBUS: Device with no driver: device/vbd/2049
[ 0.099011] XENBUS: Device with no driver: device/vbd/2050
[ 0.099016] XENBUS: Device with no driver: device/vif/0
[ 0.099031] /tmp/buildd/linux-2.6-2.6.32/debian/build/source_amd64_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[ 0.099051] Initalizing network drop monitor service
[ 0.099133] Freeing unused kernel memory: 592k freed
[ 0.099329] Write protecting the kernel read-only data: 4216k
Loading, please wait...
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... [ 0.146776] udevd version 125 started
[ 0.203644] Initialising Xen virtual ethernet driver.
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... kinit: name_to_dev_t(/dev/xvda2) = dev(202,2)
kinit: trying to resume from /dev/xvda2
[ 0.426844] PM: Starting manual resume from disk
kinit: No resume image, doing normal boot...
done.
[ 0.450912] kjournald starting. Commit interval 5 seconds
[ 0.450930] EXT3-fs: mounted filesystem with ordered data mode.
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.86 booting
Starting the hotplug events dispatcher: udevd[ 1.497236] udevd version 125 started
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Setting the system clock.
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Unable to set System Clock to: Sat Feb 19 13:46:23 UTC 2011 [33m(warning).[39;49m
Activating swap...[ 1.901850] Adding 29327352k swap on /dev/xvda2. Priority:-1 extents:1 across:29327352k SS
done.
Checking root file system...fsck 1.41.3 (12-Oct-2008)
/dev/xvda1: clean, 1082673/2117632 files, 3180334/16941056 blocks
done.
[ 2.240091] EXT3 FS on xvda1, internal journal
Setting the system clock.
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Unable to set System Clock to: Sat Feb 19 13:46:24 UTC 2011 [33m(warning).[39;49m
Cleaning up ifupdown....
Loading kernel modules...done.
Checking file systems...fsck 1.41.3 (12-Oct-2008)
done.
Setting kernel variables (/etc/sysctl.conf)...done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Setting up networking....
Configuring network interfaces...Internet Systems Consortium DHCP Client V3.1.1
Copyright 2004-2008 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/
/var/lib/dhcp3/dhclient.eth0.leases line 10: Expecting a domain string.
option domain-search ;
^
/var/lib/dhcp3/dhclient.eth0.leases line 26: Expecting a domain string.
option domain-search ;
^
/var/lib/dhcp3/dhclient.eth0.leases line 42: Expecting a domain string.
option domain-search ;
^
/var/lib/dhcp3/dhclient.eth0.leases line 58: Expecting a domain string.
option domain-search ;
^
Listening on LPF/eth0/00:16:3e:01:cd:bc
Sending on LPF/eth0/00:16:3e:01:cd:bc
Sending on Socket/fallback
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4
DHCPOFFER from 68.67.141.131
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPACK from 68.67.141.131
Invalid domain list.
Invalid domain list.
bound to 68.67.141.188 -- renewal in 12177 seconds.
Starting portmap daemon....
Starting NFS common utilities: statd.
[ 22.386603] RPC: Registered udp transport module.
[ 22.386620] RPC: Registered tcp transport module.
[ 22.386625] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 22.469714] Slow work thread pool: Starting up
[ 22.469823] Slow work thread pool: Ready
[ 22.469881] FS-Cache: Loaded
[ 22.534836] FS-Cache: Netfs 'nfs' registered for caching
done.
Starting portmap daemon...Already running..
Starting NFS common utilities: statd.
INIT: Entering runlevel: 2
Starting system logging: syslog-ng.
Starting system message bus: dbus.
Starting OpenBSD Secure Shell server: sshd.
Starting MySQL database server: mysqld . . ..
Checking for corrupt, not cleanly closed and upgrade needing tables..
Starting atop system monitor: atop.
Starting Java program runner: nailgun-common.
Starting NFS common utilities: statd.
Starting NTP server: ntpd.
Starting Postfix Mail Transport Agent: postfix[ 33.614777] warning: `ntpd' uses 32-bit capabilities (legacy support in use)
.
sadc not enabled in /etc/default/sysstat, not starting. [33m(warning).[39;49m
Starting Zabbix agent: zabbix_agentd.
Starting puppet agent.
Debian GNU/Linux 5.0 xxx.xxx.xxx.xxx hvc0
xxx.xxx.xxx.xxx login: [ 95.222920] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 4920.088064] INFO: task parse_ulogs.pl:1504 blocked for more than 120 seconds.
[ 4920.088079] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4920.088101] parse_ulogs.p D 0000000000000000 0 1504 1502 0x00000000
[ 4920.088111] ffff88037c861530 0000000000000282 0000000000000000 ffffffff811031c5
[ 4920.088123] ffff88037b781e48 000000000000f9e0 ffff88037b781fd8 0000000000015780
[ 4920.088134] 0000000000015780 ffff8803112fc6a0 ffff8803112fc998 00000002112fc6a0
[ 4920.088148] Call Trace:
[ 4920.088170] [<ffffffff811031c5>] ? mntput_no_expire+0x23/0xee
[ 4920.088184] [<ffffffff812fb049>] ? __mutex_lock_common+0x122/0x192
[ 4920.088195] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[ 4920.088204] [<ffffffff8100e242>] ? check_events+0x12/0x20
[ 4920.088211] [<ffffffff812fb171>] ? mutex_lock+0x1a/0x31
[ 4920.088219] [<ffffffff810ece76>] ? do_sys_open+0xec/0xfc
[ 4920.088231] [<ffffffff810ef3ef>] ? generic_file_llseek+0x22/0x53
[ 4920.088247] [<ffffffff810ee228>] ? sys_lseek+0x44/0x64
[ 4920.088258] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[15265.002752] hrtimer: interrupt took 17640570 ns
[530522.149567] INFO: task mysqld:10019 blocked for more than 120 seconds.
[530522.149583] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[530522.149590] mysqld D ffff8800123c8780 0 10019 9758 0x00000000
[530522.149600] ffff8800e9610000 0000000000000282 ffff8800123c47c0 000000010000000e
[530522.149613] ffff88037c6e5b38 000000000000f9e0 ffff8801120f1fd8 0000000000015780
[530522.149628] 0000000000015780 ffff88011ac6b880 ffff88011ac6bb78 00000000810bbd06
[530522.149640] Call Trace:
[530522.149654] [<ffffffff8100e242>] ? check_events+0x12/0x20
[530522.149668] [<ffffffffa0017f64>] ? log_wait_commit+0xbf/0x112 [jbd]
[530522.149677] [<ffffffff81064d16>] ? autoremove_wake_function+0x0/0x2e
[530522.149686] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[530522.149696] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[530522.149707] [<ffffffffa00203f8>] ? ext3_sync_file+0x94/0xc8 [ext3]
[530522.149717] [<ffffffff8110b880>] ? vfs_fsync_range+0x73/0x9e
[530522.149724] [<ffffffff8110b92a>] ? do_fsync+0x28/0x39
[530522.149731] [<ffffffff8110b959>] ? sys_fsync+0xb/0x10
[530522.149739] [<ffffffff81010b63>] ? sysret_check+0x17/0x5a
[530522.149746] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
[936868.824090] nfs: server isilon.lax1 not responding, still trying
[936874.344299] nfs: server isilon.lax1 OK
[1065529.000062] NOHZ: local_softirq_pending 100
[1144513.865534] nfs: server isilon.lax1 not responding, still trying
[1144513.866131] nfs: server isilon.lax1 OK
[1963932.743134] ------------[ cut here ]------------
[1963932.743148] kernel BUG at /tmp/buildd/linux-2.6-2.6.32/debian/build/source_amd64_none/arch/x86/mm/tlb.c:61!
[1963932.743156] invalid opcode: 0000 [#1] SMP
[1963932.743164] last sysfs file: /sys/devices/virtual/net/lo/operstate
[1963932.743170] CPU 2
[1963932.743174] Modules linked in: xt_recent xt_tcpudp iptable_filter ip_tables x_tables nfs lockd fscache nfs_acl auth_rpcgss sunrpc ext3 jbd mbcache xen_netfront xen_blkfront
[1963932.743209] Pid: 10679, comm: khelper Not tainted 2.6.32-bpo.5-amd64 #1
[1963932.743214] RIP: e030:[<ffffffff81036626>] [<ffffffff81036626>] leave_mm+0x10/0x44
[1963932.743228] RSP: e02b:ffff8800123f2e48 EFLAGS: 00010046
[1963932.743233] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88037c861530
[1963932.743240] RDX: ffff8800123f2e78 RSI: 0000000000000000 RDI: 0000000000000002
[1963932.743247] RBP: ffff880379d03b80 R08: dead000000100100 R09: ffffffff814d2740
[1963932.743254] R10: ffffffff8144bec0 R11: 0000000000000001 R12: ffff8800123f2e78
[1963932.743260] R13: 000000000000000e R14: 0000000000000000 R15: 0000000000000000
[1963932.743270] FS: 00007f695ec426e0(0000) GS:ffff8800123ef000(0000) knlGS:0000000000000000
[1963932.743277] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[1963932.743283] CR2: 00000000006b97f8 CR3: 000000027960c000 CR4: 0000000000002660
[1963932.743291] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1963932.743298] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1963932.743305] Process khelper (pid: 10679, threadinfo ffff880243d0c000, task ffff88008621aa60)
[1963932.743312] Stack:
[1963932.743315] 0000000000000000 ffffffff8100d420 0003d130ac03faa5 0000000000000001
[1963932.743325] <0> ffff880279613b38 ffffffff81074dd3 ffff8800123f2e78 ffff8800123f2e78
[1963932.743338] <0> ffffffff8144c3c0 ffff88037e01c540 0000000000012b10 ffffffff8100e9b0
[1963932.743352] Call Trace:
[1963932.743356] <IRQ>
[1963932.743364] [<ffffffff8100d420>] ? drop_other_mm_ref+0x24/0x55
[1963932.745958] [<ffffffff81074dd3>] ? generic_smp_call_function_single_interrupt+0xcc/0x100
[1963932.745958] [<ffffffff8100e9b0>] ? xen_call_function_single_interrupt+0xe/0x23
[1963932.745958] [<ffffffff81094628>] ? handle_IRQ_event+0x58/0x126
[1963932.745958] [<ffffffff81095d7b>] ? handle_percpu_irq+0x39/0x6a
[1963932.745958] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[1963932.745958] [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[1963932.745958] [<ffffffff811eed0e>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[1963932.745958] [<ffffffff811eedd3>] ? xen_evtchn_do_upcall+0x2e/0x41
[1963932.745958] [<ffffffff81011cfe>] ? xen_do_hypervisor_callback+0x1e/0x30
[1963932.745958] <EOI>
[1963932.745958] [<ffffffff810f32c3>] ? flush_old_exec+0x399/0x548
[1963932.745958] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[1963932.745958] [<ffffffff8112630d>] ? load_elf_binary+0x39d/0x1954
[1963932.745958] [<ffffffff810caa38>] ? vm_normal_page+0x11/0x61
[1963932.745958] [<ffffffff810cb43c>] ? follow_page+0x2ad/0x303
[1963932.745958] [<ffffffff810ccf98>] ? __get_user_pages+0x3b4/0x451
[1963932.745958] [<ffffffff810f3e7b>] ? get_arg_page+0x61/0x110
[1963932.745958] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[1963932.745958] [<ffffffff810f2b5a>] ? search_binary_handler+0xb4/0x245
[1963932.745958] [<ffffffff810f4357>] ? do_execve+0x1e4/0x2c3
[1963932.745958] [<ffffffff8100f500>] ? sys_execve+0x35/0x4c
[1963932.745958] [<ffffffff81011c28>] ? kernel_execve+0x68/0xd0
[1963932.745958] [<ffffffff81152bbe>] ? cap_cred_commit+0x0/0x1
[1963932.745958] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[1963932.745958] [<ffffffff810600c2>] ? ____call_usermodehelper+0x114/0x11e
[1963932.745958] [<ffffffff81011baa>] ? child_rip+0xa/0x20
[1963932.745958] [<ffffffff81010d61>] ? int_ret_from_sys_call+0x7/0x1b
[1963932.745958] [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[1963932.745958] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[1963932.745958] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[1963932.745958] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[1963932.745958] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[1963932.745958] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
[1963932.745958] Code: 08 ea 5c 81 00 00 00 00 66 ff 85 10 ea 5c 81 58 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 83 ec 08 65 8b 04 25 48 57 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 04 25 40 57 01 00 48 8d 90 b8 02 00 00 f0
[1963932.745958] RIP [<ffffffff81036626>] leave_mm+0x10/0x44
[1963932.745958] RSP <ffff8800123f2e48>
[1963932.745958] ---[ end trace 64d5f162f62f408d ]---
[1963932.745958] Kernel panic - not syncing: Fatal exception in interrupt
[1963932.745958] Pid: 10679, comm: khelper Tainted: G D 2.6.32-bpo.5-amd64 #1
[1963932.745958] Call Trace:
[1963932.745958] <IRQ> [<ffffffff812f9d03>] ? panic+0x86/0x143
[1963932.745958] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[1963932.745958] [<ffffffff811b719c>] ? dummycon_dummy+0x0/0x3
[1963932.745958] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[1963932.745958] [<ffffffff8100e242>] ? check_events+0x12/0x20
[1963932.745958] [<ffffffff811b719c>] ? dummycon_dummy+0x0/0x3
[1963932.745958] [<ffffffff812fca65>] ? oops_end+0xa7/0xb4
[1963932.745958] [<ffffffff810125f0>] ? do_invalid_op+0x8b/0x95
[1963932.745958] [<ffffffff81036626>] ? leave_mm+0x10/0x44
[1963932.745958] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[1963932.745958] [<ffffffff812fbbca>] ? _spin_unlock_irqrestore+0xd/0xe
[1963932.745958] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[1963932.745958] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[1963932.745958] [<ffffffff8100e160>] ? xen_vcpuop_set_next_event+0x0/0x60
[1963932.745958] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[1963932.745958] [<ffffffff8100e1ac>] ? xen_vcpuop_set_next_event+0x4c/0x60
[1963932.745958] [<ffffffff8101193b>] ? invalid_op+0x1b/0x20
[1963932.745958] [<ffffffff81036626>] ? leave_mm+0x10/0x44
[1963932.745958] [<ffffffff8100d420>] ? drop_other_mm_ref+0x24/0x55
[1963932.745958] [<ffffffff81074dd3>] ? generic_smp_call_function_single_interrupt+0xcc/0x100
[1963932.745958] [<ffffffff8100e9b0>] ? xen_call_function_single_interrupt+0xe/0x23
[1963932.745958] [<ffffffff81094628>] ? handle_IRQ_event+0x58/0x126
[1963932.917154] [<ffffffff81095d7b>] ? handle_percpu_irq+0x39/0x6a
[1963932.917154] [<ffffffff8102cdc0>] ? pvclock_clocksource_read+0x3a/0x8b
[1963932.917154] [<ffffffff8101327f>] ? handle_irq+0x17/0x1d
[1963932.917154] [<ffffffff811eed0e>] ? __xen_evtchn_do_upcall+0xd5/0x15e
[1963932.917154] [<ffffffff811eedd3>] ? xen_evtchn_do_upcall+0x2e/0x41
[1963932.917154] [<ffffffff81011cfe>] ? xen_do_hypervisor_callback+0x1e/0x30
[1963932.917154] <EOI> [<ffffffff810f32c3>] ? flush_old_exec+0x399/0x548
[1963932.917154] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[1963932.917154] [<ffffffff8112630d>] ? load_elf_binary+0x39d/0x1954
[1963932.917154] [<ffffffff810caa38>] ? vm_normal_page+0x11/0x61
[1963932.917154] [<ffffffff810cb43c>] ? follow_page+0x2ad/0x303
[1963932.917154] [<ffffffff810ccf98>] ? __get_user_pages+0x3b4/0x451
[1963932.917154] [<ffffffff810f3e7b>] ? get_arg_page+0x61/0x110
[1963932.917154] [<ffffffff81125f70>] ? load_elf_binary+0x0/0x1954
[1963932.917154] [<ffffffff810f2b5a>] ? search_binary_handler+0xb4/0x245
[1963932.917154] [<ffffffff810f4357>] ? do_execve+0x1e4/0x2c3
[1963932.917154] [<ffffffff8100f500>] ? sys_execve+0x35/0x4c
[1963932.917154] [<ffffffff81011c28>] ? kernel_execve+0x68/0xd0
[1963932.917154] [<ffffffff81152bbe>] ? cap_cred_commit+0x0/0x1
[1963932.917154] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[1963932.917154] [<ffffffff810600c2>] ? ____call_usermodehelper+0x114/0x11e
[1963932.917154] [<ffffffff81011baa>] ? child_rip+0xa/0x20
[1963932.917154] [<ffffffff81010d61>] ? int_ret_from_sys_call+0x7/0x1b
[1963932.917154] [<ffffffff8101151d>] ? retint_restore_args+0x5/0x6
[1963932.917154] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[1963932.917154] [<ffffffff8100dc35>] ? xen_force_evtchn_callback+0x9/0xa
[1963932.917154] [<ffffffff8100e22f>] ? xen_restore_fl_direct_end+0x0/0x1
[1963932.917154] [<ffffffff8104801a>] ? finish_task_switch+0x44/0xaf
[1963932.917154] [<ffffffff81011ba0>] ? child_rip+0x0/0x20
Reply to: