[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#668511: Kernel panic with bridge networking



On 21/05/2012 15:03, Jonathan Nieder wrote:
# patch applied to mainline
tags 668511 - moreinfo
quit

Sounds painful.  Here are instructions for testing against upstream
source, which might be easier.

[CUT]

You wrote at [1] that this doesn't seem to have completely solved the
bridging-related panics.  Am I correct in assuming the patch still
works as advertised, though?  Are there other patches we should pick
up?
Hi,

I'm sorry for this late reply but i had a lot of problems and i lost time reverting the kernel to a 2.6 series one to avoid panics.

Today i have installed the latest debian kernel from the testing repository (linux-image-3.2.0-2-amd64 - 3.2.20-1) and the panics are still present.
I'm attaching 2 of them grabbed through netconsole.

Then i compiled, using your instructions, the latest 3.2.21 and i had (after 5 minutes of uptime) another panic which seems different.
The panic with a 3.2.21 is attached as well.

I don't really know the cause (i'm not a kernel hacker) but it seems related to bridges and to "aliased" bridge interfaces. I'm using heartbeat for high availability and if the aliased bridged IPs are moved to the other node, the panic does not happen.

Max

P.S.: (i'll send another email to netdev@vger)

Jun 28 13:18:35 172.30.1.2 [ 295.504843] BUG: unable to handle kernel 
Jun 28 13:18:35 172.30.1.2 [ 295.520628] IP:
Jun 28 13:18:35 172.30.1.2 [ 295.534792] PGD 0 
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.538889] Oops: 0000 [#1] 
Jun 28 13:18:35 172.30.1.2 SMP
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.545471] CPU 0 
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.549169] Modules linked in:
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.691502] 
Jun 28 13:18:35 172.30.1.2 [ 295.694506] Pid: 7754, comm: kvm Not tainted 3.2.0-2-amd64 #1
Jun 28 13:18:35 172.30.1.2 /0N051F
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.711720] RIP: 0010:[<ffffffffa01c333d>] 
Jun 28 13:18:35 172.30.1.2 [ 295.730738] RSP: 0018:ffff88042fc03b18  EFLAGS: 00010293
Jun 28 13:18:35 172.30.1.2 [ 295.741389] RAX: 0000000000000000 RBX: ffff880411b9eec0 RCX: 00000000fffffba0
Jun 28 13:18:35 172.30.1.2 [ 295.755682] RDX: ffffffffa01c330f RSI: 0000000000000282 RDI: ffff880411b9eec0
Jun 28 13:18:35 172.30.1.2 [ 295.769975] RBP: ffff880226688000 R08: 0000000000000000 R09: ffff88042fc03ad0
Jun 28 13:18:35 172.30.1.2 [ 295.784268] R10: ffffffff8165ab00 R11: ffffffff8165ab00 R12: 0000000000000000
Jun 28 13:18:35 172.30.1.2 [ 295.798561] R13: ffff880225bec002 R14: ffff8804131987c0 R15: ffff880225bec000
Jun 28 13:18:35 172.30.1.2 [ 295.812854] FS:  00007fc06177d900(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
Jun 28 13:18:35 172.30.1.2 [ 295.829070] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 28 13:18:35 172.30.1.2 [ 295.840587] CR2: 0000000000000018 CR3: 0000000409d7b000 CR4: 00000000000026e0
Jun 28 13:18:35 172.30.1.2 [ 295.854880] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 28 13:18:35 172.30.1.2 [ 295.869175] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 28 13:18:35 172.30.1.2 [ 295.883469] Process kvm (pid: 7754, threadinfo ffff88040bcc6000, task ffff880424ca2100)
Jun 28 13:18:35 172.30.1.2 [ 295.899512] Stack:
Jun 28 13:18:35 172.30.1.2 [ 295.903571]  ffffffff80000000
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.918613]  ffff880425367000
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.933657]  ffff880411b9eec0
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.948699] Call Trace:
Jun 28 13:18:35 172.30.1.2 [ 295.953626]  <IRQ> 
Jun 28 13:18:35 172.30.1.2 
Jun 28 13:18:35 172.30.1.2 [ 295.957918]  [<ffffffffa01c3714>] ? br_parse_ip_options+0x3d/0x19a [bridge]
Jun 28 13:18:35 172.30.1.2 [ 295.971866]  [<ffffffffa01c3aa0>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 295.985469]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 13:18:36 172.30.1.2 [ 295.996123]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.008687]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.021245]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 13:18:36 172.30.1.2 [ 296.032418]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.044981]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.058757]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.071319]  [<ffffffffa01be887>] ? NF_HOOK.constprop.10+0x3c/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.085267]  [<ffffffffa01bea1b>] ? br_forward+0x16/0x5a [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.097480]  [<ffffffffa01bf543>] ? br_handle_frame_finish+0x1a1/0x20f [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.112141]  [<ffffffffa01c3638>] ? br_nf_pre_routing_finish+0x1ee/0x1fb [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.127146]  [<ffffffffa01c2ff7>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.140055]  [<ffffffffa01c3f91>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.153828]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 13:18:36 172.30.1.2 [ 296.164482]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.178255]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 13:18:36 172.30.1.2 [ 296.189430]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.203206]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.216982]  [<ffffffffa01bf388>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.230757]  [<ffffffff810135ad>] ? paravirt_read_tsc+0x5/0x8
Jun 28 13:18:36 172.30.1.2 [ 296.242274]  [<ffffffff81013622>] ? read_tsc+0x5/0x14
Jun 28 13:18:36 172.30.1.2 [ 296.252409]  [<ffffffffa01bf764>] ? br_handle_frame+0x1b3/0x1cb [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.265837]  [<ffffffffa01bf5b1>] ? br_handle_frame_finish+0x20f/0x20f [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.280498]  [<ffffffff8128a306>] ? __netif_receive_skb+0x324/0x41f
Jun 28 13:18:36 172.30.1.2 [ 296.293058]  [<ffffffff8128a46d>] ? process_backlog+0x6c/0x123
Jun 28 13:18:36 172.30.1.2 [ 296.304751]  [<ffffffff8128c351>] ? net_rx_action+0xa1/0x1af
Jun 28 13:18:36 172.30.1.2 [ 296.316098]  [<ffffffff8103703b>] ? test_tsk_need_resched+0xa/0x13
Jun 28 13:18:36 172.30.1.2 [ 296.328486]  [<ffffffff8104bee4>] ? __do_softirq+0xb9/0x177
Jun 28 13:18:36 172.30.1.2 [ 296.339662]  [<ffffffff8135162c>] ? call_softirq+0x1c/0x30
Jun 28 13:18:36 172.30.1.2 [ 296.350656]  <EOI> 
Jun 28 13:18:36 172.30.1.2 
Jun 28 13:18:36 172.30.1.2 [ 296.354946]  [<ffffffff8100f8e5>] ? do_softirq+0x3c/0x7b
Jun 28 13:18:36 172.30.1.2 [ 296.365597]  [<ffffffff8128c641>] ? netif_rx_ni+0x1e/0x27
Jun 28 13:18:36 172.30.1.2 [ 296.376425]  [<ffffffffa028b721>] ? tun_get_user+0x39a/0x3c2 [tun]
Jun 28 13:18:36 172.30.1.2 [ 296.388811]  [<ffffffffa028ba66>] ? tun_chr_poll+0xcd/0xcd [tun]
Jun 28 13:18:36 172.30.1.2 [ 296.400851]  [<ffffffffa028bac4>] ? tun_chr_aio_write+0x5e/0x79 [tun]
Jun 28 13:18:36 172.30.1.2 [ 296.413760]  [<ffffffff810f97f8>] ? do_sync_readv_writev+0x9a/0xd7
Jun 28 13:18:36 172.30.1.2 [ 296.426145]  [<ffffffff81036457>] ? should_resched+0x5/0x23
Jun 28 13:18:36 172.30.1.2 [ 296.437316]  [<ffffffff81036457>] ? should_resched+0x5/0x23
Jun 28 13:18:36 172.30.1.2 [ 296.448491]  [<ffffffff81162ea1>] ? security_file_permission+0x16/0x2d
Jun 28 13:18:36 172.30.1.2 [ 296.461569]  [<ffffffff810f9a5c>] ? do_readv_writev+0xaf/0x11c
Jun 28 13:18:36 172.30.1.2 [ 296.473266]  [<ffffffff8112aebe>] ? eventfd_ctx_read+0x162/0x174
Jun 28 13:18:36 172.30.1.2 [ 296.485310]  [<ffffffff8103f48f>] ? try_to_wake_up+0x197/0x197
Jun 28 13:18:36 172.30.1.2 [ 296.497001]  [<ffffffff810f9c31>] ? sys_writev+0x45/0x90
Jun 28 13:18:36 172.30.1.2 [ 296.507653]  [<ffffffff8134f3d2>] ? system_call_fastpath+0x16/0x1b
Jun 28 13:18:36 172.30.1.2 [ 296.520038] Code: 
Jun 28 13:18:36 172.30.1.2 53
Jun 28 13:18:36 172.30.1.2 48
Jun 28 13:18:36 172.30.1.2 89
Jun 28 13:18:36 172.30.1.2 fb
Jun 28 13:18:36 172.30.1.2 48
Jun 28 13:18:36 172.30.1.2 83
Jun 28 13:18:36 172.30.1.2 ec
Jun 28 13:18:36 172.30.1.2 10
Jun 28 13:18:36 172.30.1.2 66
Jun 28 13:18:36 172.30.1.2 81
Jun 28 13:18:36 172.30.1.2 7f
Jun 28 13:18:36 172.30.1.2 7e
Jun 28 13:18:36 172.30.1.2 08
Jun 28 13:18:36 172.30.1.2 06
Jun 28 13:18:36 172.30.1.2 4c
Jun 28 13:18:36 172.30.1.2 8b
Jun 28 13:18:36 172.30.1.2 a7
Jun 28 13:18:36 172.30.1.2 98
Jun 28 13:18:36 172.30.1.2 00
Jun 28 13:18:36 172.30.1.2 00
Jun 28 13:18:36 172.30.1.2 00
Jun 28 13:18:36 172.30.1.2 74
Jun 28 13:18:36 172.30.1.2 3d
Jun 28 13:18:36 172.30.1.2 e8
Jun 28 13:18:36 172.30.1.2 07
Jun 28 13:18:36 172.30.1.2 fe
Jun 28 13:18:36 172.30.1.2 ff
Jun 28 13:18:36 172.30.1.2 ff
Jun 28 13:18:36 172.30.1.2 66
Jun 28 13:18:36 172.30.1.2 3d
Jun 28 13:18:36 172.30.1.2 08
Jun 28 13:18:36 172.30.1.2 06
Jun 28 13:18:36 172.30.1.2 75
Jun 28 13:18:36 172.30.1.2 09
Jun 28 13:18:36 172.30.1.2 83
Jun 28 13:18:36 172.30.1.2 3d
Jun 28 13:18:36 172.30.1.2 91
Jun 28 13:18:36 172.30.1.2 6a
Jun 28 13:18:36 172.30.1.2 00
Jun 28 13:18:36 172.30.1.2 00
Jun 28 13:18:36 172.30.1.2 00
Jun 28 13:18:36 172.30.1.2 75
Jun 28 13:18:36 172.30.1.2 29
Jun 28 13:18:36 172.30.1.2 
Jun 28 13:18:36 172.30.1.2 f6
Jun 28 13:18:36 172.30.1.2 44
Jun 28 13:18:36 172.30.1.2 24
Jun 28 13:18:36 172.30.1.2 18
Jun 28 13:18:36 172.30.1.2 01
Jun 28 13:18:36 172.30.1.2 49
Jun 28 13:18:36 172.30.1.2 8b
Jun 28 13:18:36 172.30.1.2 6c
Jun 28 13:18:36 172.30.1.2 24
Jun 28 13:18:36 172.30.1.2 08
Jun 28 13:18:36 172.30.1.2 74
Jun 28 13:18:36 172.30.1.2 12
Jun 28 13:18:36 172.30.1.2 8a
Jun 28 13:18:36 172.30.1.2 43
Jun 28 13:18:36 172.30.1.2 7d
Jun 28 13:18:36 172.30.1.2 83
Jun 28 13:18:36 172.30.1.2 e0
Jun 28 13:18:36 172.30.1.2 f8
Jun 28 13:18:36 172.30.1.2 83
Jun 28 13:18:36 172.30.1.2 c8
Jun 28 13:18:36 172.30.1.2 
Jun 28 13:18:36 172.30.1.2 [ 296.561479] RIP 
Jun 28 13:18:36 172.30.1.2 [ 296.575830]  RSP <ffff88042fc03b18>
Jun 28 13:18:36 172.30.1.2 [ 296.582838] CR2: 0000000000000018
Jun 28 13:18:36 172.30.1.2 [ 296.589912] ---[ end trace c5e13ce92514d70f ]---
Jun 28 13:18:36 172.30.1.2 [ 296.599302] Kernel panic - not syncing: Fatal exception in interrupt
Jun 28 13:18:36 172.30.1.2 [ 296.612171] Pid: 7754, comm: kvm Tainted: G      D      3.2.0-2-amd64 #1
Jun 28 13:18:36 172.30.1.2 [ 296.625731] Call Trace:
Jun 28 13:18:36 172.30.1.2 [ 296.630786]  <IRQ> 
Jun 28 13:18:36 172.30.1.2 [ 296.642245]  [<ffffffff8134b246>] ? oops_end+0xa9/0xb6
Jun 28 13:18:36 172.30.1.2 [ 296.652680]  [<ffffffff8134362d>] ? no_context+0x1ff/0x20e
Jun 28 13:18:36 172.30.1.2 [ 296.663873]  [<ffffffff81052316>] ? __mod_timer+0x139/0x14b
Jun 28 13:18:36 172.30.1.2 [ 296.675175]  [<ffffffff8134d259>] ? do_page_fault+0x1a8/0x337
Jun 28 13:18:36 172.30.1.2 [ 296.686892]  [<ffffffffa03caf06>] ? ip_vs_conn_put+0x28/0x32 [ip_vs]
Jun 28 13:18:36 172.30.1.2 [ 296.699794]  [<ffffffffa03cd0e0>] ? ip_vs_out+0x2bd/0x432 [ip_vs]
Jun 28 13:18:36 172.30.1.2 [ 296.712179]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 13:18:36 172.30.1.2 [ 296.723614]  [<ffffffff8134a9b5>] ? page_fault+0x25/0x30
Jun 28 13:18:36 172.30.1.2 [ 296.734519]  [<ffffffffa01c330f>] ? nf_bridge_update_protocol+0x20/0x20 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.749474]  [<ffffffffa01c333d>] ? br_nf_forward_finish+0x2e/0x95 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.763549]  [<ffffffffa01c332e>] ? br_nf_forward_finish+0x1f/0x95 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.777632]  [<ffffffffa01c3714>] ? br_parse_ip_options+0x3d/0x19a [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.791715]  [<ffffffffa01c3aa0>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.805450]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 13:18:36 172.30.1.2 [ 296.816247]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.829003]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.841695]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 13:18:36 172.30.1.2 [ 296.852992]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.865687]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.879725]  [<ffffffffa01be941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.892411]  [<ffffffffa01be887>] ? NF_HOOK.constprop.10+0x3c/0x56 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.906483]  [<ffffffffa01bea1b>] ? br_forward+0x16/0x5a [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.918889]  [<ffffffffa01bf543>] ? br_handle_frame_finish+0x1a1/0x20f [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.933723]  [<ffffffffa01c3638>] ? br_nf_pre_routing_finish+0x1ee/0x1fb [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.948918]  [<ffffffffa01c2ff7>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.962024]  [<ffffffffa01c3f91>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge]
Jun 28 13:18:36 172.30.1.2 [ 296.975994]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 13:18:37 172.30.1.2 [ 296.986837]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:37 172.30.1.2 [ 297.000802]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 13:18:37 172.30.1.2 [ 297.012107]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:37 172.30.1.2 [ 297.026146]  [<ffffffffa01bf3a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 13:18:37 172.30.1.2 [ 297.040021]  [<ffffffffa01bf388>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge]
Jun 28 13:18:37 172.30.1.2 [ 297.053999]  [<ffffffff810135ad>] ? paravirt_read_tsc+0x5/0x8
Jun 28 13:18:37 172.30.1.2 [ 297.065638]  [<ffffffff81013622>] ? read_tsc+0x5/0x14
Jun 28 13:18:37 172.30.1.2 [ 297.075900]  [<ffffffffa01bf764>] ? br_handle_frame+0x1b3/0x1cb [bridge]
Jun 28 13:18:37 172.30.1.2 [ 297.089449]  [<ffffffffa01bf5b1>] ? br_handle_frame_finish+0x20f/0x20f [bridge]
Jun 28 13:18:37 172.30.1.2 [ 297.104301]  [<ffffffff8128a306>] ? __netif_receive_skb+0x324/0x41f
Jun 28 13:18:37 172.30.1.2 [ 297.116984]  [<ffffffff8128a46d>] ? process_backlog+0x6c/0x123
Jun 28 13:18:37 172.30.1.2 [ 297.128799]  [<ffffffff8128c351>] ? net_rx_action+0xa1/0x1af
Jun 28 13:18:37 172.30.1.2 [ 297.140267]  [<ffffffff8103703b>] ? test_tsk_need_resched+0xa/0x13
Jun 28 13:18:37 172.30.1.2 [ 297.152847]  [<ffffffff8104bee4>] ? __do_softirq+0xb9/0x177
Jun 28 13:18:37 172.30.1.2 [ 297.164227]  [<ffffffff8135162c>] ? call_softirq+0x1c/0x30
Jun 28 13:18:37 172.30.1.2 [ 297.175344]  <EOI> 
Jun 28 13:18:37 172.30.1.2 [ 297.187521]  [<ffffffff8128c641>] ? netif_rx_ni+0x1e/0x27
Jun 28 13:18:37 172.30.1.2 [ 297.198537]  [<ffffffffa028b721>] ? tun_get_user+0x39a/0x3c2 [tun]
Jun 28 13:18:37 172.30.1.2 [ 297.211100]  [<ffffffffa028ba66>] ? tun_chr_poll+0xcd/0xcd [tun]
Jun 28 13:18:37 172.30.1.2 [ 297.223376]  [<ffffffffa028bac4>] ? tun_chr_aio_write+0x5e/0x79 [tun]
Jun 28 13:18:37 172.30.1.2 [ 297.236417]  [<ffffffff810f97f8>] ? do_sync_readv_writev+0x9a/0xd7
Jun 28 13:18:37 172.30.1.2 [ 297.248994]  [<ffffffff81036457>] ? should_resched+0x5/0x23
Jun 28 13:18:37 172.30.1.2 [ 297.260291]  [<ffffffff81036457>] ? should_resched+0x5/0x23
Jun 28 13:18:37 172.30.1.2 [ 297.271594]  [<ffffffff81162ea1>] ? security_file_permission+0x16/0x2d
Jun 28 13:18:37 172.30.1.2 [ 297.284804]  [<ffffffff810f9a5c>] ? do_readv_writev+0xaf/0x11c
Jun 28 13:18:37 172.30.1.2 [ 297.296692]  [<ffffffff8112aebe>] ? eventfd_ctx_read+0x162/0x174
Jun 28 13:18:37 172.30.1.2 [ 297.308933]  [<ffffffff8103f48f>] ? try_to_wake_up+0x197/0x197
Jun 28 13:18:37 172.30.1.2 [ 297.320754]  [<ffffffff810f9c31>] ? sys_writev+0x45/0x90
Jun 28 13:18:37 172.30.1.2 [ 297.331532]  [<ffffffff8134f3d2>] ? system_call_fastpath+0x16/0x1b
Jun 28 13:18:37 mork mach_down[22201]: [22388]: info: Taking over resource group 172.30.1.99/24/br1/172.30.1.255
Jun 28 13:18:37 mork ResourceManager[22392]: [22414]: info: Acquiring resource group: mindy 172.30.1.99/24/br1/172.30.1.255
Jun 28 13:19:24 mork mon[12667]: failure for mail2 mxmon 1340882364 172.30.1.22
Jun 28 13:19:24 mork mon[12667]: calling alert marksolmail.down.alert for mail2/mxmon (/usr/lib/mon/alert.d/marksolmail.down.alert,-h) 172.30.1.22
Jun 28 13:20:25 mork mon[12667]: failure for mail2 mxmon 1340882425 172.30.1.22
Jun 28 13:21:25 mork mon[12667]: failure for mail2 mxmon 1340882485 172.30.1.22
Jun 28 13:22:25 mork mon[12667]: failure for mail2 mxmon 1340882545 172.30.1.22
Jun 28 13:23:12 mork ResourceManager[22956]: [22966]: info: Releasing resource group: mindy 172.30.1.99/24/br1/172.30.1.255
Jun 28 13:23:12 mork ResourceManager[22956]: [22983]: info: Running /etc/ha.d/resource.d/IPaddr 172.30.1.99/24/br1/172.30.1.255 stop
Jun 28 13:23:12 mork ResourceManager[22956]: [22984]: debug: Starting /etc/ha.d/resource.d/IPaddr 172.30.1.99/24/br1/172.30.1.255 stop
Jun 28 13:23:12 mork ResourceManager[22956]: [23030]: debug: /etc/ha.d/resource.d/IPaddr 172.30.1.99/24/br1/172.30.1.255 stop done. RC=0
Jun 28 13:23:25 mork mon[12667]: failure for mail2 mxmon 1340882605 172.30.1.22
Jun 28 14:16:03 172.30.1.2 [ 3193.846371] BUG: unable to handle kernel 
Jun 28 14:16:03 172.30.1.2 [ 3193.862035] IP:
Jun 28 14:16:03 172.30.1.2 [ 3193.876118] PGD 0 
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3193.880134] Oops: 0000 [#1] 
Jun 28 14:16:03 172.30.1.2 SMP
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3193.886590] CPU 0 
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3193.890246] Modules linked in:
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3194.029137] 
Jun 28 14:16:03 172.30.1.2 [ 3194.032099] Pid: 0, comm: swapper/0 Not tainted 3.2.0-2-amd64 #1
Jun 28 14:16:03 172.30.1.2 /0N051F
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3194.049664] RIP: 0010:[<ffffffffa01dc33d>] 
Jun 28 14:16:03 172.30.1.2 [ 3194.068599] RSP: 0018:ffff88042fc039b0  EFLAGS: 00010293
Jun 28 14:16:03 172.30.1.2 [ 3194.079206] RAX: 0000000000000000 RBX: ffff8802f335f980 RCX: 0000000000000006
Jun 28 14:16:03 172.30.1.2 [ 3194.093456] RDX: ffffffffa01dc30f RSI: 00000001000b4640 RDI: ffff8802f335f980
Jun 28 14:16:03 172.30.1.2 [ 3194.107706] RBP: ffff880424ce6000 R08: 0000000000000000 R09: ffff88042fc03968
Jun 28 14:16:03 172.30.1.2 [ 3194.121957] R10: ffffffff8165ab00 R11: ffffffff8165ab00 R12: 0000000000000000
Jun 28 14:16:03 172.30.1.2 [ 3194.136206] R13: ffff880426cc1002 R14: ffff8802e094f140 R15: ffff880426cc1000
Jun 28 14:16:03 172.30.1.2 [ 3194.150456] FS:  0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
Jun 28 14:16:03 172.30.1.2 [ 3194.166629] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jun 28 14:16:03 172.30.1.2 [ 3194.178104] CR2: 0000000000000018 CR3: 00000001c304d000 CR4: 00000000000026e0
Jun 28 14:16:03 172.30.1.2 [ 3194.192353] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 28 14:16:03 172.30.1.2 [ 3194.206602] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 28 14:16:03 172.30.1.2 [ 3194.220852] Process swapper/0 (pid: 0, threadinfo ffffffff81600000, task ffffffff8160d020)
Jun 28 14:16:03 172.30.1.2 [ 3194.237370] Stack:
Jun 28 14:16:03 172.30.1.2 [ 3194.241387]  ffffffff80000000
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3194.256222]  ffff880424d4c000
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3194.271056]  ffff8802f335f980
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3194.285891] Call Trace:
Jun 28 14:16:03 172.30.1.2 [ 3194.290776]  <IRQ> 
Jun 28 14:16:03 172.30.1.2 
Jun 28 14:16:03 172.30.1.2 [ 3194.294987]  [<ffffffffa01dc714>] ? br_parse_ip_options+0x3d/0x19a [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.308893]  [<ffffffffa01dcaa0>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.322456]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 14:16:03 172.30.1.2 [ 3194.333067]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.345586]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.358104]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 14:16:03 172.30.1.2 [ 3194.369235]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.381756]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.395489]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.408007]  [<ffffffffa01d7887>] ? NF_HOOK.constprop.10+0x3c/0x56 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.421913]  [<ffffffffa01d7a1b>] ? br_forward+0x16/0x5a [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.434086]  [<ffffffffa01d8543>] ? br_handle_frame_finish+0x1a1/0x20f [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.448703]  [<ffffffffa01dc638>] ? br_nf_pre_routing_finish+0x1ee/0x1fb [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.463664]  [<ffffffffa01dbff7>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.476529]  [<ffffffffa01dcf91>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.490260]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 14:16:03 172.30.1.2 [ 3194.500871]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.514600]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 14:16:03 172.30.1.2 [ 3194.525732]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.539466]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.553200]  [<ffffffffa01d8388>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.566933]  [<ffffffff812bd793>] ? tcp_gro_receive+0x83/0x1d8
Jun 28 14:16:03 172.30.1.2 [ 3194.578586]  [<ffffffffa01d8764>] ? br_handle_frame+0x1b3/0x1cb [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.591971]  [<ffffffffa01d85b1>] ? br_handle_frame_finish+0x20f/0x20f [bridge]
Jun 28 14:16:03 172.30.1.2 [ 3194.606588]  [<ffffffff8128a306>] ? __netif_receive_skb+0x324/0x41f
Jun 28 14:16:03 172.30.1.2 [ 3194.619106]  [<ffffffff8128bd61>] ? netif_receive_skb+0x63/0x69
Jun 28 14:16:03 172.30.1.2 [ 3194.630927]  [<ffffffff8128c233>] ? napi_gro_receive+0x1d/0x2b
Jun 28 14:16:03 172.30.1.2 [ 3194.642577]  [<ffffffff8128bddd>] ? napi_skb_finish+0x1c/0x31
Jun 28 14:16:03 172.30.1.2 [ 3194.654057]  [<ffffffffa00b6460>] ? bnx2_poll_work+0x84a/0x95d [bnx2]
Jun 28 14:16:03 172.30.1.2 [ 3194.666926]  [<ffffffff8103f47f>] ? try_to_wake_up+0x187/0x197
Jun 28 14:16:03 172.30.1.2 [ 3194.678579]  [<ffffffffa00b6716>] ? bnx2_poll+0xf6/0x1c4 [bnx2]
Jun 28 14:16:03 172.30.1.2 [ 3194.690402]  [<ffffffff8128c351>] ? net_rx_action+0xa1/0x1af
Jun 28 14:16:03 172.30.1.2 [ 3194.701706]  [<ffffffff81037d6e>] ? __wake_up+0x35/0x46
Jun 28 14:16:03 172.30.1.2 [ 3194.712143]  [<ffffffff8104bee4>] ? __do_softirq+0xb9/0x177
Jun 28 14:16:03 172.30.1.2 [ 3194.723276]  [<ffffffff8106a8e7>] ? clockevents_program_event+0xaa/0xce
Jun 28 14:16:04 172.30.1.2 [ 3194.736491]  [<ffffffff8135162c>] ? call_softirq+0x1c/0x30
Jun 28 14:16:04 172.30.1.2 [ 3194.747450]  [<ffffffff8100f8e5>] ? do_softirq+0x3c/0x7b
Jun 28 14:16:04 172.30.1.2 [ 3194.758059]  [<ffffffff8104c14c>] ? irq_exit+0x3c/0x9a
Jun 28 14:16:04 172.30.1.2 [ 3194.768320]  [<ffffffff8100f615>] ? do_IRQ+0x82/0x98
Jun 28 14:16:04 172.30.1.2 [ 3194.778239]  [<ffffffff8134a6ee>] ? common_interrupt+0x6e/0x6e
Jun 28 14:16:04 172.30.1.2 [ 3194.789886]  <EOI> 
Jun 28 14:16:04 172.30.1.2 
Jun 28 14:16:04 172.30.1.2 [ 3194.794091]  [<ffffffff811aed74>] ? timerqueue_add+0x80/0xa0
Jun 28 14:16:04 172.30.1.2 [ 3194.805397]  [<ffffffff811ec569>] ? intel_idle+0xea/0x119
Jun 28 14:16:04 172.30.1.2 [ 3194.816179]  [<ffffffff811ec548>] ? intel_idle+0xc9/0x119
Jun 28 14:16:04 172.30.1.2 [ 3194.826966]  [<ffffffff8126c573>] ? cpuidle_idle_call+0xec/0x179
Jun 28 14:16:04 172.30.1.2 [ 3194.838963]  [<ffffffff8100d248>] ? cpu_idle+0xa5/0xf2
Jun 28 14:16:04 172.30.1.2 [ 3194.849225]  [<ffffffff816aab33>] ? start_kernel+0x3b3/0x3be
Jun 28 14:16:04 172.30.1.2 [ 3194.860528]  [<ffffffff816aa140>] ? early_idt_handlers+0x140/0x140
Jun 28 14:16:04 172.30.1.2 [ 3194.872870]  [<ffffffff816aa3c4>] ? x86_64_start_kernel+0x104/0x111
Jun 28 14:16:04 172.30.1.2 [ 3194.885386] Code: 
Jun 28 14:16:04 172.30.1.2 53
Jun 28 14:16:04 172.30.1.2 48
Jun 28 14:16:04 172.30.1.2 89
Jun 28 14:16:04 172.30.1.2 fb
Jun 28 14:16:04 172.30.1.2 48
Jun 28 14:16:04 172.30.1.2 83
Jun 28 14:16:04 172.30.1.2 ec
Jun 28 14:16:04 172.30.1.2 10
Jun 28 14:16:04 172.30.1.2 66
Jun 28 14:16:04 172.30.1.2 81
Jun 28 14:16:04 172.30.1.2 7f
Jun 28 14:16:04 172.30.1.2 7e
Jun 28 14:16:04 172.30.1.2 08
Jun 28 14:16:04 172.30.1.2 06
Jun 28 14:16:04 172.30.1.2 4c
Jun 28 14:16:04 172.30.1.2 8b
Jun 28 14:16:04 172.30.1.2 a7
Jun 28 14:16:04 172.30.1.2 98
Jun 28 14:16:04 172.30.1.2 00
Jun 28 14:16:04 172.30.1.2 00
Jun 28 14:16:04 172.30.1.2 00
Jun 28 14:16:04 172.30.1.2 74
Jun 28 14:16:04 172.30.1.2 3d
Jun 28 14:16:04 172.30.1.2 e8
Jun 28 14:16:04 172.30.1.2 07
Jun 28 14:16:04 172.30.1.2 fe
Jun 28 14:16:04 172.30.1.2 ff
Jun 28 14:16:04 172.30.1.2 ff
Jun 28 14:16:04 172.30.1.2 66
Jun 28 14:16:04 172.30.1.2 3d
Jun 28 14:16:04 172.30.1.2 08
Jun 28 14:16:04 172.30.1.2 06
Jun 28 14:16:04 172.30.1.2 75
Jun 28 14:16:04 172.30.1.2 09
Jun 28 14:16:04 172.30.1.2 83
Jun 28 14:16:04 172.30.1.2 3d
Jun 28 14:16:04 172.30.1.2 91
Jun 28 14:16:04 172.30.1.2 6a
Jun 28 14:16:04 172.30.1.2 00
Jun 28 14:16:04 172.30.1.2 00
Jun 28 14:16:04 172.30.1.2 00
Jun 28 14:16:04 172.30.1.2 75
Jun 28 14:16:04 172.30.1.2 29
Jun 28 14:16:04 172.30.1.2 
Jun 28 14:16:04 172.30.1.2 f6
Jun 28 14:16:04 172.30.1.2 44
Jun 28 14:16:04 172.30.1.2 24
Jun 28 14:16:04 172.30.1.2 18
Jun 28 14:16:04 172.30.1.2 01
Jun 28 14:16:04 172.30.1.2 49
Jun 28 14:16:04 172.30.1.2 8b
Jun 28 14:16:04 172.30.1.2 6c
Jun 28 14:16:04 172.30.1.2 24
Jun 28 14:16:04 172.30.1.2 08
Jun 28 14:16:04 172.30.1.2 74
Jun 28 14:16:04 172.30.1.2 12
Jun 28 14:16:04 172.30.1.2 8a
Jun 28 14:16:04 172.30.1.2 43
Jun 28 14:16:04 172.30.1.2 7d
Jun 28 14:16:04 172.30.1.2 83
Jun 28 14:16:04 172.30.1.2 e0
Jun 28 14:16:04 172.30.1.2 f8
Jun 28 14:16:04 172.30.1.2 83
Jun 28 14:16:04 172.30.1.2 c8
Jun 28 14:16:04 172.30.1.2 
Jun 28 14:16:04 172.30.1.2 [ 3194.924067] RIP 
Jun 28 14:16:04 172.30.1.2 [ 3194.938333]  RSP <ffff88042fc039b0>
Jun 28 14:16:04 172.30.1.2 [ 3194.945299] CR2: 0000000000000018
Jun 28 14:16:04 172.30.1.2 [ 3194.952668] ---[ end trace c28d39074732fa49 ]---
Jun 28 14:16:04 172.30.1.2 [ 3194.962197] Kernel panic - not syncing: Fatal exception in interrupt
Jun 28 14:16:04 172.30.1.2 [ 3194.975155] Pid: 0, comm: swapper/0 Tainted: G      D      3.2.0-2-amd64 #1
Jun 28 14:16:04 172.30.1.2 [ 3194.989324] Call Trace:
Jun 28 14:16:04 172.30.1.2 [ 3194.994443]  <IRQ> 
Jun 28 14:16:04 172.30.1.2 [ 3195.006177]  [<ffffffff8134b246>] ? oops_end+0xa9/0xb6
Jun 28 14:16:04 172.30.1.2 [ 3195.016744]  [<ffffffff8134362d>] ? no_context+0x1ff/0x20e
Jun 28 14:16:04 172.30.1.2 [ 3195.028010]  [<ffffffff810e9e7c>] ? virt_to_slab+0x6/0x16
Jun 28 14:16:04 172.30.1.2 [ 3195.039098]  [<ffffffff810e9f1f>] ? __cache_free.isra.41+0x3d/0x198
Jun 28 14:16:04 172.30.1.2 [ 3195.051922]  [<ffffffff8134d259>] ? do_page_fault+0x1a8/0x337
Jun 28 14:16:04 172.30.1.2 [ 3195.063621]  [<ffffffffa03cbf06>] ? ip_vs_conn_put+0x28/0x32 [ip_vs]
Jun 28 14:16:04 172.30.1.2 [ 3195.076593]  [<ffffffffa03ce0e0>] ? ip_vs_out+0x2bd/0x432 [ip_vs]
Jun 28 14:16:04 172.30.1.2 [ 3195.089052]  [<ffffffffa01d7847>] ? br_dev_queue_push_xmit+0x9b/0x9f [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.103656]  [<ffffffff8134a9b5>] ? page_fault+0x25/0x30
Jun 28 14:16:04 172.30.1.2 [ 3195.114563]  [<ffffffffa01dc30f>] ? nf_bridge_update_protocol+0x20/0x20 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.129719]  [<ffffffffa01dc33d>] ? br_nf_forward_finish+0x2e/0x95 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.143939]  [<ffffffffa01dc714>] ? br_parse_ip_options+0x3d/0x19a [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.158197]  [<ffffffffa01dcaa0>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.172126]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 14:16:04 172.30.1.2 [ 3195.183162]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.195973]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.208757]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 14:16:04 172.30.1.2 [ 3195.220300]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.233150]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.247306]  [<ffffffffa01d7941>] ? __br_deliver+0xa0/0xa0 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.260078]  [<ffffffffa01d7887>] ? NF_HOOK.constprop.10+0x3c/0x56 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.274354]  [<ffffffffa01d7a1b>] ? br_forward+0x16/0x5a [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.286828]  [<ffffffffa01d8543>] ? br_handle_frame_finish+0x1a1/0x20f [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.301718]  [<ffffffffa01dc638>] ? br_nf_pre_routing_finish+0x1ee/0x1fb [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.317041]  [<ffffffffa01dbff7>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.330274]  [<ffffffffa01dcf91>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.344275]  [<ffffffff812ad0ed>] ? nf_iterate+0x41/0x77
Jun 28 14:16:04 172.30.1.2 [ 3195.355232]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.369214]  [<ffffffff812ad18b>] ? nf_hook_slow+0x68/0x101
Jun 28 14:16:04 172.30.1.2 [ 3195.380687]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.394806]  [<ffffffffa01d83a2>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.408906]  [<ffffffffa01d8388>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.423003]  [<ffffffff812bd793>] ? tcp_gro_receive+0x83/0x1d8
Jun 28 14:16:04 172.30.1.2 [ 3195.434976]  [<ffffffffa01d8764>] ? br_handle_frame+0x1b3/0x1cb [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.448742]  [<ffffffffa01d85b1>] ? br_handle_frame_finish+0x20f/0x20f [bridge]
Jun 28 14:16:04 172.30.1.2 [ 3195.463742]  [<ffffffff8128a306>] ? __netif_receive_skb+0x324/0x41f
Jun 28 14:16:04 172.30.1.2 [ 3195.476522]  [<ffffffff8128bd61>] ? netif_receive_skb+0x63/0x69
Jun 28 14:16:04 172.30.1.2 [ 3195.488591]  [<ffffffff8128c233>] ? napi_gro_receive+0x1d/0x2b
Jun 28 14:16:04 172.30.1.2 [ 3195.500581]  [<ffffffff8128bddd>] ? napi_skb_finish+0x1c/0x31
Jun 28 14:16:04 172.30.1.2 [ 3195.512350]  [<ffffffffa00b6460>] ? bnx2_poll_work+0x84a/0x95d [bnx2]
Jun 28 14:16:04 172.30.1.2 [ 3195.525547]  [<ffffffff8103f47f>] ? try_to_wake_up+0x187/0x197
Jun 28 14:16:04 172.30.1.2 [ 3195.537480]  [<ffffffffa00b6716>] ? bnx2_poll+0xf6/0x1c4 [bnx2]
Jun 28 14:16:04 172.30.1.2 [ 3195.549671]  [<ffffffff8128c351>] ? net_rx_action+0xa1/0x1af
Jun 28 14:16:04 172.30.1.2 [ 3195.561264]  [<ffffffff81037d6e>] ? __wake_up+0x35/0x46
Jun 28 14:16:04 172.30.1.2 [ 3195.572052]  [<ffffffff8104bee4>] ? __do_softirq+0xb9/0x177
Jun 28 14:16:04 172.30.1.2 [ 3195.583508]  [<ffffffff8106a8e7>] ? clockevents_program_event+0xaa/0xce
Jun 28 14:16:04 172.30.1.2 [ 3195.597050]  [<ffffffff8135162c>] ? call_softirq+0x1c/0x30
Jun 28 14:16:04 172.30.1.2 [ 3195.608366]  [<ffffffff8100f8e5>] ? do_softirq+0x3c/0x7b
Jun 28 14:16:04 172.30.1.2 [ 3195.619247]  [<ffffffff8104c14c>] ? irq_exit+0x3c/0x9a
Jun 28 14:16:04 172.30.1.2 [ 3195.629829]  [<ffffffff8100f615>] ? do_IRQ+0x82/0x98
Jun 28 14:16:04 172.30.1.2 [ 3195.640035]  [<ffffffff8134a6ee>] ? common_interrupt+0x6e/0x6e
Jun 28 14:16:04 172.30.1.2 [ 3195.652030]  <EOI> 
Jun 28 14:16:04 172.30.1.2 [ 3195.665050]  [<ffffffff811ec569>] ? intel_idle+0xea/0x119
Jun 28 14:16:04 172.30.1.2 [ 3195.676195]  [<ffffffff811ec548>] ? intel_idle+0xc9/0x119
Jun 28 14:16:04 172.30.1.2 [ 3195.687278]  [<ffffffff8126c573>] ? cpuidle_idle_call+0xec/0x179
Jun 28 14:16:04 172.30.1.2 [ 3195.699551]  [<ffffffff8100d248>] ? cpu_idle+0xa5/0xf2
Jun 28 14:16:04 172.30.1.2 [ 3195.710072]  [<ffffffff816aab33>] ? start_kernel+0x3b3/0x3be
Jun 28 14:16:04 172.30.1.2 [ 3195.721740]  [<ffffffff816aa140>] ? early_idt_handlers+0x140/0x140
Jun 28 14:16:05 172.30.1.2 [ 3195.734358]  [<ffffffff816aa3c4>] ? x86_64_start_kernel+0x104/0x111
Jun 28 14:16:05 172.30.1.2 [ 3195.747405] ------------[ cut here ]------------
Jun 28 14:16:05 172.30.1.2 [ 3195.756678] WARNING: at /build/buildd-linux_3.2.20-1-amd64-lTzScn/linux-3.2.20/arch/x86/kernel/smp.c:119 try_to_wake_up+0x158/0x197()
Jun 28 14:16:05 172.30.1.2 [ 3195.780697] Hardware name: PowerEdge R410
Jun 28 14:16:05 172.30.1.2 [ 3195.788744] Modules linked in:
Jun 28 14:16:05 172.30.1.2 
Jun 28 14:16:05 172.30.1.2 [ 3195.930822] Pid: 4659, comm: kvm Tainted: G      D      3.2.0-2-amd64 #1
Jun 28 14:16:05 172.30.1.2 [ 3195.944245] Call Trace:
Jun 28 14:16:05 172.30.1.2 [ 3195.949175]  [<ffffffff810468fd>] ? warn_slowpath_common+0x78/0x8c
Jun 28 14:16:05 172.30.1.2 [ 3195.961562]  [<ffffffff8103f450>] ? try_to_wake_up+0x158/0x197
Jun 28 14:16:05 172.30.1.2 [ 3195.973256]  [<ffffffff811071e2>] ? pollwake+0x4d/0x52
Jun 28 14:16:05 172.30.1.2 [ 3195.983560]  [<ffffffff8103f48f>] ? try_to_wake_up+0x197/0x197
Jun 28 14:16:05 172.30.1.2 [ 3195.995254]  [<ffffffff810360b2>] ? __wake_up_common+0x40/0x77
Jun 28 14:16:05 172.30.1.2 [ 3196.006949]  [<ffffffff8112ad43>] ? eventfd_write+0x1a3/0x1bc
Jun 28 14:16:05 172.30.1.2 [ 3196.018467]  [<ffffffff8103f48f>] ? try_to_wake_up+0x197/0x197
Jun 28 14:16:05 172.30.1.2 [ 3196.030162]  [<ffffffff810f947f>] ? vfs_write+0xa2/0xe9
Jun 28 14:16:05 172.30.1.2 [ 3196.040649]  [<ffffffffa022a5fb>] ? kvm_on_user_return+0x36/0x5c [kvm]
Jun 28 14:16:05 172.30.1.2 [ 3196.053728]  [<ffffffff810f965c>] ? sys_write+0x45/0x6b
Jun 28 14:16:05 172.30.1.2 [ 3196.064206]  [<ffffffff8134f3d2>] ? system_call_fastpath+0x16/0x1b
Jun 28 14:16:05 172.30.1.2 [ 3196.076589] ---[ end trace c28d39074732fa4a ]---
Jun 28 15:29:19 172.30.1.2 [ 304.578777] BUG: unable to handle kernel 
Jun 28 15:29:19 172.30.1.2 [ 304.594562] IP:
Jun 28 15:29:19 172.30.1.2 [ 304.608732] PGD 0 
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 304.612831] Oops: 0000 [#1] 
Jun 28 15:29:19 172.30.1.2 SMP
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 304.619414] CPU 0 
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 304.623112] Modules linked in:
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 304.765458] 
Jun 28 15:29:19 172.30.1.2 [ 304.768463] Pid: 4486, comm: kvm Not tainted 3.2.21 #1
Jun 28 15:29:19 172.30.1.2 /0N051F
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 304.784464] RIP: 0010:[<ffffffffa01d09d0>] 
Jun 28 15:29:19 172.30.1.2 [ 304.803483] RSP: 0018:ffff88042fc03960  EFLAGS: 00010287
Jun 28 15:29:19 172.30.1.2 [ 304.814133] RAX: 0000000000000008 RBX: ffff880422148d80 RCX: ffff88042fc03706
Jun 28 15:29:19 172.30.1.2 [ 304.828426] RDX: 0000000000000000 RSI: 0000000100003f40 RDI: ffff880422148d80
Jun 28 15:29:19 172.30.1.2 [ 304.842719] RBP: ffff880221d2c000 R08: 0000000000000000 R09: ffff88042fc03918
Jun 28 15:29:19 172.30.1.2 [ 304.857012] R10: ffffffff8180b240 R11: ffff880422148d80 R12: ffff880221bc8000
Jun 28 15:29:19 172.30.1.2 [ 304.871303] R13: ffff880418d91500 R14: ffff880221bc8002 R15: ffff880420cdf000
Jun 28 15:29:19 172.30.1.2 [ 304.885595] FS:  00007f8d47807900(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
Jun 28 15:29:19 172.30.1.2 [ 304.901811] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 28 15:29:19 172.30.1.2 [ 304.913328] CR2: 0000000000000018 CR3: 000000041c3d1000 CR4: 00000000000026e0
Jun 28 15:29:19 172.30.1.2 [ 304.927620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 28 15:29:19 172.30.1.2 [ 304.941912] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 28 15:29:19 172.30.1.2 [ 304.956205] Process kvm (pid: 4486, threadinfo ffff88040f24e000, task ffff8804215f9510)
Jun 28 15:29:19 172.30.1.2 [ 304.972247] Stack:
Jun 28 15:29:19 172.30.1.2 [ 304.976307]  0000000080000000
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 304.991349]  ffff880221d2c000
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 305.006392]  ffff880422148d80
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 305.021434] Call Trace:
Jun 28 15:29:19 172.30.1.2 [ 305.026360]  <IRQ> 
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:19 172.30.1.2 [ 305.030651]  [<ffffffffa01d0c91>] ? br_nf_forward_ip+0x214/0x231 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.044255]  [<ffffffff812c7de1>] ? nf_iterate+0x41/0x77
Jun 28 15:29:19 172.30.1.2 [ 305.054909]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.067122]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.079336]  [<ffffffff812c7f3a>] ? nf_hook_slow+0x68/0xfd
Jun 28 15:29:19 172.30.1.2 [ 305.090336]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.102552]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.113898]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.126113]  [<ffffffffa01cb96e>] ? T.963+0x3c/0x56 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.137460]  [<ffffffffa01ca753>] ? __br_fdb_get+0x10/0x83 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.150021]  [<ffffffffa01cc61d>] ? br_handle_frame_finish+0x1a7/0x211 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.164680]  [<ffffffffa01d1598>] ? br_nf_pre_routing_finish+0x222/0x22f [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.179685]  [<ffffffffa01d03b5>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.192593]  [<ffffffffa01d135b>] ? br_nf_pre_routing+0x4fc/0x517 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.206366]  [<ffffffff812ce172>] ? T.1028+0x4f/0x4f
Jun 28 15:29:19 172.30.1.2 [ 305.216322]  [<ffffffff812c7de1>] ? nf_iterate+0x41/0x77
Jun 28 15:29:19 172.30.1.2 [ 305.226975]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.238320]  [<ffffffff812c7f3a>] ? nf_hook_slow+0x68/0xfd
Jun 28 15:29:19 172.30.1.2 [ 305.249320]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.260669]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.272015]  [<ffffffffa01cc45c>] ? T.947+0x3c/0x56 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.283363]  [<ffffffffa01cc84a>] ? br_handle_frame+0x1c3/0x1d9 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.296790]  [<ffffffffa01cc687>] ? br_handle_frame_finish+0x211/0x211 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.311448]  [<ffffffff812a74f2>] ? __netif_receive_skb+0x37a/0x490
Jun 28 15:29:19 172.30.1.2 [ 305.324009]  [<ffffffff81013a01>] ? read_tsc+0x5/0x16
Jun 28 15:29:19 172.30.1.2 [ 305.334139]  [<ffffffff812a799f>] ? netif_receive_skb+0x67/0x6d
Jun 28 15:29:19 172.30.1.2 [ 305.346005]  [<ffffffff812a7f0b>] ? napi_gro_receive+0x1f/0x2d
Jun 28 15:29:19 172.30.1.2 [ 305.357697]  [<ffffffff812a7a79>] ? napi_skb_finish+0x1c/0x31
Jun 28 15:29:19 172.30.1.2 [ 305.369220]  [<ffffffffa0009a21>] ? bnx2_poll_work+0x8db/0x9f4 [bnx2]
Jun 28 15:29:19 172.30.1.2 [ 305.382128]  [<ffffffffa00092f4>] ? bnx2_poll_work+0x1ae/0x9f4 [bnx2]
Jun 28 15:29:19 172.30.1.2 [ 305.395036]  [<ffffffffa01cc45c>] ? T.947+0x3c/0x56 [bridge]
Jun 28 15:29:19 172.30.1.2 [ 305.406383]  [<ffffffffa0009cfc>] ? bnx2_poll+0x116/0x1f2 [bnx2]
Jun 28 15:29:19 172.30.1.2 [ 305.418421]  [<ffffffff812a74f2>] ? __netif_receive_skb+0x37a/0x490
Jun 28 15:29:19 172.30.1.2 [ 305.430980]  [<ffffffff812a8038>] ? net_rx_action+0xa8/0x207
Jun 28 15:29:19 172.30.1.2 [ 305.442327]  [<ffffffff8106637e>] ? hrtimer_get_next_event+0x7f/0x9a
Jun 28 15:29:19 172.30.1.2 [ 305.455060]  [<ffffffff8104f11e>] ? __do_softirq+0xc4/0x1a0
Jun 28 15:29:19 172.30.1.2 [ 305.466234]  [<ffffffff81097905>] ? handle_irq_event_percpu+0x166/0x184
Jun 28 15:29:19 172.30.1.2 [ 305.479490]  [<ffffffff8136bd6c>] ? call_softirq+0x1c/0x30
Jun 28 15:29:19 172.30.1.2 [ 305.490492]  [<ffffffff8100fa3f>] ? do_softirq+0x3f/0x79
Jun 28 15:29:19 172.30.1.2 [ 305.501142]  [<ffffffff8104eeee>] ? irq_exit+0x44/0xb5
Jun 28 15:29:19 172.30.1.2 [ 305.511446]  [<ffffffff8100f38a>] ? do_IRQ+0x94/0xaa
Jun 28 15:29:19 172.30.1.2 [ 305.521407]  [<ffffffff813647ee>] ? common_interrupt+0x6e/0x6e
Jun 28 15:29:19 172.30.1.2 [ 305.533098]  <EOI> 
Jun 28 15:29:19 172.30.1.2 
Jun 28 15:29:20 172.30.1.2 [ 305.537386]  [<ffffffff81364498>] ? _raw_spin_unlock_irqrestore+0xb/0x11
Jun 28 15:29:20 172.30.1.2 [ 305.550816]  [<ffffffff81045f07>] ? try_to_wake_up+0x181/0x190
Jun 28 15:29:20 172.30.1.2 [ 305.562510]  [<ffffffff81071764>] ? wake_futex+0x2b/0x47
Jun 28 15:29:20 172.30.1.2 [ 305.573161]  [<ffffffff81073949>] ? do_futex+0x3de/0x84a
Jun 28 15:29:20 172.30.1.2 [ 305.583814]  [<ffffffff810dfff1>] ? do_brk+0x2ca/0x326
Jun 28 15:29:20 172.30.1.2 [ 305.594119]  [<ffffffff81073ee4>] ? sys_futex+0x12f/0x147
Jun 28 15:29:20 172.30.1.2 [ 305.604945]  [<ffffffff81369b12>] ? system_call_fastpath+0x16/0x1b
Jun 28 15:29:20 172.30.1.2 [ 305.617329] Code: 
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 10
Jun 28 15:29:20 172.30.1.2 75
Jun 28 15:29:20 172.30.1.2 25
Jun 28 15:29:20 172.30.1.2 66
Jun 28 15:29:20 172.30.1.2 3d
Jun 28 15:29:20 172.30.1.2 81
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 75
Jun 28 15:29:20 172.30.1.2 1f
Jun 28 15:29:20 172.30.1.2 8b
Jun 28 15:29:20 172.30.1.2 87
Jun 28 15:29:20 172.30.1.2 c8
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 48
Jun 28 15:29:20 172.30.1.2 8b
Jun 28 15:29:20 172.30.1.2 8f
Jun 28 15:29:20 172.30.1.2 d8
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 66
Jun 28 15:29:20 172.30.1.2 81
Jun 28 15:29:20 172.30.1.2 7c
Jun 28 15:29:20 172.30.1.2 01
Jun 28 15:29:20 172.30.1.2 10
Jun 28 15:29:20 172.30.1.2 08
Jun 28 15:29:20 172.30.1.2 06
Jun 28 15:29:20 172.30.1.2 75
Jun 28 15:29:20 172.30.1.2 09
Jun 28 15:29:20 172.30.1.2 83
Jun 28 15:29:20 172.30.1.2 3d
Jun 28 15:29:20 172.30.1.2 f2
Jun 28 15:29:20 172.30.1.2 63
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 00
Jun 28 15:29:20 172.30.1.2 75
Jun 28 15:29:20 172.30.1.2 3c
Jun 28 15:29:20 172.30.1.2 f6>
Jun 28 15:29:20 172.30.1.2 42
Jun 28 15:29:20 172.30.1.2 18
Jun 28 15:29:20 172.30.1.2 01
Jun 28 15:29:20 172.30.1.2 48
Jun 28 15:29:20 172.30.1.2 8b
Jun 28 15:29:20 172.30.1.2 6a
Jun 28 15:29:20 172.30.1.2 08
Jun 28 15:29:20 172.30.1.2 74
Jun 28 15:29:20 172.30.1.2 10
Jun 28 15:29:20 172.30.1.2 8a
Jun 28 15:29:20 172.30.1.2 
Jun 28 15:29:20 172.30.1.2 [ 305.652454] ------------[ cut here ]------------
Jun 28 15:29:20 172.30.1.2 [ 305.652457] WARNING: at kernel/softirq.c:159 _local_bh_enable_ip+0x41/0x8b()
Jun 28 15:29:20 172.30.1.2 [ 305.652459] Hardware name: PowerEdge R410
Jun 28 15:29:20 172.30.1.2 [ 305.652460] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter ip_tables x_tables ip_vs_rr ip_vs nf_conntrack crc32c libcrc32c drbd lru_cache cn tun virtio_net virtio_blk virtio_rng rng_core virtio_pci virtio_ring virtio kvm_intel kvm ipmi_devintf ipmi_poweroff ipmi_si ipmi_watchdog ipmi_msghandler netconsole configfs loop bridge stp snd_pcm i7core_edac dcdbas edac_core iTCO_wdt iTCO_vendor_support processor psmouse snd_timer snd soundcore snd_page_alloc pcspkr serio_raw joydev thermal_sys button evdev ext3 mbcache jbd dm_mod sr_mod cdrom sd_mod ses usbhid hid crc_t10dif enclosure ata_generic uhci_hcd ata_piix ehci_hcd libata megaraid_sas usbcore scsi_mod usb_common bnx2 [last unloaded: scsi_wait_scan]
Jun 28 15:29:20 172.30.1.2 [ 305.652495] Pid: 4486, comm: kvm Not tainted 3.2.21 #1
Jun 28 15:29:20 172.30.1.2 [ 305.652496] Call Trace:
Jun 28 15:29:20 172.30.1.2 [ 305.652497]  <IRQ>  [<ffffffff81049810>] ? warn_slowpath_common+0x78/0x8c
Jun 28 15:29:20 172.30.1.2 [ 305.652503]  [<ffffffff8104efa0>] ? _local_bh_enable_ip+0x41/0x8b
Jun 28 15:29:20 172.30.1.2 [ 305.652508]  [<ffffffffa0000648>] ? bnx2_reg_rd_ind+0x31/0x38 [bnx2]
Jun 28 15:29:20 172.30.1.2 [ 305.652514]  [<ffffffffa0009dc8>] ? bnx2_poll+0x1e2/0x1f2 [bnx2]
Jun 28 15:29:20 172.30.1.2 [ 305.652520]  [<ffffffff812b6b62>] ? netpoll_poll_dev+0xab/0x4ad
Jun 28 15:29:20 172.30.1.2 [ 305.652525]  [<ffffffffa0006ba0>] ? bnx2_start_xmit+0x411/0x55d [bnx2]
Jun 28 15:29:20 172.30.1.2 [ 305.652531]  [<ffffffff810f693e>] ? ____cache_alloc+0xc6/0x20d
Jun 28 15:29:20 172.30.1.2 [ 305.652535]  [<ffffffff812b7075>] ? netpoll_send_skb_on_dev+0x111/0x215
Jun 28 15:29:20 172.30.1.2 [ 305.652541]  [<ffffffffa01ca6d2>] ? br_dev_xmit+0x136/0x150 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652544]  [<ffffffff812b7033>] ? netpoll_send_skb_on_dev+0xcf/0x215
Jun 28 15:29:20 172.30.1.2 [ 305.652551]  [<ffffffff811c3157>] ? delay_tsc+0x2b/0x74
Jun 28 15:29:20 172.30.1.2 [ 305.652555]  [<ffffffffa0167272>] ? write_msg+0x9f/0x102 [netconsole]
Jun 28 15:29:20 172.30.1.2 [ 305.652558]  [<ffffffff810499e6>] ? __call_console_drivers+0x75/0x86
Jun 28 15:29:20 172.30.1.2 [ 305.652561]  [<ffffffff8104a084>] ? console_unlock+0x13d/0x1e8
Jun 28 15:29:20 172.30.1.2 [ 305.652564]  [<ffffffff8104a707>] ? vprintk+0x39e/0x3e5
Jun 28 15:29:20 172.30.1.2 [ 305.652570]  [<ffffffffa01d09db>] ? br_nf_forward_finish+0x53/0xc0 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652576]  [<ffffffffa01d09a5>] ? br_nf_forward_finish+0x1d/0xc0 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652579]  [<ffffffff81362842>] ? printk+0x40/0x46
Jun 28 15:29:20 172.30.1.2 [ 305.652583]  [<ffffffff8100fff9>] ? show_registers+0x1e2/0x211
Jun 28 15:29:20 172.30.1.2 [ 305.652586]  [<ffffffff813652dd>] ? __die+0x8e/0xc9
Jun 28 15:29:20 172.30.1.2 [ 305.652589]  [<ffffffff8102fcd0>] ? no_context+0x1d6/0x20c
Jun 28 15:29:20 172.30.1.2 [ 305.652592]  [<ffffffff810f53c3>] ? virt_to_slab+0x9/0x3c
Jun 28 15:29:20 172.30.1.2 [ 305.652595]  [<ffffffff81367534>] ? do_page_fault+0x1ad/0x34c
Jun 28 15:29:20 172.30.1.2 [ 305.652601]  [<ffffffffa03fcd93>] ? ip_vs_notrack+0x6e/0xa6 [ip_vs]
Jun 28 15:29:20 172.30.1.2 [ 305.652606]  [<ffffffffa03fbdec>] ? ip_vs_conn_put+0x29/0x2f [ip_vs]
Jun 28 15:29:20 172.30.1.2 [ 305.652610]  [<ffffffffa03fde1a>] ? ip_vs_out+0x33c/0x49d [ip_vs]
Jun 28 15:29:20 172.30.1.2 [ 305.652613]  [<ffffffff812a896b>] ? dev_hard_start_xmit+0x3fd/0x571
Jun 28 15:29:20 172.30.1.2 [ 305.652617]  [<ffffffff81364ab5>] ? page_fault+0x25/0x30
Jun 28 15:29:20 172.30.1.2 [ 305.652623]  [<ffffffffa01d09d0>] ? br_nf_forward_finish+0x48/0xc0 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652628]  [<ffffffffa01d0c91>] ? br_nf_forward_ip+0x214/0x231 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652631]  [<ffffffff812c7de1>] ? nf_iterate+0x41/0x77
Jun 28 15:29:20 172.30.1.2 [ 305.652636]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652641]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652644]  [<ffffffff812c7f3a>] ? nf_hook_slow+0x68/0xfd
Jun 28 15:29:20 172.30.1.2 [ 305.652648]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652654]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652658]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652663]  [<ffffffffa01cb96e>] ? T.963+0x3c/0x56 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652668]  [<ffffffffa01ca753>] ? __br_fdb_get+0x10/0x83 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652673]  [<ffffffffa01cc61d>] ? br_handle_frame_finish+0x1a7/0x211 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652678]  [<ffffffffa01d1598>] ? br_nf_pre_routing_finish+0x222/0x22f [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652684]  [<ffffffffa01d03b5>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652689]  [<ffffffffa01d135b>] ? br_nf_pre_routing+0x4fc/0x517 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652692]  [<ffffffff812ce172>] ? T.1028+0x4f/0x4f
Jun 28 15:29:20 172.30.1.2 [ 305.652695]  [<ffffffff812c7de1>] ? nf_iterate+0x41/0x77
Jun 28 15:29:20 172.30.1.2 [ 305.652700]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652703]  [<ffffffff812c7f3a>] ? nf_hook_slow+0x68/0xfd
Jun 28 15:29:20 172.30.1.2 [ 305.652708]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652713]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652718]  [<ffffffffa01cc45c>] ? T.947+0x3c/0x56 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652723]  [<ffffffffa01cc84a>] ? br_handle_frame+0x1c3/0x1d9 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652728]  [<ffffffffa01cc687>] ? br_handle_frame_finish+0x211/0x211 [bridge]
Jun 28 15:29:20 172.30.1.2 [ 305.652731]  [<ffffffff812a74f2>] ? __netif_receive_skb+0x37a/0x490
Jun 28 15:29:20 172.30.1.2 [ 305.652734]  [<ffffffff81013a01>] ? read_tsc+0x5/0x16
Jun 28 15:29:20 172.30.1.2 [ 305.652736]  [<ffffffff812a799f>] ? netif_receive_skb+0x67/0x6d
Jun 28 15:29:20 172.30.1.2 [ 305.652739]  [<ffffffff812a7f0b>] ? napi_gro_receive+0x1f/0x2d
Jun 28 15:29:20 172.30.1.2 [ 305.652742]  [<ffffffff812a7a79>] ? napi_skb_finish+0x1c/0x31
Jun 28 15:29:21 172.30.1.2 [ 305.652747]  [<ffffffffa0009a21>] ? bnx2_poll_work+0x8db/0x9f4 [bnx2]
Jun 28 15:29:21 172.30.1.2 [ 305.652752]  [<ffffffffa00092f4>] ? bnx2_poll_work+0x1ae/0x9f4 [bnx2]
Jun 28 15:29:21 172.30.1.2 [ 305.652757]  [<ffffffffa01cc45c>] ? T.947+0x3c/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 305.652762]  [<ffffffffa0009cfc>] ? bnx2_poll+0x116/0x1f2 [bnx2]
Jun 28 15:29:21 172.30.1.2 [ 305.652765]  [<ffffffff812a74f2>] ? __netif_receive_skb+0x37a/0x490
Jun 28 15:29:21 172.30.1.2 [ 305.652768]  [<ffffffff812a8038>] ? net_rx_action+0xa8/0x207
Jun 28 15:29:21 172.30.1.2 [ 305.652771]  [<ffffffff8106637e>] ? hrtimer_get_next_event+0x7f/0x9a
Jun 28 15:29:21 172.30.1.2 [ 305.652774]  [<ffffffff8104f11e>] ? __do_softirq+0xc4/0x1a0
Jun 28 15:29:21 172.30.1.2 [ 305.652776]  [<ffffffff81097905>] ? handle_irq_event_percpu+0x166/0x184
Jun 28 15:29:21 172.30.1.2 [ 305.652780]  [<ffffffff8136bd6c>] ? call_softirq+0x1c/0x30
Jun 28 15:29:21 172.30.1.2 [ 305.652782]  [<ffffffff8100fa3f>] ? do_softirq+0x3f/0x79
Jun 28 15:29:21 172.30.1.2 [ 305.652785]  [<ffffffff8104eeee>] ? irq_exit+0x44/0xb5
Jun 28 15:29:21 172.30.1.2 [ 305.652787]  [<ffffffff8100f38a>] ? do_IRQ+0x94/0xaa
Jun 28 15:29:21 172.30.1.2 [ 305.652790]  [<ffffffff813647ee>] ? common_interrupt+0x6e/0x6e
Jun 28 15:29:21 172.30.1.2 [ 305.652792]  <EOI>  [<ffffffff81364498>] ? _raw_spin_unlock_irqrestore+0xb/0x11
Jun 28 15:29:21 172.30.1.2 [ 305.652797]  [<ffffffff81045f07>] ? try_to_wake_up+0x181/0x190
Jun 28 15:29:21 172.30.1.2 [ 305.652799]  [<ffffffff81071764>] ? wake_futex+0x2b/0x47
Jun 28 15:29:21 172.30.1.2 [ 305.652802]  [<ffffffff81073949>] ? do_futex+0x3de/0x84a
Jun 28 15:29:21 172.30.1.2 [ 305.652804]  [<ffffffff810dfff1>] ? do_brk+0x2ca/0x326
Jun 28 15:29:21 172.30.1.2 [ 305.652807]  [<ffffffff81073ee4>] ? sys_futex+0x12f/0x147
Jun 28 15:29:21 172.30.1.2 [ 305.652810]  [<ffffffff81369b12>] ? system_call_fastpath+0x16/0x1b
Jun 28 15:29:21 172.30.1.2 [ 305.652812] ---[ end trace 33ba304ebe73322b ]---
Jun 28 15:29:21 172.30.1.2 [ 306.782187] 43 
Jun 28 15:29:21 172.30.1.2 7d
Jun 28 15:29:21 172.30.1.2 83
Jun 28 15:29:21 172.30.1.2 e0
Jun 28 15:29:21 172.30.1.2 f8
Jun 28 15:29:21 172.30.1.2 83
Jun 28 15:29:21 172.30.1.2 c8
Jun 28 15:29:21 172.30.1.2 03
Jun 28 15:29:21 172.30.1.2 88
Jun 28 15:29:21 172.30.1.2 43
Jun 28 15:29:21 172.30.1.2 
Jun 28 15:29:21 172.30.1.2 [ 306.790953] RIP 
Jun 28 15:29:21 172.30.1.2 [ 306.805304]  RSP <ffff88042fc03960>
Jun 28 15:29:21 172.30.1.2 [ 306.817583] CR2: 0000000000000018
Jun 28 15:29:21 172.30.1.2 [ 306.824706] ---[ end trace 33ba304ebe73322c ]---
Jun 28 15:29:21 172.30.1.2 [ 306.834283] Kernel panic - not syncing: Fatal exception in interrupt
Jun 28 15:29:21 172.30.1.2 [ 306.847312] Pid: 4486, comm: kvm Tainted: G      D W    3.2.21 #1
Jun 28 15:29:21 172.30.1.2 [ 306.859837] Call Trace:
Jun 28 15:29:21 172.30.1.2 [ 306.865093]  <IRQ> 
Jun 28 15:29:21 172.30.1.2 [ 306.876993]  [<ffffffff81049da9>] ? kmsg_dump+0xb2/0xdd
Jun 28 15:29:21 172.30.1.2 [ 306.887798]  [<ffffffff813653c1>] ? oops_end+0xa9/0xb6
Jun 28 15:29:21 172.30.1.2 [ 306.898438]  [<ffffffff8102fcf9>] ? no_context+0x1ff/0x20c
Jun 28 15:29:21 172.30.1.2 [ 306.909769]  [<ffffffff810f53c3>] ? virt_to_slab+0x9/0x3c
Jun 28 15:29:21 172.30.1.2 [ 306.920924]  [<ffffffff81367534>] ? do_page_fault+0x1ad/0x34c
Jun 28 15:29:21 172.30.1.2 [ 306.932780]  [<ffffffffa03fcd93>] ? ip_vs_notrack+0x6e/0xa6 [ip_vs]
Jun 28 15:29:21 172.30.1.2 [ 306.945675]  [<ffffffffa03fbdec>] ? ip_vs_conn_put+0x29/0x2f [ip_vs]
Jun 28 15:29:21 172.30.1.2 [ 306.958743]  [<ffffffffa03fde1a>] ? ip_vs_out+0x33c/0x49d [ip_vs]
Jun 28 15:29:21 172.30.1.2 [ 306.971339]  [<ffffffff812a896b>] ? dev_hard_start_xmit+0x3fd/0x571
Jun 28 15:29:21 172.30.1.2 [ 306.984312]  [<ffffffff81364ab5>] ? page_fault+0x25/0x30
Jun 28 15:29:21 172.30.1.2 [ 306.995300]  [<ffffffffa01d09d0>] ? br_nf_forward_finish+0x48/0xc0 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.009564]  [<ffffffffa01d0c91>] ? br_nf_forward_ip+0x214/0x231 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.023567]  [<ffffffff812c7de1>] ? nf_iterate+0x41/0x77
Jun 28 15:29:21 172.30.1.2 [ 307.034624]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.047239]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.059782]  [<ffffffff812c7f3a>] ? nf_hook_slow+0x68/0xfd
Jun 28 15:29:21 172.30.1.2 [ 307.071240]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.083788]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.095464]  [<ffffffffa01cba5a>] ? br_deliver+0x2d/0x2d [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.108227]  [<ffffffffa01cb96e>] ? T.963+0x3c/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.119901]  [<ffffffffa01ca753>] ? __br_fdb_get+0x10/0x83 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.132796]  [<ffffffffa01cc61d>] ? br_handle_frame_finish+0x1a7/0x211 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.147785]  [<ffffffffa01d1598>] ? br_nf_pre_routing_finish+0x222/0x22f [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.163124]  [<ffffffffa01d03b5>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.176369]  [<ffffffffa01d135b>] ? br_nf_pre_routing+0x4fc/0x517 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.190541]  [<ffffffff812ce172>] ? T.1028+0x4f/0x4f
Jun 28 15:29:21 172.30.1.2 [ 307.200832]  [<ffffffff812c7de1>] ? nf_iterate+0x41/0x77
Jun 28 15:29:21 172.30.1.2 [ 307.211813]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.223490]  [<ffffffff812c7f3a>] ? nf_hook_slow+0x68/0xfd
Jun 28 15:29:21 172.30.1.2 [ 307.234827]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.246623]  [<ffffffffa01cc476>] ? T.947+0x56/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.258233]  [<ffffffffa01cc45c>] ? T.947+0x3c/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.269846]  [<ffffffffa01cc84a>] ? br_handle_frame+0x1c3/0x1d9 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.283539]  [<ffffffffa01cc687>] ? br_handle_frame_finish+0x211/0x211 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.298466]  [<ffffffff812a74f2>] ? __netif_receive_skb+0x37a/0x490
Jun 28 15:29:21 172.30.1.2 [ 307.311298]  [<ffffffff81013a01>] ? read_tsc+0x5/0x16
Jun 28 15:29:21 172.30.1.2 [ 307.321743]  [<ffffffff812a799f>] ? netif_receive_skb+0x67/0x6d
Jun 28 15:29:21 172.30.1.2 [ 307.334023]  [<ffffffff812a7f0b>] ? napi_gro_receive+0x1f/0x2d
Jun 28 15:29:21 172.30.1.2 [ 307.346044]  [<ffffffff812a7a79>] ? napi_skb_finish+0x1c/0x31
Jun 28 15:29:21 172.30.1.2 [ 307.357884]  [<ffffffffa0009a21>] ? bnx2_poll_work+0x8db/0x9f4 [bnx2]
Jun 28 15:29:21 172.30.1.2 [ 307.371057]  [<ffffffffa00092f4>] ? bnx2_poll_work+0x1ae/0x9f4 [bnx2]
Jun 28 15:29:21 172.30.1.2 [ 307.384236]  [<ffffffffa01cc45c>] ? T.947+0x3c/0x56 [bridge]
Jun 28 15:29:21 172.30.1.2 [ 307.395847]  [<ffffffffa0009cfc>] ? bnx2_poll+0x116/0x1f2 [bnx2]
Jun 28 15:29:21 172.30.1.2 [ 307.408147]  [<ffffffff812a74f2>] ? __netif_receive_skb+0x37a/0x490
Jun 28 15:29:21 172.30.1.2 [ 307.421034]  [<ffffffff812a8038>] ? net_rx_action+0xa8/0x207
Jun 28 15:29:21 172.30.1.2 [ 307.432644]  [<ffffffff8106637e>] ? hrtimer_get_next_event+0x7f/0x9a
Jun 28 15:29:21 172.30.1.2 [ 307.445688]  [<ffffffff8104f11e>] ? __do_softirq+0xc4/0x1a0
Jun 28 15:29:21 172.30.1.2 [ 307.457118]  [<ffffffff81097905>] ? handle_irq_event_percpu+0x166/0x184
Jun 28 15:29:21 172.30.1.2 [ 307.470749]  [<ffffffff8136bd6c>] ? call_softirq+0x1c/0x30
Jun 28 15:29:21 172.30.1.2 [ 307.482114]  [<ffffffff8100fa3f>] ? do_softirq+0x3f/0x79
Jun 28 15:29:21 172.30.1.2 [ 307.493077]  [<ffffffff8104eeee>] ? irq_exit+0x44/0xb5
Jun 28 15:29:21 172.30.1.2 [ 307.503654]  [<ffffffff8100f38a>] ? do_IRQ+0x94/0xaa
Jun 28 15:29:21 172.30.1.2 [ 307.513875]  [<ffffffff813647ee>] ? common_interrupt+0x6e/0x6e
Jun 28 15:29:21 172.30.1.2 [ 307.525807]  <EOI> 
Jun 28 15:29:22 172.30.1.2 [ 307.541012]  [<ffffffff81045f07>] ? try_to_wake_up+0x181/0x190
Jun 28 15:29:22 172.30.1.2 [ 307.552919]  [<ffffffff81071764>] ? wake_futex+0x2b/0x47
Jun 28 15:29:22 172.30.1.2 [ 307.563873]  [<ffffffff81073949>] ? do_futex+0x3de/0x84a
Jun 28 15:29:22 172.30.1.2 [ 307.574792]  [<ffffffff810dfff1>] ? do_brk+0x2ca/0x326
Jun 28 15:29:22 172.30.1.2 [ 307.585363]  [<ffffffff81073ee4>] ? sys_futex+0x12f/0x147
Jun 28 15:29:22 172.30.1.2 [ 307.596418]  [<ffffffff81369b12>] ? system_call_fastpath+0x16/0x1b

Reply to: