Bug#899044: Oops: 0000 [#1] SMP in skb_release_data, openvswitch related
To: netdev, dev@openvswitch
Cc: Eric Dumazet (author of ff04a771ad), debian bug
Hi,
As follow-up to my bug report at Debian [0], I'm trying to do bug triage
and find out more. I'm not the expert here, but anything could help, and
it's an opportunity to learn things.
I'm observing the attached errors ('general protection fault: 0000 [#1]
SMP' and 'BUG: unable to handle kernel paging request') on machines that
are Xen dom0 and running a 4.9.88 Debian Stretch kernel as dom0 kernel.
The errors have been happening a few times in the last few weeks. It
started after upgrading them from Jessie and 3.16 kernel to Stretch with
4.9 kernel.
The traces printed look very much alike every time.
If I look up the listed address, I get:
-$ addr2line -e /usr/lib/debug/boot/vmlinux-4.9.0-6-amd64 -i -a
ffffffff814f5c7d
0xffffffff814f5c7d
./debian/build/build_amd64_none_amd64/./include/linux/compiler.h:243
(discriminator 3)
./debian/build/build_amd64_none_amd64/./include/linux/page-flags.h:143
(discriminator 3)
./debian/build/build_amd64_none_amd64/./include/linux/mm.h:779
(discriminator 3)
./debian/build/build_amd64_none_amd64/./include/linux/skbuff.h:2592
(discriminator 3)
./debian/build/build_amd64_none_amd64/./net/core/skbuff.c:594
(discriminator 3)
583 static void skb_release_data(struct sk_buff *skb)
584 {
585 struct skb_shared_info *shinfo = skb_shinfo(skb);
586 int i;
587
588 if (skb->cloned &&
589 atomic_sub_return(skb->nohdr ? (1 << SKB_DATAREF_SHIFT)
+ 1 : 1,
590 &shinfo->dataref))
591 return;
592
593 for (i = 0; i < shinfo->nr_frags; i++)
594 -----> __skb_frag_unref(&shinfo->frags[i]); <------
595
596 /*
597 * If skb buf is from userspace, we need to notify the caller
598 * the lower device DMA has done;
599 */
600 if (shinfo->tx_flags & SKBTX_DEV_ZEROCOPY) {
601 struct ubuf_info *uarg;
602
603 uarg = shinfo->destructor_arg;
604 if (uarg->callback)
605 uarg->callback(uarg, true);
606 }
607
608 if (shinfo->frag_list)
609 kfree_skb_list(shinfo->frag_list);
610
611 skb_free_head(skb);
612 }
The most recent (well, from 2014) biggest change in this area is...
commit ff04a771ad25fc9ba91690e73465b4d34b6bf8b3
Author: Eric Dumazet <edumazet@google.com>
Date: Tue Sep 23 18:39:30 2014 -0700
net : optimize skb_release_data()
...which is not present in the 3.16.y kernel that Debian Jessie still
uses, and which does not hit this problem (however, also using older
openvswitch userspace components).
Other changes in this area mention zero copy IO, which sounds like
something openvswitch could be using.
-- background: openvswitch usage --
For networking between domUs and the outside world, we use openvswitch.
After such an error happens:
* The amount of "flows" in the kernel quickly raises to the limit,
10000, as seen in output of ovs-dpctl show.
* Network traffic that should flow through the openvswitch bridge starts
disappearing in a seemingly random way (probably because it can't handle
new traffic flows).
* The memory usage of the userspace ovs-vswitchd starts growing quickly.
* Many of the ovs commands, like to add or remove an interface or bridge
hang.
After a restart of the openvswitch-switch service, and fixing up a bunch
of configuration of connected interfaces, functionality is restored.
While most of the symptoms seem related to userspace openvswitch
processes, the cause of it all seems to be in the kernel, while the
userspace ovs-vswitchd process is receiving a network packet?
-- reproducer --
I don't have a reliable reproducer yet, except for waiting days or weeks
until it randomly happens somewhere. There's no sign of unusual amounts
of traffic / load etc when it happens.
An idea I can come up with is builing a semi-random udp packet generator
to start stressing the code path from kernel to ovs-vswitchd.
If I succeed reproducing, I can start trying other kernels or changes.
Please advice what else I could do to help resolving this issue.
Thanks,
Regards,
Hans van Kranenburg
[0] https://bugs.debian.org/899044
May 4 08:23:03 altair kernel: [83978.662075] BUG: unable to handle kernel paging request at 000000030000001f
May 4 08:23:03 altair kernel: [83978.665887] IP: [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110
May 4 08:23:03 altair kernel: [83978.669837] PGD 0
May 4 08:23:03 altair kernel: [83978.669882]
May 4 08:23:03 altair kernel: [83978.673589] Oops: 0000 [#1] SMP
May 4 08:23:03 altair kernel: [83978.677281] Modules linked in: cls_u32 sch_ingress act_mirred sch_fq_codel ifb xt_mark sch_htb xt_physdev br_netfilter bridge stp llc xen_netback xen_blkback algif_skcipher af_alg dm_service_time binfmt_misc xen_gntdev xen_evtchn openvswitch nf_nat_ipv6 libcrc32c xenfs xen_privcmd ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw dm_crypt intel_powerclamp crct10dif_pclmul crc32_pclmul iTCO_wdt iTCO_vendor_support ghash_clmulni_intel pcspkr serio_raw joydev evdev amdkfd radeon ttm drm_kms_helper drm i2c_algo_bit lpc_ich mfd_core i7core_edac hpilo
May 4 08:23:03 altair kernel: [83978.701936] sg ipmi_si hpwdt edac_core ipmi_msghandler acpi_power_meter button shpchp dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb mbcache btrfs crc32c_generic xor raid6_pq mlx4_en ptp pps_core hid_generic usbhid hid sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse ehci_pci uhci_hcd ehci_hcd usbcore usb_common hpsa scsi_transport_sas bnx2 mlx4_core devlink scsi_mod thermal
May 4 08:23:03 altair kernel: [83978.724406] CPU: 1 PID: 1486 Comm: revalidator7 Not tainted 4.9.0-6-amd64 #1 Debian 4.9.88-1
May 4 08:23:03 altair kernel: [83978.729139] Hardware name: HP ProLiant DL360 G7, BIOS P68 08/16/2015
May 4 08:23:03 altair kernel: [83978.733958] task: ffff880119e1ee80 task.stack: ffffc90042764000
May 4 08:23:03 altair kernel: [83978.738724] RIP: e030:[<ffffffff814f5c7d>] [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110
May 4 08:23:03 altair kernel: [83978.743560] RSP: e02b:ffffc90042767c78 EFLAGS: 00010206
May 4 08:23:03 altair kernel: [83978.748352] RAX: 0000000000000050 RBX: 00000002ffffffff RCX: ffffffff81ce0f40
May 4 08:23:03 altair kernel: [83978.753116] RDX: ffffffffffffffff RSI: ffff8800cc998900 RDI: ffff8800cc998900
May 4 08:23:03 altair kernel: [83978.757867] RBP: ffff8800cc998900 R08: ffff880123c00000 R09: ffff88011f220000
May 4 08:23:03 altair kernel: [83978.762598] R10: ffff8800cc998900 R11: ffff880119e10280 R12: 0000000000000002
May 4 08:23:03 altair kernel: [83978.767321] R13: ffff88011f227ec0 R14: ffff88011dea2800 R15: 0000000000000000
May 4 08:23:03 altair kernel: [83978.772000] FS: 00007fc1656cc700(0000) GS:ffff880128240000(0000) knlGS:0000000000000000
May 4 08:23:03 altair kernel: [83978.776671] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
May 4 08:23:03 altair kernel: [83978.781355] CR2: 000000030000001f CR3: 00000001212b1000 CR4: 0000000000002660
May 4 08:23:03 altair kernel: [83978.786135] Stack:
May 4 08:23:03 altair kernel: [83978.790841] ffff880120a28000 ffff8800cc998900 ffffc90042767ec0 0000000000007ea4
May 4 08:23:03 altair kernel: [83978.795898] ffffffff814f6267 ffff880120a28000 ffff8800cc998900 ffffffff814fcc91
May 4 08:23:03 altair kernel: [83978.800806] ffff880120a28000 ffffffff8153f2df ffffc90000000000 ffff8800cc998900
May 4 08:23:03 altair kernel: [83978.805723] Call Trace:
May 4 08:23:03 altair kernel: [83978.810654] [<ffffffff814f6267>] ? consume_skb+0x27/0x80
May 4 08:23:03 altair kernel: [83978.815626] [<ffffffff814fcc91>] ? skb_free_datagram+0x11/0x40
May 4 08:23:03 altair kernel: [83978.820545] [<ffffffff8153f2df>] ? netlink_recvmsg+0x19f/0x440
May 4 08:23:03 altair kernel: [83978.825426] [<ffffffff814ed4ca>] ? ___sys_recvmsg+0xda/0x1f0
May 4 08:23:03 altair kernel: [83978.830273] [<ffffffff812237fb>] ? file_update_time+0xcb/0x110
May 4 08:23:03 altair kernel: [83978.835058] [<ffffffff8120fbeb>] ? pipe_write+0x29b/0x3e0
May 4 08:23:03 altair kernel: [83978.839800] [<ffffffff812066b0>] ? new_sync_write+0xe0/0x130
May 4 08:23:03 altair kernel: [83978.844502] [<ffffffff814edf4e>] ? __sys_recvmsg+0x4e/0x90
May 4 08:23:03 altair kernel: [83978.849161] [<ffffffff81003b7d>] ? do_syscall_64+0x8d/0xf0
May 4 08:23:03 altair kernel: [83978.853779] [<ffffffff8161244e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
May 4 08:23:03 altair kernel: [83978.858397] Code: 03 48 c1 e8 37 83 e0 07 83 f8 04 74 49 41 0f b6 45 00 41 83 c4 01 44 39 e0 7e 51 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 5c 05 00 <48> 8b 43 20 48 8d 50 ff a8 01 48 0f 45 da f0 ff 4b 1c 75 bf 48
May 4 08:23:03 altair kernel: [83978.868227] RIP [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110
May 4 08:23:03 altair kernel: [83978.873017] RSP <ffffc90042767c78>
May 4 08:23:03 altair kernel: [83978.877746
May 4 22:00:22 sirius kernel: [1999361.378086] BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0
May 4 22:00:22 sirius kernel: [1999361.381804] IP: [<ffffffff814f4c7d>] skb_release_data+0x8d/0x110
May 4 22:00:22 sirius kernel: [1999361.385492] PGD 0
May 4 22:00:22 sirius kernel: [1999361.385535]
May 4 22:00:22 sirius kernel: [1999361.389145] Oops: 0000 [#1] SMP
May 4 22:00:22 sirius kernel: [1999361.392725] Modules linked in: xt_physdev br_netfilter bridge stp llc xen_netback xen_blkback algif_skcipher af_alg dm_service_time binfmt_misc openvswitch nf_nat_ipv6 libcrc32c xen_gntdev xen_evtchn xenfs xen_privcmd ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw dm_crypt intel_powerclamp crct10dif_pclmul crc32_pclmul amdkfd iTCO_wdt evdev joydev iTCO_vendor_support ghash_clmulni_intel radeon ttm serio_raw pcspkr drm_kms_helper drm i2c_algo_bit sg i7core_edac lpc_ich ipmi_si acpi_power_meter hpilo hpwdt mfd_core edac_core ipmi_msghandler button
May 4 22:00:22 sirius kernel: [1999361.416634] shpchp dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb mbcache btrfs crc32c_generic xor raid6_pq mlx4_en ptp pps_core hid_generic usbhid hid sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse ehci_pci uhci_hcd ehci_hcd usbcore usb_common mlx4_core hpsa scsi_transport_sas bnx2 devlink scsi_mod thermal
May 4 22:00:22 sirius kernel: [1999361.438322] CPU: 2 PID: 1400 Comm: revalidator9 Not tainted 4.9.0-6-amd64 #1 Debian 4.9.82-1+deb9u3
May 4 22:00:22 sirius kernel: [1999361.442773] Hardware name: HP ProLiant DL360 G7, BIOS P68 08/16/2015
May 4 22:00:22 sirius kernel: [1999361.447219] task: ffff880111c58540 task.stack: ffffc90041bcc000
May 4 22:00:22 sirius kernel: [1999361.451796] RIP: e030:[<ffffffff814f4c7d>] [<ffffffff814f4c7d>] skb_release_data+0x8d/0x110
May 4 22:00:22 sirius kernel: [1999361.456294] RSP: e02b:ffffc90041bcfc78 EFLAGS: 00010206
May 4 22:00:22 sirius kernel: [1999361.460758] RAX: 0000000000000030 RBX: 00000000000001c0 RCX: ffffffff81ce0e00
May 4 22:00:22 sirius kernel: [1999361.465261] RDX: 0000000000008100 RSI: ffff880118a94f00 RDI: ffff880118a94f00
May 4 22:00:22 sirius kernel: [1999361.469724] RBP: ffff880118a94f00 R08: ffff88011bc00000 R09: ffff8800b0218000
May 4 22:00:22 sirius kernel: [1999361.474230] R10: ffff880118a94f00 R11: ffff880111c50240 R12: 0000000000000000
May 4 22:00:22 sirius kernel: [1999361.478710] R13: ffff8800b021fec0 R14: ffff8800b8356a40 R15: 0000000000000000
May 4 22:00:22 sirius kernel: [1999361.483220] FS: 00007faa54946700(0000) GS:ffff880120280000(0000) knlGS:0000000000000000
May 4 22:00:22 sirius kernel: [1999361.487736] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
May 4 22:00:22 sirius kernel: [1999361.492181] CR2: 00000000000001e0 CR3: 0000000119ce9000 CR4: 0000000000002660
May 4 22:00:22 sirius kernel: [1999361.496661] Stack:
May 4 22:00:22 sirius kernel: [1999361.501036] ffff8801190a2800 ffff880118a94f00 ffffc90041bcfec0 0000000000007eac
May 4 22:00:22 sirius kernel: [1999361.505470] ffffffff814f5267 ffff8801190a2800 ffff880118a94f00 ffffffff814fbc91
May 4 22:00:22 sirius kernel: [1999361.509844] ffff8801190a2800 ffffffff8153e2bf ffffc90000000000 ffff880118a94f00
May 4 22:00:22 sirius kernel: [1999361.514213] Call Trace:
May 4 22:00:22 sirius kernel: [1999361.518499] [<ffffffff814f5267>] ? consume_skb+0x27/0x80
May 4 22:00:22 sirius kernel: [1999361.522818] [<ffffffff814fbc91>] ? skb_free_datagram+0x11/0x40
May 4 22:00:22 sirius kernel: [1999361.527109] [<ffffffff8153e2bf>] ? netlink_recvmsg+0x19f/0x440
May 4 22:00:22 sirius kernel: [1999361.531314] [<ffffffff814ec4ca>] ? ___sys_recvmsg+0xda/0x1f0
May 4 22:00:22 sirius kernel: [1999361.535488] [<ffffffff812221ab>] ? file_update_time+0xcb/0x110
May 4 22:00:22 sirius kernel: [1999361.539626] [<ffffffff8120e5cb>] ? pipe_write+0x29b/0x3e0
May 4 22:00:22 sirius kernel: [1999361.543790] [<ffffffff812050a0>] ? new_sync_write+0xe0/0x130
May 4 22:00:22 sirius kernel: [1999361.547989] [<ffffffff814ecf4e>] ? __sys_recvmsg+0x4e/0x90
May 4 22:00:22 sirius kernel: [1999361.552218] [<ffffffff81003b7f>] ? do_syscall_64+0x8f/0xf0
May 4 22:00:22 sirius kernel: [1999361.556467] [<ffffffff816113b8>] ? entry_SYSCALL_64_after_swapgs+0x42/0xb0
May 4 22:00:22 sirius kernel: [1999361.560791] Code: 03 48 c1 e8 37 83 e0 07 83 f8 04 74 49 41 0f b6 45 00 41 83 c4 01 44 39 e0 7e 51 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 5c 05 00 <48> 8b 43 20 48 8d 50 ff a8 01 48 0f 45 da f0 ff 4b 1c 75 bf 48
May 4 22:00:22 sirius kernel: [1999361.570202] RIP [<ffffffff814f4c7d>] skb_release_data+0x8d/0x110
May 4 22:00:22 sirius kernel: [1999361.575033] RSP <ffffc90041bcfc78>
May 4 22:00:22 sirius kernel: [1999361.579731] CR2: 00000000000001e0
May 4 22:00:22 sirius kernel: [1999361.599233] ---[ end trace de6345fc470c5362 ]---
May 18 13:49:26 omega kernel: [1213243.942643] general protection fault: 0000 [#1] SMP
May 18 13:49:26 omega kernel: [1213243.946704] Modules linked in: xt_physdev br_netfilter bridge stp llc xen_netback xen_blkback algif_skcipher af_alg dm_service_time xen_gntdev openvswitch xen_evtchn nf_nat_ipv6 libcrc32c xenfs xen_privcmd ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw dm_crypt amdkfd radeon intel_powerclamp crct10dif_pclmul iTCO_wdt crc32_pclmul iTCO_vendor_support ttm ghash_clmulni_intel hpwdt pcspkr drm_kms_helper drm serio_raw evdev i2c_algo_bit joydev sg hpilo lpc_ich mfd_core i7core_edac ipmi_si edac_core ipmi_msghandler acpi_power_meter shpchp button dm_multipath
May 18 13:49:26 omega kernel: [1213243.973478] dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb mbcache btrfs crc32c_generic xor raid6_pq mlx4_en ptp pps_core hid_generic usbhid hid sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse uhci_hcd ehci_pci ehci_hcd usbcore usb_common hpsa bnx2 mlx4_core scsi_transport_sas devlink scsi_mod thermal
May 18 13:49:26 omega kernel: [1213243.997290] CPU: 2 PID: 1582 Comm: revalidator9 Not tainted 4.9.0-6-amd64 #1 Debian 4.9.88-1
May 18 13:49:26 omega kernel: [1213244.002200] Hardware name: HP ProLiant DL360 G7, BIOS P68 08/16/2015
May 18 13:49:26 omega kernel: [1213244.007157] task: ffff8801186caf00 task.stack: ffffc90041b8c000
May 18 13:49:26 omega kernel: [1213244.012040] RIP: e030:[<ffffffff814f5c7d>] [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110
May 18 13:49:26 omega kernel: [1213244.016957] RSP: e02b:ffffc90041b8fc78 EFLAGS: 00010206
May 18 13:49:26 omega kernel: [1213244.021783] RAX: 0000000000000030 RBX: 290008a753b675a9 RCX: ffffffff81ce0f40
May 18 13:49:26 omega kernel: [1213244.026673] RDX: 0000000000008100 RSI: ffff88011a2c6200 RDI: ffff88011a2c6200
May 18 13:49:26 omega kernel: [1213244.031596] RBP: ffff88011a2c6200 R08: ffff88011bc00000 R09: ffff88011aa70000
May 18 13:49:26 omega kernel: [1213244.036422] R10: ffff88011a2c6200 R11: ffff8801186c0200 R12: 0000000000000000
May 18 13:49:26 omega kernel: [1213244.041267] R13: ffff88011aa77ec0 R14: ffff8801199da7c0 R15: 0000000000000000
May 18 13:49:26 omega kernel: [1213244.046055] FS: 00007fe5f35e2700(0000) GS:ffff880120280000(0000) knlGS:0000000000000000
May 18 13:49:26 omega kernel: [1213244.050785] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
May 18 13:49:26 omega kernel: [1213244.055488] CR2: 00007fe579f4f059 CR3: 0000000117428000 CR4: 0000000000002660
May 18 13:49:26 omega kernel: [1213244.060232] Stack:
May 18 13:49:26 omega kernel: [1213244.064896] ffff880117588800 ffff88011a2c6200 ffffc90041b8fec0 0000000000007e94
May 18 13:49:26 omega kernel: [1213244.069725] ffffffff814f6267 ffff880117588800 ffff88011a2c6200 ffffffff814fcc91
May 18 13:49:26 omega kernel: [1213244.074552] ffff880117588800 ffffffff8153f2df ffffc90000000000 ffff88011a2c6200
May 18 13:49:26 omega kernel: [1213244.079377] Call Trace:
May 18 13:49:26 omega kernel: [1213244.084123] [<ffffffff814f6267>] ? consume_skb+0x27/0x80
May 18 13:49:26 omega kernel: [1213244.089047] [<ffffffff814fcc91>] ? skb_free_datagram+0x11/0x40
May 18 13:49:26 omega kernel: [1213244.093728] [<ffffffff8153f2df>] ? netlink_recvmsg+0x19f/0x440
May 18 13:49:26 omega kernel: [1213244.098359] [<ffffffff814ed4ca>] ? ___sys_recvmsg+0xda/0x1f0
May 18 13:49:26 omega kernel: [1213244.102962] [<ffffffff812237fb>] ? file_update_time+0xcb/0x110
May 18 13:49:26 omega kernel: [1213244.107530] [<ffffffff8120fbeb>] ? pipe_write+0x29b/0x3e0
May 18 13:49:26 omega kernel: [1213244.112074] [<ffffffff812066b0>] ? new_sync_write+0xe0/0x130
May 18 13:49:26 omega kernel: [1213244.116625] [<ffffffff814edf4e>] ? __sys_recvmsg+0x4e/0x90
May 18 13:49:26 omega kernel: [1213244.121183] [<ffffffff81003b7d>] ? do_syscall_64+0x8d/0xf0
May 18 13:49:26 omega kernel: [1213244.125715] [<ffffffff8161244e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
May 18 13:49:26 omega kernel: [1213244.130196] Code: 03 48 c1 e8 37 83 e0 07 83 f8 04 74 49 41 0f b6 45 00 41 83 c4 01 44 39 e0 7e 51 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 5c 05 00 <48> 8b 43 20 48 8d 50 ff a8 01 48 0f 45 da f0 ff 4b 1c 75 bf 48
May 18 13:49:26 omega kernel: [1213244.139830] RIP [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110
May 18 13:49:26 omega kernel: [1213244.144491] RSP <ffffc90041b8fc78>
May 18 13:49:26 omega kernel: [1213244.164037] ---[ end trace c53e06696e145c33 ]---
May 22 13:10:59 sirius kernel: [1520897.781283] general protection fault: 0000 [#1] SMP
May 22 13:10:59 sirius kernel: [1520897.784780] Modules linked in: xt_physdev br_netfilter bridge stp llc xen_netback xen_blkback binfmt_misc algif_skcipher af_alg dm_service_time openvswitch nf_nat_ipv6 libcrc32c xen_gntdev xen_evtchn xenfs xen_privcmd ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_owner xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_raw dm_crypt intel_powerclamp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support amdkfd joydev evdev radeon serio_raw pcspkr ttm drm_kms_helper drm i2c_algo_bit lpc_ich i7core_edac mfd_core sg ipmi_si hpilo hpwdt edac_core acpi_power_meter ipmi_msghandler button
May 22 13:10:59 sirius kernel: [1520897.808262] shpchp dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto ecb mbcache btrfs crc32c_generic xor raid6_pq mlx4_en ptp pps_core hid_generic usbhid hid sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse ehci_pci uhci_hcd ehci_hcd usbcore usb_common hpsa scsi_transport_sas mlx4_core bnx2 devlink scsi_mod thermal
May 22 13:10:59 sirius kernel: [1520897.829713] CPU: 3 PID: 1570 Comm: revalidator8 Not tainted 4.9.0-6-amd64 #1 Debian 4.9.88-1
May 22 13:10:59 sirius kernel: [1520897.834150] Hardware name: HP ProLiant DL360 G7, BIOS P68 08/16/2015
May 22 13:10:59 sirius kernel: [1520897.838605] task: ffff880117432400 task.stack: ffffc90041c3c000
May 22 13:10:59 sirius kernel: [1520897.843037] RIP: e030:[<ffffffff814f5c7d>] [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110
May 22 13:10:59 sirius kernel: [1520897.847518] RSP: e02b:ffffc90041c3fc78 EFLAGS: 00010206
May 22 13:10:59 sirius kernel: [1520897.851953] RAX: 0000000000000030 RBX: 33efd0d4e0b4e3a6 RCX: ffffffff81ce0f40
May 22 13:10:59 sirius kernel: [1520897.856583] RDX: 0000000000008100 RSI: ffff880116496800 RDI: ffff880116496800
May 22 13:10:59 sirius kernel: [1520897.861027] RBP: ffff880116496800 R08: ffff88011bc00000 R09: ffff8800d08f8000
May 22 13:10:59 sirius kernel: [1520897.865531] R10: ffff880116496800 R11: ffff880117430200 R12: 0000000000000000
May 22 13:10:59 sirius kernel: [1520897.869988] R13: ffff8800d08ffec0 R14: ffff8800d2658040 R15: 0000000000000000
May 22 13:10:59 sirius kernel: [1520897.874478] FS: 00007fb1c173c700(0000) GS:ffff8801202c0000(0000) knlGS:0000000000000000
May 22 13:10:59 sirius kernel: [1520897.878963] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
May 22 13:10:59 sirius kernel: [1520897.883445] CR2: 00007fa9ddc3b174 CR3: 00000001195e8000 CR4: 0000000000002660
May 22 13:10:59 sirius kernel: [1520897.887955] Stack:
May 22 13:10:59 sirius kernel: [1520897.892363] ffff8800cfefb000 ffff880116496800 ffffc90041c3fec0 0000000000007e78
May 22 13:10:59 sirius kernel: [1520897.896907] ffffffff814f6267 ffff8800cfefb000 ffff880116496800 ffffffff814fcc91
May 22 13:10:59 sirius kernel: [1520897.901431] ffff8800cfefb000 ffffffff8153f2df ffffc90000000000 ffff880116496800
May 22 13:10:59 sirius kernel: [1520897.905926] Call Trace:
May 22 13:10:59 sirius kernel: [1520897.910354] [<ffffffff814f6267>] ? consume_skb+0x27/0x80
May 22 13:10:59 sirius kernel: [1520897.914875] [<ffffffff814fcc91>] ? skb_free_datagram+0x11/0x40
May 22 13:10:59 sirius kernel: [1520897.919414] [<ffffffff8153f2df>] ? netlink_recvmsg+0x19f/0x440
May 22 13:10:59 sirius kernel: [1520897.923949] [<ffffffff814ed4ca>] ? ___sys_recvmsg+0xda/0x1f0
May 22 13:10:59 sirius kernel: [1520897.928548] [<ffffffff812237fb>] ? file_update_time+0xcb/0x110
May 22 13:10:59 sirius kernel: [1520897.933258] [<ffffffff8120fbeb>] ? pipe_write+0x29b/0x3e0
May 22 13:10:59 sirius kernel: [1520897.937883] [<ffffffff812066b0>] ? new_sync_write+0xe0/0x130
May 22 13:10:59 sirius kernel: [1520897.942522] [<ffffffff814edf4e>] ? __sys_recvmsg+0x4e/0x90
May 22 13:10:59 sirius kernel: [1520897.947202] [<ffffffff81003b7d>] ? do_syscall_64+0x8d/0xf0
May 22 13:10:59 sirius kernel: [1520897.951837] [<ffffffff8161244e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
May 22 13:10:59 sirius kernel: [1520897.956458] Code: 03 48 c1 e8 37 83 e0 07 83 f8 04 74 49 41 0f b6 45 00 41 83 c4 01 44 39 e0 7e 51 49 63 c4 48 83 c0 03 48 c1 e0 04 49 8b 5c 05 00 <48> 8b 43 20 48 8d 50 ff a8 01 48 0f 45 da f0 ff 4b 1c 75 bf 48
May 22 13:10:59 sirius kernel: [1520897.966226] RIP [<ffffffff814f5c7d>] skb_release_data+0x8d/0x110
May 22 13:10:59 sirius kernel: [1520897.971014] RSP <ffffc90041c3fc78>
May 22 13:10:59 sirius kernel: [1520897.990735] ---[ end trace 70c169490927f55d ]---
Reply to: