Bug#862358: Kernel linux-image-3.16.0-4-amd64 missing Xen guest large grant table refs cast overflow bug fix
Package: linux-image-3.16.0-4-amd64
Version: 3.16.43-2
Jessie kernels, e.g. linux-image-3.16.0-4-amd64, (also applicable to
Wheezy) when used on Xen guests that require a large number of grant
table references (many vCPUs and/or vNICs in multi-queue xennet) leads
to a boot-time kernel BUG assertion and panic due to a type casting
error. On a 2-socket E5-2698v3 Xen 4.4 hypervisor
(gnttab_max_nr_frames=256) and a Debian 8 guest with (40) vCPUs, 128GB
RAM, (3) xvd block devices, (2) xennet vNICs, and xennet in the
default all CPUs multi-queue mode in both dom0 and domU, the console
output looks like this:
[ 1.317129] zswap: loaded using pool lzo/zbud
[ 1.317259] xenbus_probe_frontend: Device with no driver: device/vbd/51713
[ 1.317265] xenbus_probe_frontend: Device with no driver: device/vbd/51714
[ 1.317269] xenbus_probe_frontend: Device with no driver: device/vbd/51715
[ 1.317273] xenbus_probe_frontend: Device with no driver: device/vif/0
[ 1.317277] xenbus_probe_frontend: Device with no driver: device/vif/1
[ 1.317479] hctosys: unable to open rtc device (rtc0)
[ 1.318306] Freeing unused kernel memory: 1316K (ffffffff81b1f000 -
ffffffff81c68000)
[ 1.318315] Write protecting the kernel read-only data: 10240k
[ 1.321142] Freeing unused kernel memory: 392K (ffff88000159e000 -
ffff880001600000)
[ 1.321862] Freeing unused kernel memory: 1200K (ffff8800018d4000 -
ffff880001a00000)
Loading, please wait...
[ 1.361904] systemd-udevd[260]: starting version 215
[ 1.362768] random: systemd-udevd: uninitialized urandom read (16
bytes read, 121 bits of entropy available)
[ 1.395602] xen_netfront: Initialising Xen virtual ethernet driver
[ 1.400679] random: nonblocking pool is initialized
[ 1.604799] blkfront: xvda1: barrier or flush: disabled; persistent
grants: enabled; indirect descriptors: enabled;
[ 1.722225] ------------[ cut here ]------------
[ 1.722242] kernel BUG at
/build/kernel/orig/linux-4.4.43/drivers/net/xen-netfront.c:307!
[ 1.722255] invalid opcode: 0000 [#1] SMP
[ 1.722265] Modules linked in: xen_netfront(+) xen_blkfront(+) crc32c_intel
[ 1.722282] CPU: 8 PID: 209 Comm: xenwatch Not tainted
4.4.0-1-amd64 #1 Debian 4.4.43-1
[ 1.722298] task: ffff881f51c8b240 ti: ffff881f51c8c000 task.ti:
ffff881f51c8c000
[ 1.722310] RIP: e030:[<ffffffffa00732e5>] [<ffffffffa00732e5>]
xennet_alloc_rx_buffers+0x215/0x2b0 [xen_netfront]
[ 1.722333] RSP: e02b:ffff881f51c8fdb8 EFLAGS: 00010286
[ 1.722341] RAX: 0000000000008000 RBX: 0000000000000000 RCX: 0000000000000000
[ 1.722351] RDX: 0000000000008000 RSI: ffff881f51c6f400 RDI: ffff881f498b2bf8
[ 1.722360] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000001000
[ 1.722369] R10: ffff881f47c64540 R11: ffff881f47c64540 R12: 0000000000000000
[ 1.722378] R13: 0000000000008000 R14: ffff881f498b0e00 R15: ffff881f47c68600
[ 1.722394] FS: 0000000000000000(0000) GS:ffff881f5f500000(0000)
knlGS:ffff881f5f500000
[ 1.722409] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.722417] CR2: 00007f6eb6205095 CR3: 0000000001a0b000 CR4: 0000000000042660
[ 1.722427] Stack:
[ 1.722434] ffffffff810bcab1 ffff881f498b2bf8 ffff881f51a8ee90
ffff881f52392800
[ 1.722449] ffff881f4dbac000 ffff881f498b0e00 0000000000033000
ffff881f498b2380
[ 1.722466] 0000000000055000 ffffffffa0075528 ffff881f00000028
ffff881f52392800
[ 1.722484] Call Trace:
[ 1.722503] [<ffffffff810bcab1>] ?
__raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[ 1.722531] [<ffffffffa0075528>] ? netback_changed+0xcd8/0xd67
[xen_netfront]
[ 1.722557] [<ffffffff813b5ae0>] ? split+0xf0/0xf0
[ 1.722569] [<ffffffff813b5b6a>] ? xenwatch_thread+0x8a/0x140
[ 1.722581] [<ffffffff810b7a30>] ? wait_woken+0x90/0x90
[ 1.722596] [<ffffffff810962ff>] ? kthread+0xdf/0x100
[ 1.722608] [<ffffffff81096220>] ? kthread_park+0x50/0x50
[ 1.722623] [<ffffffff8159721f>] ? ret_from_fork+0x3f/0x70
[ 1.722633] [<ffffffff81096220>] ? kthread_park+0x50/0x50
[ 1.722641] Code: 8b 05 90 b4 a9 e1 48 8b 04 f8 48 83 f8 ff 0f 84
98 00 00 00 48 89 c2 48 b8 ff ff ff ff ff ff ff 3f 48 21 c2 e9 26 ff
ff ff 0f 0b <0f> 0b 48 b8 00 00 00 00 00 00 00 40 48 09 c2 48 3b 3d 45
b4 a9
[ 1.722745] RIP [<ffffffffa00732e5>]
xennet_alloc_rx_buffers+0x215/0x2b0 [xen_netfront]
[ 1.722762] RSP <ffff881f51c8fdb8>
[ 1.722772] ---[ end trace a266a8dd13d8465b ]---
[ 1.722780] Kernel panic - not syncing: Fatal exception in interrupt
[ 1.722909] Kernel Offset: disabled
The offending xen_netfront code is:
static void xennet_alloc_rx_buffers(struct netfront_queue *queue)
{
...
ref = gnttab_claim_grant_reference(&queue->gref_rx_head);
BUG_ON((signed short)ref < 0);
queue->grant_rx_ref[id] = ref;
...
}
Each vNIC requires 514 grant refs per tx+rx queue pair / CPU, 20560
refs for each vNIC on the guest.
This mis-casting has been fixed in later Linux 4.x kernels:
https://github.com/torvalds/linux/commit/87557efc27f6a50140fb20df06a917f368ce3c66
https://github.com/torvalds/linux/commit/269ebce4531b8edc4224259a02143181a1c1d77c
Reply to: