Bug#922182: linux-image-4.19.0-2-cloud-amd64: Kernel call trace when testing NVME on Azure
Package: src:linux
Version: 4.19.16-1
Severity: normal
The Azure team ran into the following call trace with the Debian 4.19 kernel while validating NVME devices on Microsoft Azure.
The following call trace was observed while running fio on Standard_L64s_v2 size:
[ 651.652598] rcu: INFO: rcu_sched self-detected stall on CPU
[ 651.656617] rcu: 8-....: (5250 ticks this GP) idle=4ce/1/0x4000000000000004 softirq=4630/4630 fqs=2527
[ 651.656617] rcu: (t=5256 jiffies g=69713 q=8089)
[ 651.684600] Sending NMI from CPU 8 to CPUs 0:
[ 651.694277] NMI backtrace for cpu 0
[ 651.694278] CPU: 0 PID: 10 Comm: ksoftirqd/0 Not tainted 4.19.0-0.bpo.1-cloud-amd64 #1 Debian 4.19.12-1~bpo9+1
[ 651.694279] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 05/18/2018
[ 651.694279] RIP: 0010:cpumask_next+0x16/0x20
[ 651.694281] Code: 38 22 75 97 eb d4 90 90 90 90 90 90 90 90 90 90 90 90 90 90 8d 57 01 48 89 f0 be 00 02 00 00 48 89 c7 48 63 d2 e8 2a 51 d8 ff <f3> c3 0f 1f 84 00 00 00 00 00 55 ba ff ff ff ff 53 48 89 fd 89 f3
[ 651.694282] RSP: 0018:ffff9a6adf803ac0 EFLAGS: 00000046
[ 651.694283] RAX: 0000000000000200 RBX: 0000000000000002 RCX: 0000000000000200
[ 651.694284] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9a6adf803b10
[ 651.694284] RBP: ffff9a6adf803b10 R08: ffffffffffffffff R09: 0000000000000000
[ 651.694285] R10: ffff9a6adf803ad0 R11: 0000000000000000 R12: 00000000000000fd
[ 651.694285] R13: ffff9a6adf803bc0 R14: 0000000000000046 R15: 00000000000211c0
[ 651.694286] FS: 0000000000000000(0000) GS:ffff9a6adf800000(0000) knlGS:0000000000000000
[ 651.694286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 651.694286] CR2: 00007f215b3d6220 CR3: 000000307f486000 CR4: 00000000003406b0
[ 651.694287] Call Trace:
[ 651.694287] <IRQ>
[ 651.694287] __send_ipi_mask+0x11e/0x310
[ 651.694288] __send_ipi_one+0x34/0x50
[ 651.694288] hv_send_ipi+0x10/0x30
[ 651.694288] check_preempt_curr+0x4e/0x90
[ 651.694289] ttwu_do_wakeup+0x19/0x140
[ 651.694289] try_to_wake_up+0x1ce/0x4a0
[ 651.694289] autoremove_wake_function+0x11/0x50
[ 651.694290] __wake_up_common+0x96/0x180
[ 651.694290] __wake_up_common_lock+0x7c/0xc0
[ 651.694290] aio_complete+0x17a/0x250
[ 651.694291] blkdev_bio_end_io+0x71/0x140
[ 651.694291] blk_update_request+0x91/0x2d0
[ 651.694291] blk_mq_end_request+0x1a/0xd0
[ 651.694292] nvme_irq+0x119/0x1d0 [nvme]
[ 651.694293] __handle_irq_event_percpu+0x81/0x190
[ 651.694297] handle_irq_event_percpu+0x30/0x80
[ 651.694297] handle_irq_event+0x3c/0x60
[ 651.694298] handle_edge_irq+0x94/0x1f0
[ 651.694298] handle_irq+0x1f/0x30
[ 651.694298] do_IRQ+0x49/0xe0
[ 651.694299] common_interrupt+0xf/0xf
We think this issue can be resolved with the following four upstream patches. I think these made it into the 4.20 kernel, but not 4.19:
git cherry-pick 8ffe4e61c06a48324cfd97f1199bb9838acce2f2
git cherry-pick 76f99ae5b54d48430d1f0c5512a84da0ff9761e0
git cherry-pick b82592199032bf7c778f861b936287e37ebc9f62
git cherry-pick e8da8794a7fd9eef1ec9a07f0d4897c68581c72b
-- Package-specific info:
** Kernel log: boot messages should be attached
-- System Information:
Debian Release: buster/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-1-cloud-amd64 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages linux-image-4.19.0-2-cloud-amd64 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.133
ii kmod 25-2
ii linux-base 4.5
Versions of packages linux-image-4.19.0-2-cloud-amd64 recommends:
ii apparmor 2.13.2-7
ii firmware-linux-free 3.4
ii irqbalance 1.5.0-3
Versions of packages linux-image-4.19.0-2-cloud-amd64 suggests:
pn debian-kernel-handbook <none>
ii grub-pc 2.02+dfsg1-10
pn linux-doc-4.19 <none>
Versions of packages linux-image-4.19.0-2-cloud-amd64 is related to:
pn firmware-amd-graphics <none>
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux-nonfree <none>
pn firmware-misc-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>
-- no debconf information
Reply to: