[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#922182: linux-image-4.19.0-2-cloud-amd64: Kernel call trace when testing NVME on Azure



Package: src:linux
Version: 4.19.16-1
Severity: normal

The Azure team ran into the following call trace with the Debian 4.19 kernel while validating NVME devices on Microsoft Azure.
The following call trace was observed while running fio on Standard_L64s_v2 size:

[  651.652598] rcu: INFO: rcu_sched self-detected stall on CPU
[  651.656617] rcu:       8-....: (5250 ticks this GP) idle=4ce/1/0x4000000000000004 softirq=4630/4630 fqs=2527 
[  651.656617] rcu:        (t=5256 jiffies g=69713 q=8089)
[  651.684600] Sending NMI from CPU 8 to CPUs 0:
[  651.694277] NMI backtrace for cpu 0
[  651.694278] CPU: 0 PID: 10 Comm: ksoftirqd/0 Not tainted 4.19.0-0.bpo.1-cloud-amd64 #1 Debian 4.19.12-1~bpo9+1
[  651.694279] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  05/18/2018
[  651.694279] RIP: 0010:cpumask_next+0x16/0x20
[  651.694281] Code: 38 22 75 97 eb d4 90 90 90 90 90 90 90 90 90 90 90 90 90 90 8d 57 01 48 89 f0 be 00 02 00 00 48 89 c7 48 63 d2 e8 2a 51 d8 ff <f3> c3 0f 1f 84 00 00 00 00 00 55 ba ff ff ff ff 53 48 89 fd 89 f3
[  651.694282] RSP: 0018:ffff9a6adf803ac0 EFLAGS: 00000046
[  651.694283] RAX: 0000000000000200 RBX: 0000000000000002 RCX: 0000000000000200
[  651.694284] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9a6adf803b10
[  651.694284] RBP: ffff9a6adf803b10 R08: ffffffffffffffff R09: 0000000000000000
[  651.694285] R10: ffff9a6adf803ad0 R11: 0000000000000000 R12: 00000000000000fd
[  651.694285] R13: ffff9a6adf803bc0 R14: 0000000000000046 R15: 00000000000211c0
[  651.694286] FS:  0000000000000000(0000) GS:ffff9a6adf800000(0000) knlGS:0000000000000000
[  651.694286] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  651.694286] CR2: 00007f215b3d6220 CR3: 000000307f486000 CR4: 00000000003406b0
[  651.694287] Call Trace:
[  651.694287]  <IRQ>
[  651.694287]  __send_ipi_mask+0x11e/0x310
[  651.694288]  __send_ipi_one+0x34/0x50
[  651.694288]  hv_send_ipi+0x10/0x30
[  651.694288]  check_preempt_curr+0x4e/0x90
[  651.694289]  ttwu_do_wakeup+0x19/0x140
[  651.694289]  try_to_wake_up+0x1ce/0x4a0
[  651.694289]  autoremove_wake_function+0x11/0x50
[  651.694290]  __wake_up_common+0x96/0x180
[  651.694290]  __wake_up_common_lock+0x7c/0xc0
[  651.694290]  aio_complete+0x17a/0x250
[  651.694291]  blkdev_bio_end_io+0x71/0x140
[  651.694291]  blk_update_request+0x91/0x2d0
[  651.694291]  blk_mq_end_request+0x1a/0xd0
[  651.694292]  nvme_irq+0x119/0x1d0 [nvme]
[  651.694293]  __handle_irq_event_percpu+0x81/0x190
[  651.694297]  handle_irq_event_percpu+0x30/0x80
[  651.694297]  handle_irq_event+0x3c/0x60
[  651.694298]  handle_edge_irq+0x94/0x1f0
[  651.694298]  handle_irq+0x1f/0x30
[  651.694298]  do_IRQ+0x49/0xe0
[  651.694299]  common_interrupt+0xf/0xf


We think this issue can be resolved with the following four upstream patches. I think these made it into the 4.20 kernel, but not 4.19:

  git cherry-pick 8ffe4e61c06a48324cfd97f1199bb9838acce2f2
  git cherry-pick 76f99ae5b54d48430d1f0c5512a84da0ff9761e0
  git cherry-pick b82592199032bf7c778f861b936287e37ebc9f62
  git cherry-pick e8da8794a7fd9eef1ec9a07f0d4897c68581c72b





-- Package-specific info:
** Kernel log: boot messages should be attached


-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-1-cloud-amd64 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages linux-image-4.19.0-2-cloud-amd64 depends on:
ii  initramfs-tools [linux-initramfs-tool]  0.133
ii  kmod                                    25-2
ii  linux-base                              4.5

Versions of packages linux-image-4.19.0-2-cloud-amd64 recommends:
ii  apparmor             2.13.2-7
ii  firmware-linux-free  3.4
ii  irqbalance           1.5.0-3

Versions of packages linux-image-4.19.0-2-cloud-amd64 suggests:
pn  debian-kernel-handbook  <none>
ii  grub-pc                 2.02+dfsg1-10
pn  linux-doc-4.19          <none>

Versions of packages linux-image-4.19.0-2-cloud-amd64 is related to:
pn  firmware-amd-graphics     <none>
pn  firmware-atheros          <none>
pn  firmware-bnx2             <none>
pn  firmware-bnx2x            <none>
pn  firmware-brcm80211        <none>
pn  firmware-cavium           <none>
pn  firmware-intel-sound      <none>
pn  firmware-intelwimax       <none>
pn  firmware-ipw2x00          <none>
pn  firmware-ivtv             <none>
pn  firmware-iwlwifi          <none>
pn  firmware-libertas         <none>
pn  firmware-linux-nonfree    <none>
pn  firmware-misc-nonfree     <none>
pn  firmware-myricom          <none>
pn  firmware-netxen           <none>
pn  firmware-qlogic           <none>
pn  firmware-realtek          <none>
pn  firmware-samsung          <none>
pn  firmware-siano            <none>
pn  firmware-ti-connectivity  <none>
pn  xen-hypervisor            <none>

-- no debconf information


Reply to: