Bug#968567: linux-image-4.19.0-10-amd64: kernel failure when writing on a GFS2 partition
Hi Nicolas,
Not a direct help, but some comments:
On Mon, Aug 17, 2020 at 07:12:32PM +0200, Nicolas Courtel wrote:
> Package: src:linux
> Version: 4.19.132-1
> Severity: normal
>
> Dear maintainer,
>
> After upgrading to kernel 4.19.0-10, writing to a GFS2 volume makes the kernel
> output a series of messages, and the server quickly becomes unusable. Before
> trying to write, mounting the volume and reading its content works as expected.
>
> The problem is reproductible on 2 different servers, using FC and iSCSI. They
> both work well otherwise, the normal behavior is restored after switching to
> previous kernel 4.19.0-9
>
> The kernel messages are the following:
>
> Aug 12 16:31:26 ertok kernel: [ 90.951413] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
> Aug 12 16:31:26 ertok kernel: [ 90.951415] PGD 0 P4D 0
> Aug 12 16:31:26 ertok kernel: [ 90.951418] Oops: 0002 [#1] SMP PTI
> Aug 12 16:31:26 ertok kernel: [ 90.951420] CPU: 2 PID: 2140 Comm: libvirtd Not tainted 4.19.0-10-amd64 #1 Debian 4.19.132-1
> Aug 12 16:31:26 ertok kernel: [ 90.951420] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 02/25/2012
> Aug 12 16:31:26 ertok kernel: [ 90.951433] RIP: 0010:gfs2_log_commit+0x104/0x400 [gfs2]
> Aug 12 16:31:26 ertok kernel: [ 90.951435] Code: 60 4c 8d b3 dc 08 00 00 4c 89 f7 e8 f6 90 d2 dd 48 8b 55 70 48 8d 45 70 48 39 d0 74 29 49 8b 4c 24 78 48 8b 75 70 48 8b 55 78 <48> 89 4e 08 48 89 31 49 8d 4c 24 70 48 89 0a 49 89 54 24 78 48 89
> Aug 12 16:31:26 ertok kernel: [ 90.951435] RSP: 0018:ffffad46c24dfba8 EFLAGS: 00010282
> Aug 12 16:31:26 ertok kernel: [ 90.951437] RAX: ffff9caba6487b70 RBX: ffff9cab8bc32000 RCX: 0000000000000000
> Aug 12 16:31:26 ertok kernel: [ 90.951437] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9cab8bc328dc
> Aug 12 16:31:26 ertok kernel: [ 90.951438] RBP: ffff9caba6487b00 R08: ffff9caba6487b58 R09: ffff9caba33fe4d0
> Aug 12 16:31:26 ertok kernel: [ 90.951439] R10: ffff9cab54d8b000 R11: 0000000000000000 R12: ffff9caba94f9200
> Aug 12 16:31:26 ertok kernel: [ 90.951440] R13: ffff9cab8bc327c8 R14: ffff9cab8bc328dc R15: ffffd40b0f4b1740
> Aug 12 16:31:26 ertok kernel: [ 90.951441] FS: 00007f6a7cff9700(0000) GS:ffff9cabaea80000(0000) knlGS:0000000000000000
> Aug 12 16:31:26 ertok kernel: [ 90.951442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 12 16:31:26 ertok kernel: [ 90.951443] CR2: 0000000000000008 CR3: 00000003ebfd6005 CR4: 00000000000606e0
> Aug 12 16:31:26 ertok kernel: [ 90.951443] Call Trace:
> Aug 12 16:31:26 ertok kernel: [ 90.951481] gfs2_trans_end+0x7d/0x160 [gfs2]
> Aug 12 16:31:26 ertok kernel: [ 90.951492] gfs2_dirty_inode+0x1bc/0x240 [gfs2]
> Aug 12 16:31:26 ertok kernel: [ 90.951497] ? iomap_readpage+0x85/0x110
> Aug 12 16:31:26 ertok kernel: [ 90.951506] ? gfs2_dirty_inode+0x144/0x240 [gfs2]
> Aug 12 16:31:26 ertok kernel: [ 90.951511] __mark_inode_dirty+0x1ba/0x380
> Aug 12 16:31:26 ertok kernel: [ 90.951515] generic_update_time+0xb6/0xd0
> Aug 12 16:31:26 ertok kernel: [ 90.951518] touch_atime+0xbe/0xe0
> Aug 12 16:31:26 ertok kernel: [ 90.951522] generic_file_read_iter+0x8ca/0xbc0
> Aug 12 16:31:26 ertok kernel: [ 90.951531] ? gfs2_glock_add_to_lru.part.41+0x7c/0xd0 [gfs2]
> Aug 12 16:31:26 ertok kernel: [ 90.951540] gfs2_file_read_iter+0xe3/0xf0 [gfs2]
> Aug 12 16:31:26 ertok kernel: [ 90.951544] new_sync_read+0xf8/0x160
> Aug 12 16:31:26 ertok kernel: [ 90.951547] vfs_read+0x91/0x140
> Aug 12 16:31:26 ertok kernel: [ 90.951549] ksys_read+0x57/0xd0
> Aug 12 16:31:26 ertok kernel: [ 90.951552] do_syscall_64+0x53/0x110
> Aug 12 16:31:26 ertok kernel: [ 90.951556] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> Aug 12 16:31:26 ertok kernel: [ 90.951558] RIP: 0033:0x7f6a91a43544
> Aug 12 16:31:26 ertok kernel: [ 90.951560] Code: 84 00 00 00 00 00 41 54 49 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 5b fc ff ff 4c 89 e2 48 89 ee 89 df 41 89 c0 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 38 44 89 c7 48 89 44 24 08 e8 97 fc ff ff 48
> Aug 12 16:31:26 ertok kernel: [ 90.951561] RSP: 002b:00007f6a7cff84e0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> Aug 12 16:31:26 ertok kernel: [ 90.951562] RAX: ffffffffffffffda RBX: 0000000000000016 RCX: 00007f6a91a43544
> Aug 12 16:31:26 ertok kernel: [ 90.951564] RDX: 0000000000002000 RSI: 00007f6a40104ed0 RDI: 0000000000000016
> Aug 12 16:31:26 ertok kernel: [ 90.951564] RBP: 00007f6a40104ed0 R08: 0000000000000000 R09: 00007f6a40000560
> Aug 12 16:31:26 ertok kernel: [ 90.951565] R10: 00007f6a400008d0 R11: 0000000000000246 R12: 0000000000002000
> Aug 12 16:31:26 ertok kernel: [ 90.951566] R13: 0000000000000000 R14: 0000000000000016 R15: 0000000000002001
> Aug 12 16:31:26 ertok kernel: [ 90.951568] Modules linked in: gfs2 dlm ses enclosure sctp ip_gre ip_tunnel gre openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl sb_edac ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf serio_raw pcspkr mgag200 evdev ttm iTCO_wdt iTCO_vendor_support drm_kms_helper hpilo sg drm hpwdt i2c_algo_bit ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter button ioatdma pcc_cpufreq dca ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb btrfs zstd_decompress zstd_compress xxhash dm_mod raid10 raid456
> Aug 12 16:31:26 ertok kernel: [ 90.951604] async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid raid6_pq libcrc32c raid1 raid0 multipath linear md_mod sd_mod ata_generic crc32c_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ata_piix psmouse libata hpsa uhci_hcd lpc_ich mfd_core ehci_pci scsi_transport_sas ehci_hcd scsi_mod usbcore tg3 usb_common libphy thermal
> Aug 12 16:31:26 ertok kernel: [ 90.951623] CR2: 0000000000000008
> Aug 12 16:31:26 ertok kernel: [ 90.951626] ---[ end trace eb351a15a6419c5c ]---
> Aug 12 16:31:26 ertok kernel: [ 90.951635] RIP: 0010:gfs2_log_commit+0x104/0x400 [gfs2]
> Aug 12 16:31:26 ertok kernel: [ 90.951636] Code: 60 4c 8d b3 dc 08 00 00 4c 89 f7 e8 f6 90 d2 dd 48 8b 55 70 48 8d 45 70 48 39 d0 74 29 49 8b 4c 24 78 48 8b 75 70 48 8b 55 78 <48> 89 4e 08 48 89 31 49 8d 4c 24 70 48 89 0a 49 89 54 24 78 48 89
> Aug 12 16:31:26 ertok kernel: [ 90.951637] RSP: 0018:ffffad46c24dfba8 EFLAGS: 00010282
> Aug 12 16:31:26 ertok kernel: [ 90.951638] RAX: ffff9caba6487b70 RBX: ffff9cab8bc32000 RCX: 0000000000000000
> Aug 12 16:31:26 ertok kernel: [ 90.951639] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9cab8bc328dc
> Aug 12 16:31:26 ertok kernel: [ 90.951640] RBP: ffff9caba6487b00 R08: ffff9caba6487b58 R09: ffff9caba33fe4d0
> Aug 12 16:31:26 ertok kernel: [ 90.951641] R10: ffff9cab54d8b000 R11: 0000000000000000 R12: ffff9caba94f9200
> Aug 12 16:31:26 ertok kernel: [ 90.951642] R13: ffff9cab8bc327c8 R14: ffff9cab8bc328dc R15: ffffd40b0f4b1740
> Aug 12 16:31:26 ertok kernel: [ 90.951643] FS: 00007f6a7cff9700(0000) GS:ffff9cabaea80000(0000) knlGS:0000000000000000
> Aug 12 16:31:26 ertok kernel: [ 90.951644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 12 16:31:26 ertok kernel: [ 90.951645] CR2: 0000000000000008 CR3: 00000003ebfd6005 CR4: 00000000000606e0
> Aug 12 16:32:16 ertok kernel: [ 140.610569] rcu: INFO: rcu_sched self-detected stall on CPU
> Aug 12 16:32:16 ertok kernel: [ 140.610601] rcu: 3-....: (5249 ticks this GP) idle=3be/1/0x4000000000000002 softirq=11079/11079 fqs=2624
> Aug 12 16:32:16 ertok kernel: [ 140.610636] rcu: (t=5250 jiffies g=18469 q=2151)
> Aug 12 16:32:16 ertok kernel: [ 140.610658] Sending NMI from CPU 3 to CPUs 1:
> Aug 12 16:32:16 ertok kernel: [ 140.610783] NMI backtrace for cpu 1
> Aug 12 16:32:16 ertok kernel: [ 140.610784] CPU: 1 PID: 301 Comm: kworker/1:1H Tainted: G D 4.19.0-10-amd64 #1 Debian 4.19.132-1
> Aug 12 16:32:16 ertok kernel: [ 140.610785] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 02/25/2012
> Aug 12 16:32:16 ertok kernel: [ 140.610785] Workqueue: glock_workqueue glock_work_func [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.610786] RIP: 0010:native_queued_spin_lock_slowpath+0x52/0x190
> Aug 12 16:32:16 ertok kernel: [ 140.610787] Code: 74 37 81 e6 00 ff ff ff 75 5f f0 0f ba 2f 08 8b 07 72 56 89 c2 30 e6 a9 00 00 ff ff 75 47 85 d2 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00
> Aug 12 16:32:16 ertok kernel: [ 140.610787] RSP: 0018:ffffad46c211fbd8 EFLAGS: 00000202
> Aug 12 16:32:16 ertok kernel: [ 140.610788] RAX: 0000000000000101 RBX: ffff9cab8bc32000 RCX: ffff9cab7041a548
> Aug 12 16:32:16 ertok kernel: [ 140.610788] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9cab8bc327c8
> Aug 12 16:32:16 ertok kernel: [ 140.610789] RBP: ffffd40b0f54db80 R08: ffff9cab8cc7dd89 R09: ffff9cab8cc7dd88
> Aug 12 16:32:16 ertok kernel: [ 140.610789] R10: 0000000000040000 R11: 0000000000000000 R12: ffff9cab8bc327c8
> Aug 12 16:32:16 ertok kernel: [ 140.610790] R13: ffff9cab8bc328dc R14: ffffad46c211fcd8 R15: 0000000000000001
> Aug 12 16:32:16 ertok kernel: [ 140.610790] FS: 0000000000000000(0000) GS:ffff9cabaea40000(0000) knlGS:0000000000000000
> Aug 12 16:32:16 ertok kernel: [ 140.610791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 12 16:32:16 ertok kernel: [ 140.610791] CR2: 000055f853c89f50 CR3: 000000009380a004 CR4: 00000000000606e0
> Aug 12 16:32:16 ertok kernel: [ 140.610791] Call Trace:
> Aug 12 16:32:16 ertok kernel: [ 140.610792] _raw_spin_lock+0x1c/0x20
> Aug 12 16:32:16 ertok kernel: [ 140.610792] gfs2_releasepage+0x73/0x1e0 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.610792] truncate_cleanup_page+0x6f/0xc0
> Aug 12 16:32:16 ertok kernel: [ 140.610793] truncate_inode_pages_range+0x1da/0x820
> Aug 12 16:32:16 ertok kernel: [ 140.610793] ? load_balance+0x165/0x9f0
> Aug 12 16:32:16 ertok kernel: [ 140.610793] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610794] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610794] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610794] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610795] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610795] ? native_usergs_sysret64+0x1/0x10
> Aug 12 16:32:16 ertok kernel: [ 140.610795] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610796] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610796] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610796] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610797] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610797] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610797] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610797] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610798] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610798] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610798] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610799] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610799] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610799] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.610800] inode_go_inval+0x4a/0x130 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.610800] do_xmote+0x127/0x1c0 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.610800] glock_work_func+0x5c/0x110 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.610801] process_one_work+0x1a7/0x3a0
> Aug 12 16:32:16 ertok kernel: [ 140.610801] worker_thread+0x30/0x390
> Aug 12 16:32:16 ertok kernel: [ 140.610801] ? create_worker+0x1a0/0x1a0
> Aug 12 16:32:16 ertok kernel: [ 140.610802] kthread+0x112/0x130
> Aug 12 16:32:16 ertok kernel: [ 140.610802] ? kthread_bind+0x30/0x30
> Aug 12 16:32:16 ertok kernel: [ 140.610802] ret_from_fork+0x35/0x40
> Aug 12 16:32:16 ertok kernel: [ 140.611686] NMI backtrace for cpu 3
> Aug 12 16:32:16 ertok kernel: [ 140.611717] CPU: 3 PID: 2119 Comm: gfs2_logd Tainted: G D 4.19.0-10-amd64 #1 Debian 4.19.132-1
> Aug 12 16:32:16 ertok kernel: [ 140.611753] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 02/25/2012
> Aug 12 16:32:16 ertok kernel: [ 140.611779] Call Trace:
> Aug 12 16:32:16 ertok kernel: [ 140.611791] <IRQ>
> Aug 12 16:32:16 ertok kernel: [ 140.611804] dump_stack+0x66/0x90
> Aug 12 16:32:16 ertok kernel: [ 140.612546] nmi_cpu_backtrace.cold.4+0x13/0x50
> Aug 12 16:32:16 ertok kernel: [ 140.613283] ? lapic_can_unplug_cpu.cold.31+0x37/0x37
> Aug 12 16:32:16 ertok kernel: [ 140.614014] nmi_trigger_cpumask_backtrace+0xf9/0xfb
> Aug 12 16:32:16 ertok kernel: [ 140.614743] rcu_dump_cpu_stacks+0x9b/0xcb
> Aug 12 16:32:16 ertok kernel: [ 140.615458] rcu_check_callbacks.cold.81+0x1db/0x335
> Aug 12 16:32:16 ertok kernel: [ 140.616158] ? tick_sched_do_timer+0x60/0x60
> Aug 12 16:32:16 ertok kernel: [ 140.616854] update_process_times+0x28/0x60
> Aug 12 16:32:16 ertok kernel: [ 140.617555] tick_sched_handle+0x22/0x60
> Aug 12 16:32:16 ertok kernel: [ 140.618238] tick_sched_timer+0x37/0x70
> Aug 12 16:32:16 ertok kernel: [ 140.618898] __hrtimer_run_queues+0x100/0x280
> Aug 12 16:32:16 ertok kernel: [ 140.619553] hrtimer_interrupt+0x100/0x220
> Aug 12 16:32:16 ertok kernel: [ 140.620202] ? handle_irq_event+0x47/0x5c
> Aug 12 16:32:16 ertok kernel: [ 140.620822] smp_apic_timer_interrupt+0x6a/0x140
> Aug 12 16:32:16 ertok kernel: [ 140.621418] apic_timer_interrupt+0xf/0x20
> Aug 12 16:32:16 ertok kernel: [ 140.622022] </IRQ>
> Aug 12 16:32:16 ertok kernel: [ 140.622596] RIP: 0010:native_queued_spin_lock_slowpath+0x52/0x190
> Aug 12 16:32:16 ertok kernel: [ 140.623203] Code: 74 37 81 e6 00 ff ff ff 75 5f f0 0f ba 2f 08 8b 07 72 56 89 c2 30 e6 a9 00 00 ff ff 75 47 85 d2 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00
> Aug 12 16:32:16 ertok kernel: [ 140.624500] RSP: 0018:ffffad46c22cfe38 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
> Aug 12 16:32:16 ertok kernel: [ 140.625164] RAX: 0000000000000101 RBX: ffff9cab8bc32000 RCX: 0000000000000000
> Aug 12 16:32:16 ertok kernel: [ 140.625830] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9cab8bc328dc
> Aug 12 16:32:16 ertok kernel: [ 140.626471] RBP: ffff9cab8bc32050 R08: ffff9cabaeae2ae0 R09: 0000000000000000
> Aug 12 16:32:16 ertok kernel: [ 140.627371] R10: 0000000000000000 R11: 0000001bf5fa0fe4 R12: ffff9cab8bc32000
> Aug 12 16:32:16 ertok kernel: [ 140.628022] R13: ffff9cab8bc32838 R14: ffff9cab8bc32960 R15: ffffffffc0c08200
> Aug 12 16:32:16 ertok kernel: [ 140.628703] ? gfs2_log_flush+0x6d0/0x6d0 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.629367] ? schedule_timeout+0x173/0x390
> Aug 12 16:32:16 ertok kernel: [ 140.630028] _raw_spin_lock+0x1c/0x20
> Aug 12 16:32:16 ertok kernel: [ 140.630706] gfs2_ail1_empty+0x2a/0x290 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.631382] ? __next_timer_interrupt+0xc0/0xc0
> Aug 12 16:32:16 ertok kernel: [ 140.632065] ? gfs2_log_flush+0x6d0/0x6d0 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.632753] gfs2_logd+0xa8/0x2f0 [gfs2]
> Aug 12 16:32:16 ertok kernel: [ 140.633445] ? finish_wait+0x80/0x80
> Aug 12 16:32:16 ertok kernel: [ 140.634140] kthread+0x112/0x130
> Aug 12 16:32:16 ertok kernel: [ 140.634835] ? kthread_bind+0x30/0x30
> Aug 12 16:32:16 ertok kernel: [ 140.635519] ret_from_fork+0x35/0x40
> Aug 12 16:32:43 ertok kernel: [ 167.834672] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:1H:301]
> Aug 12 16:32:43 ertok kernel: [ 167.835411] Modules linked in: gfs2 dlm ses enclosure sctp ip_gre ip_tunnel gre openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl sb_edac ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf serio_raw pcspkr mgag200 evdev ttm iTCO_wdt iTCO_vendor_support drm_kms_helper hpilo sg drm hpwdt i2c_algo_bit ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter button ioatdma pcc_cpufreq dca ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb btrfs zstd_decompress zstd_compress xxhash dm_mod raid10 raid456
> Aug 12 16:32:43 ertok kernel: [ 167.840671] async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid raid6_pq libcrc32c raid1 raid0 multipath linear md_mod sd_mod ata_generic crc32c_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ata_piix psmouse libata hpsa uhci_hcd lpc_ich mfd_core ehci_pci scsi_transport_sas ehci_hcd scsi_mod usbcore tg3 usb_common libphy thermal
> Aug 12 16:32:43 ertok kernel: [ 167.842665] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [gfs2_logd:2119]
> Aug 12 16:32:43 ertok kernel: [ 167.843836] CPU: 1 PID: 301 Comm: kworker/1:1H Tainted: G D 4.19.0-10-amd64 #1 Debian 4.19.132-1
> Aug 12 16:32:43 ertok kernel: [ 167.844951] Modules linked in: gfs2 dlm ses enclosure sctp ip_gre ip_tunnel gre openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl sb_edac ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf serio_raw pcspkr mgag200 evdev ttm iTCO_wdt iTCO_vendor_support drm_kms_helper hpilo sg drm hpwdt i2c_algo_bit ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter button ioatdma pcc_cpufreq dca ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb btrfs zstd_decompress zstd_compress xxhash dm_mod raid10 raid456
> Aug 12 16:32:43 ertok kernel: [ 167.846099] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 02/25/2012
> Aug 12 16:32:43 ertok kernel: [ 167.853595] async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic usbhid hid raid6_pq libcrc32c raid1 raid0 multipath linear md_mod sd_mod ata_generic crc32c_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper ata_piix psmouse libata hpsa uhci_hcd lpc_ich mfd_core ehci_pci scsi_transport_sas ehci_hcd scsi_mod usbcore tg3 usb_common libphy thermal
> Aug 12 16:32:43 ertok kernel: [ 167.854955] Workqueue: glock_workqueue glock_work_func [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.859166] CPU: 3 PID: 2119 Comm: gfs2_logd Tainted: G D 4.19.0-10-amd64 #1 Debian 4.19.132-1
> Aug 12 16:32:43 ertok kernel: [ 167.860625] RIP: 0010:native_queued_spin_lock_slowpath+0x52/0x190
> Aug 12 16:32:43 ertok kernel: [ 167.862066] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 02/25/2012
> Aug 12 16:32:43 ertok kernel: [ 167.862069] RIP: 0010:native_queued_spin_lock_slowpath+0x54/0x190
> Aug 12 16:32:43 ertok kernel: [ 167.863507] Code: 74 37 81 e6 00 ff ff ff 75 5f f0 0f ba 2f 08 8b 07 72 56 89 c2 30 e6 a9 00 00 ff ff 75 47 85 d2 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00
> Aug 12 16:32:43 ertok kernel: [ 167.864959] Code: 81 e6 00 ff ff ff 75 5f f0 0f ba 2f 08 8b 07 72 56 89 c2 30 e6 a9 00 00 ff ff 75 47 85 d2 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00 00 75
> Aug 12 16:32:43 ertok kernel: [ 167.866408] RSP: 0018:ffffad46c211fbd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
> Aug 12 16:32:43 ertok kernel: [ 167.869414] RSP: 0018:ffffad46c22cfe38 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
> Aug 12 16:32:43 ertok kernel: [ 167.872552] RAX: 0000000000000101 RBX: ffff9cab8bc32000 RCX: ffff9cab7041a548
> Aug 12 16:32:43 ertok kernel: [ 167.872553] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9cab8bc327c8
> Aug 12 16:32:43 ertok kernel: [ 167.874140] RAX: 0000000000000101 RBX: ffff9cab8bc32000 RCX: 0000000000000000
> Aug 12 16:32:43 ertok kernel: [ 167.874141] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9cab8bc328dc
> Aug 12 16:32:43 ertok kernel: [ 167.875733] RBP: ffffd40b0f54db80 R08: ffff9cab8cc7dd89 R09: ffff9cab8cc7dd88
> Aug 12 16:32:43 ertok kernel: [ 167.877562] RBP: ffff9cab8bc32050 R08: ffff9cabaeae2ae0 R09: 0000000000000000
> Aug 12 16:32:43 ertok kernel: [ 167.877564] R10: 0000000000000000 R11: 0000001bf5fa0fe4 R12: ffff9cab8bc32000
> Aug 12 16:32:43 ertok kernel: [ 167.879155] R10: 0000000000040000 R11: 0000000000000000 R12: ffff9cab8bc327c8
> Aug 12 16:32:43 ertok kernel: [ 167.880750] R13: ffff9cab8bc32838 R14: ffff9cab8bc32960 R15: ffffffffc0c08200
> Aug 12 16:32:43 ertok kernel: [ 167.880758] FS: 0000000000000000(0000) GS:ffff9cabaeac0000(0000) knlGS:0000000000000000
> Aug 12 16:32:43 ertok kernel: [ 167.882358] R13: ffff9cab8bc328dc R14: ffffad46c211fcd8 R15: 0000000000000001
> Aug 12 16:32:43 ertok kernel: [ 167.882359] FS: 0000000000000000(0000) GS:ffff9cabaea40000(0000) knlGS:0000000000000000
> Aug 12 16:32:43 ertok kernel: [ 167.883971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 12 16:32:43 ertok kernel: [ 167.883973] CR2: 0000559643664290 CR3: 000000009380a006 CR4: 00000000000606e0
> Aug 12 16:32:43 ertok kernel: [ 167.885573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Aug 12 16:32:43 ertok kernel: [ 167.885574] CR2: 000055f853c89f50 CR3: 000000009380a004 CR4: 00000000000606e0
> Aug 12 16:32:43 ertok kernel: [ 167.887202] Call Trace:
> Aug 12 16:32:43 ertok kernel: [ 167.888794] Call Trace:
> Aug 12 16:32:43 ertok kernel: [ 167.890407] _raw_spin_lock+0x1c/0x20
> Aug 12 16:32:43 ertok kernel: [ 167.892013] _raw_spin_lock+0x1c/0x20
> Aug 12 16:32:43 ertok kernel: [ 167.893653] gfs2_ail1_empty+0x2a/0x290 [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.895264] gfs2_releasepage+0x73/0x1e0 [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.896833] ? __next_timer_interrupt+0xc0/0xc0
> Aug 12 16:32:43 ertok kernel: [ 167.898405] truncate_cleanup_page+0x6f/0xc0
> Aug 12 16:32:43 ertok kernel: [ 167.899966] ? gfs2_log_flush+0x6d0/0x6d0 [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.901542] truncate_inode_pages_range+0x1da/0x820
> Aug 12 16:32:43 ertok kernel: [ 167.903102] gfs2_logd+0xa8/0x2f0 [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.904666] ? load_balance+0x165/0x9f0
> Aug 12 16:32:43 ertok kernel: [ 167.906211] ? finish_wait+0x80/0x80
> Aug 12 16:32:43 ertok kernel: [ 167.907747] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.909286] kthread+0x112/0x130
> Aug 12 16:32:43 ertok kernel: [ 167.910798] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.912350] ? kthread_bind+0x30/0x30
> Aug 12 16:32:43 ertok kernel: [ 167.913868] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.915411] ret_from_fork+0x35/0x40
> Aug 12 16:32:43 ertok kernel: [ 167.916948] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.916949] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.933055] ? native_usergs_sysret64+0x1/0x10
> Aug 12 16:32:43 ertok kernel: [ 167.934345] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.935613] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.936832] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.938045] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.939219] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.940367] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.941467] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.942529] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.943553] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.944531] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.945476] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.946401] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.947289] ? __switch_to_asm+0x35/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.948144] ? __switch_to_asm+0x41/0x70
> Aug 12 16:32:43 ertok kernel: [ 167.948962] inode_go_inval+0x4a/0x130 [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.949772] do_xmote+0x127/0x1c0 [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.950584] glock_work_func+0x5c/0x110 [gfs2]
> Aug 12 16:32:43 ertok kernel: [ 167.951371] process_one_work+0x1a7/0x3a0
> Aug 12 16:32:43 ertok kernel: [ 167.952151] worker_thread+0x30/0x390
> Aug 12 16:32:43 ertok kernel: [ 167.952917] ? create_worker+0x1a0/0x1a0
> Aug 12 16:32:43 ertok kernel: [ 167.953659] kthread+0x112/0x130
> Aug 12 16:32:43 ertok kernel: [ 167.954358] ? kthread_bind+0x30/0x30
> Aug 12 16:32:43 ertok kernel: [ 167.955038] ret_from_fork+0x35/0x40
> [...]
This looks similar to what a user reported in the linux-cluster
mailinglist running 5.4.58. See
https://www.redhat.com/archives/linux-cluster/2020-August/msg00000.html
Would you be able to do two things: Is e.g. running 5.8.7-1 from
unstable still exposing the issue? What about the most recent release
in the v4.19.y series?
Would you be able to bisect the upstream changes between 4.19.118 and
4.19.132 to determine which commit introduced the issue? If this can
detemrined then ideally you could report it directly to upstream
(keeping us in the loop).
Some hints are in
https://wiki.debian.org/DebianKernelReportingBugs, specifically
https://wiki.debian.org/DebianKernelReportingBugs#Identifying_when_the_bug_was_introduced
and https://wiki.debian.org/DebianKernel/GitBisect .
Regards,
Salvatore
Reply to: