[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#1120718: linux-image-6.12.43+deb13-amd64: hung tasks on mutex acquisition during rtnetlink ops



Control: tags -1 + moreinfo

On Fri, Nov 14, 2025 at 11:40:13PM -0800, Nick Owens wrote:
> Package: src:linux
> Version: 6.12.43-1
> Severity: important
> 
> Dear Maintainer,
> 
>    * What led up to the situation?
> 	machine is a server on commodity hardware. it runs some vms, and runs prometheus/node-exporter/grafana.
>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
> 	the machine is left on like a server for many days at a time.
>    * What was the outcome of this action?
> 	after some days, the machine locks up. i can't login over ssh and the console is unusable. there are many reports of hung tasks in the kernel that appear to be blocked on mutex acquisition.
>    * What outcome did you expect instead?
> 	no hang.
> 
> 
> -- Package-specific info:
> ** Version:
> Linux version 6.12.43+deb13-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC Debian 6.12.43-1 (2025-08-27)
> 
> ** Command line:
> BOOT_IMAGE=/boot/vmlinuz-6.12.43+deb13-amd64 root=/dev/mapper/root-root ro console=tty0 console=ttyS0,115200 rcu_nocbs=0-15
> 
> ** Not tainted
> 
> ** Kernel log:
> (note: i am attaching logs of a boot that hung here)
> 
> Sep 28 16:17:22 sokoban kernel: INFO: task systemd-network:1260549 blocked for more than 120 seconds.
> Sep 28 16:17:22 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:22 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:22 sokoban kernel: task:systemd-network state:D stack:0     pid:1260549 tgid:1260549 ppid:1      flags:0x00000006
> Sep 28 16:17:22 sokoban kernel: Call Trace:
> Sep 28 16:17:22 sokoban kernel:  <TASK>
> Sep 28 16:17:22 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:22 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:22 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:22 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:22 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:22 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:22 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:22 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:22 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:22 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:22 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:22 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:22 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:22 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:22 sokoban kernel:  __sys_sendto+0x1df/0x1f0
> Sep 28 16:17:22 sokoban kernel:  __x64_sys_sendto+0x24/0x30
> Sep 28 16:17:22 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:22 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:22 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:22 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:22 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:22 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:22 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:22 sokoban kernel: RIP: 0033:0x7fcfe471e643
> Sep 28 16:17:22 sokoban kernel: RSP: 002b:00007ffe0821c038 EFLAGS: 00000202 ORIG_RAX: 000000000000002c
> Sep 28 16:17:22 sokoban kernel: RAX: ffffffffffffffda RBX: 000055a3f48a2c90 RCX: 00007fcfe471e643
> Sep 28 16:17:22 sokoban kernel: RDX: 0000000000000020 RSI: 000055a3f48f3670 RDI: 0000000000000003
> Sep 28 16:17:22 sokoban kernel: RBP: 000055a3f48f3240 R08: 00007ffe0821c040 R09: 0000000000000080
> Sep 28 16:17:22 sokoban kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffe0821c104
> Sep 28 16:17:22 sokoban kernel: R13: 00007ffe0821c170 R14: 000055a3f48a0f50 R15: 0000000000000003
> Sep 28 16:17:22 sokoban kernel:  </TASK>
> Sep 28 16:17:22 sokoban kernel: INFO: task prometheus-node:1270917 blocked for more than 121 seconds.
> Sep 28 16:17:23 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:23 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:23 sokoban kernel: task:prometheus-node state:D stack:0     pid:1270917 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:23 sokoban kernel: Call Trace:
> Sep 28 16:17:23 sokoban kernel:  <TASK>
> Sep 28 16:17:23 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:23 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:23 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:23 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:23 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:23 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:23 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:23 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:23 sokoban kernel:  __sys_sendto+0x1df/0x1f0
> Sep 28 16:17:23 sokoban kernel:  __x64_sys_sendto+0x24/0x30
> Sep 28 16:17:23 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? release_sock+0x1d/0xa0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? netlink_insert+0x64/0x4a0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? __wake_up+0x44/0x60
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? netlink_bind+0x32c/0x390
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? __sys_bind+0xba/0xf0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? __count_memcg_events+0x58/0xf0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? count_memcg_events.constprop.0+0x1a/0x30
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? handle_mm_fault+0xae/0x320
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? do_user_addr_fault+0x33a/0x6a0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:23 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:23 sokoban kernel: RSP: 002b:000000c0002a95e0 EFLAGS: 00000216 ORIG_RAX: 000000000000002c
> Sep 28 16:17:23 sokoban kernel: RAX: ffffffffffffffda RBX: 000000000000000f RCX: 0000000000483e4e
> Sep 28 16:17:23 sokoban kernel: RDX: 0000000000000011 RSI: 000000c0002314e8 RDI: 000000000000000f
> Sep 28 16:17:23 sokoban kernel: RBP: 000000c0002a9620 R08: 000000c0002314dc R09: 000000000000000c
> Sep 28 16:17:23 sokoban kernel: R10: 0000000000000000 R11: 0000000000000216 R12: 000000c0002314e8
> Sep 28 16:17:23 sokoban kernel: R13: 0000000000000000 R14: 000000c0002f3dc0 R15: 00000000ffffffff
> Sep 28 16:17:23 sokoban kernel:  </TASK>
> Sep 28 16:17:23 sokoban kernel: INFO: task prometheus-node:1270936 blocked for more than 121 seconds.
> Sep 28 16:17:23 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:23 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:23 sokoban kernel: task:prometheus-node state:D stack:0     pid:1270936 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:23 sokoban kernel: Call Trace:
> Sep 28 16:17:23 sokoban kernel:  <TASK>
> Sep 28 16:17:23 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:23 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:23 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:23 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:23 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:23 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:23 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:23 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:23 sokoban kernel:  __sys_sendto+0x1df/0x1f0
> Sep 28 16:17:23 sokoban kernel:  __x64_sys_sendto+0x24/0x30
> Sep 28 16:17:23 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? __sys_bind+0xba/0xf0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:23 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:23 sokoban kernel: RSP: 002b:000000c0004695e0 EFLAGS: 00000216 ORIG_RAX: 000000000000002c
> Sep 28 16:17:23 sokoban kernel: RAX: ffffffffffffffda RBX: 000000000000000b RCX: 0000000000483e4e
> Sep 28 16:17:23 sokoban kernel: RDX: 0000000000000011 RSI: 000000c000411530 RDI: 000000000000000b
> Sep 28 16:17:23 sokoban kernel: RBP: 000000c000469620 R08: 000000c000411524 R09: 000000000000000c
> Sep 28 16:17:23 sokoban kernel: R10: 0000000000000000 R11: 0000000000000216 R12: 000000c000411530
> Sep 28 16:17:23 sokoban kernel: R13: 0000000000000000 R14: 000000c000105880 R15: 000000001fffffff
> Sep 28 16:17:23 sokoban kernel:  </TASK>
> Sep 28 16:17:23 sokoban kernel: INFO: task prometheus-node:1270938 blocked for more than 121 seconds.
> Sep 28 16:17:23 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:23 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:23 sokoban kernel: task:prometheus-node state:D stack:0     pid:1270938 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:23 sokoban kernel: Call Trace:
> Sep 28 16:17:23 sokoban kernel:  <TASK>
> Sep 28 16:17:23 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:23 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:23 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:23 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:23 sokoban kernel:  ? __alloc_skb+0xde/0x1a0
> Sep 28 16:17:23 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:23 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:23 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:23 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:23 sokoban kernel:  ____sys_sendmsg+0x399/0x3d0
> Sep 28 16:17:23 sokoban kernel:  ? copy_msghdr_from_user+0x7d/0xc0
> Sep 28 16:17:23 sokoban kernel:  ___sys_sendmsg+0x9a/0xe0
> Sep 28 16:17:23 sokoban kernel:  __sys_sendmsg+0x7a/0xd0
> Sep 28 16:17:23 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? __count_memcg_events+0x58/0xf0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? count_memcg_events.constprop.0+0x1a/0x30
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? handle_mm_fault+0xae/0x320
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? do_user_addr_fault+0x33a/0x6a0
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:23 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:23 sokoban kernel: RSP: 002b:000000c0001e31f8 EFLAGS: 00000206 ORIG_RAX: 000000000000002e
> Sep 28 16:17:23 sokoban kernel: RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 0000000000483e4e
> Sep 28 16:17:23 sokoban kernel: RDX: 0000000000000000 RSI: 000000c0001e3358 RDI: 0000000000000008
> Sep 28 16:17:23 sokoban kernel: RBP: 000000c0001e3238 R08: 0000000000000000 R09: 0000000000000000
> Sep 28 16:17:23 sokoban kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
> Sep 28 16:17:23 sokoban kernel: R13: 000000c000100428 R14: 000000c00067ae00 R15: 0000000000000000
> Sep 28 16:17:23 sokoban kernel:  </TASK>
> Sep 28 16:17:23 sokoban kernel: INFO: task prometheus-node:1728523 blocked for more than 121 seconds.
> Sep 28 16:17:23 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:23 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:23 sokoban kernel: task:prometheus-node state:D stack:0     pid:1728523 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:23 sokoban kernel: Call Trace:
> Sep 28 16:17:23 sokoban kernel:  <TASK>
> Sep 28 16:17:23 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:23 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:23 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:23 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:23 sokoban kernel:  ? __alloc_skb+0xde/0x1a0
> Sep 28 16:17:23 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:23 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:23 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:23 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:23 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:23 sokoban kernel:  ____sys_sendmsg+0x399/0x3d0
> Sep 28 16:17:23 sokoban kernel:  ? copy_msghdr_from_user+0x7d/0xc0
> Sep 28 16:17:23 sokoban kernel:  ___sys_sendmsg+0x9a/0xe0
> Sep 28 16:17:23 sokoban kernel:  __sys_sendmsg+0x7a/0xd0
> Sep 28 16:17:23 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:23 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:23 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:23 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:23 sokoban kernel: RSP: 002b:000000c0004671f8 EFLAGS: 00000206 ORIG_RAX: 000000000000002e
> Sep 28 16:17:23 sokoban kernel: RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 0000000000483e4e
> Sep 28 16:17:23 sokoban kernel: RDX: 0000000000000000 RSI: 000000c000467358 RDI: 0000000000000009
> Sep 28 16:17:23 sokoban kernel: RBP: 000000c000467238 R08: 0000000000000000 R09: 0000000000000000
> Sep 28 16:17:23 sokoban kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
> Sep 28 16:17:23 sokoban kernel: R13: 000000c000212428 R14: 000000c0002f2700 R15: 0000000000000000
> Sep 28 16:17:23 sokoban kernel:  </TASK>
> Sep 28 16:17:23 sokoban kernel: INFO: task prometheus-node:2036056 blocked for more than 122 seconds.
> Sep 28 16:17:23 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:23 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:24 sokoban kernel: task:prometheus-node state:D stack:0     pid:2036056 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:24 sokoban kernel: Call Trace:
> Sep 28 16:17:24 sokoban kernel:  <TASK>
> Sep 28 16:17:24 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:24 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:24 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:24 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:24 sokoban kernel:  ? __alloc_skb+0xde/0x1a0
> Sep 28 16:17:24 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:24 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:24 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:24 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:24 sokoban kernel:  __sys_sendto+0x1df/0x1f0
> Sep 28 16:17:24 sokoban kernel:  __x64_sys_sendto+0x24/0x30
> Sep 28 16:17:24 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __sys_socket+0xd4/0x110
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __sys_socket+0xd4/0x110
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:24 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:24 sokoban kernel: RSP: 002b:000000c0008595e0 EFLAGS: 00000216 ORIG_RAX: 000000000000002c
> Sep 28 16:17:24 sokoban kernel: RAX: ffffffffffffffda RBX: 0000000000000012 RCX: 0000000000483e4e
> Sep 28 16:17:24 sokoban kernel: RDX: 0000000000000011 RSI: 000000c000315b30 RDI: 0000000000000012
> Sep 28 16:17:24 sokoban kernel: RBP: 000000c000859620 R08: 000000c000315b24 R09: 000000000000000c
> Sep 28 16:17:24 sokoban kernel: R10: 0000000000000000 R11: 0000000000000216 R12: 000000c000315b30
> Sep 28 16:17:24 sokoban kernel: R13: 0000000000000000 R14: 000000c0002f2540 R15: 0000000019def97b
> Sep 28 16:17:24 sokoban kernel:  </TASK>
> Sep 28 16:17:24 sokoban kernel: INFO: task prometheus-node:2036080 blocked for more than 122 seconds.
> Sep 28 16:17:24 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:24 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:24 sokoban kernel: task:prometheus-node state:D stack:0     pid:2036080 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:24 sokoban kernel: Call Trace:
> Sep 28 16:17:24 sokoban kernel:  <TASK>
> Sep 28 16:17:24 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:24 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:24 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:24 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:24 sokoban kernel:  ? __alloc_skb+0xde/0x1a0
> Sep 28 16:17:24 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:24 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:24 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:24 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:24 sokoban kernel:  ____sys_sendmsg+0x399/0x3d0
> Sep 28 16:17:24 sokoban kernel:  ? copy_msghdr_from_user+0x7d/0xc0
> Sep 28 16:17:24 sokoban kernel:  ___sys_sendmsg+0x9a/0xe0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  __sys_sendmsg+0x7a/0xd0
> Sep 28 16:17:24 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __sys_getsockname+0xd7/0xf0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __wake_up+0x44/0x60
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? netlink_bind+0x32c/0x390
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __sys_bind+0xba/0xf0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:24 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:24 sokoban kernel: RSP: 002b:000000c0004651f8 EFLAGS: 00000206 ORIG_RAX: 000000000000002e
> Sep 28 16:17:24 sokoban kernel: RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 0000000000483e4e
> Sep 28 16:17:24 sokoban kernel: RDX: 0000000000000000 RSI: 000000c000465358 RDI: 0000000000000015
> Sep 28 16:17:24 sokoban kernel: RBP: 000000c000465238 R08: 0000000000000000 R09: 0000000000000000
> Sep 28 16:17:24 sokoban kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
> Sep 28 16:17:24 sokoban kernel: R13: 000000c000078728 R14: 000000c0002f2c40 R15: 0000000000000000
> Sep 28 16:17:24 sokoban kernel:  </TASK>
> Sep 28 16:17:24 sokoban kernel: INFO: task prometheus-node:2036081 blocked for more than 122 seconds.
> Sep 28 16:17:24 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:24 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:24 sokoban kernel: task:prometheus-node state:D stack:0     pid:2036081 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:24 sokoban kernel: Call Trace:
> Sep 28 16:17:24 sokoban kernel:  <TASK>
> Sep 28 16:17:24 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:24 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:24 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:24 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:24 sokoban kernel:  ? __alloc_skb+0xde/0x1a0
> Sep 28 16:17:24 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:24 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:24 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:24 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:24 sokoban kernel:  ____sys_sendmsg+0x399/0x3d0
> Sep 28 16:17:24 sokoban kernel:  ? copy_msghdr_from_user+0x7d/0xc0
> Sep 28 16:17:24 sokoban kernel:  ___sys_sendmsg+0x9a/0xe0
> Sep 28 16:17:24 sokoban kernel:  __sys_sendmsg+0x7a/0xd0
> Sep 28 16:17:24 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? add_wait_queue+0x68/0xa0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? ep_ptable_queue_proc+0x68/0x90
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? release_sock+0x1d/0xa0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? netlink_insert+0x64/0x4a0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __wake_up+0x44/0x60
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? netlink_bind+0x32c/0x390
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __sys_bind+0xba/0xf0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __x64_sys_fcntl+0x85/0xd0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:24 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:24 sokoban kernel: RSP: 002b:000000c0008531f8 EFLAGS: 00000206 ORIG_RAX: 000000000000002e
> Sep 28 16:17:24 sokoban kernel: RAX: ffffffffffffffda RBX: 000000000000001a RCX: 0000000000483e4e
> Sep 28 16:17:24 sokoban kernel: RDX: 0000000000000000 RSI: 000000c000853358 RDI: 000000000000001a
> Sep 28 16:17:24 sokoban kernel: RBP: 000000c000853238 R08: 0000000000000000 R09: 0000000000000000
> Sep 28 16:17:24 sokoban kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
> Sep 28 16:17:24 sokoban kernel: R13: 000000c000078e28 R14: 000000c0002fc700 R15: 0000000000000000
> Sep 28 16:17:24 sokoban kernel:  </TASK>
> Sep 28 16:17:24 sokoban kernel: INFO: task prometheus-node:2036082 blocked for more than 122 seconds.
> Sep 28 16:17:24 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:24 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:24 sokoban kernel: task:prometheus-node state:D stack:0     pid:2036082 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:24 sokoban kernel: Call Trace:
> Sep 28 16:17:24 sokoban kernel:  <TASK>
> Sep 28 16:17:24 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:24 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:24 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:24 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:24 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:24 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:24 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:24 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:24 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:24 sokoban kernel:  __sys_sendto+0x1df/0x1f0
> Sep 28 16:17:24 sokoban kernel:  __x64_sys_sendto+0x24/0x30
> Sep 28 16:17:24 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? __fdget+0xb6/0xe0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? __sys_recvmsg+0xab/0xd0
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:24 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:24 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? __sys_getsockopt+0x83/0xe0
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? __sys_getsockopt+0xd0/0xe0
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? kmem_cache_free+0x3dd/0x400
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:25 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:25 sokoban kernel: RSP: 002b:000000c000b595e0 EFLAGS: 00000216 ORIG_RAX: 000000000000002c
> Sep 28 16:17:25 sokoban kernel: RAX: ffffffffffffffda RBX: 000000000000001d RCX: 0000000000483e4e
> Sep 28 16:17:25 sokoban kernel: RDX: 0000000000000011 RSI: 000000c0003e79f8 RDI: 000000000000001d
> Sep 28 16:17:25 sokoban kernel: RBP: 000000c000b59620 R08: 000000c0003e79ec R09: 000000000000000c
> Sep 28 16:17:25 sokoban kernel: R10: 0000000000000000 R11: 0000000000000216 R12: 000000c0003e79f8
> Sep 28 16:17:25 sokoban kernel: R13: 0000000000000000 R14: 000000c000308c40 R15: 000003f7ffffffff
> Sep 28 16:17:25 sokoban kernel:  </TASK>
> Sep 28 16:17:25 sokoban kernel: INFO: task prometheus-node:2036180 blocked for more than 123 seconds.
> Sep 28 16:17:25 sokoban kernel:       Tainted: G           OE      6.10.3-amd64 #1 Debian 6.10.3-1
> Sep 28 16:17:25 sokoban kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 28 16:17:25 sokoban kernel: task:prometheus-node state:D stack:0     pid:2036180 tgid:1270917 ppid:1      flags:0x00000006
> Sep 28 16:17:25 sokoban kernel: Call Trace:
> Sep 28 16:17:25 sokoban kernel:  <TASK>
> Sep 28 16:17:25 sokoban kernel:  __schedule+0x3eb/0xb40
> Sep 28 16:17:25 sokoban kernel:  schedule+0x27/0xf0
> Sep 28 16:17:25 sokoban kernel:  schedule_preempt_disabled+0x15/0x30
> Sep 28 16:17:25 sokoban kernel:  __mutex_lock.constprop.0+0x339/0x630
> Sep 28 16:17:25 sokoban kernel:  ? __alloc_skb+0xde/0x1a0
> Sep 28 16:17:25 sokoban kernel:  netlink_dump+0x338/0x3b0
> Sep 28 16:17:25 sokoban kernel:  __netlink_dump_start+0x1dd/0x2a0
> Sep 28 16:17:25 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:25 sokoban kernel:  rtnetlink_rcv_msg+0x2ac/0x3f0
> Sep 28 16:17:25 sokoban kernel:  ? __pfx_rtnl_dumpit+0x10/0x10
> Sep 28 16:17:25 sokoban kernel:  ? __pfx_rtnl_dump_ifinfo+0x10/0x10
> Sep 28 16:17:25 sokoban kernel:  ? __pfx_rtnetlink_rcv_msg+0x10/0x10
> Sep 28 16:17:25 sokoban kernel:  netlink_rcv_skb+0x5b/0x110
> Sep 28 16:17:25 sokoban kernel:  netlink_unicast+0x19c/0x290
> Sep 28 16:17:25 sokoban kernel:  netlink_sendmsg+0x222/0x490
> Sep 28 16:17:25 sokoban kernel:  __sys_sendto+0x1df/0x1f0
> Sep 28 16:17:25 sokoban kernel:  __x64_sys_sendto+0x24/0x30
> Sep 28 16:17:25 sokoban kernel:  do_syscall_64+0x82/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? kmem_cache_alloc_noprof+0x118/0x330
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? apparmor_file_alloc_security+0x44/0x1d0
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? security_file_alloc+0x61/0xf0
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? file_init_path+0x74/0x140
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? alloc_file_pseudo+0xbf/0x110
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? __sys_socket+0xd4/0x110
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? syscall_exit_to_user_mode+0x77/0x210
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? do_syscall_64+0x8e/0x190
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  ? srso_return_thunk+0x5/0x5f
> Sep 28 16:17:25 sokoban kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> Sep 28 16:17:25 sokoban kernel: RIP: 0033:0x483e4e
> Sep 28 16:17:25 sokoban kernel: RSP: 002b:000000c000b5d5e0 EFLAGS: 00000216 ORIG_RAX: 000000000000002c
> Sep 28 16:17:25 sokoban kernel: RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 0000000000483e4e
> Sep 28 16:17:25 sokoban kernel: RDX: 0000000000000011 RSI: 000000c0000d6378 RDI: 0000000000000018
> Sep 28 16:17:25 sokoban kernel: RBP: 000000c000b5d620 R08: 000000c0000d636c R09: 000000000000000c
> Sep 28 16:17:25 sokoban kernel: R10: 0000000000000000 R11: 0000000000000216 R12: 000000c0000d6378
> Sep 28 16:17:25 sokoban kernel: R13: 0000000000000000 R14: 000000c0004d4e00 R15: 0000000003ffffff
> Sep 28 16:17:25 sokoban kernel:  </TASK>
> Sep 28 16:17:25 sokoban kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings

Is this regressing from an earlier kernel? If yes can you isolate the
range more closely?

Additionally 6.12.43-1 is not the current trixie kernel, do the hangs
are reproducible with 6.12.48-1 (and the today to be released via the
13.2 point release) 6.12.57-1?

Regards,
Salvatore


Reply to: