AW: Debugging initramfs, server hangs during boot process

["thahn01@t-online.de" <thahn01@t-online.de>]

I cannot catch the user space output on the broken machine 

To get netconsole going I have edited  /usr/share/initramfs-tools/scripts/init-top/udev

First to load the network module.
2nd to load the netconsole:

---snip----
echo modprobe comes next ...
sleep 9
set -x
modprobe r8169
lsmod|grep r8169

modprobe netconsole netconsole="6666@192.168.178.62/enp3s0,6666@192.168.178.54/04:92:26:d9:4e:9e"
res=$?
if [ "$res" -eq 0 ]; then
       echo modprobe worked
else
        echo modprobe failed eith $res
fi
---snap----

The netconsole output from the kernel:

[   28.109866] r8169 0000:03:00.0: firmware: direct-loading firmware rtl_nic/rtl8125b-2.fw
[   28.141085] RTL8226B_RTL8221B 2.5Gbps PHY r8169-0-300:00: attached PHY driver (mii_bus:phy_addr=r8169-0-300:00, irq=MAC)
[   28.341236] r8169 0000:03:00.0 enp3s0: Link is Down
[   30.013941] r8169 0000:03:00.0 enp3s0: Link is Up - 100Mbps/Full - flow control rx/tx
[   30.014760] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
[   30.021222] printk: console [netcon0] enabled
[   30.024682] netconsole: network logging started
[   68.102701] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0
[   68.102771] ACPI: button: Sleep Button [SLPB]
[   68.102874] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[   68.102964] ACPI: button: Power Button [PWRB]
[   68.103014] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[   68.103079] ACPI: button: Power Button [PWRF]
[   68.158125] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[   68.160436] i2c i2c-0: 4/4 memory slots populated (from DMI)
[   68.160795] i2c i2c-0: Successfully instantiated SPD at 0x50
[   68.161131] i2c i2c-0: Successfully instantiated SPD at 0x51
[   68.161513] i2c i2c-0: Successfully instantiated SPD at 0x52
[   68.161895] i2c i2c-0: Successfully instantiated SPD at 0x53
[   68.166974] ACPI: bus type USB registered
[   68.166994] usbcore: registered new interface driver usbfs
[   68.166998] usbcore: registered new interface driver hub
[   68.167054] usbcore: registered new device driver usb
[   68.170281] SCSI subsystem initialized
[   68.183217] cryptd: max_cpu_qlen set to 1000
[   68.183493] ACPI: bus type drm_connector registered
[   68.195886] nvme nvme0: pci function 0000:02:00.0
[   68.196051] xhci_hcd 0000:00:14.0: xHCI Host Controller
[   68.196193] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[   68.196443] nvme nvme1: pci function 0000:04:00.0
[   68.197314] libata version 3.00 loaded.
[   68.197503] xhci_hcd 0000:00:14.0: hcc params 0x20007fc1 hci version 0x120 quirks 0x0000000200009810
[   68.197931] xhci_hcd 0000:00:14.0: xHCI Host Controller
[   68.198112] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[   68.198305] xhci_hcd 0000:00:14.0: Host supports USB 3.2 Enhanced SuperSpeed
[   68.198544] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.04
[   68.198762] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   68.198986] usb usb1: Product: xHCI Host Controller
[   68.199214] usb usb1: Manufacturer: Linux 6.4.0-3-amd64 xhci-hcd
[   68.199452] usb usb1: SerialNumber: 0000:00:14.0
[   68.199775] AVX2 version of gcm_enc/dec engaged.
[   68.200044] AES CTR mode by8 optimization enabled
[   68.200049] hub 1-0:1.0: USB hub found
[   68.200580] hub 1-0:1.0: 16 ports detected
[   68.200680] nvme nvme0: missing or invalid SUBNQN field.
[   68.200681] nvme nvme1: missing or invalid SUBNQN field.
[   68.200757] nvme nvme1: Shutdown timeout set to 10 seconds
[   68.202337] nvme nvme1: 8/0/0 default/read/poll queues
[   68.203130] nvme nvme1: Ignoring bogus Namespace Identifiers
[   68.204269] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.04
[   68.204570] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   68.204625]  nvme1n1: p1 p2 p3 p4 p5 p6
[   68.204875] usb usb2: Product: xHCI Host Controller
[   68.205508] usb usb2: Manufacturer: Linux 6.4.0-3-amd64 xhci-hcd
[   68.205837] usb usb2: SerialNumber: 0000:00:14.0
[   68.206236] hub 2-0:1.0: USB hub found
[   68.206585] hub 2-0:1.0: 6 ports detected
[   68.224577] nvme nvme0: 12/0/0 default/read/poll queues
[   68.229866]  nvme0n1: p1
[   68.235064] ahci 0000:00:17.0: version 3.0
[   68.245738] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[   68.246127] ahci 0000:00:17.0: flags: 64bit ncq sntf clo only pio slum part ems deso sadm sds 
[   68.309605] scsi host0: ahci
[   68.310264] scsi host1: ahci
[   68.310866] scsi host2: ahci
[   68.311437] scsi host3: ahci
[   68.312021] scsi host4: ahci
[   68.312594] scsi host5: ahci
[   68.313025] ata1: SATA max UDMA/133 abar m2048@0x81402000 port 0x81402100 irq 148
[   68.313454] ata2: SATA max UDMA/133 abar m2048@0x81402000 port 0x81402180 irq 148
[   68.313869] ata3: SATA max UDMA/133 abar m2048@0x81402000 port 0x81402200 irq 148
[   68.314272] ata4: SATA max UDMA/133 abar m2048@0x81402000 port 0x81402280 irq 148
[   68.314665] ata5: SATA max UDMA/133 abar m2048@0x81402000 port 0x81402300 irq 148
[   68.315049] ata6: SATA max UDMA/133 abar m2048@0x81402000 port 0x81402380 irq 148
[   68.461091] usb 1-3: new low-speed USB device number 2 using xhci_hcd
[   68.549897] Console: switching to colour dummy device 80x25
[   68.550154] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[   68.550195] i915 0000:00:02.0: firmware: direct-loading firmware i915/kbl_dmc_ver1_04.bin
[   68.550693] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4)
[   68.552138] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[   68.552868] ACPI: video: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
[   68.553270] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input3
[   68.565172] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   68.628180] ata2: SATA link down (SStatus 4 SControl 300)
[   68.628223] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   68.628265] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   68.629030] ata5.00: ATA-9: CT480BX300SSD1, M2CR010, max UDMA/133
[   68.629074] ata6.00: ATA-9: CT480BX300SSD1, M2CR010, max UDMA/133
[   68.629090] ata5.00: 937703088 sectors, multi 1: LBA48 NCQ (depth 32), AA
[   68.629174] ata6.00: 937703088 sectors, multi 1: LBA48 NCQ (depth 32), AA
[   68.629625] ata5.00: Features: Dev-Sleep
[   68.629641] usb 1-3: New USB device found, idVendor=045e, idProduct=00db, bcdDevice= 1.73
[   68.629657] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   68.629664] usb 1-3: Product: Natural® Ergonomic Keyboard 4000
[   68.629670] usb 1-3: Manufacturer: Microsoft
[   68.629705] ata6.00: Features: Dev-Sleep
[   68.630350] ata6.00: configured for UDMA/133
[   68.630371] ata5.00: configured for UDMA/133
[   68.631892] ata4: SATA link down (SStatus 4 SControl 300)
[   68.631934] ata3: SATA link down (SStatus 4 SControl 300)
[   68.631952] ata1: SATA link down (SStatus 4 SControl 300)
[   68.633155] scsi 4:0:0:0: Direct-Access     ATA      CT480BX300SSD1   010  PQ: 0 ANSI: 5
[   68.633604] scsi 5:0:0:0: Direct-Access     ATA      CT480BX300SSD1   010  PQ: 0 ANSI: 5
[   68.635890] hid: raw HID events driver (C) Jiri Kosina
[   68.638070] sd 5:0:0:0: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[   68.638076] sd 5:0:0:0: [sda] 4096-byte physical blocks
[   68.638092] sd 5:0:0:0: [sda] Write Protect is off
[   68.638098] sd 5:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   68.638115] sd 5:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   68.638134] sd 5:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[   68.638161] sd 4:0:0:0: [sdb] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[   68.638190] sd 4:0:0:0: [sdb] 4096-byte physical blocks
[   68.638252] sd 4:0:0:0: [sdb] Write Protect is off
[   68.638258] sd 4:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[   68.638304] sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   68.638324] sd 4:0:0:0: [sdb] Preferred minimum I/O size 4096 bytes
[   68.639878]  sda: sda1
[   68.639926] sd 5:0:0:0: [sda] Attached SCSI disk
[   68.640097]  sdb: sdb1
[   68.640139] sd 4:0:0:0: [sdb] Attached SCSI disk
[   68.647111] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   68.654680] usbcore: registered new interface driver usbhid
[   68.654685] usbhid: USB HID core driver
[   68.657704] input: Microsoft Natural® Ergonomic Keyboard 4000 as /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:045E:00DB.0001/input/input4
[   68.659288] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   68.717243] microsoft 0003:045E:00DB.0001: input,hidraw0: USB HID v1.11 Keyboard [Microsoft Natural® Ergonomic Keyboard 4000] on usb-0000:00:14.0-3/input0
[   68.717576] input: Microsoft Natural® Ergonomic Keyboard 4000 as /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:045E:00DB.0002/input/input5
[   68.757125] usb 1-4: new full-speed USB device number 3 using xhci_hcd
[   68.777340] microsoft 0003:045E:00DB.0002: input,hidraw1: USB HID v1.11 Device [Microsoft Natural® Ergonomic Keyboard 4000] on usb-0000:00:14.0-3/input1
[   68.908309] usb 1-4: New USB device found, idVendor=145f, idProduct=026d, bcdDevice= 1.00
[   68.908339] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   68.908354] usb 1-4: Product: GXT 144 Gaming Mouse
[   68.908364] usb 1-4: Manufacturer: SINOWEALTH
[   68.912256] input: SINOWEALTH GXT 144 Gaming Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.0/0003:145F:026D.0003/input/input6
[   68.912547] hid-generic 0003:145F:026D.0003: input,hidraw2: USB HID v1.11 Mouse [SINOWEALTH GXT 144 Gaming Mouse] on usb-0000:00:14.0-4/input0
[   68.917393] input: SINOWEALTH GXT 144 Gaming Mouse Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.1/0003:145F:026D.0004/input/input7
[   68.977606] hid-generic 0003:145F:026D.0004: input,hiddev0,hidraw3: USB HID v1.11 Keyboard [SINOWEALTH GXT 144 Gaming Mouse] on usb-0000:00:14.0-4/input1
[   69.105124] usb 1-9: new high-speed USB device number 4 using xhci_hcd
[   69.254887] usb 1-9: New USB device found, idVendor=05e3, idProduct=0608, bcdDevice=60.90
[   69.254915] usb 1-9: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[   69.254929] usb 1-9: Product: USB2.0 Hub
[   69.256630] hub 1-9:1.0: USB hub found
[   69.256981] hub 1-9:1.0: 4 ports detected
[   69.258124] usb: port power management may be unreliable
[   69.389128] usb 1-11: new full-speed USB device number 5 using xhci_hcd
[   69.538356] usb 1-11: New USB device found, idVendor=1462, idProduct=7d18, bcdDevice= 0.01
[   69.538385] usb 1-11: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   69.538400] usb 1-11: Product: MYSTIC LIGHT 
[   69.538410] usb 1-11: Manufacturer: MSI
[   69.538420] usb 1-11: SerialNumber: A02021081805
[   69.541808] input: MSI MYSTIC LIGHT  as /devices/pci0000:00/0000:00:14.0/usb1/1-11/1-11:1.0/0003:1462:7D18.0005/input/input9
[   69.542258] hid-generic 0003:1462:7D18.0005: input,hiddev1,hidraw4: USB HID v1.10 Device [MSI MYSTIC LIGHT ] on usb-0000:00:14.0-11/input0
[   69.669124] usb 1-14: new full-speed USB device number 6 using xhci_hcd
[   69.819394] usb 1-14: New USB device found, idVendor=8087, idProduct=0026, bcdDevice= 0.02
[   69.819423] usb 1-14: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[  182.141933] sd 5:0:0:0: [sda] Synchronizing SCSI cache
[  182.142749] sd 4:0:0:0: [sdb] Synchronizing SCSI cache

The last USB device in this list is a bluetooth card. I have blacklisted btusb, 
but this didn't help, still hangs.
At 182 seconds I pressed CRTL_ALT_DEL and the times without sleep statements 
come down to like 11 seconds when it hangs.



-----Original-Nachricht-----
Betreff: Re: AW: Debugging initramfs, server hangs during boot process
Datum: 2023-08-28T11:42:19+0200
Von: "Michel Verdier" <mv524@free.fr>
An: "debian-user@lists.debian.org" <debian-user@lists.debian.org>

On 2023-08-28, thahn01@t-online.de wrote:

> It hangs in /usr/share/initramfs-tools/scripts/init-top/udev
>
> The 2nd last udev call hangs my box
> udevadm trigger --type=devices --action=add

Perhaps add before
udevadm trigger --verbose --dry-run --type=devices --action=add