Adding some more data from logs and stuff now that I got the machine to finish installation successfully.
This is the first os-prober run:
Apr 21 08:35:48 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:35:48 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:35:48 kernel: [ 953.534341] FAT-fs (dm-4): FAT read failed (blocknr 6158)
Apr 21 08:35:48 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: mounting /dev/mapper/osprober-linux-nvme0n1p1 on /var/lib/os-prober/mount failed: Input/output error
Apr 21 08:35:48 50mounted-tests: debug: remove device mapper device /dev/mapper/osprober-linux-nvme0n1p1
This is the second run:
Apr 21 08:35:53 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:35:53 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:38:58 init: starting pid 260, tty '/dev/tty2': '-/bin/sh'
Apr 21 08:40:41 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: special device /dev/mapper/osprober-linux-nvme0n1p1 does not exist
You can see that I logged in there and took a few minutes to gather data and then unblock it. This is what I did (with some information obscured)
Trailing part of ps fax:
31842 ? S+ 0:00 \_ /bin/sh /etc/grub.d/30_os-prober
31848 ? S+ 0:00 \_ /bin/sh /etc/grub.d/30_os-prober
31849 ? S+ 0:00 \_ /bin/sh /usr/bin/os-prober
31931 ? S+ 0:00 | \_ /bin/sh /usr/lib/os-probes/50mounted-tests /dev/nvme0n1p1
31938 ? S+ 0:00 | \_ /bin/sh /usr/lib/os-probes/50mounted-tests /dev/nvme0n1p1
31943 ? S+ 0:00 | \_ dmsetup create -r osprober-linux-nvme0n1p1
31850 ? S+ 0:00 \_ tr ^
31851 ? S+ 0:00 \_ paste -s -d
$ dmsetup table
osprober-linux-nvme0n1p1: 0 2048 linear 259:1 0
<LVM volume1>: 0 747470848 linear 253:0 117188608
<LVM volume2>: 0 117186560 linear 253:0 2048
nvme0n1p4_crypt: 0 998707200 crypt aes-xts-plain64 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0 259:4 4096
<LVM swap>: 0 134045696 linear 253:0 864659456
$ ipcs -s
------ Semaphore Arrays --------
key semid owner perms nsems
0x0d4ddd11 0 root 600 1
And then I unblocked the process by doing (inside the chroot):
$ dmsetup udevcomplete 0x0d4ddd11
These are the third and fourth os-prober runs:
Apr 21 08:42:49 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:42:49 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:42:49 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: special device /dev/mapper/osprober-linux-nvme0n1p1 does not exist
Apr 21 08:43:53 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:43:53 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:43:53 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: special device /dev/mapper/osprober-linux-nvme0n1p1 does not exist
I have a machine where I can reproduce this every time (since the partition never gets written to), so let me know if you want me to gather any additional information.
Thanks!