[SOLVED] potential bug? - was Re: Problem with cryptsetup-initramfs

[Ian Molton <debian_user-ian@molton.org.uk>]

Hi all,


I have solved the boot issue. I got physical access to the machine, and 
connected a monitor and keyboard to it.

As usual, it booted the initramfs, and started dropbear, but this time, 
I entered the disk key on the keyboard.

The machine attempted to boot, but failed, dropping back to the 
initramfs prompt - however, crucially, the video console logged a useful 
error - the rootfs could not be mounted (despite mounting just fine, 
without errors, in read write mode, over the ssh connection previously).

I ran a fsck in the initramfs environment, and rebooted it.

The machine booted and I again unlocked it via the (keyboard) console.

This resulted in a clean boot with no untoward errors.

I shut it down, and rebooted, and now it boots successfully even if I 
unlock it via the ssh connection.

I feel like this is a bug in cryptroot-initramfs - surely it would be 
critically important for the aforementioned filesystem error to have 
been logged to the ssh console, rather than it silently failing to boot? 
afterall - it DOES log the errors to the video console.

Or have I been daft and somehow misconfigured it? I don't think I 
deviated from the standard configuration.

Is there a simple way to have errors logged to the console AND to the 
ssh session? Have I been thick?

Thanks,

-Ian


On 24/07/2024 00:14, Ian Molton wrote:
> Hi all,
>
> I'm having problems with one of my machines. it's a Pine RockPro64.
>
> Debian bookworm has been running very stably on it for some time. I 
> rebooted it a couple of weeks ago for maintainance, having applied 
> updates, after 108 days up. I have an encrypted LVM volume, containing 
> root, swap, and data LVs, with /boot on a MMC card, and I use 
> cryptsetup-initramfs to allow me to log in and unlock the volume at 
> boot time, via ssh.
>
> I rebooted it this morning, due to a crash, and it didn't come back up.
>
> I can still connect to dropbear (running in the initramfs context), 
> and the MoTD (as always) prompts me to run cryptroot-unlock, which 
> appears to do its job (ie. the LVM volume is unlocked / decrypted), 
> however it does not proceed to switch root (and drop the ssh 
> connection), as it used to, with complete reliability.
>
> At first, I suspected a problem with the rootfs, however this appears 
> not to be the issue - the volume is present at the expected location, 
> and can be manually mounted.
>
> Executing the following allows me to enter a chroot on the rootfs:
>
> mount -text4 /dev/vg0/root /root
> mount --bind /dev /root/dev
> mount --bind /proc /root/proc
> mount --bind /sys /root/sys
> chroot /root /bin/bash --login
>
> and following this I can run:
>
> mount /boot
>
> which correctly mounts the MMC card containing the /boot partition in 
> the chrooted environment.
>
> Inspecting /boot, everything appears to be in order. I have issued 
> update-initramfs a few times, even completely removing the existing 
> initramfs and recreating it.  I have also inspected the initramfs 
> built by update-initramfs, and can see nothing out of the ordinary. 
> crypttab is copied from the host, and the UUID matches that displayed 
> by lsblk -f - which is not surprising given that executing 
> cryptroot-unlock does, in fact, decrypt the volume.
>
> Once chrooted, I can see that /sbin/init is a symlink to 
> /lib/systemd/systemd, which exists and is executable, but obviously 
> cannot be executed as anything other than PID 1. Attempting to execute 
> it results in it complaining of a missing argument, or, if one is 
> provided, an error that it is ignoring the request due to running in a 
> chroot.
>
> The kernel command line (cat /proc/cmdline) contains a correct root= 
> entry, which points at /dev/mapper/vg0-root
>
> I'm stumped - I cannot see why the initramfs environment fails to 
> mount the rootfs and execute init.
>
> I have run an additional apt update / upgrade / dist-upgrade, whilst 
> under chroot, in the hope that it will magically fix everything, but 
> to no avail.
>
> I was using a custom dtb that enabled PCIe x4 on the board, but have 
> removed that and reverted to the debian-supplied .dtb file just in case.
>
> Any ideas? I have several machines using this configuration, both 
> arm64 and amd64, and I'm now a little uneasy about rebooting any of 
> them, in case there has been a breaking change somewhere which they, 
> too, are likely to fall afoul of.
>
> Any thoughts? I've never really had to debug the init process (ie. 
> PID1) and am not sure how to proceed.
>
> Thanks,
>
> -Ian