KVM Debian guests fail to boot after update-initramfs
(I have already posted to the libguestfs mailing list, without any
luck)
I run a number of virtualisation hosts, the host OS is Debian 10 (tho
this issue has also been present with 9) and the virtualisation tech is
KVM. All packages are current & from 'stable'.
I am encountering a problem where Debian guests, specifically built
using `virt-resize`, work flawlessly until the first time `update-
initramfs` is run, when they then fail to boot with the following
error:
"mount: mounting /dev/vda1 on /root failed: No such device"
after which the system fails to the initramfs Busybox debug shell. If I
use the `break=premount` kernel param to drop out of boots to the debug
shell before and after `update-initramfs` is run, and then compare the
output of `dmesg` it is clear that the necessary IO drivers are not
being loaded after the initramfs has been rebuilt:
BEFORE `update-initramfs`
[ 0.736291] Run /init as init process
[ 0.795060] lpc_ich 0000:00:1f.0: I/O space for GPIO uninitialized
[ 0.800259] PCI Interrupt Link [GSIA] enabled at IRQ 16
[ 0.800363] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[ 0.811428] cryptd: max_cpu_qlen set to 1000
[ 0.821407] AVX2 version of gcm_enc/dec engaged.
[ 0.821407] AES CTR mode by8 optimization enabled
[ 0.823702] ACPI: bus type USB registered
[ 0.823712] usbcore: registered new interface driver usbfs
[ 0.823717] usbcore: registered new interface driver hub
[ 0.823726] usbcore: registered new device driver usb
[ 0.829766] SCSI subsystem initialized
[ 0.830240] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input3
[ 0.830430] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input2
[ 0.844882] virtio_blk virtio2: [vda] 41943040 512-byte logical
blocks (21.5 GB/20.0 GiB)
[ 0.854231] vda: vda1 vda2
[ 0.857633] xhci_hcd 0000:02:00.0: xHCI Host Controller
[ 0.857638] xhci_hcd 0000:02:00.0: new USB bus registered, assigned
bus number 1
[ 0.857900] xhci_hcd 0000:02:00.0: hcc params 0x00087001 hci version
0x100 quirks 0x0000000000000010
[ 0.859088] virtio_net virtio0 enp1s0: renamed from eth0
[ 0.860055] usb usb1: New USB device found, idVendor=1d6b,
idProduct=0002, bcdDevice= 4.19
[ 0.860056] usb usb1: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
[ 0.860056] usb usb1: Product: xHCI Host Controller
[ 0.860057] usb usb1: Manufacturer: Linux 4.19.0-9-amd64 xhci-hcd
[ 0.860057] usb usb1: SerialNumber: 0000:02:00.0
...
AFTER `update-initramfs`
[ 0.743940] Run /init as init process
[ 0.801833] lpc_ich 0000:00:1f.0: I/O space for GPIO uninitialized
[ 0.814263] cryptd: max_cpu_qlen set to 1000
[ 0.817594] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input3
[ 0.817785] input: VirtualPS/2 VMware VMMouse as
/devices/platform/i8042/serio1/input/input2
[ 0.820087] PCI Interrupt Link [GSIA] enabled at IRQ 16
[ 0.820187] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[ 0.829008] AVX2 version of gcm_enc/dec engaged.
[ 0.829008] AES CTR mode by8 optimization enabled
[ 0.839844] virtio_blk virtio2: [vda] 41943040 512-byte logical
blocks (21.5 GB/20.0 GiB)
[ 0.843734] vda: vda1 vda2
[ 0.846319] virtio_net virtio0 enp1s0: renamed from eth0
(In the BEFORE case I have snipped the output of `dmesg` as it just
goes on to load the expected drivers, however in the case of AFTER,
that is the end of the `dmesg` output)
My workflow to create guests is as follows:
Initially manually install a minimal 'reference' guest to a small LVM
backed disk containing two partitions (/ & swap). The rest is then
scripted each time I want to 'clone' a new guest from this reference:
* take an LVM snapshot of the reference host disk volume
* `kpartx -a` on the snapshot, to gain access to the partitions, mount
/ and modify two files to set hostname and IP
* `kpartx -d`
* create a new LVM volume at the desired size to be used as the disk
for the finished guest
* `virt-resize` to copy the partitions from the snapshot to the new
volume, resizing swap to a fixed size and letting `virt-resize` grow /
to fill the remaining space
* `kpartx -a` on the new volume and then `mkswap` on the new swap
partition
* `kpartx -d`
* define the new host in KVM using the new volume as the backing disk
* `virt-sysprep -d <hostname> --enable bash-
history,customize,logfiles,mail-spool,package-manager-cache,ssh-
hostkeys,tmp-files` to clean up ahead of first boot
At this point the guest will operate flawlessly, will boot & reboot
without issue. However when update-initramfs first runs (usually as
part of a kernel upgrade, but this happens if it is run directly as
well), the problem is then present on next boot. CentOS guests built
the same way do not encounter this problem.
If I skip the `virt-resize` step and either use the snapshot as the
backing disk for the new guest, or `dd` the reference disk volume to
another volume and use that, Debian guests work fine in all cases. This
problem is present when both Debian 9 or 10 are used for either the
host or the guests.
So it appears that `virt-resize` triggers a behavior where `update-
initramfs` does not build the initramfs correctly.
Could anybody give me any pointers as to either what may be wrong, or
the best route to troubleshoot this further?
Many thanks in advance.
Reply to: