Bug#783620: initramfs-tools: initramfs broken on first boot into Jessie, "Unable to mount root fs on unknown-block(0, 0)"

[Bernhard Schmidt <berni@birkenwald.de>]

Package: initramfs-tools
Version: 0.120
Severity: important

Dear Maintainer,

I have a hard time wrapping my head around this bug, feel free to assign somewhere
else.

We have started upgrading some of our production VMs to Jessie. The testsystems worked
fine, but I have hit the following bug for the second time on a production VM now.

- dist-upgrade works flawlessly
- on first boot into Jessie I get an immediate (<1s) kernel-panic (see attached
  screenshot) about being unable to find the root fs. Unfortunately I'm unable to
  get the full boot log, since I don't have a serial console there and kernel
  messages scroll by too fast.
- To fix the issue I have to boot into the old Wheezy kernel (3.2.0-4-amd64) in
  grub and regenerate the initrd for the Jessie kernel

	# update-initramfs -k 3.16.0-4-amd64 -u

  Then it works fine.

Now comes the interesting part ... I have saved the broken initrd for later 
analysis

The compressed size is marginally different (broken being 3k smaller)

-rw-r--r--  1 root root 14339199 Apr 28 13:59 initrd.img-3.16.0-4-amd64
-rw-r--r--  1 root root 14338898 Apr 28 13:58 initrd.img-3.16.0-4-amd64.broken

The uncompressed size is the same

root@lxmhs63:/tmp# zcat /boot/initrd.img-3.16.0-4-amd64.broken > initrd.img-3.16.0-4-amd64.broken
root@lxmhs63:/tmp# zcat /boot/initrd.img-3.16.0-4-amd64.broken > /tmp/initrd.img-3.16.0-4-amd64.broken
root@lxmhs63:/tmp# ls -la /tmp/initrd.img-3.16.0-4-amd64*
-rw-r--r-- 1 root root 45304832 Apr 28 14:44 /tmp/initrd.img-3.16.0-4-amd64
-rw-r--r-- 1 root root 45304832 Apr 28 14:44 /tmp/initrd.img-3.16.0-4-amd64.broken

The checksum is different

root@lxmhs63:/tmp# md5sum /tmp/initrd.img-3.16.0-4-amd64*
7b24aa901b697dc5dfdbad03bd199072  /tmp/initrd.img-3.16.0-4-amd64
5e467c0a49afa4ddae315cc6e818d7ac  /tmp/initrd.img-3.16.0-4-amd64.broken

Now comes the puzzling part ... the _content_ of the initrd is exactly the same

root@lxmhs63:/tmp# mkdir broken && cd broken && cpio -id < ../initrd.img-3.16.0-4-amd64.broken  
88486 blocks
root@lxmhs63:/tmp/broken# cd ..
root@lxmhs63:/tmp# mkdir ok && cd ok && cpio -id < ../initrd.img-3.16.0-4-amd64                     
88486 blocks
root@lxmhs63:/tmp/ok# cd ..
root@lxmhs63:/tmp# diff -urN broken ok

I will try to capture a screenlog on the next upgrades, maybe there is something 
interesting in there. 

Bernhard