initramfs broken on Jessie upgrade

[Bernhard Schmidt <berni@birkenwald.de>]

Hi,

has anyone observed something similar to
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=783620 on their
Upgrade from Wheezy to Jessie? I'm still trying to figure out what's
happening, and I don't really know where to look.

I was unable to attach the screenshot so far (mail is accepted but
never makes it to the BTS), I've put the screenshot here:

http://users.birkenwald.de/~berni/volatile/783620.png

---snip---
Dear Maintainer,

I have a hard time wrapping my head around this bug, feel free to assign
somewhere else.

We have started upgrading some of our production VMs to Jessie. The
testsystems worked fine, but I have hit the following bug for the second
time on a production VM now.

- dist-upgrade works flawlessly
- on first boot into Jessie I get an immediate (<1s) kernel-panic (see
  attached screenshot) about being unable to find the root fs.
  Unfortunately I'm unable to get the full boot log, since I don't have
  a serial console there and kernel messages scroll by too fast.
- To fix the issue I have to boot into the old Wheezy kernel
  (3.2.0-4-amd64) in grub and regenerate the initrd for the Jessie
  kernel

	# update-initramfs -k 3.16.0-4-amd64 -u

  Then it works fine.

Now comes the interesting part ... I have saved the broken initrd for
later analysis

The compressed size is marginally different (broken being 3k smaller)

-rw-r--r--  1 root root 14339199 Apr 28 13:59 initrd.img-3.16.0-4-amd64
-rw-r--r--  1 root root 14338898 Apr 28 13:58 initrd.img-3.16.0-4-amd64.broken

The uncompressed size is the same

root@lxmhs63:/tmp# zcat /boot/initrd.img-3.16.0-4-amd64.broken > initrd.img-3.16.0-4-amd64.broken
root@lxmhs63:/tmp# zcat /boot/initrd.img-3.16.0-4-amd64 > /tmp/initrd.img-3.16.0-4-amd64
root@lxmhs63:/tmp# ls -la /tmp/initrd.img-3.16.0-4-amd64*
-rw-r--r-- 1 root root 45304832 Apr 28 14:44 /tmp/initrd.img-3.16.0-4-amd64
-rw-r--r-- 1 root root 45304832 Apr 28 14:44 /tmp/initrd.img-3.16.0-4-amd64.broken

The checksum is different

root@lxmhs63:/tmp# md5sum /tmp/initrd.img-3.16.0-4-amd64*
7b24aa901b697dc5dfdbad03bd199072  /tmp/initrd.img-3.16.0-4-amd64
5e467c0a49afa4ddae315cc6e818d7ac  /tmp/initrd.img-3.16.0-4-amd64.broken

Now comes the puzzling part ... the _content_ of the initrd is exactly
the same

root@lxmhs63:/tmp# mkdir broken && cd broken && cpio -id < ../initrd.img-3.16.0-4-amd64.broken
88486 blocks
root@lxmhs63:/tmp/broken# cd ..
root@lxmhs63:/tmp# mkdir ok && cd ok && cpio -id < ../initrd.img-3.16.0-4-amd64
88486 blocks
root@lxmhs63:/tmp/ok# cd ..
root@lxmhs63:/tmp# diff -urN broken ok

I will try to capture a screenlog on the next upgrades, maybe there is
something interesting in there. 
----snip----

Best Regards,
Bernhard