Turns out this is a known issue with initramfs-tools. It's simply not waiting for the root devices properly. Adding rootdelay=30 to kernel parameters was good enough for me. While this isn't a debian-installer bug, it would be nice if debian-installer provided workaround to all RAID/LVM users by default until it's fixed in initramfs-tools.