Bug#958311: cloud kernel 5.5.0-2 does not boot under xen

[Andy Smith <andy@strugglers.net>]

Hi Samuel,

On Thu, Feb 16, 2023 at 03:59:09PM +0100, Samuel Thibault wrote:
> Andy Smith, le jeu. 09 juin 2022 15:32:38 +0000, a ecrit:
> > If you're using pvgrub2 to boot PV mode then the bad news is that it
> > seems to be largely abandoned as nobody wants to alter it to support
> > different kernel compression methods.
> 
> Uh... I wonder how it is that it's not just orthogonal to whether
> booting in native/PV/PVH...

It's because:

- "native" booting is the xen hypervisor itself booting your Debian
  kernel from out of its own filesystem, so the hypervisor needs to
  understand all kernel compression methods and historically it has
  lagged behind upstream kernel compression types. Grub is not
  involved here.

- PV grub and PVH grub are both grub binaries that are booted by the
  hypervisor, so the hypervisor just needs to know how to boot that
  grub image, which is (a) a slower-moving target and (b) something
  that admin of the bare metal host can recompile easily without
  interfering with guests.

  HOWEVER

  - The PV part of grub is quite old and from what I understand
    implemented in a strange way that no one wants to maintain any
    more, so this part of grub is stuck without ability to
    understand the newer kernel compressions.

  - The PVH part of grub is more modern and uses grub's own
    facilities for loading the kernel file, so as long as grub
    understands a compression for normal Linux, it works with the
    PVH part of grub too, which is obviously a lot more
    maintainable.

So in summary:

For Xen there's two different places for implementing the
understanding of loading a Linux kernel, that being the hypervisor
or the grub bootloader. The hypervisor is slower upstream to support
new kernel compressions so there has been times when for example
Ubuntu or Fedora has by default been unbootable directly without
getting a pre-release version of Xen hypervisor (or un/repacking
the guest kernel compression).

The grub method is preferred so that guests can manage their own
kernels, and that has two different code paths in grub depending
upon PV mode or PVH mode. The PV mode part doesn't seem to be
maintained. The PVH part uses more of core grub functionality so is
easier to maintain.

I would recommend defaulting to PVH mode for guests these days,
unless you are doing HVM. There are still people who want to use
old kernels that don't support PVH mode, though. I don't think any of
those old kernels are supported by Debian at this stage, but still…

Cheers,
Andy