Re: How to build XEN dom0 and domU kernels based on 2.6.25?

[Daniel Widenfalk <Daniel@lilla-graen.c.se>]

Ian Campbell wrote:
> On Sun, 2008-05-11 at 11:32 +0200, Daniel Widenfalk wrote:
> > Ian Campbell wrote:
> > > On Thu, 2008-05-08 at 14:16 +0200, Daniel Widenfalk wrote:
> > > 	
> > > > Ok, so dropping back a step. Let's assume that I build the 3.2.0 XEN
> > > > hypervisor and dom0 kernel using 2.6.18 as base. I should then be able to
> > > > build domU kernel(s) using the linux-source-2.6.25 files? How?
> > > >
> > > > I can't seem to get CONFIG_XEN set in my .config file. Running "make
> > > > menuconfig" does not show any xen-specific options.
> > > Have you enabled CONFIG_PARAVIRT? You might also need to select a 686
> > > kernel flavour for the option to appear. I've attached my config in case
> > > it is useful, it's PAE+XEN.
> > >
> > > Also that that kernel only has i386 support for Xen, no 64 bit yet.
> > >
> > > Ian.
> > Many thanks all,
> >
> > I've decided to do the following:
> >
> > Upgrade to Xen 3.2.0 using the 2.6.18 kernel (debian tree) for
> > my dom0 machines and build 2.6.25 kernels for my domUs. I can't
> > see any major howlers with this as the domUs won't rely on
> > anything linux/version specific in the dom0 kernel?
>
> No that should be perfectly fine and is pretty much what I do (I use the
> XenSource 2.6.18-xen tree though).
>
> > My major problem with 2.6.25 was that I was trying to build a
> > 64-bit kernel which caused the config system to disable Xen.
> > Quite explainable if xen is not yet available for 64-bit in the
> > current linux tree...
>
> Yes, there is a tree at
> http://git.et.redhat.com/?p=xen-pvops-64.git;a=summary which is the
> in-progress tree but it needs pretty specific configuration settings:
> http://lists.xensource.com/archives/html/xen-devel/2008-05/msg00117.html
>
> Ian.

Hi all,

I'd just like to come back with some good news! I've (finally)
had some success with a 64-bit 2.6.18 dom0 together with a
32-bit 2.6.25 domU.

I started on an Amd64 machine running (an old) Xen 3.0 with
custom built kernels. Here is how I did it:

0) Save all changes to the network initialization scripts used
   by xen!

1) Shut down all domU machines before updating Xen from 3.0 to
   3.2. The reason for this is that Xen 3.0 is bound to Python
   2.4 whereas Xen 3.2 is bound to Python 2.5.

   If you don't shut down your domU guests before updating Xen,
   then you won't be able to shut them down gracefully afterwards!
   (Learned that the hard way ;-) If you forget, then you have
   to go in and redirect the "python" soft link in /usr/bin to
   point at python2.4 instead, then run "xm shutdown -wa" and then
   point is back at python2.5.

2) Port your changes to the network initialization scripts!

   It seems that the default bridge name is no longer
   "xenbr0", it will be "eth0". I had some trouble with this
   as I'm running a DMZ-bridge for use by the web and email
   servers. Had to update my guest configuration files to
   refer to eth0 instead of xenbr0...

3) Reboot your machine using your old dom0 kernel. This should
   work without a hitch as long as you're got the network
   scripts fixed OK. You may run into problems if your old
   kernel had cpu frequency adaption enabled. This shouldn't
   be the case as Xen 3.0 doesn't support cpu frequence adaption
   but I had applied a patch to allow this which has caused me
   a lot of problems.

4) Download the Xen distribution and also the latest 2.6.18
   kernel from xensource. Configure and build a new dom0
   kernel using the Xen sources.

   IMPORTANT: Don't use gcc-4.3 for this! This is what tripped
   me up. The kernels I built using gcc-4.3 crashed on boot.

   > make linux-2.6-xen-config CONFIGMODE=menuconfig (or xconfig)
   > make linux-2.6-xen-build CC=gcc-4.2
   > sudo cp dist/install/boot/config* /boot
   > sudo cp dist/install/boot/System.map* /boot
   > sudo cp dist/install/vmlinuz* /boot

   Update your grub-menu with an additional entry using your
   newly built kernel.

5) Reboot your machine with your new dom0 kernel. This should,
   hopefully, work like a charm. You're now running Xen 3.2
   together with a dom0 kernel based on Xen 3.2.

6) Download the linux-tree-2.6.25 (and build dependencies, of
   course) package and build/configure your domU kernel:

   > MAKEFLAGS="CC=gcc-4.2" make-kpkg --rootcmd fakeroot \
     --revision 1 --added-patches debian --append-to-version \
     my-32-xenu --cross-compile - --arch i386 \
     --config=menuconfig kernel_image

   IMPORTANT: Note the use of gcc-4.2!

   IMPORTANT: Enable PAE and 64Gbyte memory or it won't work!

   The important point here is to remember:

     "--cross-compile - --arch i386"

   This tells make-kpkg to build a 32-bit image using the native
   amd64 gcc. You'll need to install the "gcc-multilib" package!

7) Install your newly created image

8) Create disk images using, e.g., xen-create-image. Remember to
   specify "--arch i386" or you'll get an Amd64 image and that
   won't boot!

9) Mount your root image and edit the fstab file. The virtual
   block devices are named xvda1, xvda2 etc and not sda1, sda2!

   You probably also want to make all other important changes
   in etc now before booting your machine for the first time. I
   find I usually have to update the network configuration to
   get it right the first time.

   Remember to unmount your image!

10) Open your Xen domain config file and add a line:

       extra = 'console=hvc0'

    I've also added a line:

       pae = 1

    which I'm not 100% sure if it's actually needed.

    Make sure your root entry uses xvda1 instead of sda1!

11) Launch your 32-bit guest!

This seems to work fine for me. I haven't yet transitioned
any of my production servers so I can't say anything about
stability and such, but it seems to run fine.

I hope this helps someone!

Regards
/Daniel Widenfalk