[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

NFS root with initrd and GRUB

Following the advice from some people here (thanks in particular to Thierry 
Laronde) I have got initrd and NFS root working so that I can use the same 
kernel binary for hard drives, RAID, and NFS root without making the kernel 
excessively large.

I have written a document on what I have done because there seems no other 
good documentation on some of these things.  It is attached.  Any suggestions 
on improving the document (or how I could have done things differently to 
save myself some effort) will be appreciated.

http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page
Title: NFS Root With INITRD, GRUB, and Devfs

GRUB Configuration

The first thing to do is compile GRUB with support for your network card. Typical binaries of GRUB (such as the Debian package of GRUB) come without any support for network booting. I imagine that part of the reason for this is the fact that probing for the wrong network card has the potential to lock the machine, and also for the tulip card there are two versions of the driver (only the old version works for my card). I am going to request the creation of grub packages for network booting to be created for Debian.
Anyway the way to build GRUB with network support is to run ./configure with the option for your network card and then run make. For my work I tried to build GRUB in a temporary directory and install it from there, but it seemed to get some files from the Debian GRUB package I had installed so I ended up changing the debian/rules file and building and installing my own package of GRUB for network booting.

Once GRUB is compiled you have to install it to a floppy. There is a script that just writes the stage1 and stage2 files to the raw floppy device, AFAIK it is not possible to use a menu.lst file with this so it won't work for anything other than a recovery disk where you type every command entirely.
So I created a minix format floppy disk and installed GRUB to it, and then put the following in boot/grub/menu.lst:

# give me 10 seconds to interrupt the boot and change things
timeout 10
# boot from the first image by default
default 0
# set the title that is displayed in the menu
title net
# I am not using dhcp/bootp so we have an ifconfig command to set our IP address
# to and the tftp server is
# use the dhcp, bootp, or rarp directives instead for a dynamic IP
ifconfig --address= --server=
# the root file system is a network disk
root (nd)
# specify the kernel file to load (/vmlinuz), --no-mem-option means not to
# pass mem= to the kernel (I trust the kernel to find it's own memory)
# load the root file system from /dev/ram (the initrd) and make /linuxrc the
# init program.
kernel --no-mem-option /vmlinuz root=/dev/ram init=/linuxrc
# load the initrd from /boot/initrd on the tftp server
initrd /boot/initrd
# now boot the sucker!

# now we've got an option to boot from the old version.
title net-old
ifconfig --address= --server=
root (nd)
kernel --no-mem-option /vmlinuz.old root=/dev/ram init=/linuxrc
initrd /boot/initrd.old

Initrd configuration

The first problem of initrd is that in the default setup the kernel will mount the root file system after the /linuxrc process exits, however the kernel options nfsroot= and ip= don't seem to operate when the NFS client code is loaded as modules (may not even be compiled into the module). This means that the regular operation of initrd won't work. So I tell the kernel it's a regular ram disk and that /linuxrc is the init program. This means that I had to put in some script code to mount the root file system and then use pivot_root to make change the root file system.

This is the script I put in /etc/mkinitrd/scripts/nfs to set the initrd options for NFS. In Debian there are scripts to automatically create an initrd in the initrd-tools package. As part of this creation process the scripts in /etc/mkinitrd/scripts are all run, and the variable $INITRDDIR specifies the temporary directory that contains the new file system image.

# make a directory for mounting the NFS root file system
mkdir $INITRDDIR/new-root
# Create this script to be run on the initrd after modules have been loaded
cat > $SCRIPT << END
ifconfig eth0
mount -n /new-root -orsize=8192,wsize=8192,lock,ro,port=2049,mountport=32790
mount -n none -t devfs /new-root/dev
cd /new-root
pivot_root . initrd
umount /initrd/dev
chmod 755 $SCRIPT

# my script needs ifconfig, pivot_root, and mount which aren't installed by
# default, change ifconfig to dhclient, pump, etc for dynamic IP addresses
cp /sbin/ifconfig /sbin/pivot_root $INITRDDIR/sbin
cp /bin/mount $INITRDDIR/bin

# tell /linuxrc to run init because the kernel won't do it, have to use exec
# so that init gets PID=1
echo "exec init" >> $INITRDDIR/linuxrc

Here's the created script with some extra comments:

# assign a static IP address to the interface, replace it with a call to
# dhclient, pump, etc for dynamic IP addresses
ifconfig eth0
# mount the new root file system, specify the rsize and wsize options for
# performance, specify the port numbers for NFS3 so that we don't need portmap
# running
mount -n /new-root -orsize=8192,wsize=8192,lock,ro,port=2049,mountport=32790
# mount devfs under the new root. If the kernel was doing the pivot it would
# manage this, but it's not so we have to do it.
mount -n none -t devfs /new-root/dev
# change to the new root (remove one reference to the old root)
cd /new-root
# make the current directory the new root, and remount the old root directory
# under /initrd on the new root
pivot_root . initrd
# I'd like to do this but actually it'll fail. :(
umount /initrd/dev

Debian has a file named /etc/mkinitrd/mkinitrd.conf to configure the process of making an initrd, for all my initrd systems I use the directive MODULES=none as I specify exactly the modules I need. Also for NFS we need ROOT= (IE we set it to an empty string). This means that the scripts to create the initrd don't try and figure out where the root file system is to load modules etc. This is necessary because we aren't following the regular initrd proceedure, and because I usually run the creation script on the NFS server machine (not on the diskless workstation).

Debian has the file /etc/mkinitrd/modules to specify which modules are to be loaded via modprobe during the start of the initrd process, I use nfs and tulip to load the network driver and the NFS client code. I also load usbcore, usb-uhci, input, keybdev, usbkbd, and hid so that I can use my USB keyboard before the real root file system is mounted.

I use the following in /etc/mkinitrd/scripts/devfs to remove un-needed (for devfs) entries in /dev:

rm -rf $INITRDDIR/dev/*

I use the following in /etc/mkinitrd/scripts/copy-needed-modules to parse the modules.dep file and install all modules that are needed by the modules I have specified into the lib/modules directory in the initrd:


open(DEP, "$ENV{MODULEDIR}/modules.dep") or die "Can't open modules.dep";
$/ = "\n\n";
# map of name to full path
my %names;
# map of path to dependencies
my %deps;
  $_ =~ s/\\\n/ /;
  my @line = split(':', $_);
  my $name = $line[0];
  $name =~ s/^.*\///;
  $name =~ s/\..*$//;
  $names{$name} = $line[0];
  $deps{$line[0]} = $line[1];
$/ = "\n";
open(MODULES, "grep -v ^# /etc/mkinitrd/modules | grep .|") or die "Can't open modules";
  foreach my $n ($names{$_}, split('\t', $deps{$names{$_}}) )
    if(length($n) > 0)
      $n =~ s/[ \t]*//g;
      my $dir = $n;
      $dir =~ s/[a-z0-9\-\.]*$//;
      system("mkdir -p $ENV{INITRDDIR}$dir");
      system("cp $n $ENV{INITRDDIR}$n");

Setting up the server

If using rarp, bootp, or DHCP you have to set the server up appropriately (NB the rarp, bootp, or DHCP server does NOT have to the the same machine as the TFTP or NFS server).

To run the TFTP server I used the tftpd-hpa server. This is because it supports PXE network boots (which I plan to use at some future time). For what I am doing so far any TFTP server should work. Here is the configuration file for RLINETD (my favourite inetd), any inetd should work just as well:

service "tftp" {
    protocol udp;
    port "tftp";
    user "root";
    exec "/usr/sbin/in.tftpd -s /var/tftpboot";
    nice 5;
    instances 1;
The -s /var/tftpboot says that I want the /var/tftpboot to be the root of the TFTP server.

Then I put the following in /etc/exports:

After running exportfs -r to export that everything was working.
Reply to: