[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Configuring foreign packages without qemu in order to boot a system

On Sun, 2 Sep 2012 01:40:45 +0100
Wookey <wookey@wookware.org> wrote:
> On Sun, 2 Sep 2012 01:07:11 +0200
> Paul Boddie <paul@boddie.org.uk> wrote:

> > This configuration step appears to be the necessary measure to permit the init 
> > mechanism and various related activities to function.

Principally it is to run ldconfig, to setup the system shell and to
initialise some data caches. Take a look
at /var/lib/dpkg/info/*.postinst to see what some of the packages need
to do in this phase. Note that the preinst scripts are *not* run by
multistrap (whether using qemu or not) unless your configscript
explicitly calls them (as the wiki recommends it does for at least the
dash preinst). Note that many packages will simply fail to operate if
ldconfig has not been run and many shared libraries have more than just
a call to ldconfig as their postinst.


I'm not aware of a safe & robust method to restrict ldconfig to only
certain shared libraries from out of a much larger set of shared
libraries awaiting configuration on the system. Configuring all shared
libraries via dpkg to suit ldconfig will involve running the maintainer
scripts for all shared libraries which will themselves require
executables from other packages to be configured.

dpkg can work all this out but only when given the entire set.

> > I suppose that not all 
> > packages need to be configured, but some configuration seems to make the 
> > difference between a functioning and non-functioning system.

In order for this to work, you have to at least have the ability to
chroot into the semi-configured rootfs *and* you need enough
non-packaging data (/etc/inittab, /etc/fstab) provided by the
setupscript to allow the kernel to use the rootfs for init. That
generally means that shared libraries have to be configured (ldconfig),
a shell must be fully operational (dash preinst and postinst) and dpkg
itself as well as all of the dependencies of these.

Note: It is *not* advisable to manually force these steps. dpkg does
not take kindly to someone second-guessing the *order* in which
packages are configured even if you manage to identify the set itself.

dpkg assumes that the configure step is either:

a) system-wide with no external ordering, or
b) single-upgrade mode where everything that this package needs
is already configured or within the scope of the current upgrade.

Breaking these assumptions will do horrible things to the system in
ways that might be recoverable by reinstalling the package but as this
is completely untested maybe not ....

It's one of the principle differences between multistrap and other
bootstrap tools. Multistrap can handle all the dependencies because it
handles all packages at the same time. debootstrap breaks the system
into two halves and fully configures the first before trying to go back
for the later packages without full dependency calculation support.
debootstrap does allow that later packages have a fully configured base
beneath them but this means that debootstrap *cannot* offer the chance
to trim the base system itself as it has no way of knowing what the
later packages will need. Multistrap combines both phases and gives a
smaller package set at the cost of having to do all processes in a
single step. It is this single-step approach which allows multistrap to
work with multiple repositories at the same time - apt needs to know
about all the packages at the same time for dependency calculations
just as dpkg needs to be able to work on all packages in whatever
order it calculates.

Multistrap can easily take a rootfs all the way up to a full package
set with a UI and customisations and give you a tarball with absolutely
everything setup and ready to run. The cost of that is that the rather
arbitrary division line with which people have become familiar from the
debootstrap model simply doesn't exist. This has advantages in package
selection but also has requirements about package configuration.

The imposition of another arbitrary line between "first level packages"
and "second level packages" in a system where *all* packages are not
fully configured is explicitly unsupported by dpkg. We have the
debootstrap way of two completely separate stages where one level has
no idea what the requirements are of the next level or we have the
multistrap single stage model where everything happens as a whole.
Either way, each run of dpkg needs to consider the entire system.

Now that's not to say that you have to put the entire rootfs into the
multistrap config. One of my regular use-cases for multistrap provides
a -platform package which only adds the specific packages that the
device needs which are not usually part of a chroot, then a later
script (entirely outside multistrap) takes data from a purchase order
and populates the top level package set based on that specific order.
This isn't the same as the debootstrap model but is done in order to
support multiple variants from a single rootfs tarball to reduce
overall programming time.

> > Obviously, in my situation, I can't boot into Emdebian without 
> > having first performed the configuration step, so this supposedly leaves me 
> > with the fairly undesirable option of using qemu.

... or a rescue init system stuffed somewhere else. A second partition
maybe. A minimal system which is created once, packed up as a tarball
and then used repeatedly. This is roughly equivalent to Debian
Installer except that that image is created in RAM. It only needs to be
enough for the kernel to be able to sort out init and chroot in.
buildroot is often used for this. It only needs to be 4Mb with
buildroot (uClibc & busybox) or you can have a second Emdebian system
with ~50Mb. Recent versions of buildroot (>= 2011-05) are much improved
and build and run without the need for third-party patches.

That rescue system can then be scripted to mount the other partition,
chroot in, run all of the configuration stages, remove the script from
the rescue system to take itself out of the boot sequence and reboot -
just like Debian Installer - directly into the full system.

> > I did wonder, however, whether there are any known techniques for configuring 
> > just enough of a system on the build host so that the init mechanism can 
> > complete its work on the target system and then allow the proper 
> > configuration to occur. 

"just enough of a system to run init (and chroot)" sounds a lot like
a description of buildroot. 

> Multistrap provides a (fairly low-tech) mechanism for this. In your
> multistrap config provide a
> setupscript=/path/to/setup.sh
> which does any tweaking necessary to make the image bootable (e.g. setting
> /etc/fstab, inittab, /etc/modules to something useful).
Whether packages work correctly when unconfigured - even with support
from static files created in /etc/ - is a little bit of a wild guess.
It's completely untested because packages are not required to operate
when in the state installed-ok-unpacked, only in installed-ok-installed
and dpkg only sets that state after the configure step.

You may find it works, you may find that some packages misbehave. It
might be enough to get a chroot but it is also inherently fragile. Used
with only packages from stable and always with the same package set and
it will be deterministic - change the package set or allow package
versions to change by using testing or unstable and you introduce

> > Alternatively, I wondered whether there might be any way of deploying a "first 
> > boot" script to perform configuration on the target system.
> That doesn't help if it won't even boot. So there is a second option:
> configscript=/path/to/config.sh which specifies a script to copy into
> the image for 'first run' use.
> There are examples of these scripts in the package. Some details are
> given in the 'Machine:variant support' and 'Root Filesystem
> Configuration' sections of the multistrap man page. And also on
> http://wiki.debian.org/Multistrap/CustomisingRootfs


Getting the kernel to boot is the first barrier. Once that is done,
getting the kernel to use the init available and successfully hand-off
to it is the next barrier. Once you're at that point, the configscript
can be executed by that init process and that process could be

> > I see a script 
> > apparently suggested for this kind of thing...
> > 
> > http://wiki.debian.org/Multistrap#Debconf_and_pre-seeding
> That is a typical example of a first-run 'configscript'.
> > I would certainly appreciate any advice on this topic. Emdebian promises to 
> > make a lot of things easier when developing for embedded devices like the 
> > NanoNote - just the availability of numerous packages for different 
> > architectures is a real time-saver - and I'd like to be able to offer some 
> > concrete documentation to help people start using it without there being too 
> > many complications involved.

It's embedded, there will always be room for complications just as
there is always room for documentation.

The key is to not break the underlying models in order to simplify the
documentation/setup at the cost of making the system fragile.


Neil Williams

Attachment: pgpB6DVbYXVGw.pgp
Description: PGP signature

Reply to: