strap1: an alternative to debootstrap
Dear Debian developers,
I have just written this paper about strap1, a component of hackable:1
(also described below). I think it can be of interest for both the
-embedded and -boot lists, my apologies otherwise.
I have also uploaded this paper here:
http://people.defora.org/~khorben/papers/hackable1/strap1.txt
Good read,
-- khorben
strap1: an alternative to debootstrap
=====================================
Synopsis
--------
strap1 is part of hackable:1, a GNU/Linux community distribution for
hackable devices. hackable:1 is based on Debian, with a strong focus on
GNOME Mobile technologies integration. The idea behind it is to make
embedded development easier, while benefiting of the many strength of
Debian as a portable Operating System.
The target platforms are hackable devices, hardware designed in an Open
Source fashion. Any such hardware able to run the Linux kernel is a
potential target for hackable:1. The first focus is on the Openmoko
Freerunner phone, a modern, touch-based mobile phone developed for the
Open Source community.
An advantage of hackable devices is that they can run any software its
owner might like. However, it is not trivial to tune and generate
filesystem images that might fulfil given purposes directly.
strap1 addresses this exact problem, generating final, ready to run
software images from a number of Debian repositories. It resembles
Debian's debootstrap utility in many ways, which are highlighted at the
end of this paper.
1. About strap1
---------------
1.1 Requirements
----------------
strap1 should run as-is on any GNU/Linux distribution, and even on other
Unix systems (like BSD). It depends on these specific packages in
particular:
- binutils (for ar and strip)
- fakeroot (optional)
- grep
- gzip
- mtdutils (depending on the target hardware)
- sed
- sh
- sudo (optional)
- tar
- wget
The core component of strap1 consists of a simple POSIX shell script,
currently containing less than a thousand lines of code.
1.2 Obtaining strap1
--------------------
strap1 is found inside hackable:1's source code management system. It
uses SVN, and can be obtained this way:
$ svn co svn://trac.hackable1.org/hackable1
It is then found inside the trunk/build directory inside hackable:1's
SVN tree.
There is no official release of strap1 yet.
2. Project structure
--------------------
strap1 consists of the following components:
- build.sh, the script invoked to create images
- profiles, defining the target hardware images for their respective
purposes
- packages, containing the packages post-install configuration scripts
- depends, a staging directory where dependencies are resolved
- destdir, another staging directory where the filesystem is prepared
2.1 build.sh
------------
build.sh is a shell script, joining the different pieces together. It
accepts the following syntax:
$ ./build.sh
Usage: build.sh [option=value...] target...
Targets:
archive Create an archive of an image contents
clean Remove temporary data
config Configure installed packages
help Display this help screen
image Create a native image file
list List the available profiles
Options:
VENDOR (Openmoko)
MODEL (Freerunner)
PURPOSE (user)
A profile consists of a VENDOR-MODEL-PURPOSE combination. Any internal
variable from build.sh, or within the given profile can also be
overriden as an option.
Some typical examples:
To build a JFFS2 flash image for the Openmoko Freerunner:
$ ./build.sh VENDOR=Openmoko MODEL=Freerunner image
To build an image archive for the Openmoko Freerunner, with a developer
environment:
$ ./build.sh VENDOR=Openmoko MODEL=Freerunner PURPOSE=developer archive
Useful options include:
DESTDIR The staging directory to use while building the filesystem
MODEL The model name of the target hardware platform
PURPOSE The intended purpose of the image to generate
USERNAME The name of the user account on the image
VENDOR The vendor name of the target hardware platform
VERSION The version number or string to use for this image
More advanced options and variables are also detailed below.
2.2 Profiles
------------
The profiles directory contains one file per supported
VENDOR-MODEL-PURPOSE combination (with the ".profile" extention), which
in turn may include common files (with the ".include" extention).
They typically only define a number of variables, of which the most
relevant are:
CLEAN_APT "yes" removes the packages database
CLEAN_DEVEL "yes" removes include files and development libraries
CLEAN_DOC "yes" removes any documentation
CLEAN_KERNEL "yes" removes the kernel image
CLEAN_LOCALES "yes" removes the locales information
CLEAN_STRIP "yes" strips binaries and libraries
DEBIAN_ARCH the Debian port to use
DISTRIBUTIONS the Debian repositories to use
PACKAGES_BLACKLIST a list of packages not to install
PACKAGES_PRIORITY packages with either one of these priorities are installed
PACKAGES an additional list of packages to install
These can also be overriden on the command line with build.sh.
2.3 Packages configuration
--------------------------
As packages are simply extracted into the staging directory before
building the actual image, they are configured with their default
settings, or sometimes not at all. Each package can be configured inside
the packages directory, within a shell script of the same name.
These shell scripts rely on a number of variables. The options defined
above can be used to alter the behaviour of the scripts, and hence the
final configuration of the images to be generated (eg the
VENDOR-MODEL-PURPOSE profile).
Other essential commands are available, and can be overriden as well:
AR Create, analyze and extract archives
CHMOD Change the permissions of files and directories
CHOWN Change the ownership of files and directories
CP Copy files and directories
CUT Process text data on the command-line
DU Query the disk space used by files and directories
GREP Look for given text patterns in files or on the command-line
GUNZIP Decompress files
LN Create and modify hard and symbolic links
MKDIR Create directories
MKNOD Create device nodes
MV Rename and move files and directories
RMDIR Delete directories
RM Delete files and directories
SED Replace content in files or on the command-line
STRIP Remove debugging
TAR Create, analyze and extract archives
TOUCH Create files and alter timestamps
WGET Download files over HTTP or FTP
Some examples follow.
Within packages/base-passwd:
#/etc/passwd
$SUDO$MKDIR "$DESTDIR/home/$USERNAME" || exit 2
$SUDO$CHMOD 0750 "$DESTDIR/home/$USERNAME" || exit 2
$SUDO$CHOWN 1000:1000 "$DESTDIR/home/$USERNAME" || exit 2
${SUDO}sh -c "cat > \"$DESTDIR/etc/passwd\"" << EOF
root:x:0:0:root:/root:/bin/sh
$USERNAME:x:1000:1000:Hackable1 user:/home/hackable1:/bin/sh
EOF
Within packages/bluez-utils:
#/etc/rc?.d/S25bluetooth
for i in 2 3 4 5; do
$SUDO$MKDIR "$DESTDIR/etc/rc$i.d" || exit 2
$SUDO$LN -s "../init.d/bluetooth"
"$DESTDIR/etc/rc$i.d/S25bluetooth" \
|| exit 2
done
Within packages/fakeroot:
#/usr/bin/fakeroot
$SUDO$MKDIR "$DESTDIR/etc/alternatives" "$DESTDIR/usr/bin" || exit 2
$SUDO$LN -s "/usr/bin/fakeroot-tcp" "$DESTDIR/etc/alternatives/fakeroot" \
|| exit 2
$SUDO$LN -s "/etc/alternatives/fakeroot" "$DESTDIR/usr/bin/fakeroot" \
|| exit 2
3. Generating images
--------------------
The operational mode of strap1 is fairly simple, and can be decomposed
this way:
1. Resolve and cache dependencies
2. Extract the required packages
3. Create the device nodes
4. Configure the packages installed
5. Clean up the filesystem as required
6. Generate the final image
3.1 Packages dependencies
-------------------------
Debian packages repositories contain a Packages file, gathering the
meta-data from the different packages in one convenient place. This is
particularly welcome when resolving dependencies, and therefore listing
the packages to be installed. There are a few relevant parameters there:
- priorities: the essential packages have a particular "Priority"
header, like "required", "important" or "standard"; if it matches
PACKAGES_PRIORITY, the packages gets selected;
- dependencies: the "Depends", "Pre-Depends" and "Recommends" headers
are also used to select packages.
Two files are created during this process:
- depends/VENDOR-MODEL-PURPOSE.Depends
- depends/VENDOR-MODEL-PURPOSE.Packages
The "Depends" file contains the list of the packages selected for
inclusion, while the "Packages" file gathers all of the meta-data in one
file. The latter resembles dpkg's "available" file.
3.2 Packages extraction
-----------------------
Debian packages consist of an ar archive, with a meta-data archive
member, and an optional archive to decompress. The extraction is
performed without the dpkg tool, using the ar, gzip, bzip2 and tar
utilities as required.
The packages database is completed at the same time, always appending
information to the "available" database, and appending information to
the "status" database for the packages extracted. The list of files
extracted, and the package meta-data is also placed in
"/var/lib/dpkg/info". Minor modifications to ar's output are also
necessary to format the "package.list" file properly.
3.3 Device nodes creation
-------------------------
At this stage, device nodes are created. The regular device node
creation script from Debian is used in this process, with the
appropriate architecture specified on the command line.
This phase is only performed if the sbin/MAKEDEV script was installed on
the target platform. Otherwise, it has to be performed from within the
packages configuration scripts.
This phase is the only one currently known not to be portable. It is
automatically skipped by Debian's device node creation script when it
detects a non-Linux system.
3.4 Configure the packages installed
------------------------------------
As packages get extracted, they also need to be configured in the
context of the image generated. Many packages also actually require some
scripts and executables to be ran right before or after extraction.
Unfortunately, in the case of hackable:1, the staging filesystem is
often compiled for a different architecture than the native host. It is
therefore impossible to run the configuration scripts. Instead, the
necessary steps are reproduced within the "packages" directory.
For each configuration script found in the "packages" directory, it is
checked if the package of the same name was actually installed on the
filesystem (by testing the presence of the
"/var/lib/dpkg/info/package.list" file). If it is the case, the
configuration script is sourced from within the build.sh script.
It is possible to reproduce this phase at will, using the "config"
target of the build.sh script. Beware that this can only work if the
packages database information was kept (eg without the CLEAN_APT option
set to "yes").
3.5 Clean up the filesystem as required
---------------------------------------
Some operations can then be conducted in order to gain space. They were
mentioned in the 2.2 section:
CLEAN_APT Deletes everything in /var/lib/dpkg
CLEAN_DEVEL Deletes /usr/include, /usr/lib/pkgconfig, /usr/lib/*.a...
CLEAN_DOC Deletes /usr/share/doc, /usr/share/man...
CLEAN_KERNEL Deletes /boot
CLEAN_LOCALES Deletes /usr/share/locales
CLEAN_STRIP Strips all the binaries found in /usr/bin, /usr/lib...
3.6 Generate the final image
----------------------------
Lastly, the filesystem is either saved as a compressed tar archive (the
"archive" target), or packed as a ready-to-flash software image (the
"image" target). Unfortunately, this phase is currently hard-coded
within the build.sh script.
4. Comparison with debootstrap
------------------------------
In fact, strap1 implements parts of deboostrap, and extends it in some
regards.
4.1 Why debootstrap is not enough
---------------------------------
deboostrap is also useful to create Debian systems from scratch, without
requiring dpkg or apt either. It is actually used by the Debian
installer to initialize the system being installed. debootstrap consists
of several thousands of lines of perl, of which cdebootstrap is an
alternative implementation in C.
However, both assume that the installer will resume the installation by
chrooting to the target system, and then install and configure packages
using the native tools. In many cases, this is not likely during
embedded development.
4.2 Limitations of debootstrap
------------------------------
First, in Debian packages the configuration process is relying on the
possibility to run native binary programs from within the target system.
This is used to either automatically detect, or ask the user
specifically the settings to apply.
Moreover, debootstrap can not conveniently be used to generate systems
from multiple source repositories. It is also not able to initialize the
packages database as well as apt does itself.
Finally, debootstrap stops at the packages extraction phase. It does not
provide filesystem images ready to flash, nor a way to conveniently
automatically reproduce images from updated packages.
4.3 Advantages of strap1
------------------------
strap1 addresses some of these issues already:
- it can use multiple source repositories at once
- it can generate valid packages databases, almost as complete as what
apt does itself
- it can configure packages cross-platform
- it can generate final images.
It is also very portable, and easy to extend: it consists only of a
simple core script, along with individual profiles and configuration files.
4.4 Remaining issues
--------------------
Dependencies are not resolved recursively at the moment (this problem is
also found in debootstrap). The images generated may therefore be
incomplete in some cases.
Device nodes creation is not portable across Unix systems. This can
sometimes be work-arounded with a tar archive of the necessary nodes,
however it cannot be conveniently forced during the process at the moment.
The stock Debian packages may not always fit in an embedded environment.
More cooperation with the Emdebian project is certainly desirable.
The overall performance could be improved.
5. Conclusion
-------------
Even with these current limitations in mind, and with still too few
supported hardware platforms supported at the time, strap1 has the
potential to widen the range of devices running Debian. You are welcome
to join the hackable:1 project, and help us contribute to Debian's
community. We are of course found on the web:
- http://www.hackable1.org/
- http://trac.hackable1.org/ (for users and developers alike)
You can also join us on IRC:
- #hackable1 on the Freenode network
We will all welcome your feedback and ideas!
--
khorben
Reply to: