[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

strap1: an alternative to debootstrap

			Dear Debian developers,

I have just written this paper about strap1, a component of hackable:1 (also described below). I think it can be of interest for both the -embedded and -boot lists, my apologies otherwise.

I have also uploaded this paper here:

Good read,
-- khorben

strap1: an alternative to debootstrap


strap1 is part of hackable:1, a GNU/Linux community distribution for hackable devices. hackable:1 is based on Debian, with a strong focus on GNOME Mobile technologies integration. The idea behind it is to make embedded development easier, while benefiting of the many strength of Debian as a portable Operating System.

The target platforms are hackable devices, hardware designed in an Open Source fashion. Any such hardware able to run the Linux kernel is a potential target for hackable:1. The first focus is on the Openmoko Freerunner phone, a modern, touch-based mobile phone developed for the Open Source community.

An advantage of hackable devices is that they can run any software its owner might like. However, it is not trivial to tune and generate filesystem images that might fulfil given purposes directly.

strap1 addresses this exact problem, generating final, ready to run software images from a number of Debian repositories. It resembles Debian's debootstrap utility in many ways, which are highlighted at the end of this paper.

1. About strap1

1.1 Requirements

strap1 should run as-is on any GNU/Linux distribution, and even on other Unix systems (like BSD). It depends on these specific packages in particular:

- binutils (for ar and strip)
- fakeroot (optional)
- grep
- gzip
- mtdutils (depending on the target hardware)
- sed
- sh
- sudo (optional)
- tar
- wget

The core component of strap1 consists of a simple POSIX shell script, currently containing less than a thousand lines of code.

1.2 Obtaining strap1

strap1 is found inside hackable:1's source code management system. It uses SVN, and can be obtained this way:

$ svn co svn://trac.hackable1.org/hackable1

It is then found inside the trunk/build directory inside hackable:1's SVN tree.

There is no official release of strap1 yet.

2. Project structure

strap1 consists of the following components:

- build.sh, the script invoked to create images
- profiles, defining the target hardware images for their respective purposes
- packages, containing the packages post-install configuration scripts
- depends, a staging directory where dependencies are resolved
- destdir, another staging directory where the filesystem is prepared

2.1 build.sh

build.sh is a shell script, joining the different pieces together. It accepts the following syntax:

$ ./build.sh
Usage: build.sh [option=value...] target...
  archive       Create an archive of an image contents
  clean         Remove temporary data
  config        Configure installed packages
  help          Display this help screen
  image         Create a native image file
  list          List the available profiles
  VENDOR        (Openmoko)
  MODEL         (Freerunner)
  PURPOSE       (user)

A profile consists of a VENDOR-MODEL-PURPOSE combination. Any internal variable from build.sh, or within the given profile can also be overriden as an option.

Some typical examples:

To build a JFFS2 flash image for the Openmoko Freerunner:
$ ./build.sh VENDOR=Openmoko MODEL=Freerunner image

To build an image archive for the Openmoko Freerunner, with a developer environment:
$ ./build.sh VENDOR=Openmoko MODEL=Freerunner PURPOSE=developer archive

Useful options include:

DESTDIR		The staging directory to use while building the filesystem
MODEL		The model name of the target hardware platform
PURPOSE		The intended purpose of the image to generate
USERNAME	The name of the user account on the image
VENDOR		The vendor name of the target hardware platform
VERSION		The version number or string to use for this image

More advanced options and variables are also detailed below.

2.2 Profiles

The profiles directory contains one file per supported VENDOR-MODEL-PURPOSE combination (with the ".profile" extention), which in turn may include common files (with the ".include" extention).

They typically only define a number of variables, of which the most relevant are:

CLEAN_APT		"yes" removes the packages database
CLEAN_DEVEL		"yes" removes include files and development libraries
CLEAN_DOC		"yes" removes any documentation
CLEAN_KERNEL		"yes" removes the kernel image
CLEAN_LOCALES		"yes" removes the locales information
CLEAN_STRIP		"yes" strips binaries and libraries
DEBIAN_ARCH		the Debian port to use
DISTRIBUTIONS		the Debian repositories to use
PACKAGES_BLACKLIST	a list of packages not to install
PACKAGES_PRIORITY	packages with either one of these priorities are installed
PACKAGES		an additional list of packages to install

These can also be overriden on the command line with build.sh.

2.3 Packages configuration

As packages are simply extracted into the staging directory before building the actual image, they are configured with their default settings, or sometimes not at all. Each package can be configured inside the packages directory, within a shell script of the same name.

These shell scripts rely on a number of variables. The options defined above can be used to alter the behaviour of the scripts, and hence the final configuration of the images to be generated (eg the VENDOR-MODEL-PURPOSE profile).

Other essential commands are available, and can be overriden as well:

AR	Create, analyze and extract archives
CHMOD	Change the permissions of files and directories
CHOWN	Change the ownership of files and directories
CP	Copy files and directories
CUT	Process text data on the command-line
DU	Query the disk space used by files and directories
GREP	Look for given text patterns in files or on the command-line
GUNZIP	Decompress files
LN	Create and modify hard and symbolic links
MKDIR	Create directories
MKNOD	Create device nodes
MV	Rename and move files and directories
RMDIR	Delete directories
RM	Delete files and directories
SED	Replace content in files or on the command-line
STRIP	Remove debugging
TAR	Create, analyze and extract archives
TOUCH	Create files and alter timestamps
WGET	Download files over HTTP or FTP

Some examples follow.

Within packages/base-passwd:

$SUDO$MKDIR "$DESTDIR/home/$USERNAME"                           || exit 2
$SUDO$CHMOD 0750 "$DESTDIR/home/$USERNAME"                      || exit 2
$SUDO$CHOWN 1000:1000 "$DESTDIR/home/$USERNAME"                 || exit 2
${SUDO}sh -c "cat > \"$DESTDIR/etc/passwd\"" << EOF
$USERNAME:x:1000:1000:Hackable1 user:/home/hackable1:/bin/sh

Within packages/bluez-utils:

for i in 2 3 4 5; do
        $SUDO$MKDIR "$DESTDIR/etc/rc$i.d"                       || exit 2
$SUDO$LN -s "../init.d/bluetooth" "$DESTDIR/etc/rc$i.d/S25bluetooth" \
                                                                || exit 2

Within packages/fakeroot:

$SUDO$MKDIR "$DESTDIR/etc/alternatives" "$DESTDIR/usr/bin"      || exit 2
$SUDO$LN -s "/usr/bin/fakeroot-tcp" "$DESTDIR/etc/alternatives/fakeroot" \
                                                                || exit 2
$SUDO$LN -s "/etc/alternatives/fakeroot" "$DESTDIR/usr/bin/fakeroot" \
                                                                || exit 2

3. Generating images

The operational mode of strap1 is fairly simple, and can be decomposed this way:

1. Resolve and cache dependencies
2. Extract the required packages
3. Create the device nodes
4. Configure the packages installed
5. Clean up the filesystem as required
6. Generate the final image

3.1 Packages dependencies

Debian packages repositories contain a Packages file, gathering the meta-data from the different packages in one convenient place. This is particularly welcome when resolving dependencies, and therefore listing the packages to be installed. There are a few relevant parameters there:

- priorities: the essential packages have a particular "Priority" header, like "required", "important" or "standard"; if it matches PACKAGES_PRIORITY, the packages gets selected; - dependencies: the "Depends", "Pre-Depends" and "Recommends" headers are also used to select packages.

Two files are created during this process:
- depends/VENDOR-MODEL-PURPOSE.Depends
- depends/VENDOR-MODEL-PURPOSE.Packages

The "Depends" file contains the list of the packages selected for inclusion, while the "Packages" file gathers all of the meta-data in one file. The latter resembles dpkg's "available" file.

3.2 Packages extraction

Debian packages consist of an ar archive, with a meta-data archive member, and an optional archive to decompress. The extraction is performed without the dpkg tool, using the ar, gzip, bzip2 and tar utilities as required.

The packages database is completed at the same time, always appending information to the "available" database, and appending information to the "status" database for the packages extracted. The list of files extracted, and the package meta-data is also placed in "/var/lib/dpkg/info". Minor modifications to ar's output are also necessary to format the "package.list" file properly.

3.3 Device nodes creation

At this stage, device nodes are created. The regular device node creation script from Debian is used in this process, with the appropriate architecture specified on the command line.

This phase is only performed if the sbin/MAKEDEV script was installed on the target platform. Otherwise, it has to be performed from within the packages configuration scripts.

This phase is the only one currently known not to be portable. It is automatically skipped by Debian's device node creation script when it detects a non-Linux system.

3.4 Configure the packages installed

As packages get extracted, they also need to be configured in the context of the image generated. Many packages also actually require some scripts and executables to be ran right before or after extraction.

Unfortunately, in the case of hackable:1, the staging filesystem is often compiled for a different architecture than the native host. It is therefore impossible to run the configuration scripts. Instead, the necessary steps are reproduced within the "packages" directory.

For each configuration script found in the "packages" directory, it is checked if the package of the same name was actually installed on the filesystem (by testing the presence of the "/var/lib/dpkg/info/package.list" file). If it is the case, the configuration script is sourced from within the build.sh script.

It is possible to reproduce this phase at will, using the "config" target of the build.sh script. Beware that this can only work if the packages database information was kept (eg without the CLEAN_APT option set to "yes").

3.5 Clean up the filesystem as required

Some operations can then be conducted in order to gain space. They were mentioned in the 2.2 section:

CLEAN_APT		Deletes everything in /var/lib/dpkg
CLEAN_DEVEL		Deletes /usr/include, /usr/lib/pkgconfig, /usr/lib/*.a...
CLEAN_DOC		Deletes /usr/share/doc, /usr/share/man...
CLEAN_KERNEL		Deletes /boot
CLEAN_LOCALES		Deletes /usr/share/locales
CLEAN_STRIP		Strips all the binaries found in /usr/bin, /usr/lib...

3.6 Generate the final image

Lastly, the filesystem is either saved as a compressed tar archive (the "archive" target), or packed as a ready-to-flash software image (the "image" target). Unfortunately, this phase is currently hard-coded within the build.sh script.

4. Comparison with debootstrap

In fact, strap1 implements parts of deboostrap, and extends it in some regards.

4.1 Why debootstrap is not enough

deboostrap is also useful to create Debian systems from scratch, without requiring dpkg or apt either. It is actually used by the Debian installer to initialize the system being installed. debootstrap consists of several thousands of lines of perl, of which cdebootstrap is an alternative implementation in C.

However, both assume that the installer will resume the installation by chrooting to the target system, and then install and configure packages using the native tools. In many cases, this is not likely during embedded development.

4.2 Limitations of debootstrap

First, in Debian packages the configuration process is relying on the possibility to run native binary programs from within the target system. This is used to either automatically detect, or ask the user specifically the settings to apply.

Moreover, debootstrap can not conveniently be used to generate systems from multiple source repositories. It is also not able to initialize the packages database as well as apt does itself.

Finally, debootstrap stops at the packages extraction phase. It does not provide filesystem images ready to flash, nor a way to conveniently automatically reproduce images from updated packages.

4.3 Advantages of strap1

strap1 addresses some of these issues already:

- it can use multiple source repositories at once
- it can generate valid packages databases, almost as complete as what apt does itself
- it can configure packages cross-platform
- it can generate final images.

It is also very portable, and easy to extend: it consists only of a simple core script, along with individual profiles and configuration files.

4.4 Remaining issues

Dependencies are not resolved recursively at the moment (this problem is also found in debootstrap). The images generated may therefore be incomplete in some cases.

Device nodes creation is not portable across Unix systems. This can sometimes be work-arounded with a tar archive of the necessary nodes, however it cannot be conveniently forced during the process at the moment.

The stock Debian packages may not always fit in an embedded environment. More cooperation with the Emdebian project is certainly desirable.

The overall performance could be improved.

5. Conclusion

Even with these current limitations in mind, and with still too few supported hardware platforms supported at the time, strap1 has the potential to widen the range of devices running Debian. You are welcome to join the hackable:1 project, and help us contribute to Debian's community. We are of course found on the web:

- http://www.hackable1.org/
- http://trac.hackable1.org/ (for users and developers alike)

You can also join us on IRC:

- #hackable1 on the Freenode network

We will all welcome your feedback and ideas!

Reply to: