Bootstrappable Debian - Report 3 + N + 1
This is a report for the "Bootstrappable Debian" project  mentored
by Wookey and co-mentored by Jonathan Austin.
There seems to be some confusion about the ordinal number of this week's
report; the schedule numbers it fifth, but the previous report was the
third. So I've numbered mine 3 + N + 1, where N is either 0 or 1
according to reader preference. :)
Copies of this report are sent to the debian-bootstrap  and
My first three reports were organized primarily into "Work Done" and
"Next Steps". I've organized this one primarily by the areas of work
I've done and plan to do yet.
A number of package build tools (especially dpkg and sbuild, for our
testing purposes) need to be modified to support reduced dependency
information in those source packages that will be buildable in "staged"
Back in May, Guillem Jover suggested  an alternative to
"Build-Depends-Stage1", etc. fields. He referred to a document  he'd
written that proposes methods of conditionally reducing build
dependencies. Two such methods are "build profiles" (with fields like
"Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !bootstrap>, tiny")
and "purpose overrides" (with fields like "Build-Depends[embedded
On 2012-07-10, Wookey wrote  that purpose overrides "seems a lot
nicer than adding lots of Build-Depends-StageN fields". Purpose
overrides would offer the same functionality that the previously
considered new fields would but also offer a more generic syntax.
Guillem then noted  that he meant to propose build profiles rather
than purpose overrides. The latter proposal, as do the
Build-Depends-StageN fields, involves the duplication of build
Out of curiosity, I implemented  build profiles in dpkg and found
profile specification parsing quite simple to add (much simpler than
pattern-based fields were to add – I had dpkg-checkbuilddeps handling
profiles in only about half an hour).
The patch makes dpkg-checkbuilddeps and Dpkg::Deps parse and reduce
build profiles (e.g. "<!stage1 !nodocs>") and makes dpkg-gencontrol add
a new "Build-Profile" field to packages built with a profile.
The interfaces for specifying a build profile are modelled after those
for specifying a host architecture. They include a "-P<profile>" option
to dpkg-buildpackage and dpkg-checkbuilddeps and a "DEB_BUILD_PROFILE"
environment variable to be used by debian/rules, dpkg-checkbuilddeps,
Wookey, Johannes Schauer, and I agree that build profiles appear to be
the best solution, but we're not yet sure if all of the staged
dependency information can be easily expressed in the profiles syntax.
So I wrote a Perl script called dpkg-prof2purp.pl  to convert build
dependency fields in a debian/control file from the profiles syntax to
the purpose overrides one, should we eventually need to convert my work.
I've been testing my build profiles patch against packages (see
dependency removal work below) and will continue to do so. Pending
further testing and a future discussion on the debian-devel mailing
list, this patch should eventually be included in dpkg. Unfortunately
it seems we missed the chance to get support for staged packages into
Many other programs will have to be modified to support profile
specifications, including sbuild (see below) and APT.
I patched  sbuild to support build profiles. Cross building
packages that have profile specifications won't work yet however,
because sbuild uses APT to get the list of cross build dependencies and
APT doesn't yet understand profile specifications.
With this patch, the user specifies a profile to build using the
"--profile" option. Wookey and I decided that sbuild should also obey
the environment variable that specifies the profile. I plan to add that
behavior to the patch.
As with the dpkg one, this patch will be submitted for inclusion in
sbuild pending further testing.
Workflow Assistant Script
I briefly mentioned in my previous report that I had begun working on a
script written in shell command language to partially automate my cross
building workflow. I since realized that it can have the same effect on
my dependency cycle breaking workflow.
I spent a few more hours on the script to "finish" it. It could be
improved, but it works well enough for my needs so far and makes my work
It allows me to easily:
* Download Debian source packages (for sid – or any distribution I
configure – even though I have only wheezy in my APT sources),
* Unpack downloaded source packages for editing,
* Build edited source packages (and patches generated by debdiff),
* Build binary packages from the modified source packages (with a
configurable build command), and
* Upload all my work to my server.
The script may be found in a Git repository  on my server. It
lacks documentation for now, so anyone interested can just ask me any
questions about it they might have.
Dependency Cycle Breaking
The core of this project is to break build dependency cycles – to remove
edges (dependencies) and vertices (packages) from the cyclic dependency
As Johannes Schauer mentioned in his report , we have a number of
strategies for removing edges (and hopefully vertices) from the cyclic
I previously focused on cross building, which makes use of packages
already built for the build architecture to break dependency cycles and
build packages for a host architecture. See my previous report  for
more information on my work to cross build the base build system.
At the advice of Wookey, I've moved away from cross building for now and
focused instead on staged building – the building of packages with
reduced build dependencies. I do this by adding negative profile
specifications such as "<!stage1 !nodocs>" to the "Build-Depends" fields
of source packages and building the packages in the "stage1" profile
using my patched sbuild and dpkg packages.
Johannes suggested  that some dependency cycles could be avoided
simply by ignoring "Build-Depends-Indep" fields in packages, since
architecture-independent packages need not be built when bootstrapping a
Debian architecture. Wookey explained  that Build-Depends-Indep is
still a relatively new feature in source packages and in build tools. I
observed that recent versions of sbuild correctly handle the field; and
Wookey, Johannes, and I all agreed  that moving dependencies from
Build-Depends into Build-Depends-Indep is a useful strategy for breaking
Finding Dependencies to Remove
I started out by attempting to remove dependencies that occur in the
greatest number of cycles. As Johannes noted on IRC, this is
inefficient because there are billions of dependency cycles in Debian
Going forward, I will look at:
* Vertices (packages) in the cyclic dependency graph with the smallest
degrees, since removing the edges (dependencies) connected to them
can easily remove entire vertices from the graph, and
* Small cycles, since these contain dependencies that must be removed
and breaking small cycles can simultaneously break some larger
cycles (while the reverse is not true).
I've so far removed 27 build dependencies from 10 source packages :
atk1.0, atlas, avahi, dbus, gnutls26 krb5, libgcrypt11, libtasn1-3,
poppler, and qt4-x11.
I'll summarize my work rather than explain the changes I made to each
In atlas, dbus, gnutls26, and libgcrypt11, I moved build dependencies
into Build-Depends-Indep. (I have not yet been able to fully test my
changes to atlas, as the software takes a long time to build and the
previous build was interrupted.)
I added a "stage1" build profile to avahi, krb5, and qt4-x11. I added
"stage1" and "nodocs" profiles to atk1.0, libtasn1-3, and poppler. I
was even able to remove an unused dependency on libgtk2.0-dev from
Find on my server patches  and build logs  for all source
packages. (So far there's no patch for qt4-x11, because debdiff was
unable to unpack the large original tar archive in my small /tmp
filesystem.) These files are managed by the workflow assistant script I
I'm gradually submitting bug reports  with patches to move build
dependencies to Build-Depends-Indep and remove any unused dependencies.
"Sources" File Patch and Weak Build Dependencies List
At Johannes's request, I began maintaining :
* A patch  (automatically generated  twice a day) against the
Debian sid "Sources" file to add my changes to Build-Depends and
Build-Depends-Indep fields and
* A list  of "weak" build dependencies (binary packages used by
source packages for non-critical things like documentation
generation), which I will eventually push into Johannes's bootstrap
tools Git repository .
In my original project schedule , I overestimated the amount of work
I needed to do myself.
My progress was rather slow in the first half of the program in part
because I underestimated the scope and complexity of some tasks (such as
patching package build tools).
I did the cross and staged building tasks in the order opposite that
which I had originally planned. Additionally, I had planned to do more
work than can reasonably be accomplished in one summer; I've had to put
aside my cross building work to focus on staged building (in some ways
the most important part of this project).
I plan to (more casually) continue after the summer to help make more
Debian packages support cross building.
P. J. McDermott (_/@\_) ,--.
http://www.pehjota.net/ o < o o > / oo \
http://www.pehjota.net/contact.html o \ `-/ | <> |.
o o o "~v /_\--/_/