Debian port bootstrap build ordering tool status
Hi,
On Fri, Nov 23, 2012 at 03:48:04AM +0000, peter green wrote:
> >Since yesterday, my tools can now finally turn the whole dependency
> >graph
> Does this "whole dependency graph" include the implicit
> build-dependency every package has on build-essential?
For source packages for native compilation, yes. What is not included is
Build-Depends-Indep. I have been told that there are many problems with
Build-Depends-Indep but not considering those dependencies tremendously
helps to reduced the amount of cyclic dependency hell, so they should be
made working for better bootstrappability.
For binary packages, the implicit dependency on priority:essential is
kept.
For source packages for cross compilation, afaik wookey currently uses
an implicit dependency on crossbuild-essential-$arch but this is not yet
policy of course and has to be discussed still.
> >The above case for example has no alternative solution as the cycle
> >is of length two and has no other way of braking it than building
> >pkg-config without libglib2.0-dev. Since this is unlikely to be
> >possible
> I don't see why it would be impossible to hack up the glib source
> package to not rely on pkg-config. Whether that is a good idea or
> not is another matter.
It is not a dependency loop between the pkg-config source package and
the glib2.0 source package.
It is a dependency loop between src:pkg-config (we use the src: prefix
to never confuse source package names with binary package names) and
libglib2.0-dev. There is a loop because libglib2.0-dev has a binary
dependency on pkg-config but the source package that pkg-config builds
from (src:pkg-config) cannot be built without libglib2.0-dev. So even if
the glib2.0 source package can be built and therefor make libglib2.0-dev
available, libglib2.0-dev can never be installed because it has a binary
dependency on pkg-config. pkg-config builds from src:pkg-config but
src:pkg-config can never be compiled because it has a build dependency
on libglib2.0-dev, which cannot be installed -> loop.
The assumption that this specific loop cannot trivially (without cross
compilation) be broken depends on the following two assumptions:
- binary dependencies should never be modified
- src:pkg-config cannot be built without libglib2.0-dev
> > and since the assumption is that only
> >build dependencies might be dropped when necessary but not binary
> >dependencies, a possible solution might be cross compilation.
> It seems pretty clear to me that there is a "core" of software that
> will need to be cross-built as the first stage of bootstrapping a
> port. Obviously "essential" and "build-essential" fall into this
> category but while i'm sure there are ways one could hack away the
> cycles and make things like pkg-config and debhelper natively
> bootstrapable I don't think there is much point in doing so.
It is always the call of the user to decide what is easier:
- to bootstrap package A natively by braking cycles or
- to make all source packages that need to be cross compiled to have
package A installable cross compile
Both tasks (finding out if the necessary source packages can be built
with reduced build dependencies and checking how easy it is to make them
cross compilable) can only be solved by a human.
My tools help the user in that task by allowing to analyze the
dependency graph situation for native compilation and also allow to
investigate the list of source packages that have otherwise to be cross
compiled to make the package installable. Comparing these two options
allows the user to make a better call when he has to decide for either
of those two choices.
But right now, debhelper is indeed taken as part of the packages to be
cross compiled to avoid a horribly messy initial dependency graph.
> What i'd ideally like to see is for a tool to be able to generate a
> directed acyclic graph of "build jobs" (some cross, some native, there
> should be an option in the tool as to whether to preffer native or
> cross-build jobs) that takes the user from having no packages for the
> target architecture to having a set of bootstrap packages that can be
> used to seed the "regular" building process.
This is the final goal of my project.
What is currently done is the native part of the above.
Given a minimal build system (build-essential, priority:essential) it
can devise said order of "build jobs" by braking the graph into a
directed acyclic graph (DAG). The tool helps the user to find build
dependencies that make most sense to be broken and uses a list of
brake-able build dependencies supplied by the user to break the graph
into DAG with a close to minimal number of source packages that have to
be built with reduced build dependencies.
Once Debian understands "build profiles" (see my last email) those will
of course be taken into account as well. Until this is the case, my tool
can help to identify those source packages that make sense to have build
dependencies dropped.
In a state where Debian understands "build profiles", my current code
can be used to find useful build profiles and to check if the available
build profiles are enough to bootstrap Debian and if not, point out
which source packages should best be modified to make it bootstrappable
again.
Coming back to the cross case.
My tool is using dose3 for dependency resolution. Since my summer
project, dose3 can also be used to resolve cross build dependencies. The
thing that is currently stopping me from generating a cross build
dependency graph and brake it into a DAG just as I do for native
compilation, is the current status of Multi-Arch.
For multiarch cross building to successfully resolve cross build
dependencies, more binary packages must be properly multiarch-ed. The
last time I checked the status of multiarch was a month ago. I heard
from wookey that since then he made considerable progremms with his
arm64 port, so I will look at the cross build situation again.
Once all necessary packages have sufficient multiarch properties, it
will be possible to build a cross build dependency graph and brake it
using the exact same algorithms that I already developed and
successfully tested for the native case.
Then, it can be done what you outline above, to devise a build order
that goes from zero to everything that is needed to build all of Debian
fully.
You also raise the point of having the option to "prefer" native or
cross build jobs. Sometimes (with our current assumptions) a native
build dependency cycle cannot be broken other than by cross compiling
source packages. For the other cycles there exist both options: braking
build dependencies or cross compilation.
I think if the algorithm knows (using build profiles) that enough build
dependencies can be broken, native compilation should usually be
preferred, right?
On the other hand, if the current set of build profiles doesnt provide
enough droppable dependencies for a cycle to be broken, the algorithm
(this is is what it already can do) will present the user with the
choice of either braking some build dependencies (suggestions are given)
or by cross compiling a number of packages.
What would be very nice: if my algorithm had a way of knowing which
source packages are supposed to be cross compilable. But there is
currently no way to mark a package as being cross-compilable. This makes
sense because Debian packages are supposed to be compiled natively. On
the other hand, in many bootstrapping scenarios there is no other choice
than native compilation (if one does not want to involve another
distribution like Gentoo or Openembedded).
So maybe Debian will want to include such a flag in the source package
description at some point so that future bootstrappers have an easier
time to pick packages for cross compilation. Additionally, the packages
needed for a minimal build system (priority:essential + build-essential)
should require this flag to be set to true. And additionally if a source
package has set this flag to true, it should be considered as a bug if
the source package does not cross compile. The flag should then be
removed (if it is not part of the minimal system) or the problem be
fixed.
My algorithm could then make suggestions like:
"here is a native build dependency cycle that cannot be broken with the
current knowledge I have about droppable build dependencies, but I see
that the source packages required to break the cycle at a specific point
are all supposedly cross compilable, so you might want to mark them to
be cross compiled"
Thoughts?
cheers, josch
Reply to: