dh --parallel (was: Re: RFS: lebiniou)
On Mon, Oct 17, 2011 at 12:31:06PM +0800, Paul Wise wrote:
> You might want to use dh --parallel.
I really wonder why it's not debhelper's default. I understand, it can
break old packages with buggy makefiles, but it'd be nice to have --parallel
both in the examples and as compat 9 default.
These days you can't get 1-way regular machines without looking deep, and
even phones get more cores. Having package builds utilize only a small part
of the machine is pretty ridiculous.
Right now, two parties have to opt in: the maintainer (by enabling
--parallel or doing the equivalent for non-dh), and the user. That's two
too many. Are you really supposed to edit every package only to have it
finish build before the next day? Are you supposed to have to set env vars
on every machine you use for something that could be done out of the box?
If a performance-important package was 6-8 times slower than it could be,
there'd be quite an uproar. Yet something as vital as building most
packages suffers from this and yet no one seems to mind.
I'd be really nice to rethink parallelization. This was discussed in depth
on debial-devel a long time ago, my stab at implementing what seemed to be
the consensus then is:
http://lists.debian.org/debian-devel/2006/07/msg00260.html
(doesn't obey DEB_BUILD_OPTIONS=parallel=X as it predates that, also "120MB
RAM per job" could have been expressed without two divisions).
The only argument against mentioned was that buildds usually want single
threaded builds as they can build several packages concurrently without
thinking whether they support parallelism or not. This can be done by
setting the env var explicitely -- there are tens of thousands of users who
rebuild packages[1] but only a few tens buildds, and ran by competent admins
who can read upgrade notes.
The other problem are buggy makefiles. That's why the maintainer would need
to opt-in, but it should be tied with something everyone eventually does
(like upgrading to the next debhelper compat level[2]). If someone enables
make -j with undeclared dependencies, a test rebuild with a version of make
that reverses the order would reveal 99.9% of such bugs. The rest, real
races, can be expected to be damn rare:
true: tf.c
$(CC) tf.c -DRESULT=0
cp a.out true
false: tf.c
$(CC) tf.c -DRESULT=1
cp a.out false
In the short term, I propose two things:
1. let's change all dh examples/boilerplates to use --parallel
2. let's think about having --parallel the default in dh9
Sadly, this won't make parallel=`grep ^processor /proc/cpu|wc -l` the
default but would work without disrupting existing things.
[1]. A wild guesstimate.
[2]. Except Manoj, but as the make maintainer I guess he doesn't need our
help about how to use make -j :)
--
1KB // Yo momma uses IPv4!
Reply to: