Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 03:42:07PM +0200, Wouter Verhelst wrote:
> SMP buildd systems currently run multiple instances of buildd at
> the same time, rather than expecting a package to specify make -j
> itself. Having three packages that specify 'make -j 4' on a
> multiprocessor buildd host that _already_ runs six builds in
> parallel is going to be devastating.
I admit, this is news to me.
> While parallel builds can indeed help speed up things even on
> uniprocessor systems, they do this at a price; they require a _lot_
> more RAM than non-parallel builds.
Modular C code can require an order of magnitude less memory than
templated C++. What I propose is parallelizing only those packages
which are not heavy on RAM, on the maintainer's discretion.
If all packages had a similar memory profile, this would be an
optimization as it would equalize buildd loads somewhat.
> A uniprocessor parallel build will go faster for as long as your
> buffercaches can keep make, the compiler, and the most often used
> header files in memory; as soon as those need to be swapped out,
> your performance will go down dramatically. Since not all machines
> have the same amount of RAM, it is impossible to say in a generic
> way when this point is going to be.
Right, that is why the debian/rules snippet I suggested checks just
the amount of RAM without even looking at how many CPUs are present.
Indeed, it would fail to recognize several instances of buildd
running at once, but this is such a rare case that it can be handled
with an env variable we can agree upon.
> In any case, I have nothing against doing parallel builds by
> default (at least, not as long as it can be properly disabled and
> regulated),
Cool! This is exactly what this thread is about.
What do you think about going with Don Armstrong's suggestion
($CONCURRENCY_LEVEL), while handling the default (no env var) my
way (decent memory => parallel, little memory => -j 1) instead of
Ingo's (-j 1 unless explicitely set)?
I think that the sample script I uploaded in kbtin would do the job;
if it is not good enough, it can be improved.
> but do not claim that it's a good thing "also for the buildd's",
> because you are sadly mistaken.
Still, my way is already an improvement over gem: gem used
unconditional -j4 and it happens to be a C++ package.
(Well, well. Because this happened just days after I uploaded a
version with unconditional -j4 to mentors, it's likely it's me who's
the real culprit. Gem's maintainer could have looked at my package
and copied -j4, the choice of 4 instead of 2 kind of suggests that.)
Another idea is to use -l; it trims concurrency down to 1 whenever system
load exceeds a given value.
Anyway, it's possible to come to an agreement that will make everyone
happy, and I believe we're close.
Meep?
--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.
Reply to: