Re: make -j in Debian packages
Adam Borowski <kilobyte@angband.pl> writes:
> On Wed, Jun 28, 2006 at 03:17:27AM +0200, Henning Makholm wrote:
>> > If package maintainer wants to build it faster on their own machine, I
>> > would imagine that checking for an environment variable (DEB_MAKE_OPTS
>> > or something, perhaps?) and using that would be the way to go. By
>> > default, build with a single processor.
>
> This would affect every single package, and you can't do that. While
> a vast majority of C code will build correctly, not every package is
> SMP-safe. [1]
>
>> If I understand the problem correctly, it is not even necessary to
>> modify debian/rules to get this behavior. If the interdependencies
>> are properly declared,
>>
>> $ debian/rules -j42 binary
>>
>> should do the trick, as GNU make is smart enough to pass the option
>> down to sub-makes that it starts.
>
> Actually, this is a bad idea; debian/rules are specifically the kind
> of makefiles that typically rely on the order in which dependencies
> are built. This is a bug as it breaks make -k, but as -k hardly ever
> makes sense with regard to debian/rules, this is nearly totally
> untested.
What relies on the order and why should it?
Every target in debian/rules should have the correct dependencies
listed and force make to build targets in the order required. That
means binary* depends on install*, install on build, build on
configure or whatever combination of targets are employed.
I think it is sane to demand that all debian/rules files are
concurency save and it is trivial to do so.
The same can't be said for upstream makefiles though. Many sources
don't build with -j option. I'm not sure if debian/rules should
somehow enforce -j1 in those cases or if only packages that benefit
from -jX should add support for some DEB_BUILD_OPTION or
CONCURENCY_LEVEL env var. The later would always be the save
option. The former would have lots of hidden bugs that prop up
suddenly on rebuilds or security fixes.
Maybe someone should do a complete archive rebuild with -j1 and -j4 on
a smp system and compare the amount of failures to get an overview how
bad it would be.
> On the other hand, making builds significantly faster is not
> something that you can shake a stick at. Typically make -jX is faster
> even on uniprocessor, and I don't need to tell you why it's much
> faster on SMP.
> Too bad, a C++ build where every file takes 1GB memory obviously
> should not be parallelized. Also, no one but the maintainer knows
> whether a package is SMP-clean or not. You cannot guess this in an
> automated way.
That would point to using an env varibale
> Thus, my counter-proposal:
> Let's allow maintainers to use make -jX according to their common
> sense, requiring obeying an env variable to opt out.
and let the maintainer pass that env variable down a -jX.
How about this option:
We write a tool "concurency-helper" that gets a set of requirements of
the source and outputs a suitable concurency level for current build
host. Requirements could include:
--package pkg
Let the tool know what we build. The tool can have overrides in
place for packages in case special rules apply.
--max-concurency X || --non-concurent
Limit the concurency, possibly to 1, for sources that have problems
with it. Although such sources probably just shouldn't support this.
--ram-estimate X [Y]
Give some indication of ram usage. If the host has too little ram
the concurency will be tuned down to prevent swapping. A broad
indication +- a factor of 2 is probably sufficient. The [Y] would
be to indicate ram usage for 32bit and 64bit archs seperately.
Given the pointer size ram usage can vary between them a lot.
--more-concurent
Indicate that there are lots of small files that greatly benefit
from interleaving I/O with cpu time. Try to use more concurency
than cpus.
The tool would look at those indicators and the hosts resources in
both ram and cpus and figure out a suitable concurency level for the
package from there.
What do you think?
> Rationale:
> Nearly every buildd and nearly every user building the packages on
> his own will benefit from -j2 [2], even on non-SMP. Unless it's a
> piece of heavily-templated code, any modern box will have enough
> memory to handle it. The maintainer know whether the code is heavily
> templated or not.
Mips, mipsel, arm and m68k won't benefit. The ram requirement just
leads to poor cache performance or even excessive swapping in
general. Sources and gcc are growing and growing and the ram of the
buildds stays the same.
On the other hand any modern system will build the source fast and the
buildd will be idle most of the time so -j2 or not hardly matters.
Wouldn't that indicate a preference to -j1?
MfG
Goswin
Reply to: