Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 03:17:27AM +0200, Henning Makholm wrote:
> > If package maintainer wants to build it faster on their own machine, I
> > would imagine that checking for an environment variable (DEB_MAKE_OPTS
> > or something, perhaps?) and using that would be the way to go. By
> > default, build with a single processor.
This would affect every single package, and you can't do that. While
a vast majority of C code will build correctly, not every package is
SMP-safe. [1]
> If I understand the problem correctly, it is not even necessary to
> modify debian/rules to get this behavior. If the interdependencies
> are properly declared,
>
> $ debian/rules -j42 binary
>
> should do the trick, as GNU make is smart enough to pass the option
> down to sub-makes that it starts.
Actually, this is a bad idea; debian/rules are specifically the kind
of makefiles that typically rely on the order in which dependencies
are built. This is a bug as it breaks make -k, but as -k hardly ever
makes sense with regard to debian/rules, this is nearly totally
untested.
On the other hand, making builds significantly faster is not
something that you can shake a stick at. Typically make -jX is faster
even on uniprocessor, and I don't need to tell you why it's much
faster on SMP.
Too bad, a C++ build where every file takes 1GB memory obviously
should not be parallelized. Also, no one but the maintainer knows
whether a package is SMP-clean or not. You cannot guess this in an
automated way.
Thus, my counter-proposal:
Let's allow maintainers to use make -jX according to their common
sense, requiring obeying an env variable to opt out.
Rationale:
Nearly every buildd and nearly every user building the packages on
his own will benefit from -j2 [2], even on non-SMP. Unless it's a
piece of heavily-templated code, any modern box will have enough
memory to handle it. The maintainer know whether the code is heavily
templated or not.
Giving the choice to someone who doesn't know the package (upstream
and/or the maintainer) will lead to _random_ FTBFSes.
Having DEB_MAKE_NON_PARALLEL [3] disable this behavior would make
m68k happy, while giving a reasonable default for everyone else.
And yeah, I'm guilty of using unconditional make -j4 in one of my
packages (not in the archive yet), too.
[1] A real-world example:
======================================== [ make_commands ]
#!/bin/sh
rm -f commands.h
rm -f load_commands.h
ARGS="_command(char *arg, struct session *ses)"
while read FUNC
do
if [ -z "$FUNC" ]
then
continue
fi
case $FUNC in
\;*) ;;
\#*) echo "$FUNC" >>commands.h;echo "$FUNC" >>load_commands.h
;;
\**) FUNC=`echo "$FUNC"|sed 's/^\*//'`
echo "extern struct session *$FUNC$ARGS;" >>commands.h
echo "add_command(c_commands, \"$FUNC\", (t_command)${FUNC}_command);" >>load_commands.h
;;
*) echo "extern void $FUNC$ARGS;" >>commands.h
echo "add_command(commands, \"$FUNC\", ${FUNC}_command);" >>load_commands.h
;;
esac
done
==========================================================
This script produces two files, commands.h and load_commands.h, both
of which should be regenerated every time the input data changes.
A naive Makefile will just have both of these depend on an
invocation of make_commands. This will work correctly and be
unnoticed for years until someone gives "-j" to make.
Would you bet that every single of your upstreams knows to avoid such
subtle errors?
[2] I benchmarked this with an old gcc several years ago on an
uniprocessor, and -j4 (!) gave the best results. Today, with gcc-4.2
-j2 won.
[3] This name sucks. Give a better one if you care :p
--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.
Reply to: