Re: make -j in Debian packages
On Fri, Jul 07, 2006 at 02:46:15PM +0200, Goswin von Brederlow wrote:
> The point of the helper is to remove the decision from the package
> alone to a central place that is easily configurable for a wide range
> of cases.
Ok, here goes my stab at the helper:
(attached)
Usage:
$(MAKE) -j`debian/concurrency-helper`
(note: you'll need to either chmod +x or insert "perl " after the
first backtick)
Maintainer's manipulation:
on command line
User's manipulation:
ENV var
joeyh's manipulation:
upgrade clause
I see that the rules I used are probably way too cowardly memory-wise
but way too fork-happy where the number of CPUs is concerned. But oh
well, it's not me who is supposed to set the defaults anyway :p
This time an example isn't really needed, but just in case I've used
kbtin as the test package. mentors.debian.net seems to be down, so:
deb-src http://angband.pl/debian unstable main
dget http://angband.pl/debian/kbtin/kbtin_1.0.7-1.dsc
(in an uploadeable state, in case you'll want to test it against real
buildds :p)
Cheers,
--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.
#!/usr/bin/perl -w
use strict;
use integer;
#Upgrade clause:
my $dh='/usr/bin/dh_concurrency';
exec {$dh} $dh,@ARGV if -x $dh;
=head1 NAME
concurrency-helper - guess a reasonable concurrency level for make
=head1 SYNOPSIS
$(MAKE) -j`B<concurrency-helper> [S<I<options>>]`
=head1 DESCRIPTION
concurrency-helper is a program that will guess whether running more parallel
I<make> jobs would speed up the build, and write a reasonable argument for I<-j>
on the standard output.
If the environment variable B<CONCURRENCY_LEVEL> is set, it will override any
guesses.
=head1 OPTIONS
This is the interface for the package maintainer.
=over 4
=item B<--ram-estimate> I<X> [I<Y>]
Give some indication of ram usage, in megabytes. If the host has too
little ram the concurency will be tuned down to prevent swapping. A
broad indication +- a factor of 2 is probably sufficient. The [Y] would
be to indicate ram usage for 32bit and 64bit archs seperately. Given the
pointer size ram usage can vary between them a lot.
=item B<--more-concurrent>
Indicate that there are lots of small files that greatly benefit from
interleaving I/O with cpu time. Try to use more concurency than cpus.
=item B<--max-concurrency> I<X>
Limit the concurrency to no more than X. Shouldn't be ever needed, you're
either SMP-safe or not.
=back
=head1 CAVEATS
This helper can't detect multiple builds running on the same host. In
that case, just set B<CONCURRENCY_LEVEL> appropiately; to 1 if you want
to disable any parallelism altogether.
=head1 SEE ALSO
L<make(1)>
=head1 AUTHOR
Adam Borowski <kilobyte@angband.pl>
=cut
#Leaving no output would lead to bare -j, that is, -j INFINITY.
sub die_1($)
{
print "1\n" unless -t STDOUT;
die "concurrenct-helper: $_[0]";
}
if (defined $ENV{'CONCURRENCY_LEVEL'})
{
$ENV{'CONCURRENCY_LEVEL'}=~/^(\d+)$/ && $1>0
or die_1 "E: CONCURRENCY_LEVEL malformed.\n";
print "$1\n";
exit;
}
my ($mem,$cpus); #machine's specs
my ($ram32,$ram64,$limit,$extra); #arguments
my $ram;
open FREE, "free|" or die_1 "W: can't run 'free'.\n";
while(<FREE>)
{
$mem=$1/1024 if /^Mem:\s+(\d+)/;
}
close FREE;
$mem or die_1 "W: can't get the amount of memory.\n";
if (open CPUS, "</proc/cpuinfo")
{
$cpus=scalar grep(/^processor\s/,<CPUS>);
close CPUS;
}
$cpus=1 unless $cpus;
while(@ARGV)
{
$_=shift;
if (/^--max-concur(|r)ency$/)
{
$_=shift
or die_1 "E: --max-concurrency requires an argument.\n";
/^(\d+)$/ && $1>0
or die_1 "E: --max-concurrency expects a positive integer.\n";
$limit=$1;
}
elsif (/^--ram-estimate$/)
{
$_=shift
or die_1 "E: --ram-estimate expects an argument or two.\n";
/^(\d+)$/
or die_1 "E: --ram-estimate: ram32 malformed.\n";
$ram32=$1;
#The second argument is optional.
if (@ARGV && $ARGV[0]=~/^(\d+)$/)
{
$ram64=$1;
shift;
}
else
{
$ram64=$ram32;
}
}
elsif (/^--more-concur(|r)en(cy|t)$/)
{
$extra=1;
}
else
{
die_1 "E: invalid argument \"$_\"\n";
}
}
$ram=(length(pack('l!',0)) <= 4)? $ram32:$ram64;
####### Here the fun starts.
# Edit this part to fine-tune the logic.
#Default to 24MB per job.
$ram=24 unless $ram;
#Be a coward, use no more than 1/5 of the memory.
$_=$mem/5/$ram;
#Don't have too many jobs waiting for CPU.
my $cap=$cpus+1;
$cap*=2 if $extra;
$_=$cap if $_>$cap;
#Obey --max-concurrency.
$_=$limit if (defined $limit && $_>$limit);
#But use at least _one_ job. We want to get the build done, don't we?
$_=1 if $_<1;
print "$_\n";
Reply to: