[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: make -j in Debian packages

On Fri, Jul 07, 2006 at 02:46:15PM +0200, Goswin von Brederlow wrote:
> The point of the helper is to remove the decision from the package
> alone to a central place that is easily configurable for a wide range
> of cases.

Ok, here goes my stab at the helper:

  $(MAKE) -j`debian/concurrency-helper`

  (note: you'll need to either chmod +x or insert "perl " after the
  first backtick)

Maintainer's manipulation:
  on command line

User's manipulation:
  ENV var

joeyh's manipulation:
  upgrade clause

I see that the rules I used are probably way too cowardly memory-wise
but way too fork-happy where the number of CPUs is concerned.  But oh
well, it's not me who is supposed to set the defaults anyway :p

This time an example isn't really needed, but just in case I've used
kbtin as the test package.  mentors.debian.net seems to be down, so:

deb-src http://angband.pl/debian unstable main
dget http://angband.pl/debian/kbtin/kbtin_1.0.7-1.dsc
(in an uploadeable state, in case you'll want to test it against real
buildds :p)

1KB		// Microsoft corollary to Hanlon's razor:
		//	Never attribute to stupidity what can be
		//	adequately explained by malice.
#!/usr/bin/perl -w
use strict;
use integer;

#Upgrade clause:
my $dh='/usr/bin/dh_concurrency';
exec {$dh} $dh,@ARGV if -x $dh;

=head1 NAME

concurrency-helper - guess a reasonable concurrency level for make


$(MAKE) -j`B<concurrency-helper> [S<I<options>>]`


concurrency-helper is a program that will guess whether running more parallel
I<make> jobs would speed up the build, and write a reasonable argument for I<-j>
on the standard output.

If the environment variable B<CONCURRENCY_LEVEL> is set, it will override any

=head1 OPTIONS

This is the interface for the package maintainer.

=over 4

=item B<--ram-estimate> I<X> [I<Y>]

   Give some indication of ram usage, in megabytes.  If the host has too
   little ram the concurency will be tuned down to prevent swapping.  A
   broad indication +- a factor of 2 is probably sufficient.  The [Y] would
   be to indicate ram usage for 32bit and 64bit archs seperately.  Given the
   pointer size ram usage can vary between them a lot.

=item B<--more-concurrent>

   Indicate that there are lots of small files that greatly benefit from
   interleaving I/O with cpu time.  Try to use more concurency than cpus.

=item B<--max-concurrency> I<X>

   Limit the concurrency to no more than X.  Shouldn't be ever needed, you're
   either SMP-safe or not.


=head1 CAVEATS

This helper can't detect multiple builds running on the same host.  In
that case, just set B<CONCURRENCY_LEVEL> appropiately; to 1 if you want
to disable any parallelism altogether.

=head1 SEE ALSO


=head1 AUTHOR

Adam Borowski <kilobyte@angband.pl>


#Leaving no output would lead to bare -j, that is, -j INFINITY.
sub die_1($)
    print "1\n" unless -t STDOUT;
    die "concurrenct-helper: $_[0]";

if (defined $ENV{'CONCURRENCY_LEVEL'})
    $ENV{'CONCURRENCY_LEVEL'}=~/^(\d+)$/ && $1>0
        or die_1 "E: CONCURRENCY_LEVEL malformed.\n";
    print "$1\n";

my ($mem,$cpus); #machine's specs
my ($ram32,$ram64,$limit,$extra); #arguments
my $ram;

open FREE, "free|" or die_1 "W: can't run 'free'.\n";
    $mem=$1/1024 if /^Mem:\s+(\d+)/;
close FREE;
$mem or die_1 "W: can't get the amount of memory.\n";

if (open CPUS, "</proc/cpuinfo")
    $cpus=scalar grep(/^processor\s/,<CPUS>);
    close CPUS;
$cpus=1 unless $cpus;

    if (/^--max-concur(|r)ency$/)
            or die_1 "E: --max-concurrency requires an argument.\n";
        /^(\d+)$/ && $1>0
            or die_1 "E: --max-concurrency expects a positive integer.\n";
    elsif (/^--ram-estimate$/)
            or die_1 "E: --ram-estimate expects an argument or two.\n";
            or die_1 "E: --ram-estimate: ram32 malformed.\n";
        #The second argument is optional.
        if (@ARGV && $ARGV[0]=~/^(\d+)$/)
    elsif (/^--more-concur(|r)en(cy|t)$/)
        die_1 "E: invalid argument \"$_\"\n";

$ram=(length(pack('l!',0)) <= 4)? $ram32:$ram64;

####### Here the fun starts.
# Edit this part to fine-tune the logic.

#Default to 24MB per job.
$ram=24 unless $ram;

#Be a coward, use no more than 1/5 of the memory.

#Don't have too many jobs waiting for CPU.
my $cap=$cpus+1;
$cap*=2 if $extra;
$_=$cap if $_>$cap;

#Obey --max-concurrency.
$_=$limit if (defined $limit && $_>$limit);

#But use at least _one_ job.  We want to get the build done, don't we?
$_=1 if $_<1;

print "$_\n";

Reply to: