Re: make -j in Debian packages
Adam Borowski <kilobyte@angband.pl> writes:
> On Sun, Jul 02, 2006 at 11:57:50AM +0200, Wouter Verhelst wrote:
>> Additionally, it puzzles me how you think a maintainer will be able to
>> accurately predict how much RAM a certain build is going to use. There
>> are so many variables, that I think anything but 'this is the fastest
>> way to build it on my machine' is going to be unfeasible.
Ram usage between architectures should not vary more than a factor of
4. Much less in normal cases.
> Let's say:
> program X consist of a number of C files; it seems like compiling
> every file takes around 24MB, with the final link taking much more[1].
> I guess this can be called typical, in C you need to store just the
> current file and the headers you use.
>
> Now, let's assume we use not the simple snippet I wrote [2] but a
> "concurrency-helper" with the interface Goswin described, with some
> unknown logic inside.
>
> The maintainer thus declares that package X takes 24MB, and says it's
> good to use heavy concurrency:
> concurrency-helper --ram-estimate 24 --more-concurrent
>
> The machine is a mid-range user box, with 512MB ram.
>
> Thus, if the helper decides to go with -j4, the safety margin is _5_
> times. I guess you can trust people to be at least within _that_
> error range. And even if they fail, you can always force the build
> to use -j1.
The amount of paging can also be gathered during a build and included
in the buildd log. Too bad avgtext / avgdata / maxresident accounting
is broken and always gives 0 (e.g. run /usr/bin/time ls).
> If, let's say, the machine is a high-end one with 2GB ram, it runs 4
> buildds at once and the admin didn't specify his preferences, using
> -j4 won't be any worse than on the user box mentioned above.
And the concurrency-helper could easily know about 4 buildds being run
in parallel and assume only 512MB ram and 1/4 the number of cpus being
available per buildd for its calculations.
The point of the helper is to remove the decision from the package
alone to a central place that is easily configurable for a wide range
of cases.
> [1]. I was once forced to do a kernel compile on a critically memory
> starved box. Going from .c to .o went quite smoothly, but the final
> link was an unholy swappeathon that took hours.
>
> [2]. My idea was to simply go with -j1 if the machine has less than X
> memory, or with a given constant otherwise.
>
> Cheers,
MfG
Goswin
Reply to: