Bug#793404: massive waste of CPU time in debian/rules by inline commands
On Thu, 23 Jul 2015 19:43:46 +0200 Eduard Bloch <edi@gmx.de> wrote:
> please tell me that I am wrong or that I start fighting the windmills if
> that's the case, but I have a general impression that something smells
> in lots of debian/rules files nowadays and we need a concept to improve
> that.
>
> The problem: I see lots of $(shell ...) stuff. In boost, there are about
> 12 such calls. And they run dpkg-architecture or dpkg-parsechangelogs or
> similar commands. When this was done a just couple of times (i.e. before
> dh(7)), that's acceptable. But now, it looks like debian/rules is called
> many, many times through dh. Making many, many calls of that inline
> commands. Wasting many, many CPU cycles. All that just to retrieve the
> same information all over again.
>
> In the emulated m68k environment, it spends about half an hour (guessed,
> not measured) before starting the actual build, doing things like:
>
> | \_ /usr/bin/perl -w /usr/bin/dh build --with python2 --with python3
> | \_ /usr/bin/make -f debian/rules override_dh_auto_configure
> | \_ /bin/sh -c dpkg-parsechangelog | grep Version | cut -d' ' -f2
> | \_ /usr/bin/perl /usr/bin/dpkg-parsechangelog
> | | \_ /usr/bin/perl /usr/lib/dpkg/parsechangelog/debian -ldebian/changelog --file debia
I think you're right. $(shell ...) isn't a good idea in make, as it
always runs regardless of the target. Worse yet, those shell commands
are not a trivial amount of processing. Ideally, this information
should be obtained *once* at the start of the build, cached in a file
that depends on the necessary bits (e.g. parse the version from the
changelog into a file that depends on debian/changelog), and then used
from there. Whether that's done via make, or by having commands like
dpkg-parsechangelog cache the relevant bits of thir output and check
mtime on the files they parse.
We should try to reduce the number of such things that are actually
needed in debian/rules; if they're only needed in a particular target,
or in some specific command, let's put them in that target or command.
Also, things like "grep Version | cut ..." should really be replaced
with things like "dpkg-parsechangelog -SVersion". It's unfortunate that
that make can't run a command without using the shell.
For that matter, all the override_* targets should ideally be handled in
some way that doesn't involve a full invocation of make, if the target
doesn't actually exist. That's harder, but it should be possible to run
make once, parse the list of targets, and use those. If we can reduce
the number of invocations of make, that would help as well.
- Josh Triplett
Reply to: