Re: Makefile help

[Greg Wooledge <greg@wooledge.org>]

On Fri, Jul 23, 2021 at 04:10:10PM +0200, Grzesiek wrote:
> I have two directories: source_dir and dest_dir. In the directory source_dir
> I have number of sub-directories and files. My goal is:
> 
> 1.Recreate the sub-directory structure of the source_dir inside of the
> dest_dir
> 
> 2. Each file found in source_dir should be converted to a new format using
> some command CMD in the following way:
> CMD source_dir/<path to file>/<file name>.src dest_dir/<path to file>/<file
> name>.dst

> Is it possible to do it using make

Not really.  It's a job for a shell script or some other program written
in a scripting language.


On Fri, Jul 23, 2021 at 04:29:24PM +0200, Grzesiek wrote:
> The number of files is rather large so make -j8 could save lot of time.

Ah, the good old X-Y strikes again.  OK, at least now we know the real
goal -- you want to parallelize the conversions.

The obvious way to do that is with GNU xargs -0 -P.  So, we're going to
generate a NUL-delimited stream of filenames to be processed, and feed
that to xargs -0 -P 8 -n 1 something.

Now that we know the middle piece, we only have to write the beginning
piece (the thing that generates the stream of NUL-delimited filenames),
and the ending piece (the "something" that xargs runs to perform each
conversion).

The beginning is simple: we'll be using find -print0 to generate the stream
of filenames.  find source_dir -type f -print0
You can add additional restrictions to this if needed (e.g. -iname '*.wav').

The ending is basically what you specified above, except that we need to
wrap it in a miniature shell script to generate the destination pathname
from the source pathname.  We also need to create the directory part of
the destination pathname, because presumably CMD doesn't do that for us.

So,

#!/bin/sh

find source_dir -type f -print0 |
xargs -0 -P 8 -n 1 sh -c '
    dest=dest_dir/${1#*/}
    dest=${dest%.src}.dst
    dir=${dest%/*}
    mkdir -p "$dir"
    CMD "$1" "$dest"
' x


And, testing it:

unicorn:~$ printf %s\\0 {1..12} | xargs -0 -P 8 -n 1 sh -c 'echo "$$ processing $1 at $(date +%H:%M:%S)"; sleep 3' x
56062 processing 1 at 10:56:52
56063 processing 2 at 10:56:52
56065 processing 3 at 10:56:52
56067 processing 4 at 10:56:52
56074 processing 6 at 10:56:52
56072 processing 5 at 10:56:52
56076 processing 7 at 10:56:52
56079 processing 8 at 10:56:52
56086 processing 9 at 10:56:55
56089 processing 11 at 10:56:55
56091 processing 12 at 10:56:55
56087 processing 10 at 10:56:55

The first 8 ran simultaneously, then after a 3 second delay, the last 4
ran simultaneously.  Looks good to me.  Obviously I didn't test the string
manipulations that generate the destination filename and subdirectory,
but I'm sure you can handle it from here.