Re: need help making shell script use two CPUs/cores

[Bob Proulx <bob@proulx.com>]

Stan Hoeppner wrote:
> I'm not very skilled at writing shell scripts.
> 
> #! /bin/sh
> for k in $(ls *.JPG); do convert $k -resize 1024 $k; done

First off don't use ls to list files matching a pattern.  Instead let
the shell match the pattern.

  #! /bin/sh
  for k in *.JPG; do convert $k -resize 1024 $k; done

I never like to resize in place.  Because then if I mess things up I
can lose resolution.  So I recommend doing it to a named resolution
file.  Do anything you like but this would be the way I would go.

  for k in *.JPG; do
    convert $k -resize 1024 $(basename $k .JPG).1024.jpg
  done

And not wanting to do the same work again and again:

  for k in *.JPG; do
    base=$(basename $k .JPG)
    test -f $base.1024.jpg && continue  # skip if already done
    convert $k -resize 1024 $base.1024.jpg
  done

> I use the above script to batch re-size digital camera photos after I
> dump them to my web server.  It takes a very long time with lots of new
> photos as the server is fairly old, even though it is a 2-way SMP,
> because the script only runs one convert process at a time serially,
> only taking advantage of one CPU.  The convert program is part of the
> imagemagick toolkit.
> 
> How can I best modify this script so that it splits the overall job in
> half, running two simultaneous convert processes, one on each CPU?
> Having such a script should cut the total run time in half, or nearly
> so, which would really be great.

GNU xargs has an extension to run jobs in parallel.  This is already
installed on your system.  (But won't work on other Unix systems.)

  for k in *.JPG; do echo $k; done | xargs -I{} -P4 echo convert {} -resize 1024 {}

Verify that does what you want and then remove the echo.

unfortunately that simple approach is harder to do with my renaming
scheme.  So I would probably write a helper script that did the
options to convert and renamed the file and so forth.

  for k in *.JPG; do
    base=$(basename $k .JPG)
    test -f $base.1024.jpg && continue  # skip if already done
    echo $k;
  done | xargs -L1 -P4 echo my-convert-helper

And my-convert-helper could take the argument and apply the options in
the order needed and so forth.

Adjust 4 in the above to be the number of jobs you want to run on your
multicore system.  Note that in Sid in the latest coreutils there is a
new command 'nproc' to print out the number of cores.  Or you could
get it from grep.

  grep -c ^processor /proc/cpuinfo

All of the above is off the top of my head and needs to be tested.
YMMV.  But hopefully it will give you some ideas.

HTH,
Bob

Attachment: signature.asc
Description: Digital signature