Re: need help making shell script use two CPUs/cores

[vogelke+debian@pobox.com (Karl Vogel)]

>> On Sun, 09 Jan 2011 10:05:43 -0600, 
>> Stan Hoeppner <stan@hardwarefreak.com> said:

S> #! /bin/sh
S> for k in $(ls *.JPG); do convert $k -resize 1024 $k; done

   Someone was ragging on you to let the shell do the file expansion.  I
   like your way better because most scripting shells aren't smart enough
   to realize that when there aren't any .JPG files, I don't want the
   script to echo '*.JPG' as if that's actually useful.

S> I use the above script to batch re-size digital camera photos after I
S> dump them to my web server.  It takes a very long time with lots of new
S> photos as the server is fairly old, even though it is a 2-way SMP,
S> because the script only runs one convert process at a time serially,
S> only taking advantage of one CPU.

   First things first: are you absolutely certain that running two parallel
   jobs will exercise both CPUs?  I've seen SMP systems that don't exactly
   live up to truth-in-advertising.  If you stuff two "convert" jobs in the
   background and then run "top" (or the moral equivalent) do you SEE both
   CPUs being worked?

   Second: do you have "taskset" installed?  If the work isn't being
   divided up the way you like, you can bind a process to a desired core:
   http://planet.admon.org/how-to-bind-a-certain-process-to-specified-core/

   And last: if you're not using something like LVM, can you do anything to
   make sure you're not hitting the same disk?  If all your new photos are
   on the same drive, any CPU savings you get from parallel processing will
   probably be erased by disk contention.  Better yet, do you have enough
   memory to do the processing on a RAM-backed filesystem?

-- 
Karl Vogel                      I don't speak for the USAF or my company

If you're searching for the cause of a ghastly noise and find out that it's
not the cat, leave the area immediately.      --how to survive a horror movie