[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: need help making shell script use two CPUs/cores



Karl Vogel wrote:
> >> Stan Hoeppner said:
> S> for k in $(ls *.JPG); do convert $k -resize 1024 $k; done
> 
>    Someone was ragging on you to let the shell do the file expansion.  I
>    like your way better because most scripting shells aren't smart enough
>    to realize that when there aren't any .JPG files, I don't want the
>    script to echo '*.JPG' as if that's actually useful.

:-) I thought about saying something about .JPG instead of .jpg.  Unix
is all about lower case after all.  But I restrained myself.  :-) :-)

  $ for i in *.doesnotexist; do echo $i; done
  *.doesnotexist

As to your comment about shells passing the glob off when it doesn't
match, that is a good comment.  That wasn't really too much in my
style of coding and so had missed it.  Thanks for keeping me honest!
Just to push on it a little bit more I could do this:

  set -- *.doesnotexist
  for i in "$@"; do
    echo $i
  done

That avoids the problem and still avoids spawning another process.
And depending upon what I was doing I might do something completely
different.

>    First things first: are you absolutely certain that running two parallel
>    jobs will exercise both CPUs?  I've seen SMP systems that don't exactly
>    live up to truth-in-advertising.  If you stuff two "convert" jobs in the
>    background and then run "top" (or the moral equivalent) do you SEE both
>    CPUs being worked?

When was that in terms of kernel versions?  Was Intel hyperthreading
also involved?  Because your description matches very closely running
a two core system with Intel hyperthreading on an older Linux kernel.

Here is the problem I know about.  On a dual cpu system with
hyperthreading the Linux kernel saw four cores and numbered them 0, 1,
2, 3.  But of course zero and one were on one core and two and three
were on the other core.  The first process would run on, say, zero.
The second process would run on the next core, say, one.  To Linux of
that day it thought it had allocated those processes onto different
cpus.  But of course both were running on the same cpu, each getting
half of it and taking twice as long to run, and the other cpu was
idle.  A big problem.

I always disabled Intel hyperthreading to avoid that problem.  It was
more trouble than it was worth.  Also my benchmarks showed that HT
would slightly slow down single threaded simulation processes (mostly
Spice and other simulations) that we were running.

But as far as I know this has now been addressed and the Linux kernel
now knows about hyperthreaded cpus.  It seems that with recent kernels
that the cpu allocation works okay even in the presence of fake cpus
through hyperthreading.  So I think the problem you describe is now
behind us.

Bob

Attachment: signature.asc
Description: Digital signature


Reply to: