[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: need help making shell script use two CPUs/cores



Stan Hoeppner wrote:
> Camaleón put forth:
> > real	1m44.038s
> > user	2m5.420s
> > sys	1m17.561s
> > 
> > It uses 2 "convert" proccesses so the files are being run on pairs.
> > 
> > And you can even get the job done faster if using -P8:
> > 
> > real	1m25.255s
> > user	2m1.792s
> > sys	0m43.563s
> 
> That's an unexpected result.  I would think running #cores*2^x with an
> increasing x value would start yielding lower total run times within a few
> multiples of #cores.

If you have enough memory (which is critical) then increasing the
number of processes above the number of compute units *a little bit*
is okay and increases overall throughput.

You are processing image data.  That is a large amount of disk data
and won't ever be completely cached.  At some point the process will
block on I/O waiting for the disk.  Perhaps not often but enough.  At
that moment the cpu will be idle until the disk block becomes
available.  When you are runing four processes on your two cpu machine
that means there will always be another process in the run queue ready
to go while waiting for the disk.  That allows processing to continue
when otherwise it would be waiting for the disk.  I believe what you
are seeing above is the result of being able to compute during that
small block on I/O wait for the disk interval.

On the negative side having more processes in the run queue does
consume a little more overhead for process scheduling.  And launching
a lot of processes consumes resources.  So it definitely doesn't make
sense to launch one process per image.  But being above the number of
cpus does help a small amount.

Another negative is that other tasks then suffer.  With excess compute
capacity you always have some cpu time for the desktop side of life.
Moving windows, rendering web pages, other user tasks, delivering
email.  Sometimes squeezing that last percentage point out of
something can really kill your interactive experience and end up
frustrating you more.  So as a hint I wouldn't push too hard on it.

> > No need to have a quad core with HT. Nice :-)

My benchmarks show that hyperthreading (fake cpus) actually slow down
single thread processes such as image conversions.  HT seems like a
marketing breakthrough to me.  Although having the effective extra
registers available may benefit a highly threaded application.  I just
don't have any performance critical highly threaded applications.  I
am sure they exist somewhere along with unicorns and other good
sources of sparkles.

Bob

Attachment: signature.asc
Description: Digital signature


Reply to: