Re: need help making shell script use two CPUs/cores
On Tue, 11 Jan 2011 15:58:45 -0600, Stan Hoeppner wrote:
> Camaleón put forth on 1/11/2011 9:38 AM:
>
>> I supposed you wouldn't care much in getting a script to run faster
>> with all the available core "occupied" if you had a modern (<4 years)
>> cpu and plenty of speedy ram because the routine you wanted to run it
>> should not take many time... unless you were going to process
>> "thousand" of images :-)
>
> That's a bit ironic. You're suggesting the solution is to upgrade to a
> new system with a faster processor and memory.
Why did you get that impression? No, I said I thought you were running a
resource-scarce machine so in order to simulate your environment I made
the tests under my VM... nothing more.
> However, all the newer processors have 2, 4, 6, 8, or 12 cores. So
> upgrading simply for single process throughput would waste all the
> other cores, which was the exact situation I found myself in.
But of course! I would not even think in upgrade the whole computer just
to get one concrete task done a few more seconds faster.
> The ironic part is that parallelizing the script to maximize performance
> on my system will also do the same for the newer chips, but to an even
> greater degree on those with 4, 6, 8, or 12 cores. Due to the fact that
> convert doesn't eat 100% of a core's time during its run, and the idle
> time in between one process finishing and xargs starting another, one
> could probably run 16-18 parallel convert processes on a 12 core Magny
> Cours with this script before run times stop decreasing.
I think the script should also work very well with single-core cpus.
> The script works. It cut my run time by over 50%. I'm happy. As I
> said, this system's processing power is complete overkill 99% of the
> time. It works beautifully with pretty much everything I've thrown at
> it, for 8 years now. If I _really_ wanted to maximize the speed of this
> photo resizing task I'd install Win32 ImageMagick on my 2GHz Athlon XP
> workstation with dual channel memory nForce2 mobo, convert them on the
> workstation, and copy them to the server.
>
> However, absolute maximum performance of this task was not, and is not
> my goal.
> My goal was to make use of the second CPU, which was sitting idle in
> the
> server, to speed up the task completion. That goal was accomplished. :)
Yeah, and tests are there to demonstrate the gain.
>>>> Running more processes than real cores seems fine, did you try it?
>>>
>>> Define "fine".
>>
>> Fine = system not hogging all resources.
>
> I had run 4 (2 core machine) and run time was a few seconds faster than
> 2 processes, 3 seconds IIRC. Running 8 processes pushed the system into
> swap and run time increased dramatically. Given that 4 processes only
> have a few seconds faster than two, yet consumed twice as much memory,
> the best overall number of processes to run on this system is two.
Maybe the "best number of processes" is system-dependant (old processors
could work better with a conservative value but newer ones can get some
extra seconds with a higher one and without experiencing any significant
penalty).
Greetings,
--
Camaleón
Reply to: