Re: distributed batch processing
Just a few things to clarify:
With SGE 5.3, the FIFO is the default scheduling
policy. However, you can configure the scheduler to
run in SGE EE (Enterprise Edition) mode, which has the
4 additional scheduling policies (deadline, share
tree, override, and functional). With SGE 6.0, you get
resource reservation and urgency. There are also
"projects" and "departments" that you can setup to
group the jobs.
All of those are opensource (even the EE mode) and can
be downloaded from the SGE homepage:
http://gridengine.sunsource.net
IMO, Maui is better for backfilling parallel jobs, but
SGE is better for the "compute farm" type of
workloads.
Also, the "array job" functionality parameterizes the
job group. To run 200 tasks, all you need to do is a
qsub:
% qsub -t 1-200 <job script>
And inside the jobscript:
foo -i file$SGE_TASK_ID.data -p 0.1
And you can monitor all the subtasks as one job.
Another advantage is that SGE offers failover
functionality, which allows you to have one or more
standby masters.
-Ron
Discover Yahoo!
Have fun online with music videos, cool games, IM and more. Check it out!
http://discover.yahoo.com/online.html
Reply to: