[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: canu



On 02/11/2023 10:54, George Marselis wrote:
 >  I'm just undecided because slurm itself needs
a bit more than just install the client package and you have immediate
advantage without more administration.

if I may: maybe gnu parallel would be a good suggestion, as well?

Hi, George.

GNU Parallel is a way of running 'embarrasingly' parallel jobs (i.e. running jobs that don't communicate with each other) concurrently. It's an improvement on "xargs" that also allows you to run jobs across a cluster of computers. This is useful when jobs consume only a small proportion of the available resources, but there are many jobs to run.

However, "canu" jobs typically require very large resources so "Slurm" is used to queue them up for execution. The problem I encountered was that the saved Slurm queue state was stale after an upgrade, resulting in a failure of "canu" to submit a job to the "Slurm" job-queue.

I didn't realise until later that "canu" will automatically use "Slurm" (or other resource managers like GE) if it detects that it is installed and, in principle, available.

I agree that GNU Parallel is good, but I wouldn't use it for "canu".

Andreas is right that "Slurm" requires non-trivial configuration before it can be used and, for that reason, I don't recommend adding it as a "suggests" to the Debian-Med "canu" package.

Bye,

  Tony.

--
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK)
tel. +44(0)19755 63548                    http://minke-informatics.co.uk
mob. +44(0)7985 078324        mailto:tony.travis@minke-informatics.co.uk


Reply to: