Re: canu

[Tony Travis <tony.travis@minke-informatics.co.uk>]

On 02/11/2023 10:54, George Marselis wrote:
>  >  I'm just undecided because slurm itself needs
> a bit more than just install the client package and you have immediate
> advantage without more administration.
>
> if I may: maybe gnu parallel would be a good suggestion, as well?

Hi, George.

GNU Parallel is a way of running 'embarrasingly' parallel jobs (i.e. 
running jobs that don't communicate with each other) concurrently. It's 
an improvement on "xargs" that also allows you to run jobs across a 
cluster of computers. This is useful when jobs consume only a small 
proportion of the available resources, but there are many jobs to run.

However, "canu" jobs typically require very large resources so "Slurm" 
is used to queue them up for execution. The problem I encountered was 
that the saved Slurm queue state was stale after an upgrade, resulting 
in a failure of "canu" to submit a job to the "Slurm" job-queue.

I didn't realise until later that "canu" will automatically use "Slurm" 
(or other resource managers like GE) if it detects that it is installed 
and, in principle, available.

I agree that GNU Parallel is good, but I wouldn't use it for "canu".

Andreas is right that "Slurm" requires non-trivial configuration before 
it can be used and, for that reason, I don't recommend adding it as a 
"suggests" to the Debian-Med "canu" package.

Bye,

  Tony.

--
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK)
tel. +44(0)19755 63548                    http://minke-informatics.co.uk
mob. +44(0)7985 078324        mailto:tony.travis@minke-informatics.co.uk