Re: Any input for some talk about usage of Debian in HPC
On 19/05/2024 12:12, Andreas Tille wrote:
Hi,
I have an invitation to have some talk with the title
Debian GNU/Linux for Scientific Research
Abstract:
Over the past decade, Enterprise Linux has dominated large-scale
research computing infrastructure. However, recent developments have
sparked increased interest in community-led alternatives. Debian
GNU/Linux, a long-standing choice among researchers for supporting
scientific work, is experiencing a renewed interest for High-Throughput
Computing (HTC) and High-Performance Computing (HPC) applications. This
presentation will provide an overview of how Debian is being utilized to
support scientific research and will include a case study showcasing the
migration of HTC operations from Enterprise Linux 7 (EL7) to Debian.
While I could talk about Debian Science and Debian Med in general it
would be cool to reference to some real life examples where Debian is
used in Science and what might be the reason to use Debian.
Hi, Andreas.
The Sanger Centre in the UK use Ubuntu + OpenStack + Ceph:
https://www.sanger.ac.uk/group/core-software-services/
I realise that it's not Debian, but it is based on Debian. I went there
many years ago when they were running Debian on DEC Alpha AXP's, but
they moved to CentOS because many other Academic HPC centres were using
it, including ours when I worked at the University of Aberdeen.
This was not a good experience, and they decided to change to Ubuntu
mainly because of the support provided by Canonical for OpenStack and
Ceph. However, in my opinion, CentOS/RHEL is not a good platform for
bioinformatics because the 'Enterprise' approach stifles innovation.
You can't ignore the host OS when you talk about HPC applications and
the HEP (High Energy Physics) community put a lot of effort into
developing good node provisioning systems and job-scheduling for HPC.
Consequently, there was a significant bias towards support for HEP
applications running under CentOS and less support for bioinformatics.
This was partly the motivation underlying our development of Bio-Linux
in order to provide biologists with an alternative platform running on
their own hardware instead of struggling to get the IT department to
port the software they wanted to use to CentOS. In that respect the
Debian-Med project was fundamentally important in helping biologists do
their work outside of the centrally managed 'Enterprise' oriented IT
policy imposed on us by Universities and Research Institutes.
The Sanger Centre provide a centrally managed HPC that is
'biologist-friendly' and, I think, is an excellent model of how it
should be done. However, it does not support the view that Debian should
be the HPC OS because the main reason they chose Ubuntu was the
commercial support for OpenStack and Ceph provided by Canonical.
I personally would like to stress the "we package what we use" aspect
and the "we mentor upstream to merge competence of the program with
packaging skills" idea. Any input would be welcome to cover more ideas.
As you might remember, I built and I advocate the use of 'departmental'
or 'research-group' clusters. These are much more powerful than an
individual biologists personal laptop, but are under the administrative
control of the department or research group that funded their purchase.
In the past, I've used various HPC node-provisioning, cluster filesystem
and job submission systems running under one version of another of
Bio-Linux, now using your "med-bio" meta-package to provide
bioinformatics software instead of the discontinued Bio-Linux packages.
However, I've recently set up a 3-node 'Proxmox-VE' cluster:
https://www.proxmox.com/en/proxmox-virtual-environment/overview
[Proxmox is a GPL server management system based on Debian]
I'm using the Proxmox cluster for a bioinformatics in schools project
with the University of Edinburgh:
https://4273pi.org/
I'm also planning to use it for a new project with the IAEA in Vienna.
I think that giving biologists the choice of running the software they
want under the OS they choose is very important when innovation is the
priority of an organisation rather than centralisation of IT systems to
reduce cost. You can, of course use Proxmox-VE as the node-provisioning
and shared filesystem of an HPC cluster. Or, simply provide biologists
with VMs running their OS of choice, administered by themselves e.g. a
Bio-Linux VM or vanilla Debian etc. etc.
Finally, don't forget about Amdahl's Law:
https://en.wikipedia.org/wiki/Amdahl%27s_law
There is really no such thing as an HPC or HTP 'application', because
it's the underlying resource management system of an HPC cluster that
provides the 'HP'. In my experience, most bioinformatics applications
are 'embarrassingly' parallel and in this case processes do not
communicate with each other. The 'HP' is achieved by managing the
workflow efficiently using e.g. "Slurm" or "[Sun]Grid-Engine".
MPI-parallel processes are subject to diminishing returns as the number
of processes increases as described by Amdahl's Law. Other 'Map-Reduce'
workflows are also 'embarrassingly' parallel and GPGPU workflows while
parallel on a given device are not HPC in the conventional sense despite
nVidia's claims to the contrary, but are GPGPU-accelerated applications
that are, typically, executed as embarrassingly parallel work-flows
across clusters of servers with GPGPUs. All 'SLI' does is to combine
GPGPUs locally on a server which, again, is not conventional HPC.
All of these technologies have fantastic potential, but I've been very
frustrated and disappointed many times how difficult it is to utilise
GPGPU for bioinformatics. It's only recently that e.g. Oxford Nanopore
Technology and Pacbio have made effective use of GPGPU for base-calling
long reads. The basic problem is that existing 'classic' bioinformatics
analyses require re-writing to execute in parallel and it is only a few
new bioinformatics algorithms that are worthwhile designing for GPGPU.
Bye,
Tony.
--
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK)
tel. +44(0)19755 63548 http://minke-informatics.co.uk
mob. +44(0)7985 078324 mailto:tony.travis@minke-informatics.co.uk
Reply to: