[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Debian Med in CentOS HPC environment - how?

On 05/31/2016 12:29 PM, Michael Crusoe wrote:
Reminder: you can make fake chroots without root privileges (can also be used to convert Docker images and run software within them): https://github.com/common-workflow-language/common-workflow-language/wiki/Userspace-Container-Review#getting-userspace-containers-working-on-ancient-rhel

If your kernel version is new enough and your security people are fine with Docker then there are various efforts to ease administration of Docker on top of HPC schedulers.
For Docker, I have created a script that creates a Docker image for all DebianMed packages.
they are available at https://docker-ui.genouest.org/app/#/  with container debian/xxx

Basically we fetch debianmed packages and create a docker with a simple "apt-get install". I also extract a few meta-data that are added to Dockerfile labels for additional info.


On Tue, May 31, 2016 at 11:10 AM, Steffen Möller <steffen_moeller@gmx.de> wrote:

If I can remember correctly, one reason we are doing this Debian
packaging is that we find not all but many packages not ultimately
straight-forward to compile. It is not the "configure && make install"
that bothers us, but the manifold dependencies and dependencies of those
dependencies that also need to be maintained. And this effort can be
shared, obviously.

Now, for a single binary I would not mind, but we are about to
transition all our "evolving routine" workflows over to the University's
central HPC cluster . And the suggestion is to just recompile what one
needs and automate that recompilation if it is any cumbersome - somewhat
surprisingly they did not say "package it".  It is some Jessie-analogous
CentOS, i.e. not a .deb distro and they feature the typical module
environments to get functionality in and out, together with slurm. Would
it be worthwhile to try installing the packages to $HOME/debroot as with |
dpkg-deb -x app.deb $HOME/debroot|
and fiddle with the $PATH environment not too different from how module
is doing it? This would certainly raise my interest in backports.

Our next  (first real) meeting is on Thursday. Docker and chroot
environments will certainly be discussed. We could also just not
completely migrate everything but constrain ourselves, e.g., on short
read alignments, and self-compile those "typical suspect" binaries only.
Not ideal. What do you folks do out there? Just not bother and use some
commercial cloud service? There would be single VMware instances for us,
but not tens or hundreds as in OpenStack or Eucalyptus. I presume that
this situation is rather common and I would very much like to hear your
opinions/experiences about it.



Michael R. Crusoe
Community Engineer & Co-founder
Common Workflow Language project
+32 (0) 2 808 25 58
+1 480 627 9108

Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE

gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438

Reply to: