Re: clusters, infrastructures, and package tools
On Tue, Feb 08, 2000 at 11:09:05PM +0100, Bud P. Bruegger wrote:
>
> I'm interested in the administration of clusters and infrastructures (i.e.,
> very large, dishomogeneous installations) and in related debian package and
> administration tools.
I've been thinking of automating the DQS queue complexes (node resource
lists) to add an entry for each Debian package. I've only been thinking,
not doing though. :) A hook into dpkg would be useful to update the
resources when the package list changes. Perhaps I could tie it into the
menu or doc systems. Jobs submitted to a heterogenous cluster could specify
their package, cpu, memory, and disk requirements ahead of time and only run
on the appropriate systems. This can be done now, but the queue complexes
have to be manually created. In my own small cluster I find the complexes
are always out of date, so I have a need to do this for myself.
It's unfortunate that DQS is non-free. PBS has now been released under a
BSD license though, and almost certainly has similar capabilities. If
someone packages it let me know and I'll make qsub,qdel,qconf,... into dpkg
alternatives so we can share the namespace. Or just conflict with dqs,
since there aren't many good reasons to run multiple batch systems on the
same machine anyway.
There have been a number of problems posted to debian-devel that are
begging for a batch system, but instead create their own incompatible,
underfeatured, and duplicated systems. The autobuilders, the archive
maintenance on master, mirroring, the daily log rotations and local index
construction,... Ideally I'd like to see all of these sorts of tasks scheduled via a
common batch system. GNU queue is quite small, unfortunately it doesn't use
the standard posix command set and is apparently broken. With a standard
queueing system available on all systems (even if it's just a shallow
wrapper around exec) we could modify all packages that run regular tasks to
run via the queueing system and avoid thrashing between all the different
simultaneously scheduled tasks.
>
> * debian's stability and well-done package tools are great for professional
> use--but the package system is too much geared towards single machines. It
> would be a petty not to add cluster features to make Debian more used in
> larger environments!
debconf can (or will?) provide much of the automatic configuration
required by a large installation, but it hasn't been around long enough yet
to be mature and have widespread support in packages. I believe it is
discussed on the debian-admin mailing list, in case you haven't already
found it.
-Drake
Reply to: