Bug#1061255: ITP: custodian -- flexible just-in-time job management framework in Python
Package: wnpp
Severity: wishlist
Owner: Drew Parsons <dparsons@debian.org>
X-Debbugs-Cc: debian-devel@lists.debian.org, debian-python@lists.debian.org, debian-science@lists.debian.org, debichem-devel@lists.alioth.debian.org
* Package name : custodian
Version : 2024.1.9
Upstream Contact: Shyue Ping Ong <ongsp@ucsd.edu>
* URL : https://github.com/materialsproject/custodian
* License : MIT/X
Programming Lang: Python
Description : flexible just-in-time job management framework in Python
Custodian is a simple, robust and flexible just-in-time (JIT) job
management framework written in Python. Using custodian, you can
create wrappers that perform error checking, job management and error
recovery. It has a simple plugin framework that allows you to develop
specific job management workflows for different applications.
Error recovery is an important aspect of many high-throughput projects
that generate data on a large scale. When you are running on the order
of hundreds of thousands of jobs, even an error-rate of 1% would mean
thousands of errored jobs that would be impossible to deal with on a
case-by-case basis.
The specific use case for custodian is for long running jobs, with
potentially random errors. For example, there may be a script that
takes several days to run on a server, with a 1% chance of some IO
error causing the job to fail. Using custodian, one can develop a
mechanism to gracefully recover from the error, and restart the job
with modified parameters if necessary.
The current version of Custodian also comes with several sub-packages
for error handling for Vienna Ab Initio Simulation Package (VASP),
NwChem, QChem, FEFF, Lobster and CP2K calculations.
Custodian has been developed by the Materials Project team responsible
for pymatgen, and is used to manage tests for emmet-core etc. It is a
general python package, but designed for computational chemistry. It
could arguably be managed by the Debian Python Team, but probably best
to keep it alongside pymatgen managed by the Debichem team.
Reply to: