Re: h5py and hdf5-mpi
On 2019-08-14 18:05, Steffen Möller wrote:
On 13.08.19 06:01, Drew Parsons wrote:
To reiterate, having h5py-mpi available will be transparent to a user
interacting with hdf as a serial library. It doesn't break serial use,
it just provides the capability to also run multicpu jobs.
This sounds like an omission not to feature, then. Please go for it.
h5py 2.9.0-3 will migrate to testing in a day or two, we can proceed
with the mpi then.
How do autotests work for MPI?
We simply configure the test script to invoke the same tests using
mpirun.
I am somewhat uncertain that Debian needs to be the instance testing
this. But given all the hick-ups that are possibly introduced by
parallelization - would be good to test it. And Debian should then take
some pride in it and announce that.
Once we've got mpi activated in h5py, we can check whether the
parallelisation does in fact improve your own workflow. Even on a laptop
or desktop, most come with at least 4 cpus these days. Even mobile
phones. Do you deal with GB-size hdf5 datasets, data for which access
time is noticeable? Ideally your data handling will speed up according
to the number of cpus added.
I don't think switching on mpi in h5py is itself such a big deal. But
if we can demonstrate that it measurably improves performance for a real
workflow, then that is worth crowing about.
Does Debian have any mechanisms to indicate that a software can run in
parallel? I am thinking about all the automation that now controls
workflows - like toil and/or cwl - or the testing of reverse
dependencies on some buildd. These can check for the presence for a
binary but don't immediately know if they should start it with mpirun.
No specific mechanism, since normally we known if the program is
intended to be mpi enabled or not.
But at the level of the package, we can look at dependencies, e.g.
apt-cache depends python3-scipy | grep mpi
apt-cache depends python3-dolfin | grep mpi
At the level of a given library or executable, objdump can be helpful,
e.g.
objdump -p /usr/lib/x86_64-linux-gnu/libsuperlu.so | grep mpi
objdump -p /usr/lib/x86_64-linux-gnu/libsuperlu_dist.so | grep mpi
For autopkgtest, it's our own tests so we already know if the program is
compiled with mpi or not. It wouldn't really make sense for the scripts
in debian/tests to check whether the program being tested was compiled
with mpi.
Drew
Reply to: