Re: Inclusion of the ducc0 Python package?
Dear Ole,
thanks for your reply!
in principle, I think it would be a good addition. However, I would like
to have the package having a bit more "strengtened" focus. Currently, it
looks like a random collection of algorithms, which are finally hard to
find for a developer.
There is a method to the madness, at least for most of the components :)
The SHT and gridding parts depend on the FFT, the total convolution part
in turn depends on SHT, and the Healpix component traditionally has very
strong ties to the SHT component as well.
All of this could be separated into individual packages of course, but
given the fact that the entire C++ sources are comparable in size to a
single typical configure script, I'm really not sure this would be
beneficial.
For example, if one would want to use fft, they would probably just
stick to scipy and not even search for something else (unless there is a
specific need). Therefore, the best here would probably be, instead of
maintaining your own package, to try to include it into scipy with a
pull request (on https://github.com/scipy/scipy). This would also have
the advantage that you get a review of your code by other experienced
people.
That's exactly how scipy got its current FFT implementation :)
Unfortunately I cannot do the same with the more recent versions since
scipy will not (yet) accept C++ sources that require a standard beyond
C++11.
Similar things can be said for the sht and the convolution algorithms.
That's not possible due to licensing conditions; I want to keep the
(non-FFT) parts of the code under GPLv2+, and that is not accepted by
the scipy maintainers.
For Healpix, I suggest to have a look into the Healpix package of
astropy, https://github.com/astropy/astropy-healpix, and to help them
instead of maintaining your own package.
Licensing incompatibilities again, I'm afraid.
And for the gridding algorithm, from the README it turns out that it is
already integrated in wsclean, so probably it is better to cooperate
with this package instead of having an extra one.
Wsclean contains a copy of the relevant source files from my own repo,
and I synchronize these every few months; this is a pragmatic solution
we have adopted until ducc's C++ interface has stabilized. Since wsclean
is a C++ code, it lacks the Python interface to the gridder which I
consider important for ease of use.
It is really better to improve the existing software packages instead to
start writing your own. Aside from the often fruitful discussion during
integration, the long-term maintainance is much better ensured and the
visibility is much higher. And for the software community, it is easier
to have a canonical place for an algorithm instead of searching a number
of "Useful Code Collections". Fragmentation of software is also
fragmentation of development power.
Does this make sense for you? I am however happy to discuss this if
(resp. where) you disagree.
It makes sense to me, and I agree to a certain extent, but I'd argue
that these algorithms can be applied in such broad contexts that
incorporating them into specific other packages actually makes them
harder to use in unforeseen contexts. In fact, the evolution of the
package has been almost exactly the reverse of what you are proposing.
For example, the FFT algorithm started as part of the official Healpix
package (around 15 years ago). Only after I decided to separate it from
this package, I could propose it for inclusion into numpy and later scipy.
The SHT component was developed as part of Healpix as well. Now that it
has been separated out, it is used as an optional back-end in SHTOOLS
(https://github.com/SHTOOLS/SHTOOLS) and SSHT
(https://github.com/astro-informatics/ssht).
The total convolution code is now also used in TOAST (see
https://github.com/hpc4cmb/toast/pull/387).
I think that this kind of re-use has been greatly facilitated by having
a low-level standalone package with a very flexible interface rather
than integrating tightly with particular existing packages.
Best regards,
Martin
Best regards
Ole
Martin Reinecke <martin@MPA-Garching.MPG.DE> writes:
Hi,
I wanted to ask whether it might make sense to include the "ducc0"
package of numerical algorithms into debian-astro.
Main page: https://gitlab.mpcdf.mpg.de/mtr/ducc
This package contains C++17 implementations of
- a pretty fast FFT (the one used in scipy, but with several improvements)
- spherical harmonic transforms (an evolution of the libsharp library)
- algorithms for 4pi convolution on the sphere (an evolution of the
"totalconvolver" and "conviqt" codes)
- a wide-field gridding/degridding algorithm for radio astronomy
- beginnings of a Python wrapper for Healpix-related functionality.
with a pybind11-based Python interface. The current goal is to provide a
stable Python interface, while the internal C++ interfaces are still in
flux. (More details including references etc. directly in the README.md)
The code has minimal external dependencies (pybind11) and is quite
portable (tested on Linux x86_64, Linux arm32, MacOS x86_64 and Windows
x86_64).
At the moment, not many packages are explicitly depending on it, but
healpy will probably switch to this as a back-end in the future, the
Pixell library might use it as well, and other packages like wsclean use
it by incorporating parts of its source code.
Do you think this package is sufficiently "on topic" for Debian astro,
and if yes, how would we proceed?
Thanks,
Martin
Reply to: