Re: Some progress about the packaging of nextflow
Hi Pierre,
Am 01.06.2021 um 11:46 schrieb Andreas Tille:
Hi Pierre,
thanks a lot for keeping us up to date.
On Tue, Jun 01, 2021 at 10:49:11AM +0200, Pierre Gruet wrote:
Those last weeks I spent quite a lot of time pushing forward the packaging
of nextflow. I wrote a few patches and pushed 5 dependencies into NEW.
Very cool!
I need to have a closer look, still, but, yes, very cool, indeed!
Now there remains quite a lot to do.
The biggest tasks would be to package the SDK of AWS [0] (which would also
benefit igv) and of Microsoft Azure [1]. We would also need to package
Apache Ignite [2].
After the freeze I shall discuss this on the ML of the Debian Java team. I
cannot really figure out how hard packaging those whole SDK would be. I also
have a poor idea of the maintenance burden over time.
[0] https://github.com/aws/aws-sdk-java
[1] https://github.com/Azure/azure-sdk-for-java
[2] https://github.com/apache/ignite/
These sound a bit scary for a single person (just guessing from the
names that it might be huge) but in the end it will probably be used by
lots of other software which in turn might mean that there are more
people who might join the effort to package these.
Yip. Both for packaging and for maintenance. I CCed our cloud team which
may have some extra opinion/direction for us.
Yet nextflow only relies on those in its plugins, maybe it can be built and
packaged without them, to begin with.
*@Steffen*: do you think it is worth having nextflow in Debian if the
plugins related to Amazon, Azure and Ignite are not inside?
Absolutely!
Also, we would need to upgrade the dependency libkryo-java, of which version
2.20 is currently in Debian. Yet there were lot of ABI changes between
version 2.20 and current upstream version 5.1.1, and even today gradle
depends on a version before the ABI changes. So maybe it should be worth
introducing a new package, say libkryo5-java, so that gradle still relies on
libkryo-java but nextflow can depend on libkryo5-java. I shall also discuss
this on the Debian Java ML after the release.
I agree that sometimes several versions need to be packaged. If gradle
needs something old it is probably hard to avoid having two versions.
The C++ world does not understand sonames, and the Java world somehow
got lost in their folder structures.
If anyone here has some experience with the Amazon or Azure SDK, please
kindly share it as I am willing to determine if packaging them is feasible.
I stumbled upon this *name* before. However, if nextflow could be
sensibly used without those extra plugins it might possibly a shortcut
here which can be extended once more effort can be put into those
heavy sdks.
For our immediate Covid-19 aims, having nextflow on single machines or
local clusters is just good enough. For Bioinformatics at large, I am
afraid, we better be compatible. Azure has a lower priority than AWS
from what I observe.
Nextflow blocks so many workflows for us, this would be really great to
have with us, and if it is only for local executions as a start. Thank
you very much for your effort!
Best,
Steffen
Reply to: