Re: OpenHPC on Debian
Hi Renato,
Thanks for opening this discussion ! I think co-ordination between
OpenHPC and Debian is overdue, but haven't had the time to do anything
myself,
so I'm glad someone else did.
On 22/07/2018 23:06, Afif Elghraoui wrote:
على الأحد 22 تـمـوز 2018 10:27، كتب Renato Golin:
On Sun, 22 Jul 2018 at 00:50, Afif Elghraoui <afif@debian.org> wrote:
If I understand correctly, you mean doing some of your work within
Debian officially?
Hi Afif,
That's not entirely clear to me, yet.
Some of those packages may even conflict with the distro's own
packages (due to library and version dependencies), so we try not to
get that close.
For example, we have GCC 7.1 and GCC 7.1 compiled packages. Those are
clearly incompatible with CentOS' GCC 5.4 compiled packages. We also
have meta packages, that installs bundles and/or set environment
configurations, which may break existing Debian packages? (I'm
guessing here).
This is a challenge. For me, Debian is a place where we _integrate_ free
software, to make a working whole.
Adding a new compiler is an example of an invasive change - the code
generated may not be compatible with existing code
in the distro. Testing out a whole distro like Debian shakes out issues
(see the work of the gcc8 transition happening now,
See e.g.
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=ftbfs-gcc-8;users=debian-gcc@lists.debian.org
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904998
But we've very good integration testing, especially with autopkgtest,
debci for Continous Integration:
https://ci.debian.net
triggering retests of a package when one of its dependencies change.
This can be useful information for deciding the state of play; e.g. is
gcc 8.2, OpenMPI 3.1.1 fit for release, or does it break
too many other packages ? this information can be co-ordinated between
Debian and OpenHPC.
In that sense, Debian would be the wrong place to *have* those
packages. It would create too many conflict and make our lives harder
for little benefit.
Well, at least metapackages that install bundles would be fine.
But some of those packages already exist in the HPC group at Debian,
so if I start building my own packages that override yours, I'm bound
to create problems for both of us.
I can see a few scenarios:
1. Complement: I only build the packages that you don't have. This
would create the problem of releases coming out at different times and
would potentially conflict versions and build flags.
2. Segregate: I take the source packages you have, add my own and
build in house. This would inhibit potential collaboration between our
efforts.
I think two good models for this, from the Debian perspective, are
(1) backports: building new releases of packages within an existing distro.
(2) Dare I say it: Ubuntu and similar derivatives, that snapshot debian
testing
( and the related concept of 'CUT: Continuously Usable Testing')
For the applicable packages you need that can go into Debian, if we use
the same git repositories and host on salsa.debian.org, the packages are
maintained within Debian, and you simply rebuild the sources for your
own releases, I think that would be enough collaboration.
3. Merge: We release the packages in Debian, trying to have the same
cadence of releases. This would complicate the release process on both
ends.
We could probably find a common ground, and use different strategies
for different package classes, but the more complex this is, the
harder it will be to maintain.
This I think we should try for: aiming for synchronising release versions,
so that version numbers (and build profiles) match as near as possible,
where possible.
Shaking out _why_ they are different can be a useful exercise: e.g.
avoiding major library increments
(having that discussion with upstream, sync. any major release with
ours, etc.)
The upside for us is getting more useful packages into Debian, so I hope
we can at least get something. In the process, you may end up knowing
enough to check for yourself, maybe becoming a Debian Maintainer [1] to
get upload rights for the set of packages you maintain.
We have a small user base and we usually only release what we can test.
We already add/update packages to our repos in that manner, but we
update close to 50% of our packages every few months.
If we are to have a Debian port, I would have to have the same thing
in our private repos, so not a big change to do that in Debian's own.
But it would (maybe) put pressure on your group to release more often
/ out-of-sync with your releases.
We should be putting the latest upstream releases of packages into
Debian Unstable regularly anyway, even if they already get replaced by
the time the next Stable release comes around, so this isn't a problem.
Also, Ubuntu is periodically pulling things in from Debian and would
benefit.
I'm willing to do the maintenance and packaging for as long as it's
approved within Linaro, but I'd have to convince the rest of the
OpenHPC (and Debian HPC) communities to keep it up in case Linaro
drops support.
Given I can't control any of that, I'd rather start with a simple and
pragmatic approach and grow organically, than create a massive repo
and release schedule and then see that I'm the only one really
interested. :)
Maybe the first step would be to try installing an OpenHPC-like system
from our PDF and propose adding the missing packages to make it work.
They're obvious and usually interesting to have anyway.
If that works, and people start using, we can grow more. If not,
Debian HPC would have gained a few packages that can get validated
with Ansible.
Do you have docs on how you use Ansible, vs day
Debian autopkgtest / debci
(https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html) ?
Ok, great! Looks like a plan.
Thanks and regards
Afif
Thanks
Alastair
--
Alastair McKinstry, <alastair@sceal.ie>, <mckinstry@debian.org>, https://diaspora.sceal.ie/u/amckinstry
Commander Vimes didn’t like the phrase “The innocent have nothing to fear,”
believing the innocent had everything to fear, mostly from the guilty but in the longer term
even more from those who say things like “The innocent have nothing to fear.”
- T. Pratchett, Snuff
Reply to: