Re: HDF5 linking issues with new NetCDF packages
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi Gilles,
Thanks for the feedback, and enjoy your vacation. I've started mine
with this work in preparation of the netcdf transition. :-)
On 31-07-15 10:51, Gilles Filippini wrote:
> Hi Bas,> > Sebastiaan Couwenberg a écrit le 30/07/2015 16:38 :
>> As mentioned in the 'netCDF Transition' thread [1] we're
>> preparing the netcdf transition in which we'll update from NetCDF
>> 4.1.3 to 4.4.0.
>>
>> One of the significant changes is the switch to the HDF5 MPI
>> variant. I'd like your opinion about the HDF5 related issues that
>> have surfaced.
>>
>> There is a common issue with linking the HDF5 libraries the new
>> netcdf library is built with. Several netcdf dependents FTBFS due
>> to link issues such as this:
>>
>> gcc -std=gnu99 -shared -L/usr/lib/R/lib -Wl,-z,relro -o ncdf4.so
>> \ ncdf.o ncdf2.o ncdf3.o src_ncdf4.o -L/usr/lib -lnetcdf
>> -lhdf5_hl \ -lhdf5 -lz -ldl -lm -lcurl -L/usr/lib/R/lib -lR
>> /usr/bin/ld: cannot find -lhdf5_hl /usr/bin/ld: cannot find
>> -lhdf5 collect2: error: ld returned 1 exit status
>>
>> As you documented in the hdf5 transition (#755539) [2] adding
>> the HDF5 library and include paths is the way to fix these
>> issues.
>>
>> The best way to fix this for the new netcdf package is probably
>> to patch nc-config & netcdf.pc to include the HDF5 library and
>> include paths too.
>
> Indeed.
There results were much better after implementing the above, adding
the HDF5 library & include paths in nc-config & netcdf.pc fixed the
FTBFS for all packages that failed to find the hdf5 libraries before.
I've not uploaded the netcdf (1:4.4.0~rc2-1~exp4) revision containing
these changes yet, I'll update it to switch back to the HDF5 serial
variant by default, and build with the MPI variant in addition.
>> Another issue with the switch to the HDF5 MPI variant for the new
>> netcdf package is that it's the only option it's built with.
>> Because most (if not all) of our packages are built with the HDF5
>> serial variant by default, it may not be wise to not have the
>> option in netcdf too.
>
> IIUC MPI shouldn't be the only option or you'll have to use mpirun
> for every executable depending on netcdf.
This may very well be a hidden issue. Nearly all reverse dependencies
build succesfully with the new netcdf package built only with the HDF5
MPI variant, but they may not work correctly at run time. I've only
built the packages, not installed and tested them.
>> I'm considering changing the new netcdf package to be more like
>> the hdf5 package, and default to the serial variant, and have the
>> MPI variants available too (with mpich for only a few
>> architectures).
>
> I can't think of a better solution. If you want to provide an MPI
> variant of netcdf you'll have to ship the serial variant as well.
As mentioned above I'll reintroduce the build with the HDF5 serial
variant, that will likely solve the ncview (2.1.5+ds1-1) FTBFS which
is caused by mpicc not deemed acceptable. And probably also the gmt
(5.1.2+dfsg1-1) FTBFS.
>> What do you think about these issues?
>
> Sorting out the libhdf5 build process to have all variant
> co-installable was really painful, but it was worth it. We also
> added symbols versioning to be able - in theory - to link an
> executable with both serial and mpi variants.
>
> I'd be pleased to give you a hand in this process, but I'll be in
> vac from today until the 25th of august. I'll keep an eye on my
> emails anyway.
There is a bit of time pressure to get the new netcdf package ready
for the transition, because of the GCC 5 related transitions that were
planned to start today. If building with the HDF5 MPI variant in
addition turns out to be too problematic, I'll stick to only the
serial variant for the time being (and reopen #708638). That's
probably the best course of action.
Kind Regards,
Bas
- --
GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146 50D1 6750 F10A E88D 4AF1
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCgAGBQJVu1IVAAoJEGdQ8QrojUrxdK4P/3ZI0vMxAaWfXbqe2n+nmXao
lPTB82JqfH+mE+A2isCQO0zjg/ubTZyhRVLzCEeL3BX7OtsA1Pc76rcb4i9WsmVe
x9ZTJPfoQJQFU7wa/2+j9y6N11aLXU2U1yplIFJw9KSQHa1Q2ojNuAIGJtXomTg9
Gtj6oR+xwrQ1ot6lMLfplwWpXtIaC+frRlp6LvZZvpa+M+tTWN+9370hVMG+9Mmz
iAprtZOYhyykldmWPKPorvK7k/lz64s5EM2OZ31yarzLgyydiqpcLdlwL/zsohXJ
6GSl1X47oMqOaZw9OlJe4vFHdO44qfW6XcJptOvgGHSy/+cehdtHrA0rOtDG69io
MaKO/L1YA5fT3XCY29uM29XCaJaaH1WMIOb0ZjRvlIgUcWutPCO7WKbqm9jwCGMc
j+coliO66d78OXjrjueJBTy834sBdeGL0Fn/hwPBT0eiu9j2e17mDWiZV4EbZTqz
uZkz283eQI5nsNEbpT1Rdk2RBkqx4O4Q+cm2WdGbzQwVbxw5rfWsrv6X/XkBjnLG
v8wCWHKSt5+aviSBqH9iP4v1DcibWcjUBnyDVzh2KOhKO5Bumx7YfZe+88l6X2yt
zON2zjelog3gn0zbUxHRnY1lbqhKShRc8WTjfCY0EFgm2f1BJ62COdlzzsRK76gi
wv73pvJLNeKk/a7jbMdv
=54Ig
-----END PGP SIGNATURE-----
Reply to: