Ports that depend on hdf5-18 should switch to hdf5

[mf2k (Frank Schima)]

Ports that depend on hdf5-18 should be updated and tested to work with the newer hdf5 port.

See also #52289.

Port	Maintainer	status
h4h5tools	-	8136a51881ec689cc5ff2a11dafa5bd4b240f0a5/macports-ports
hdfeos5	tenomoto	changeset:259fb59a028dcbb823a616c554215e9aa0a325a4/macports-ports
netcdf	tenomoto	#54177
octave-devel	MarcusCalhoun-Lopez	Was fixed at some point
py-tables	mamoll	changeset:1fa97da43102e3502c9381c05999f5bd8dceee61/macports-ports
ncarg	tenomoto	changeset:8a1abdde9a46af2b26b4f301a72394764e06a4e3/macports-ports

[mamoll (Mark Moll)]

In 1fa97da43102e3502c9381c05999f5bd8dceee61/macports-ports:

py-tables: update to version 3.4.2, switch from hdf5-18 to hdf5 (see #54101)

[Dave-Allured (Dave Allured)]

Please recall the original reasons for hdf5-18. It is a little hard to tell from the original tickets (ticket:51089 etc), but this alternative port solved two problems:

1. All hdf5 versions starting with 1.10.0 (2016 March 30) could easily generate new hdf5 data files that were backward incompatible with all older, non-updated hdf5 applications. Hdf5-18 prevented this, and wrote only backward compatible files. The correct long term fix is to (a) insert a specific hdf5 compatibility function call into the calling application; then and only then, (b) upgrade to hdf5 1.10.x. For example, here is a discussion and compatibility fix that was applied in Netcdf version 4.4.1 (2016 June 28). The single function call to H5Pset_libver_bounds did the trick:

​https://github.com/Unidata/netcdf-c/issues/250 (2016 April 7)

2. Hdf5 versions 1.10.0 and 1.10.0-patch1 could not access network-mounted files from Macs. I am told this was specifically due to new usage of the system flock function, and shortcomings of NFS version 3 on Macs. This of course would not affect any Mac users that used only local disk files. Hdf5-18 avoided this problem because it never used flock. Hdf5 version 1.10.1 (2017 April 27) appears to fix this, using a special run-time environment variable. IMO the correct upgrade path is to link only with hdf5 1.10.1 or later, and then remind users to use the environment variable when needed. The 1.10.1 update is forthcoming in ticket:54139, which includes details about the environment variable setting.

I recommend holding off on switching hdf5-18 dependencies to hdf5 until either (a) the format compatibility fix is inserted in the calling packages; or (b) there is some kind of consensus that original package developers and users no longer care about generating backward-incompatible hdf5 data files.

Furthermore, because of (2), I recommend waiting for hdf5 1.10.1 (ticket:54139) before switching any more hdf5-18 dependencies back to hdf5.

[aaschwanden (Andy Aschwanden)]

I just re-installed macports from scratch, and only the ports I really need (e.g. hdf5, hdf5-18, netcdf, py27-netcdf4, gdal, ...) and I'm already stuck with the usual hdf5 vs hdf5-18 library mismatch, this time when using netCDF4python. So far I have never been able to have a working macports installation that has hdf5 and hdf5-18 along side. So far my only working solution has been to overwrite all ports that use hdf5-18 and link them against hdf5. Not pretty and it only worked because I haven't use py-tables in a while. Maybe I have to compile things from scratch again.

Let's hope that hdf5 gets bumped up to 1.10.1 and we can get rid of hdf5-18 (I'd happily re-install macports again after that, well worth the trouble.)

[seanfarley (Sean Farley)]

Now that hdf5 is 1.10.1 and that netcdf works with it, is there anything preventing netcdf from being updated? @tenomoto?

[Dave-Allured (Dave Allured)]

Replying to seanfarley:

I concur that netcdf (only) is ready to update to netcdf version 4.4.1.1 (the latest), as well as its dependency to hdf5 1.10.1. Ticket #54177 is in place for this purpose.

Netcdf version 4.4.1.1 (and its predecessor 4.4.1) satisfied my complaint (1) above, backward compatible files. Hdf5 1.10.1 itself resolves my complaint (2), network mounted files.

[seanfarley (Sean Farley)]

Replying to Dave-Allured:

But other ports that depend on hdf5-18 still need to put in the H5Pset_libver_bounds, correct?

[Dave-Allured (Dave Allured)]

Replying to seanfarley:

Yes indeed. Thanks for asking.

Each package should be individually verified to be using H5Pset_libver_bounds so as to ensure writing backward compatible files, unless the relevant user community does not care. This is particularly important for packages like Netcdf that are commonly used for public distribution of large volumes of HDF5-based files.

This should be the responsibility of the original package developers, not MacPorts, IMO. See, for example, what the Netcdf developers did:

​​https://github.com/Unidata/netcdf-c/issues/250 (2016 April 7)

[tenomoto (Takeshi Enomoto)]

netcdf in [6b3cb0cd3a6a8345bc91f950cf528c4801bfe7f5/macports-ports].

[tenomoto (Takeshi Enomoto)]

hdffeos5 in [259fb59a028dcbb823a616c554215e9aa0a325a4/macports-ports].

I had problem building ncarg (and h4h5tools) with hdf5.

[tenomoto (Takeshi Enomoto)]

ncarg in [8a1abdde9a46af2b26b4f301a72394764e06a4e3/macports-ports].

[tenomoto (Takeshi Enomoto)]

h4h5tools builds fine with configure.cflags-append -DH5_USE_18_API. It appears that h4toh5 sets the version to the latest.

utils/h4toh5/h4toh5main.c: if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST) < 0)

I'll wait for a few days and commit the changes if no objections.

[mf2k (Frank Schima)]

In 8136a51881ec689cc5ff2a11dafa5bd4b240f0a5/macports-ports:

h4h5tools: Update to version 2.2.3

• Switch to hdf5

Fixes: #54101