hdf5 @1.14.2_0 build failure

[kwolcott]

hdf5 build from source (initiated by "sudo port -v -s upgrade outdated") MacOS Ventura 13.5.2 M1 chip I had just updated xCode tools; could that be a cause?

[jmroot (Joshua Root)]

:info:build ld: unknown options: -commons 

[Dave-Allured (Dave Allured)]

@kwolcott, I can not read your log file with XZ compression. Please post small logs uncompressed. Thanks.

[Dave-Allured (Dave Allured)]

Thank you. Yes, this might be related to your Xcode tools update. Your log file, line 5:

:debug:sysinfo Xcode 14.1, CLT 15.0.0.0.1.1694021235

I am not an Xcode expert, but this looks like a mismatch of Xcode vs. Command Line Tools. Try updating either one so that they match. For best results, you might want to match the same versions used in the latest build on the Macports Ventura ARM64 builder:

:debug:sysinfo Xcode 14.3, CLT 14.3.1.0.1.1683849156

Also I am not sure whether HDF5 has ever been tested with Xcode 15, so staying within the 14.x series might be helpful.

[kwolcott]

I really wish that Apple would not push beta things when I don't have beta enabled for autoupdate :-(

Since McOS Sonoma is going to be released soon I'll wait to do anything further with xCode until then.

I'm hoping that after the OS upgrade is finished installing and I update the xCode and related tools, but hdf5 will build from source correctly.

Thanks, Ken

[Dave-Allured (Dave Allured)]

Copy that. In that case, I suggest just try downgrading your CLT to 14.1, and leave Xcode right where it is. Then HDF5 etc. Xcode is a heavy load, CLT is relatively light. My guess is 14.1 will work just as well as 14.3 for this.

[news24lor]

I was updating all my installed software and updating the library "hdf5" from version 1.14.1-2 to version 1.14.2 I had several compilation errors and the library did not update. This library is important because it is linked to GDAL and consequently also to QGIS3 and many other software.

I have last "Xcode 15" and Command Line Tools updated with macOS Ventura 13.5.2.

My Mac has an Apple Silicon M2Pro Processor.

The problem is with "hdf5" 1.14.2. There was no problem with "hdf5" 1.14.1.

[Dave-Allured (Dave Allured)]

@newslor, thank you. This ain't no Xcode version mismatch! Extracting some version info from your main.2.log:

:debug:sysinfo macOS 13.5.2 (darwin/22.6.0) arch arm
:debug:sysinfo MacPorts 2.8.1
:debug:sysinfo Xcode 15.0, CLT 15.0.0.0.1.1694021235
:debug:sysinfo SDK 13
:debug:sysinfo MACOSX_DEPLOYMENT_TARGET: 13.0

:debug:configure Using compiler 'Xcode Clang'
:debug:configure CC='/usr/bin/clang'
:debug:configure CFLAGS='-pipe -Os -arch arm64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk'
:debug:configure MACOSX_DEPLOYMENT_TARGET='13.0'
:info:configure checking build system type... arm-apple-darwin22.6.0
:info:configure checking host system type... aarch64-apple-darwin22.6.0
:info:configure compiler '/usr/bin/clang' is clang-15.0.0
:info:configure compiler '/usr/bin/clang++' is clang-15.0.0
:info:configure checking dynamic linker characteristics... darwin22.6.0 dyld
:info:configure checking for ld used by /usr/bin/clang... /Library/Developer/CommandLineTools/usr/bin/ld

From the HDF5 config summary:

:info:configure  LDFLAGS: -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -arch arm64 -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk
:info:configure  H5_LDFLAGS:  -Wl,-commons,use_dylibs

The error and offending command:

:info:build libtool: link: /usr/bin/clang -std=c99 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3 -pipe -Os -arch arm64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -Wl,-commons -Wl,use_dylibs -Wl,-headerpad_max_install_names -Wl,-rpath -Wl,/opt/local/lib/libgcc -arch arm64 -Wl,-syslibroot -Wl,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -o H5detect H5detect.o  -L/opt/local/lib/libaec/lib -L/opt/local/lib -lsz -lz -ldl -lm
:info:build ld: unknown options: -commons 
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)

[Dave-Allured (Dave Allured)]

I find that -Wl,-commons,use_dylibs is used successfully in the current 13.arm64 Macports build for HDF5-1.14.2. As already mentioned, that build used Xcode 14.3 and CLT 14.3.

Also I find -Wl,-commons,use_dylibs is present in the configure file in both HDF5 versions 1.14.1 and 1.14.2. The builder log for 1.14.1 on 13.arm64 has expired, so I can't easily check whether -commons was used in that one.

It looks like the linker included in CLT 15.0 is choking on -commons. This could be tested by substituting HDF5 1.14.1_2 for 1.14.2, and rebuild on one of these CLT 15.0 systems. But you would need a Macports environment to be sure. A user could test this by extracting the previous HDF5 1.14.1_2 Portfile from github history, install in place of the new 1.14.2 Portfile, port clean hdf5, then port install hdf5.

[Dave-Allured (Dave Allured)]

Hmmm. I find that -Wl,-commons,use_dylibs comes from configure.ac in HDF5 source, more than one version. This is fortran related. That is obscure enough that Apple might have forgotten to re-test it. You might try disabling the fortran variant with port install hdf5 -fortran.

[news24lor]

Hi Dave

I have rebuilt hdf5 without fortran port install hdf5 -fortran .

It works!!!

But after it requires new rebuilt. One of these is netcdf @4.9.2+dap+netcdf4+universal requires hdf5 and it rebuilts hdf5 with fortran

hdf5-1.14.2_0+cxx+fortran+gfortran+hl+universal.darwin_22.arm64-x86_64.tbz2

And I have the error.

It's a loop.

Is the link to fortran the problem?

You can see new main.log

[Dave-Allured (Dave Allured)]

Lorenzo (@news24lor), first of all, thank you for testing hdf5 -fortran. This is excellent information, knowing that the HDF5 1.14.2 core C library builds correctly with CLT 15.0.

Something in your port configuration is responsible for requesting the fortran variant when you proceed to add other ports. I can confirm that HDF5 fortran is NOT required to build either netcdf OR netcdf-fortran. At this time I do not have enough Macports experience to be able to trace that fortran variant requirement. From here, I can think of two approaches:

(1) Locate the port configuration that is requesting the fortran variants, and disable that request.

(2) Disable the -commons,use_dylibs option in the HDF5 build. That option is found in the HDF5 configuration variable H5_LDFLAGS (see log file). I think this can be done directly with portfile commands, without needing a new patch file for configure. But once again, I do not have enough portfile experience to know how to do that. Investigate portfile commands relating to configure options.

[Dave-Allured (Dave Allured)]

This issue is being tracked at HDF5.
"hdf5 fails to build with Xcode 15 / macOS Sonoma"
​https://github.com/HDFGroup/hdf5/issues/3571

It is the same issue. "Apple ships with Xcode 15 and macOS Sonoma a new linker, which does not accept the -commons option."

[Dave-Allured (Dave Allured)]

Homebrew has reported a workaround. The explanation is "We've worked around the issue by forcing use of the old linker".

​https://github.com/Homebrew/homebrew-core/pull/144559

    # Work around incompatibility with new linker (FB13194355)
    # https://github.com/HDFGroup/hdf5/issues/3571
    ENV.append "LDFLAGS", "-Wl,-ld_classic" if DevelopmentTools.clang_build_version >= 1500

Let's see if we can work that into the Macports portfile. Help requested.

[Dave-Allured (Dave Allured)]

I investigated HDF5 library source code. It looks like -commons is now completely unnecessary for building HDF5 versions 1.10.x and newer, so it can be patched out of the Mac ports build. I have not tested this yet. More details here:
​https://github.com/HDFGroup/hdf5/issues/3571

[Dave-Allured (Dave Allured)]

Started pull request: ​https://github.com/macports/macports-ports/pull/20553

Need testing help. X86, M1, or M2. If you have updated to Xcode 15.0 or CLT 15.0, please test this PR. Copy the new Portfile and two patch files into your local Macports tree, under science/hdf5. Patch files go into the "files" subdirectory. Then port install hdf5. Report results back on this trac ticket.

[ryandesign (Ryan Carsten Schmidt)]

Replying to Dave-Allured:

@kwolcott, I can not read your log file with XZ compression. Please post small logs uncompressed. Thanks.

Dave, our ​ticket guidelines recommend users compress logs larger than 256K, which this one is. The guidelines say to use bzip2 or gzip, but xz compresses better and can be decompressed by the macOS Archive Utility (or with the unxz program that's part of the xz port) so I see no problem with posting xz-compressed logs.

[Dave-Allured (Dave Allured)]

Ryan, my mistake. I fully agree, longer log files should be compressed. I misinterpreted this error message while trying to read the original posted XZ file:

mac839:~/temp 68> tar xf hdf5_build_from_source_fails.log.xz 
tar: Error opening archive: Unrecognized archive format

I think this is actually a damaged XZ file, because my tar version has read many other XZ files with no problem. In this case only, I appreciated receiving the second log file from Ken.

[kwolcott]

I didn't use tar, just xz.

[Dave-Allured (Dave Allured)]

That explains it. I can read your original file with xz. There must be some XZ format update between Apple default tar, and command line xz. I suggest use "tar cJpf file.tar file" to reach a wider audience. With J you still get XZ compression.

[Dave-Allured (Dave Allured)]

New pull request: ​https://github.com/macports/macports-ports/pull/20572

[Dave-Allured (Dave Allured)]

An alternative fix was recently merged in commit [3d830f34ca5b20269400aa9f19231a8bb3d36c29/macports-ports]. You should be able to get the new fix right now, with port selfupdate. Xcode/CLT 15.0 users, if you have the time, please test and report here.

[kwolcott]

Seems to build now. Thanks.

[Dave-Allured (Dave Allured)]

Great. Whomever has the right permission, please close this ticket as completed.

[ryandesign (Ryan Carsten Schmidt)]

Replying to Dave-Allured:

That explains it. I can read your original file with xz. There must be some XZ format update between Apple default tar, and command line xz. I suggest use "tar cJpf file.tar file" to reach a wider audience. With J you still get XZ compression.

There's no reason to use tar when one is only compressing a single file.