netcdf-fortran @4.6.1: Build failures on 10.7, 10.8, 10.9, 14-x86, 14-arm64

[Dave-Allured (Dave Allured)]

Netcdf-fortran 4.6.1 on builders 10.7, 10.8, 10.9 were working a few months ago, but now fail in configure. What could have changed to cause this?

configure: error: cannot compute sizeof (off_t)

See attached stdio.txt. (Main.log was not available.)

I am not sure where this is coming from. A quick internet search suggests this might be caused by not correctly linking to the underlying netcdf library, but it could be anything.

[Dave-Allured (Dave Allured)]

stdio.txt from builder 10.9

[jmroot (Joshua Root)]

4.6.1 has never built successfully on these OS versions; the last successful builds were for 4.6.0. It's failing to determine the size of several other types before it gets to off_t, e.g.

checking size of int... 0

The information about why would be in the config.log.

[Dave-Allured (Dave Allured)]

The information about why would be in the config.log.

Ah. Thanks for spotting those other type size errors. How might I get a copy of config.log for 10.7, 10.8, or 10.9? I do not find that in any of the builder summaries. I do not have a local machine running any of those versions.

​https://build.macports.org/builders/ports-10.7_x86_64-builder/builds/158097
​https://build.macports.org/builders/ports-10.8_x86_64-builder/builds/146524
​https://build.macports.org/builders/ports-10.9_x86_64-builder/builds/244191

[ryandesign (Ryan Carsten Schmidt)]

You (or someone) would need a local machine running such an OS version. The contents of the buildbot work directories are deleted after a build.

[Dave-Allured (Dave Allured)]

You (or someone) would need a local machine running such an OS version.

Hmmm. In that case, I may need to give up on this issue until someone else can make an appropriate test run.

In general, I rely on builder feedback to help me work on older OS related issues. It seems that saving config.log for failed builds would be a relatively modest addition. This could be helpful in many situations. Has this been considered before? Where would be a good forum to propose this?

[ryandesign (Ryan Carsten Schmidt)]

I tried building on 10.7 on my 2012 15" MacBook Pro with Retina Display and I did not see this problem. (I saw a different build error for which I'll file a separate ticket later.)

Yes, making the config.log file(s) (and equivalents for non-autotools build systems) available via the web interface is a good idea that I'm sure has come up countless times over the years but nobody has implemented it yet. If there is not a ticket you could file one.

[Dave-Allured (Dave Allured)]

Update. The type size errors are merely side effects of failing configure tests, under the hood. By comparing the 10.9 log file with other recent builder logs, I found an earlier problem:

checking netCDF-4 present... no

This should be "yes". This comes from netcdf_meta.h in the netcdf dependency. I confirmed this contains #define NC_HAS_HDF5 1 which is correct. So this is another under the hood config test that is failing.

Here is another anomaly, even earlier.

checking for /usr/bin/clang option to enable C11 features... -std=gnu11

This is blank in successful builds for 10.6 and 10.10. What is causing this to be added? Could this be breaking the configure tests on 10.9 etc.?

[Dave-Allured (Dave Allured)]

OS 14-x86 fails in configure with approximately the same symptoms. Netcdf-4 not present, cannot compute size of off_t, etc. See attached stdio.log for OS 14-x86.

[kencu (Ken)]

there are many google references for errors building netcdf-fortran with the error cannot compute size of off_t

in all cases, the solution is to fix whatever is causing the netcdf library not to be found, ie fix : checking netCDF-4 present... no

[Dave-Allured (Dave Allured)]

Replying to kencu:

there are many google references for errors building netcdf-fortran with the error cannot compute size of off_t

in all cases, the solution is to fix whatever is causing the netcdf library not to be found, ie fix : checking netCDF-4 present... no

Unfortunately those error messages are not accurate. They only mean that a configure test failed, not necessarily library not found. I need to somehow get a copy of a failed config.log, as discussed in comments 1-3. I do not have a matching test machine for any of the failed builds.

[kencu (Ken)]

Replying to Dave-Allured:

Unfortunately those error messages are not accurate.

That was my point, actually -- the sum of internet wisdom is that the off_t issue is likely not the real problem, rather whatever is making the library not seem to be found when it is there is the real trouble, so that is where attention should be spent.

I was going to fix it for you, as I have a few spare minutes and I have every MacOS system Apple ever made running, most in hardware, some only as VMs.

I thought I would try the oldest of the failures, as that is usually going to be the hardest one to fix, but lo-and-behold, it just builds through on 10.7 without touching a single thing in any Portfile:

$ port -v installed netcdf-fortran
The following ports are currently installed:
  netcdf-fortran @4.6.1_3+gcc13 (active) requested_variants='' platform='darwin 11' archs='x86_64' date='2023-12-25T15:41:32-0800'

No problems finding the netCDF-4 library:

checking netCDF logging present... yes
checking netCDF v2 API present... yes
checking netCDF-4 present... yes

or finding off_t or any of the related types:

checking for _Bool... yes
checking for stdbool.h that conforms to C99... yes
checking for size_t... yes
checking for working alloca.h... yes
checking for alloca... yes
checking for size_t... (cached) yes
checking for off_t... yes
checking for ssize_t... yes
checking for ptrdiff_t... yes

so as everything here seems 100%, I guess I won't be able to help on 10.7 at least, as I can't reproduce it.

If I get time I will look at one of the other systems that failed.

[Dave-Allured (Dave Allured)]

but lo-and-behold, it just builds through on 10.7 without touching a single thing

I am impressed. So I suppose something in the dependency tree was updated?

If I get time I will look at one of the other systems that failed.

In particular, I am interested in the new case for Sonoma.

[kencu (Ken)]

I rescheduled a build on the 10.7 buildbot, and it failed again, just the same.

Must be some missing dependency I guess. Let me see if I can figure out what one, using trace mode (which still works on 10.7).

[Dave-Allured (Dave Allured)]

The latest rebuild attempt for 14_x86/Sonoma, following a hopeful HDF5 update, has failed in the same way as before.
​https://build.macports.org/builders/ports-14_x86_64-builder/builds/22374

checking netCDF-4 present... no

It would help me if someone with Sonoma could test the current portfile and provide a copy of config.log if it fails like this on your system.

[Dave-Allured (Dave Allured)]

Also fails on 14.arm64 in the same way.
​https://build.macports.org/builders/ports-14_arm64-builder/builds/5790

checking netCDF-4 present... no

[kencu (Ken)]

installed for me with no problem on Sonoma arm64

 % port -v installed netcdf-fortran
The following ports are currently installed:
  netcdf-fortran @4.6.1_3+gcc13 (active) requested_variants='' platform='darwin 23' archs='arm64' date='2024-01-22T20:00:53-0800'

[kencu (Ken)]

successful main.log on sonoma arm64

[kencu (Ken)]

successful config.log on Sonoma arm64

[Dave-Allured (Dave Allured)]

@kencu, thanks. Would you please trigger rebuilds for 14.x86 and 14.arm on Macports builders? Let's see whether some underlying change has fixed those failing builds. I have my suspicions, but do not want to further speculate.

If either fails, please catch config.log if possible, and post here. That is the part where I have no visibility.

[Dave-Allured (Dave Allured)]

As expected, your successful main.log shows:

checking netCDF-4 present... yes

... so this is what we are still working on.

[kencu (Ken)]

I don't know how to get the config.logs from the buildbot builders -- I don't think you can.

but I can trigger a rebuild.

yes, this is mysterious one. If you look at the configure script to see what it is trying to do, it is really gobbledygook mysterious.

[Dave-Allured (Dave Allured)]

Okay, never mind about config.log from builders. I look forward to a new builder result.

On your successful build on Sonoma arm64, did you build your own dependencies, or download prebuilt binaries?

[richarddray]

It failed for me, on Sonoma 14.3 Intel Xcode 15.2, after doing as clean an install as possible: sudo port -fp uninstall installed; sudo port install gcc13; sudo port select --set gcc mp-gcc13; sudo port install netcdf-fortran

I just added the config.log (at least I think I did); I hope it helps somebody figure this out.

[Dave-Allured (Dave Allured)]

@richarddray, this is very helpful. Would you also please attach main.log from the same build?

[Dave-Allured (Dave Allured)]

@richarddray, never mind. I think I got it. Comparing Ken's config.log to yours, this shows up and is unexpected:

dyld[92894]: Library not loaded: /opt/local/lib/libbz2.1.0.dylib
  Referenced from: <0416D6CF-FC05-318C-8F92-09AD8B456287> /opt/local/lib/libnetcdf.19.dylib
  Reason: tried: '/opt/local/lib/libbz2.1.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/local/lib/libbz2.1.0.dylib' (no such file), '/opt/local/lib/libbz2.1.0.dylib' (no such file)
./configure: line 2087: 92894 Abort trap: 6           ./conftest$ac_exeext

I will need to look into this. It looks like libbz2 is supposed to be a plugin for libnetcdf, not a hard link as shown here. So, missing dependency, but the context is not right. Hmmm.

[richarddray]

This main log goes with my previous config log.

[Dave-Allured (Dave Allured)]

@richarddray, thanks for main.2.log. Port netcdf has several missing dependencies, including bzip2 which is the current error. If convenient, please try my updated portfile for netcdf (NOT netcdf-fortran). Let us know how it goes.

Draft commit:
​https://github.com/Dave-Allured/macports-ports/commit/5addc753092a5fe1c5b9d8cb9f2611affd1951f1

1. Download updated netcdf Portfile from my test branch:
   

​https://github.com/Dave-Allured/macports-ports/raw/netcdf.deps.1/science/netcdf/Portfile

2. Copy into your local ports tree for netcdf, overwriting the Macports version:
   

/opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/science/netcdf/Portfile
(Use your custom prefix if you are using one.)

3. DO NOT run selfupdate after step 2, otherwise you will need to recopy the new Portfile back into your ports tree.

4. port uninstall netcdf

5. port -v install netcdf-fortran

There are variations on this procedure, such as skipping step 4, or starting over from scratch with the updated Portfile. Ask questions if you have any problems. This will help with my testing of the new Portfile.

[kencu (Ken)]

It's so much harder to find these now that trace mode does not (easily) work on arm Macs....hopefully we fix that soon.

[Dave-Allured (Dave Allured)]

Ken, this explains rather nicely why there are different results with netcdf-fortran on different systems. Any OS, Arm or Intel, this all works out about the same.

If you had bzip2 installed by coincidence, then netcdf-fortran would build correctly and not know anything was wrong. For a fresh install from nothing, bzip2 would *not* get installed automatically, and netcdf-fortran build would fail. That failure would be hidden inside config.log, and the usual stdio and main.log would throw inaccurate confusion like "netcdf-4 NO" and "cannot compute sizeof (off_t)". This is exactly what we are seeing here.

[richarddray]

Dave-Allured, your Portfile worked fine. The library installed, and I tested it by recompiling an old fortran program, which then executed properly. Thanks!! (Do you need the log files for this? If so, please remind me where to find them, as I don't see them at first search. And with that -v option, everything zoomed by me so fast I couldn't read anything!)

[richarddray]

Oh, one thing occurs to me. Last night I downloaded many other packages, which could themselves have installed "missing" libraries. So your test was no longer on a clean install. If you need it, I can uninstall everything and do a clean install of just gcc13 and netcdf-fortran. Let me know.

[Dave-Allured (Dave Allured)]

@richarddray, there is no need for the log files. Thanks for mentioning your other downloads. You are right, you might have installed a missing dependency by coincidence. A clean test from scratch would be optimal.

However, your successful build of netcdf-fortran is sufficient to qualify my pull request to Macports. So let's skip your extra build. Netcdf-fortran will be re-tested in clean context by the Macports Sonoma builders. If there is a lingering problem, it will be caught there. Thanks for your help on this. I have wanted to get this ticket moving for a long time.

[Dave-Allured (Dave Allured)]

PR ​https://github.com/macports/macports-ports/pull/22502 in progress.

[Dave-Allured (Dave Allured)]

In 2523f70706b262fe5abbc84be03fb8df27b8105c/macports-ports (master):

netcdf @4.9.2: Add missing dependencies

.

• Hard linked dependencies were causing downstream failures, e.g. netcdf-fortran.
• Reference: #68124
• Add missing dependencies which may have been opportunistically hard linked.
• Missing libxml2 added for DAP.
• Missing bzip2, zlib, zstd added for compression filters.
• Add blosc as external library for compression filter.

[Dave-Allured (Dave Allured)]

These build failures were generally fixed by:

[2523f70706b262fe5abbc84be03fb8df27b8105c/macports-ports]
[c68044728a12d2ab8561dfd6a2c1e5b0a6db595e/macports-ports]

There are lingering issues in one or two cases, but the missing dependencies problem is solved. Please close this ticket.