qgis3 fails to build on macOS Sonoma ARM

[alanterra (Alan Harper)]

I have tried many times to build qgis3, and it fails. It seems that if I install other ports first, then it fails to build py311-scipy, but if I do a clean install of qgis3 it fails building p5.34-io-compress-brotli. I ran

  sudo port -fp uninstall installed
  port install qgis3
  ...
  Error: Failed to build p5.34-io-compress-brotli: command execution failed

Before I upgraded to Sonoma, qgis3 built without problems. I don't know if the code base changed.

[alanterra (Alan Harper)]

/opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_perl_p5-io-compress-brotli/p5.34-io-compress-brotli/main.log

[ryandesign (Ryan Carsten Schmidt)]

The log says:

:info:build make: *** read jobs pipe: Resource temporarily unavailable.  Stop.

so that would suggest too many things running at one time on your computer, and if you try again later when you have fewer things running it might work.

I see that you're building with 24 parallel jobs. MacPorts wouldn't choose that unless you had at least 24 processor cores and at least 24 GB of memory. That's been pretty uncommon until recently so that may explain why we haven't run into this much before. You can try with fewer jobs, such as:

sudo port build p5.34-io-compress-brotli build.jobs=8

If this proves to be a common problem you can make a lower number of jobs the default by uncommenting buildmakejobs in /opt/local/etc/macports/macports.conf and changing its value.

[ryandesign (Ryan Carsten Schmidt)]

The problem could be MacPorts base. When you install a port that has a lot of dependencies (according to port rdeps qgis3 | wc -l there are 670 other ports that need to be installed before qgis3 can be installed) MacPorts needs to examine all of those other ports and put them in the right order before building them. It's possible MacPorts is keeping one or more file handles open for each of those ports, which would reduce the number of file handles available for other things like actually compiling. If so, this may be a bug; MacPorts base may have forgotten to close a file handle after it was done with it, or there may be a more optimal way to arrange the code that doesn't require so many files to be kept open. It also makes sense that you see the problem with a different port after installing some other ports: if the other ports you installed were dependencies of qgis3, then when you install qgis3 there are fewer ports remaining to be installed for which MacPorts needs to keep files open, leaving more files available for the actual build.

The lsof utility may be able to help us verify whether too many open files is the problem, though I can't advise you on its usage.

Prior to Sonoma, you were probably receiving most of qgis3's dependencies as pre-built archives from our build system and MacPorts didn't have to compile them from source on your system so you wouldn't have seen problems that only occur at build time. We have not yet begun building archives for Sonoma on arm64 so you have to build everything from source for now.

[ryandesign (Ryan Carsten Schmidt)]

Instead of decreasing the number of jobs, you can try installing ports in smaller batches. For example, pasting this command into your terminal will tell MacPorts to install a random one of qgis3's dependences that you haven't installed yet:

sudo port -N install --unrequested $(port echo depof:qgis3 and not active | sort -R | head -n1)

After doing that some number of times, you should be able to install qgis3 and its remaining dependencies successfully (unless there is also another problem).

[alanterra (Alan Harper)]

/opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-scipy/py311-scipy/main.log

[alanterra (Alan Harper)]

Thank you, Ryan. I think that your diagnosis is at least partly correct. And, also, now I know why my builds are so much slower than before.

But it looks like there may be a "hard" (repeatable) error in py311-scipy. I am in way over my head, so I don't understand the error message, but I attached the log for your perusal.

Luckily, QGIS is the one component I don't need to build in macports. There is a version on the web that will serve, and I'll download it next. So I'll probably not come back to this ticket, as I don't think I need to build qgis3, and all my other ports are (finally) building without any problems.

[nilason (Nicklas Larsson)]

If you build with -plugin_support311 , it should work (for the time being).

[nilason (Nicklas Larsson)]

#68329

[bal-agates]

I was able to build qgis3 on macOS 14.2 arm64 today. The py311-scipy build problems seem to have been solved by an OpenBLAS change. I never encountered an error building p5.34-io-compress-brotli.

Likely this issue can be closed.

[BryanWall (Bryan Wall)]

I experienced the issue today with the QGIS build failing because of an issue building p5.34-io-compress-brotli. I tried some of the suggestions above which mention Macports possibly keeping too many file handles open and was eventually able to get QGIS to build successfully.

This was on a 2021 Macbook Pro, M1 Pro CPU, 16 GB RAM. I was also trying to migrate Macports after upgrading to Sonoma. All of my Macports config is at the defaults, AFAIK.

I'm wondering if there a reason that it would consistently fail on the same package out of thousands. If the issue does have something to do with running out of file handles, I would expect the failure point to be somewhat random on different machines with different amounts of RAM, different installed packages, etc.

[ryandesign (Ryan Carsten Schmidt)]

Replying to BryanWall:

I experienced the issue today with the QGIS build failing because of an issue building p5.34-io-compress-brotli.

That's #65496.