OpenBLAS compilation fails on MacBook Pro MacOS 10.14

[Tim-at-AST]

This may be a duplicate of Ticket #57912 (ticket) but that seems to have been resolved for the user (ticket:57912#comment:30), so iI don't know for sure.

I am trying to install py37-scipy which has OpenBlas as a dependency, and this latter install fails.

 Warning: the chosen compiler cannot handle advanced optimisation instructions.
         AVX instructions are disabled.
Error: Failed to build OpenBLAS: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port openblas failed

[jmroot (Joshua Root)]

:debug:sysinfo Xcode none

Do you in fact have Xcode installed? What output do you get from xcodebuild -version?

[Tim-at-AST]

I have the command line tools only.

$ xcodebuild -version
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance

$ xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" to install updates

[NicosPavlov]

Then it is most likely the issue. Having Xcode installed is part of the installation of Macports. See ​Installing Macports

[Tim-at-AST]

I don't think the XCode is the problem; all my other similar macports install perfectly (e.g. numpy, numba etc), and I have installed scipy before on a different computer, but same setup (no XCode Gui, just command line tools), last year.

Besides which, the error message seems to indicate something else is the problem.

[kencu (Ken)]

Here is your error:

:info:build /opt/local/bin/gfortran-mp-8 -O3 -m128bit-long-double -Wall -frecursive -m64   -o sblat1 sblat1.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran 
:info:build ld: library not found for -lSystem
:info:build collect2: error: ld returned 1 exit status
:info:build make[1]: *** [sblat1] Error 1
:info:build make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.5/test'

You can see for yourself what it happening, if you like.

Go into the noted directory:

cd /_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.5/test

open up the permissions a bit

sudo chmod -R a+rw .

and then run your command:

/opt/local/bin/gfortran-mp-8 -O3 -m128bit-long-double -Wall -frecursive -m64   -o sblat1 sblat1.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran

it will most likely fail with the noted inability to find -lSystem (which is a basic system library that is apparently not being found where gcc8 is expecting it to be found.

Then you can get more info by doing something like this, to see where gcc8 and the linker are looking:

/opt/local/bin/gfortran-mp-8 -v -Wl,-v -O3 -m128bit-long-double -Wall -frecursive -m64   -o sblat1 sblat1.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran

[Tim-at-AST]

Thanks for the further analysis. Where do I go from here? I really do need XCode gui? I can see that it can't find lSystem, but don't know whether it's because I don't have lSystem at all, or because it's looking in the wrong place.

[kencu (Ken)]

That's why - if you don't want to install Xcode -- you'll need to run this command from the referenced directory above:

/opt/local/bin/gfortran-mp-8 -v -Wl,-v -O3 -m128bit-long-double -Wall -frecursive -m64   -o sblat1 sblat1.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran

and see what it outputs. Then you can at least have an idea what's going wrong.

[Tim-at-AST]

OK, installed XCode, still the same problem.

$ sudo port install py37-scipy
Password:
--->  Computing dependencies for py37-scipy
The following dependencies will be installed: 
 OpenBLAS
 swig
 swig-python
Continue? [Y/n]: Y
--->  Building OpenBLAS
Error: Failed to build OpenBLAS: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port py37-scipy failed
$ xcodebuild -version
Xcode 10.1
Build version 10B61

[kencu (Ken)]

OK. We'll get you up and running, I think.

Please do

sudo port clean openblas
sudo port selfupdate
sudo port -v upgrade outdated
sudo port -v install openblas

and if it fails again, please post up a new main.log so we can see what happened.

Ken

[Tim-at-AST]

newer version of error log.

[kencu (Ken)]

So now the linker finds libSystem, but can't understand it

:info:build ld: unexpected token: !tapi-tbd-v3 file '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/lib/libSystem.tbd' for architecture x86_64

The linker looks wrong. It should understand that tbd file. Please report back the result of:

port -v installed | grep ld64

to see what linker we're dealing with here...

[Tim-at-AST]

$ port -v installed | grep ld64
  ld64 @3_1 (active) platform='darwin 18' archs='x86_64' date='2019-02-01T14:37:19+1100'
  ld64-latest @274.2_2+llvm50 (active) platform='darwin 18' archs='x86_64' date='2019-02-01T14:37:16+1100'
  ld64-xcode @2_1 (active) platform='darwin 18' archs='x86_64' date='2019-02-03T20:48:00+1100'
$

[kencu (Ken)]

OK. I think we got it now, then. You need to match mine:

$ port -v installed | grep ld64
  ld64 @3_1+ld64_xcode (active) platform='darwin 18' archs='x86_64' date='2018-11-05T18:01:53-0800'
  ld64-xcode @2_1 (active) platform='darwin 18' archs='x86_64' date='2018-11-05T18:01:53-0800'

so

sudo port -f uninstall ld64 ld64-latest
sudo port -v install ld64 +ld64_xcode

and then all should be well. Follow that with

sudo port clean openblas
sudo port -v install openblas

[kencu (Ken)]

You have stumbled across a current issue we have wherein Apple's latest released open-source version of ld64-latest is behind the ld64-xcode version, due to some internal Apple issues that are being sorted out.

We have the default variants set up for ld64 to get you to the +xcode variant, but somehow, you don't have that. I'm not sure how or why you don't, but anyway, that fix should do it for you.

[Tim-at-AST]

Yes, great , all working now!

Thank you for your time and effort. I appreciate it very much.

Tim

[kencu (Ken)]

+1 !

[kencu (Ken)]

I think the reason you wound up with the wrong ld64 variant (ie not the +xcode variant) has to do with the fact you didn't have Xcode installed. The variant is enabled automatically if the xcode version test is >= 9 -- but I'll be the fact you didn't have Xcode installed made that test not trigger, and so you wound up without the default variant.

There are many such places in MacPorts where it is assumed that the standard installation is present, and testing for all possible variations that people might have is not feasible for the manpower involved.

[Tim-at-AST]

Understood.

[dsavransky (Dmitry Savransky)]

main.log from failed openblas +native install

[dsavransky (Dmitry Savransky)]

I'm having the exact same problem after updating to Xcode 11. After the update I had a lot of issues with previously installed command line tools version, so I ended up removing those and am now trying to rely on Xcode only (which annoyingly only has the 10.15 SDK). In any case, trying to install openblas +native produces the same "library not found for -lSystem" error, even though my ld64 port looks correct:

ld64 @3_1+ld64_xcode (active) platform='darwin 18' archs='x86_64' date='2019-09-26T14:00:14-0400' ld64-xcode @2_1 (active) platform='darwin 18' archs='x86_64' date='2019-09-26T14:00:14-0400'

Attaching the log. Any ideas? Thanks.

[JohnPritchard (Johnny P)]

I have the same problem with macOS 10.14+Xcode 11+MacPorts-2.6.0. Simply by adding -L/usr/lib to a failing compile/link command allows it to work, e.g.:

sudo -u macports /opt/local/bin/gfortran-mp-9 -m128bit-long-double -Wall -frecursive -fno-optimize-sibling-calls -m64  -mavx2  -o sblat1 sblat1.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran
ld: library not found for -lSystem
collect2: error: ld returned 1 exit status

But the following succeeds...

sudo -u macports /opt/local/bin/gfortran-mp-9 -m128bit-long-double -Wall -frecursive -fno-optimize-sibling-calls -m64  -mavx2  -o sblat1 sblat1.o -L/usr/lib ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran

[JohnPritchard (Johnny P)]

I seem to have solved the problem (at least for OpenBLAS and py-numpy) by rebuilding/installing gcc9 (which provides gfortran needed by OpenBLAS) from source.

sudo port uninstall gcc9
sudo port clean gcc9
sudo port -s install gcc9
sudo port -s install OpenBLAS

So it seems that there is some sort of incompatibility between gcc9 (and probably many other) binaries built under Xcode-10 which are then run in an Xcode-11 environment.

It would probably be safest to rebuild/install from source ALL ports in such an installation, well that's what I'll probably have my computer doing over the weekend...

Hope this was helpful.

[jmroot (Joshua Root)]

This is indeed not an openblas issue but a gcc one. Currently gcc cannot be built against the 10.15 SDK. I think there's another ticket covering that already.

[dsavransky (Dmitry Savransky)]

Ok. Confirming that gcc9 fundamentally cannot be built right now against 10.15. However, after restoring command line tools (which install SDK 10.14 - does anyone understand what Apple is doing here?) John's method of building everything from source works fine. Thanks very much John and Joshua.

[adamcrussell (Adam Russell)]

I had the same issue as the OP and the suggested fix

sudo port uninstall gcc9 sudo port clean gcc9 sudo port -s install gcc9 sudo port -s install OpenBLAS

worked for me as well.

[TP75]

Replying to dsavransky:

Ok. Confirming that gcc9 fundamentally cannot be built right now against 10.15. However, after restoring command line tools (which install SDK 10.14 - does anyone understand what Apple is doing here?) John's method of building everything from source works fine. Thanks very much John and Joshua.

One should consider the opportunity to repair some apparent slip of Apple XCode by a workaround such as the simple provision of an appropriate symbolic link by help of the command-line (CLI) as described below:

$ cd /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/

$ sudo ln -s MacOSX.sdk MacOX10.14.sdk

Hope this helps.