gcc11 on arm64 is completely broken: non-functional due to linking error

[jeffrey-hokanson (Jeffrey M. Hokanson)]

I am attempting to build OpenBLAS for use with py39-numpy. I ran into a compile issue that may be related to #63072.

:info:build /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/compwrap/fc/opt/local/bin/gfortran-mp-11 -Os -m64 -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8-a -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8-a -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -arch arm64 -o sblat1 sblat1.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran 
:info:build ld: library not found for -lgcc_s.1.1
:info:build collect2: error: ld returned 1 exit status

[jeffrey-hokanson (Jeffrey M. Hokanson)]

Build log

[michaelld (Michael Dickens)]

OpenBLAS is at 0.3.15 and about to be upgraded to 0.3.16 ( ​https://github.com/macports/macports-ports/pull/11576 ). I just tried building OpenBLAS 0.3.16 +native on 11.5beta & it works fine for me. Once the upgrade happens please try it & see if it works ... and if not then please attach a new build log.

[michaelld (Michael Dickens)]

OpenBLAS 0.3.16 and current OpenBLAS-devel both build for me on my ARM64 / M1 Mac ... but that's with +gccdevel ... if I try this with +gcc11 (the default) it fails as noted. yes it looks like gcc11 is broken ... doesn't provide libgcc_s.1.1.dylib ...

[jeffrey-hokanson (Jeffrey M. Hokanson)]

Thanks for following up with this; I was able to successfully build with +gccdevel.

[michaelld (Michael Dickens)]

adding Ken as I believe the issue here is with GCC11 rather than NumPy ... maybe he can shed some thought on the matter!

[michaelld (Michael Dickens)]

Replying to jeffrey-hokanson:

Thanks for following up with this; I was able to successfully build with +gccdevel.

Great! I'm glad we have a viable workaround ... though we need +gcc11 to be working, since we should not in general rely on gcc-devel.

[kencu (Ken)]

Last I looked, I thought our gcc11 appeared quite broken on arm64 due to the @rpath thing.

Michael, can you try building even a simple "hello, world" with it and see if it works at all?

[michaelld (Michael Dickens)]

Oh right ... I need to get you access to my Arm Mac ... sorry about that & I'll email you directly with that info once I get there (hopefully soon!)!!!

[michaelld (Michael Dickens)]

simple program:

#include <stdio.h>

int main ()
{
  printf ("hello world!\n");
  return (0);
}

compiled via:

% gcc-mp-11 -v -o hello_world_gcc11_arm64 hello_world_gcc11_arm64.c
Using built-in specs.
COLLECT_GCC=gcc-mp-11
COLLECT_LTO_WRAPPER=/opt/local/libexec/gcc/arm64-apple-darwin20/11.1.0/lto-wrapper
Target: arm64-apple-darwin20
Configured with: /opt/local/var/macports/build/_opt_sources_MacPorts_ports_github_macports_lang_gcc11/gcc11/work/gcc-11.1.0-arm-20210504/configure --prefix=/opt/local --build=arm64-apple-darwin20 --enable-languages=c,c++,objc,obj-c++,lto,fortran,jit --libdir=/opt/local/lib/gcc11 --includedir=/opt/local/include/gcc11 --infodir=/opt/local/share/info --mandir=/opt/local/share/man --datarootdir=/opt/local/share/gcc-11 --with-local-prefix=/opt/local --with-system-zlib --disable-nls --program-suffix=-mp-11 --with-gxx-include-dir=/opt/local/include/gcc11/c++/ --with-gmp=/opt/local --with-mpfr=/opt/local --with-mpc=/opt/local --with-isl=/opt/local --enable-stage1-checking --disable-multilib --enable-lto --enable-libstdcxx-time --with-build-config=bootstrap-debug --with-bugurl=https://trac.macports.org/newticket --enable-host-shared --disable-tls --without-build-config --with-as=/opt/local/bin/as --with-ld=/opt/local/bin/ld --with-ar=/opt/local/bin/ar --with-pkgversion='MacPorts gcc11 11.1.0_2' --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.1.0 (MacPorts gcc11 11.1.0_2)
COLLECT_GCC_OPTIONS='-v' '-o' 'hello_world_gcc11_arm64' '-mmacosx-version-min=11.4.0' '-asm_macosx_version_min=11.4' '-mlittle-endian' '-mabi=lp64'
 /opt/local/libexec/gcc/arm64-apple-darwin20/11.1.0/cc1 -quiet -v -D__DYNAMIC__ hello_world_gcc11_arm64.c -fPIC -quiet -dumpbase hello_world_gcc11_arm64.c -dumpbase-ext .c -mmacosx-version-min=11.4.0 -mlittle-endian -mabi=lp64 -version -o /var/folders/l6/vl_7wvbd77z1wk5k44n43hz00000gn/T//ccmFf4I4.s
GNU C17 (MacPorts gcc11 11.1.0_2) version 11.1.0 (arm64-apple-darwin20)
	compiled by GNU C version 11.1.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.22.1-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/opt/local/include"
ignoring nonexistent directory "/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/../../../../../arm64-apple-darwin20/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/include
 /opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/include-fixed
 /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/System/Library/Frameworks
End of search list.
GNU C17 (MacPorts gcc11 11.1.0_2) version 11.1.0 (arm64-apple-darwin20)
	compiled by GNU C version 11.1.0, GMP version 6.2.1, MPFR version 4.1.0, MPC version 1.2.1, isl version isl-0.22.1-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: bfc010515fd6e9ff990e91e318ffddbb
COLLECT_GCC_OPTIONS='-v' '-o' 'hello_world_gcc11_arm64' '-mmacosx-version-min=11.4.0'  '-mlittle-endian' '-mabi=lp64'
 /opt/local/bin/as -arch arm64 -v -mmacosx-version-min=11.4 -o /var/folders/l6/vl_7wvbd77z1wk5k44n43hz00000gn/T//ccdeyaFL.o /var/folders/l6/vl_7wvbd77z1wk5k44n43hz00000gn/T//ccmFf4I4.s
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: aarch64-apple-darwin20.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1as -triple arm64-apple-macosx11.4.0 -filetype obj -main-file-name ccmFf4I4.s -target-cpu apple-a12 -target-feature +v8.3a -target-feature +fp-armv8 -target-feature +neon -target-feature +crc -target-feature +crypto -target-feature +fullfp16 -target-feature +ras -target-feature +lse -target-feature +rdm -target-feature +rcpc -target-feature +zcm -target-feature +zcz -target-feature +sha2 -target-feature +aes -fdebug-compilation-dir /opt/local/var/macports/build/_opt_sources_MacPorts_ports_github_macports_science_volk/volk/work/volk-2.5.0/cpu_features/build -dwarf-debug-producer "Apple clang version 12.0.5 (clang-1205.0.22.11)" -dwarf-version=4 -mrelocation-model pic -mllvm -disable-aligned-alloc-awareness=1 -o /var/folders/l6/vl_7wvbd77z1wk5k44n43hz00000gn/T//ccdeyaFL.o /var/folders/l6/vl_7wvbd77z1wk5k44n43hz00000gn/T//ccmFf4I4.s
COMPILER_PATH=/opt/local/libexec/gcc/arm64-apple-darwin20/11.1.0/:/opt/local/libexec/gcc/arm64-apple-darwin20/11.1.0/:/opt/local/libexec/gcc/arm64-apple-darwin20/:/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/:/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/
LIBRARY_PATH=/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/:/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/../../../
COLLECT_GCC_OPTIONS='-v' '-o' 'hello_world_gcc11_arm64' '-mmacosx-version-min=11.4.0'  '-mlittle-endian' '-mabi=lp64' '-dumpdir' 'hello_world_gcc11_arm64.'
 /opt/local/libexec/gcc/arm64-apple-darwin20/11.1.0/collect2 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/ -dynamic -arch arm64 -macosx_version_min 11.4.0 -weak_reference_mismatches non-weak -o hello_world_gcc11_arm64 -L/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0 -L/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/../../.. /var/folders/l6/vl_7wvbd77z1wk5k44n43hz00000gn/T//ccdeyaFL.o -lgcc_s.1.1 -lgcc -lSystem -lgcc -no_compact_unwind -rpath @loader_path -rpath /opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0 -rpath /opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/../../.. -v
collect2 version 11.1.0
/opt/local/bin/ld -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/ -dynamic -arch arm64 -macosx_version_min 11.4.0 -weak_reference_mismatches non-weak -o hello_world_gcc11_arm64 -L/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0 -L/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/../../.. /var/folders/l6/vl_7wvbd77z1wk5k44n43hz00000gn/T//ccdeyaFL.o -lgcc_s.1.1 -lgcc -lSystem -lgcc -no_compact_unwind -rpath @loader_path -rpath /opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0 -rpath /opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0/../../.. -v
@(#)PROGRAM:ld  PROJECT:ld64-650.9
BUILD 13:09:13 May 28 2021
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
Library search paths:
	/opt/local/lib/gcc11/gcc/arm64-apple-darwin20/11.1.0
	/opt/local/lib/gcc11
	/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/lib
Framework search paths:
	/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/System/Library/Frameworks/
ld: library not found for -lgcc_s.1.1
collect2: error: ld returned 1 exit status

I don't recall if I built GCC11 from source or used the precompile binaries ... I'll build from source & try again, and if that fails then I'll try the precompiled binaries.

[michaelld (Michael Dickens)]

FYI this is not an issue for gcc-devel ... it works! So the @rpath issue isn't at least for gcc-devel

[kencu (Ken)]

I think it's the @rpath thing, but it will take some sleuthing. It could turn out to be something else in the end.

[michaelld (Michael Dickens)]

gcc11 built from source or from binary both show the same error:

% gcc-mp-11 -o hello_world_gcc11_arm64 hello_world_gcc11_arm64.c
ld: library not found for -lgcc_s.1.1
collect2: error: ld returned 1 exit status

[michaelld (Michael Dickens)]

is there a Trac issue for the GCC RPATH issue? (Or is this the CMake RPath issue?)

[kencu (Ken)]

There is no trac ticket I know of for this; I suspected an issue, but it's only proven as of now, so I guess we will need one.

I believe we will need to bring gcc back to the non-@rpath fold for MacPorts in some fashion, rather than adapt MacPorts to support @rpath linkages for gcc/libgcc. Iain was going to give us a configure arg to disable @rpath, or a separate non-@rpath branch to pull from if we wanted that -- if those do not exist (I haven't seen them yet), then we either need to revert the @rpath commits (yuck) or we need to rewrite the libraries to have full paths on installation and possibly tweak gcc to let it know we did that (probably best, I believe).

The cmake issue is separate, but is probably going to become more and more of an issue (unless we have avoided it with our cmake PortGroup settings, which would be a lucky break). So far, the cmake issue is only hitting Tiger I believe.

[kencu (Ken)]

I believe this has nothibg to do with arm, so anyone interested in coming up with a fix could build Iain's gcc11 branch on Intel and work on this.

The easiest fix would be to leave rpath usage as it is and fix where gcc11 is looking, I believe. Or move/symlink the libraries to where it is looking.

The long-term solution might be to strip out the @rpath stuff post-destroot, maybe.

[kencu (Ken)]

oh wait. gcc-devel is using Iain's branch, and apparently works.

gcc11 on arm is using FX's branch, and is broken.

Worth knowing that homebrew carries a specific patch of their own creation to change the location of some of their libraries, in their gcc formula.

[kencu (Ken)]

Just thought I'd flag this a little harder. This is the primary gcc / libgcc version for BigSur on arm64, and it's been broken for some months now, I think pretty much since gcc11 for arm was changed to use FX's gcc11 branch.

[cjones051073 (Chris Jones)]

gcc11 on arm has always used there FX branch. Probably the best option for now is to remove gcc11 from the known/allowed lists for arm, which will mean reverting back to using the devel gcc build.

[michaelld (Michael Dickens)]

At least the gcc-devel port seems to have been quite good on ARM64 for a while now ... and now that I have a Mac Mini M1 I'm able to keep it up to date ... honestly ;)

[kencu (Ken)]

Michael, can you check to see if gcc-devel on arm64 is presently generating @rpath links to it's libraries, or full path links?

[michaelld (Michael Dickens)]

@kencu : is this what you're looking for? If not, please provide code to test.

% uname -a
Darwin Mac-Mini-M1 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:27 PDT 2021; root:xnu-7195.141.2~5/RELEASE_ARM64_T8101 arm64

% cat hello_world.c
#include <stdio.h>

int main ()
{
  printf ("hello world!\n");
  return (0);
}

% gcc-mp-devel -o hello_world -Wall hello_world.c

% ./hello_world
hello world!

% otool -L hello_world
hello_world:
	@rpath/libgcc_s.1.1.dylib (compatibility version 1.0.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)

[kencu (Ken)]

Yeah -- @rpath links.

So it works, but ultimately we will have to fix this (I believe) to be full path links and then rebuild everything that was built by gcc-devel.

OR -- we learn to love @rpath in MacPorts. This is what upstream is hoping we do, but I am not sure we can live with that.

[cooljeanius (Eric Gallager)]

Iain Sandoe recently did a commit to fix up rpath support for gcc on darwin: ​https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=15bdae016654f63a36e49a37c9d26282bebb1da9

It'll be in GCC 12; I wonder if it can be backported?

[kencu (Ken)]

Aren't we already using that commit in gcc-devel? (which is what we use on arm).

[michaelld (Michael Dickens)]

Yes that commit is in gcc-devel already for months now

[cjones051073 (Chris Jones)]

Latest builds should be ok on arm now.