ld64-latest (and others?): default variant llvm38 breaks linking with pre-3.8-clang

[Ionic (Mihai Moldovan)]

Recently, I changed my variants on ld64-latest and cctools from whatever I used before (probably llvm33) to the new default variant (llvm38) on OS X 10.9+ (with me being on 10.9.5.)

While installing clang-3.8 as a dependency of something else, I hit this error:

-- Check for working C compiler: /opt/local/bin/clang-mp-3.7 -- broken
CMake Error at /opt/local/share/cmake-3.6/Modules/CMakeTestCCompiler.cmake:61 (message):
  The C compiler "/opt/local/bin/clang-mp-3.7" is not able to compile a
  simple test program.

  It fails with the following output:

   Change Dir: /opt/local/var/macports/build/_opt_local_var_macports_sources_macports.rsync.ionic.de_release_ports_lang_llvm-3.8/clang-3.8/work/build/CMakeFiles/CMakeTmp



  Run Build Command:"/usr/bin/make" "cmTC_b11b8/fast"

  /Applications/Xcode.app/Contents/Developer/usr/bin/make -f
  CMakeFiles/cmTC_b11b8.dir/build.make CMakeFiles/cmTC_b11b8.dir/build

  Building C object CMakeFiles/cmTC_b11b8.dir/testCCompiler.c.o

  /opt/local/bin/clang-mp-3.7 -pipe -Os -arch x86_64
  -mmacosx-version-min=10.9 -o CMakeFiles/cmTC_b11b8.dir/testCCompiler.c.o -c
  /opt/local/var/macports/build/_opt_local_var_macports_sources_macports.rsync.ionic.de_release_ports_lang_llvm-3.8/clang-3.8/work/build/CMakeFiles/CMakeTmp/testCCompiler.c


  Linking C executable cmTC_b11b8

  /opt/local/bin/cmake -E cmake_link_script
  CMakeFiles/cmTC_b11b8.dir/link.txt --verbose=1

  /opt/local/bin/clang-mp-3.7 -pipe -Os -arch x86_64
  -mmacosx-version-min=10.9 -Wl,-search_paths_first
  -Wl,-headerpad_max_install_names -L/opt/local/lib
  -Wl,-headerpad_max_install_names -Wl,-rpath,@loader_path
  -Wl,-rpath,@loader_path/ CMakeFiles/cmTC_b11b8.dir/testCCompiler.c.o -o
  cmTC_b11b8

  dyld: Library not loaded: @executable_path/../lib/libLTO.dylib

    Referenced from: /opt/local/libexec/llvm-3.7/bin/ld
    Reason: Incompatible library version: ld requires version 1.0.0 or later, but libLTO.dylib provides version 0.0.0

  clang: error: unable to execute command: Trace/BPT trap: 5

  clang: error: linker command failed due to signal (use -v to see
  invocation)

Further investigation revealed:

root@nopileos~# ls -ldh /opt/local/libexec/llvm-3.7/bin/ld
lrwxr-xr-x 1 root 26 Jul 27 23:42 /opt/local/libexec/llvm-3.7/bin/ld -> /opt/local/libexec/ld64/ld

So when using clang-3.7, the system-default linker as installed by ld64/the other ld64-${version} ports are used, good.

root@nopileos~# otool -L /opt/local/libexec/llvm-3.7/lib/libLTO.dylib
/opt/local/libexec/llvm-3.7/lib/libLTO.dylib:
	/opt/local/libexec/llvm-3.7/lib/libLTO.dylib (compatibility version 0.0.0, current version 0.0.0)
	/opt/local/libexec/llvm-3.7/lib/libLLVM-3.7.dylib (compatibility version 0.0.0, current version 0.0.0)
	[...]

llvm-3.7's libLTO has compat version 0.0.0, current version 0.0.0.

root@nopileos~# otool -L /opt/local/libexec/llvm-3.8/lib/libLTO.dylib
/opt/local/libexec/llvm-3.8/lib/libLTO.dylib:
	@rpath/libLTO.dylib (compatibility version 1.0.0, current version 3.8.1)
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 3.8.1)
	[...]

llvm-3.8's libLTO changed the compat version to 1.0.0 and the current version to 3.8.1.

root@nopileos~# otool -L /opt/local/libexec/ld64/ld
/opt/local/libexec/ld64/ld:
	@executable_path/../lib/libLTO.dylib (compatibility version 1.0.0, current version 3.8.1)
	[...]

And the binary installed by ld64-latest (and symlinked via ld64) needs compat version 1.0.0.

clang-3.7 invokes /opt/local/libexec/llvm-3.7/bin/ld, so the expanded path to the library through @executable_path is /opt/local/libexec/llvm-3.7/lib/libLTO.dylib, which has (see above) a lower compat version of 0.0.0.

Breakage.

A fix might be to not use @executable_path, but rewrite the libLTO library dependency to always use the libLTO library as installed by the selected llvm*-variant version.

I haven't looked at the source code and how difficult that would be, though.

[jeremyhu (Jeremy Huddleston Sequoia)]

$ otool -L llvm-3.*/lib/libLTO.dylib 
llvm-3.7/lib/libLTO.dylib:
	/opt/local/libexec/llvm-3.7/lib/libLTO.dylib (compatibility version 0.0.0, current version 0.0.0)
	/opt/local/libexec/llvm-3.7/lib/libLLVM-3.7.dylib (compatibility version 0.0.0, current version 0.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1237.0.0)
	/opt/local/lib/libffi.6.dylib (compatibility version 7.0.0, current version 7.4.0)
	/opt/local/lib/libedit.0.dylib (compatibility version 1.0.0, current version 1.54.0)
	/opt/local/lib/libncurses.6.dylib (compatibility version 6.0.0, current version 6.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
llvm-3.8/lib/libLTO.dylib:
	@rpath/libLTO.dylib (compatibility version 1.0.0, current version 3.8.1)
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 3.8.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)
llvm-3.9/lib/libLTO.dylib:
	@rpath/libLTO.dylib (compatibility version 1.0.0, current version 3.9.0)
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 3.9.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

It would've been nice to have known about this before I revbumped all the llvm ports... sigh...

Basically two options:

1. Change the compat version to 0.0.0 in libLTO in llvm-3.8+
2. Change the compat and current versions in libLTO in llvm-3.7-

[Ionic (Mihai Moldovan)]

I think number 2 is cleaner.

To be fair, the old 0.0.0/0.0.0 behavior looks more like a bug than a carefully crafted compatibility-ensurance thing. It bites back for us, though.

But should ld use an llvm-specific libLTO in the first place? Shouldn't the "MacPorts system version" link against the libLTO version as selected by the llvm* variant? We seem to replace the llvm-provided linker binaries with a symlink to the ld64-provided symlink in the first place to ensure that "our" system linker is used. Pulling in different libLTO binaries dependent upon which clang version is being used sounds... wrong?

[jeremyhu (Jeremy Huddleston Sequoia)]

No, this is the correct approach.

${prefix}/bin/ld does use the one provided by the llvm variant.

${prefix}/libexec/ld64/ld uses the one provided by the toolchain it is being used with.

[jeremyhu (Jeremy Huddleston Sequoia)]

And yes, the unset compat version (resulting in 0.0.0) was a bug in older llvm. The issue here wasn't so much that it was consciously fixed so much as it was fixed by nature of the change to cmake as the build system. I could see an argument being made for setting the compat version to 0 for compatibility with the older autoconf-built versions, but the flip side is that they were certainly wrong and should be fixed as well.

[jeremyhu (Jeremy Huddleston Sequoia)]

So yeah, I think I lean towards patching all the older llvm versions to use '-compatibility_version 1.0 -current_version 3.y.z' when linking libLTO.dylib and revbumping ld64 to force a relink.

[Ionic (Mihai Moldovan)]

If that's intended and correct, than okay, one issue less. :)

Hmm... number 1 would mean less rebuilds, but we don't want to patch that around forever neither.

[Ionic (Mihai Moldovan)]

Yep, I totally agree with that.

[jeremyhu (Jeremy Huddleston Sequoia)]

Testing out a solution now...

[Ionic (Mihai Moldovan)]

trunk/dports/lang/llvm-3.3/Portfile

@@ -379 +383 @@
 }
 
-if {${os.platform} eq "darwin" && ${os.major} > 15} {
+if {${os.platform} eq "darwin" && ${os.major} > 25} {
     depends_lib
     depends_run

You should probably revert this change again?

[jeremyhu (Jeremy Huddleston Sequoia)]

Lol, yeah. I did that to test on Sierra and forgot to revert back, thanks.

[jeremyhu (Jeremy Huddleston Sequoia)]

r150761

r150776