clang-11: targeted OS version does not support -reexported_symbols_list on 10.6

[mojca (Mojca Miklavec)]

When trying to install clang-11 on 10.6 I'm getting a build failure saying

ld: targeted OS version does not support -reexported_symbols_list

[ 25%] Linking CXX shared library ../../../lib/libc++.dylib
cd /path/to/clang-11/work/build/projects/libcxx/src && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/cxx_shared.dir/link.txt --verbose=ON
/opt/local/bin/clang++-mp-9.0 -pipe -Os  -isysroot/ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers  -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -arch x86_64 -mmacosx-version-min=10.6 -dynamiclib -Wl,-headerpad_max_install_names -L/opt/local/lib -Wl,-headerpad_max_install_names /opt/local/lib/libMacportsLegacySupport.a -Wl,-syslibroot,/ -compatibility_version 1.0.0 -current_version 1.0.0 -o ../../../lib/libc++.1.0.dylib -install_name @rpath/libc++.1.dylib CMakeFiles/cxx_shared.dir/algorithm.cpp.o CMakeFiles/cxx_shared.dir/any.cpp.o CMakeFiles/cxx_shared.dir/atomic.cpp.o CMakeFiles/cxx_shared.dir/barrier.cpp.o CMakeFiles/cxx_shared.dir/bind.cpp.o CMakeFiles/cxx_shared.dir/charconv.cpp.o CMakeFiles/cxx_shared.dir/chrono.cpp.o CMakeFiles/cxx_shared.dir/condition_variable.cpp.o CMakeFiles/cxx_shared.dir/condition_variable_destructor.cpp.o CMakeFiles/cxx_shared.dir/debug.cpp.o CMakeFiles/cxx_shared.dir/exception.cpp.o CMakeFiles/cxx_shared.dir/functional.cpp.o CMakeFiles/cxx_shared.dir/future.cpp.o CMakeFiles/cxx_shared.dir/hash.cpp.o CMakeFiles/cxx_shared.dir/ios.cpp.o CMakeFiles/cxx_shared.dir/iostream.cpp.o CMakeFiles/cxx_shared.dir/locale.cpp.o CMakeFiles/cxx_shared.dir/memory.cpp.o CMakeFiles/cxx_shared.dir/mutex.cpp.o CMakeFiles/cxx_shared.dir/mutex_destructor.cpp.o CMakeFiles/cxx_shared.dir/new.cpp.o CMakeFiles/cxx_shared.dir/optional.cpp.o CMakeFiles/cxx_shared.dir/random.cpp.o CMakeFiles/cxx_shared.dir/random_shuffle.cpp.o CMakeFiles/cxx_shared.dir/regex.cpp.o CMakeFiles/cxx_shared.dir/shared_mutex.cpp.o CMakeFiles/cxx_shared.dir/stdexcept.cpp.o CMakeFiles/cxx_shared.dir/string.cpp.o CMakeFiles/cxx_shared.dir/strstream.cpp.o CMakeFiles/cxx_shared.dir/system_error.cpp.o CMakeFiles/cxx_shared.dir/thread.cpp.o CMakeFiles/cxx_shared.dir/typeinfo.cpp.o CMakeFiles/cxx_shared.dir/utility.cpp.o CMakeFiles/cxx_shared.dir/valarray.cpp.o CMakeFiles/cxx_shared.dir/variant.cpp.o CMakeFiles/cxx_shared.dir/vector.cpp.o CMakeFiles/cxx_shared.dir/filesystem/operations.cpp.o CMakeFiles/cxx_shared.dir/filesystem/directory_iterator.cpp.o CMakeFiles/cxx_shared.dir/filesystem/int128_builtins.cpp.o  -Wl,-rpath,@loader_path/../lib -nodefaultlibs -lSystem -lc ../../../lib/libc++abi.1.0.dylib -Wl,-unexported_symbols_list,/path/to/clang-11/work/llvm-project/llvm/projects/libcxx/src/../lib/libc++unexp.exp -Wl,-reexported_symbols_list,/path/to/clang-11/work/llvm-project/llvm/projects/libcxx/src/../lib/libc++abi.v2.exp -Wl,-force_symbols_not_weak_list,/path/to/clang-11/work/llvm-project/llvm/projects/libcxx/src/../lib/notweak.exp -Wl,-force_symbols_weak_list,/path/to/clang-11/work/llvm-project/llvm/projects/libcxx/src/../lib/weak.exp -Wl,-reexported_symbols_list,/path/to/clang-11/work/llvm-project/llvm/projects/libcxx/src/../lib/libc++abi-exceptions.exp
ld: targeted OS version does not support -reexported_symbols_list
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[kencu (Ken)]

Hi Mojca!

clang/flang/llvm-11 all build on 10.6.8 (not lldb-11), but I admit I haven't tried doing it from a "buildbot cold" start recently.

My day-to-day SnowLeopard system is enhanced in two manual ways.

I documented the process here <​https://trac.macports.org/wiki/LibcxxOnOlderSystems#SnowLeopard> and I recommend it to all SnowLeopard users.

I use ld64-latest on it:

$ port -v installed | grep ld64
  ld64 @3_3+universal-ld64_127-ld64_236-ld64_97 (active) platform='darwin 10' archs='i386 x86_64' date='2020-05-01T12:11:16-0700'
  ld64-127 @127.2_14+llvm34+universal (active) platform='darwin 10' archs='i386 x86_64' date='2020-01-27T18:52:07-0800'
  ld64-latest @450.3_0+llvm90+universal-llvm34 (active) platform='darwin 10' archs='i386 x86_64' date='2020-01-28T20:08:53-0800'

$ which ld
/opt/local/bin/ld

$ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-450.3
BUILD 20:06:30 Jan 28 2020
configured to support archs: i386 x86_64 x86_64h armv6 armv7 armv7s armv7m armv7k arm64
LTO support using: LLVM version 9.0.1 (static support for 24, runtime is 24)
TAPI support using: Apple TAPI version 2.0.0

I use the upgraded libcxx with the thread_local_storage functionality added:

$ port -v installed libcxx
The following ports are currently installed:
  libcxx @5.0.1_4+emulated_tls+universal (active) platform='darwin 10' archs='i386 x86_64' date='2020-06-02T09:56:15-0700'

The steps to do this manually I'm sure you are very familiar with, but they are documented in the link above.

[kencu (Ken)]

We have, for the past two or three years, bandied about various ways we might automate this process on the buildbot, but so far I have not, in the end, settled on one to the point of actually implementing it.

[kencu (Ken)]

Hmmm.... it looks like this is not actually working, even with the newer ld64 version.

No doubt this is happening because I enabled building the libc++ and libc++abi shared libraries on the last update to clang-11, and now 10.6.8 doesn't like that.

I guess I will have to either fix that somehow, or turn that off on 10.6.8.

[kencu (Ken)]

It's coming from ld64:

<​https://opensource.apple.com/source/ld64/ld64-274.2/src/ld/Options.cpp.auto.html>

	// -reexported_symbols_list can only be used with -dynamiclib
	if ( !fReExportSymbols.empty() ) {
		if ( fOutputKind != Options::kDynamicLibrary )
			throw "-reexported_symbols_list can only used used when created dynamic libraries";
		if ( !minOS(ld::mac10_7, ld::iOS_4_2) )
			throw "targeted OS version does not support -reexported_symbols_list";
	}

[kencu (Ken)]

In our libcxx Port, the buildit script sets slightly different linker options for 10.7+ vs 10.6 and less:

<​https://github.com/macports/macports-ports/blob/edaa2043d3962512d250c12fdd1559b35341e382/lang/libcxx/files/1001-Revert-Remove-the-buildit-and-testit-scripts-they-ha.patch#L78>

we'll have to duplicate that logic if we want to build libc++ on < 10.7 using the llvm's cmake logic instead of the buildit script.

[kencu (Ken)]

Relevant:

​https://reviews.llvm.org/D54595

[kencu (Ken)]

fixed:

h$ port -v installed clang-11
The following ports are currently installed:
  clang-11 @11.0.1_0+analyzer+defaultlibcxx+emulated_tls+libstdcxx (active) platform='darwin 10' archs='x86_64' date='2021-02-20T18:04:58-0800'

I'm just going to run some tests for a few days before I push the fix.

[mojca (Mojca Miklavec)]

Can you please share the patches?

[kencu (Ken)]

I'd like to get it right before I push a final fix, but for today this will turn off the new libc++.dylib feature and get you building again:

% diff -u Portfile.orig Portfile
--- Portfile.orig	2021-02-21 10:37:55.000000000 -0800
+++ Portfile	2021-02-21 10:41:11.000000000 -0800
@@ -225,8 +225,8 @@
         -DCLANG_ENABLE_ARCMT=OFF \
         -DDARWIN_PREFER_PUBLIC_SDK=ON \
         -DLLVM_BUILD_RUNTIME=ON \
-        -DLIBCXX_ENABLE_SHARED=ON \
-        -DLIBCXX_INSTALL_LIBRARY=ON
+        -DLIBCXX_ENABLE_SHARED=OFF \
+        -DLIBCXX_INSTALL_LIBRARY=OFF
 
 } elseif {${subport} eq "flang-${llvm_version}"} {
 
@@ -389,7 +389,6 @@
         system "cd ${destroot.dir}/tools/clang && ${destroot.cmd} ${destroot.pre_args} ${destroot.target} ${destroot.post_args}"
         system "cd ${destroot.dir}/projects/compiler-rt && ${destroot.cmd} ${destroot.pre_args} ${destroot.target} ${destroot.post_args}"
         system "cd ${destroot.dir}/projects/libcxx && ${destroot.cmd} ${destroot.pre_args} ${destroot.target} ${destroot.post_args}"
-        system "cd ${destroot.dir}/projects/libcxxabi && ${destroot.cmd} ${destroot.pre_args} ${destroot.target} ${destroot.post_args}"
 
         delete ${destroot}${sub_prefix}/bin/clang
         file rename ${destroot}${sub_prefix}/bin/clang-${clang_executable_version} ${destroot}${sub_prefix}/bin/clang

The (working but needs tweaking) patch for libcxx I'm tweaking looks like this, but I think the not_weak and weak symbol exports still need to be included in the final product, so needs a bit of tweaking still. This patch is formatted for including in lilbcxx directly, and would need path tweaking for our clang-11 port's patch directory setup.

--- src/CMakeLists.txt.orig	2021-02-20 13:49:46.000000000 -0800
+++ src/CMakeLists.txt	2021-02-20 13:56:36.000000000 -0800
@@ -209,21 +209,7 @@
   if (LIBCXX_OSX_REEXPORT_LIBCXXABI_SYMBOLS)
     target_link_libraries(cxx_shared PRIVATE
       "-Wl,-unexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++unexp.exp"
-      "-Wl,-reexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++abi.v${LIBCXX_LIBCPPABI_VERSION}.exp"
-      "-Wl,-force_symbols_not_weak_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/notweak.exp"
-      "-Wl,-force_symbols_weak_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/weak.exp")
-
-    if (LIBCXX_ENABLE_EXCEPTIONS)
-      if ("${CMAKE_OSX_ARCHITECTURES}" MATCHES "^(armv6|armv7|armv7s)$")
-        target_link_libraries(cxx_shared PRIVATE "-Wl,-reexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++abi-exceptions.sjlj.exp")
-      else()
-        target_link_libraries(cxx_shared PRIVATE "-Wl,-reexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++abi-exceptions.exp")
-      endif()
-    endif()
-
-    if (NOT LIBCXX_ENABLE_NEW_DELETE_DEFINITIONS)
-      target_link_libraries(cxx_shared PRIVATE "-Wl,-reexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++abi-new-delete.exp")
-    endif()
+      "-Wl,-reexport_library,${CMAKE_OSX_SYSROOT}/usr/lib/libc++abi.dylib")
   endif()
 
   # Generate a linker script in place of a libc++.so symlink.

[kencu (Ken)]

I used the word "tweaking" four times in two sentences. See what watching "Who's Got Mail" does to you :>

[kencu (Ken)]

In 177a893176b291040f9db7e0b730fba2647abd79/macports-ports (master):

clang-11: fix build of shared libc++.dylib on < 10.7

reexported_symbols_list is not accepted on < 10.7
but reexport_library is, and that is what was used previously

closes: #62329

[mascguy (Christopher Nielsen)]

Replying to kencu:

In 177a893176b291040f9db7e0b730fba2647abd79/macports-ports (master):

clang-11: fix build of shared libc++.dylib on < 10.7

reexported_symbols_list is not accepted on < 10.7
but reexport_library is, and that is what was used previously

closes: #62329

Ken, with this fix, should clang-11 build successfully on our 10.6 buildbots?

[kencu (Ken)]

No -- to make the buildbot built it, we have to either:

• turn off the part of clang that wants thread_local_storage (easy, but I have just been avoiding doing this) OR
• make the buildbot finally install libc++ with thread_local_storage enabled

I was hoping to do the latter, but so far the only way I can get that to happen on the buildbot is to just install a pre-built libc++ binary with thread_local_storage enabled.

I think that is fine to do, but it has been (so far) disallowed.

All the other tricky machinations that would be needed to make the system bootstrap to that thread_local_storage enabled libc++.dylib are just frankly too much of a PITA to actually consider doing.

So we linger.

My clang-11 (and 12, and devel, and ...) all built just fine on 10.6.8, as do all the 10.6.8 users' clangs, as (pretty much) every 10.6.8 user has done the 30 second procedure to upgrade their systems to the libc++ that has thread_local_storage enabled (it's a one-liner).

BUT -- the buildbot is unable to do this, so the buildbot is uniquely broken.