libc++: using a newer libc++ to build software on older macos systems

[kencu (Ken)]

As libc++ evolves, software uses new libc++ features.

Older systems have a system libc++ version in /usr/lib/libc++.dylib and a matching /usr/lib/libc++abi.dylib there as well. These versions cease being updated by Apple at a certain point in the lifespan of the OS version.

By default, the clang linker driver will use this libc++, assuming -stdlib=libc++ has been set (it is set by default by the linker driver if no other stdlib option is specified).

The current libcxx port has the ability to build a new libc++ (presently corresponding to llvm-5.0). On SnowLeopard and earlier (Leopard, TIger), this libc++.dylib is used, and in so doing, provides those OS versions with a quite new libc++.dylib.

On newer systems, there is a variant available for libcxx:

$ port variants libcxx
libcxx has the variants:
   replacemnt_libcxx: EXPERTS ONLY: Build a replacement libcxxabi and libcxx even if it is already part of the base OS.

this builds a libc++/libc++abi that could use to replace the libc++ in /usr/lib/libc++*.dylib to get newer libc++ features.

I have done this, on Lion, and it does work, although it is not particularly well tested to be sure.

More reasonable would be a parallel libc++ that could be used if needed (much like we have a libstdc++ installed with newer versions of libgcc that can be used if needed). This is undoubtedly more palatable than replacing the system libc++.dylib to the vast majority of users.

To facilitate this, a newer libc++ is now installed with the clang-11 port, here by default:

$ port contents clang-11 | grep libc++
  /opt/local/libexec/llvm-11/lib/libc++.1.0.dylib
  /opt/local/libexec/llvm-11/lib/libc++.1.dylib
  /opt/local/libexec/llvm-11/lib/libc++.a
  /opt/local/libexec/llvm-11/lib/libc++.dylib
  /opt/local/libexec/llvm-11/lib/libc++abi.1.0.dylib
  /opt/local/libexec/llvm-11/lib/libc++abi.1.dylib
  /opt/local/libexec/llvm-11/lib/libc++abi.a
  /opt/local/libexec/llvm-11/lib/libc++abi.dylib
  /opt/local/libexec/llvm-11/lib/libc++experimental.a

By default, by the llvm build system, these dylibs have @rpath linkages:

$ otool -L /opt/local/libexec/llvm-11/lib/libc++.dylib
/opt/local/libexec/llvm-11/lib/libc++.dylib:
	@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	@rpath/libc++abi.1.dylib (compatibility version 1.0.0, current version 1.0.0)

(This type of linkage could easily be changed to a full path linkage with one or two lines of code in the portfile, should it prove beneficial to do so.)

This ticket will explore how we might use this newer libc++.dylib in ports that require newer libc++ features on older systems.

[kencu (Ken)]

filesystem test

[kencu (Ken)]

For a start, we'll try to build the attached cpp file that uses the new std::filesystem features.

We'll do this on Mojave, which does not have these features in the OS-supplied libc++.dylib.

Using the os-supplied Xcode clang, we don't get far:

$ /usr/bin/clang++ fstest.cpp
fstest.cpp:9:21: error: expected namespace name
namespace fs = std::filesystem;
               ~~~~~^
fstest.cpp:12:28: error: use of undeclared identifier 'fs'
void DisplayPathInfo(const fs::path& pathToShow)
                           ^
fstest.cpp:16:13: warning: 'auto' type specifier is a C++11 extension [-Wc++11-extensions]
        for (const auto& part : pathToShow)
                   ^
... and more

of course, we need c++17 for this to work so we add that, but that gives us different errors:

$ /usr/bin/clang++ -std=c++17 fstest.cpp
fstest.cpp:12:32: error: 'path' is unavailable: introduced in macOS 10.15
void DisplayPathInfo(const fs::path& pathToShow)
                               ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/filesystem:739:24: note: 'path' has been
      explicitly marked unavailable here
class _LIBCPP_TYPE_VIS path {
                       ^
fstest.cpp:15:39: error: 'operator<<' is unavailable: introduced in macOS 10.15
        cout << "Displaying path info for: " << pathToShow << "\n";
                                             ^
... and more

Apple has marked these std::filesystem features as not being available in the system libc++.dylib by blocking them in the clang c++ headers, with this file:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config

This can be overridden by passing a specific define:

// Decide whether to use availability macros.
#if !defined(_LIBCPP_BUILDING_LIBRARY) &&                                      \
    !defined(_LIBCXXABI_BUILDING_LIBRARY) &&                                   \
    !defined(_LIBCPP_DISABLE_AVAILABILITY) &&                                  \
    __has_feature(attribute_availability_with_strict) &&                       \
    __has_feature(attribute_availability_in_templates) &&                      \
    __has_extension(pragma_clang_attribute_external_declaration)
#  ifdef __APPLE__
#    define _LIBCPP_USE_AVAILABILITY_APPLE
#  endif
#endif

and so we add that define to disable the availability tests, and then we actually see that the link fails because the symbols are not in /usr/lib/libc++.dylib

$ /usr/bin/clang++ -std=c++17 -D_LIBCPP_DISABLE_AVAILABILITY fstest.cpp
Undefined symbols for architecture x86_64:
  "std::__1::__fs::filesystem::path::__filename() const", referenced from:
      std::__1::__fs::filesystem::path::filename() const in fstest-049dac.o
      std::__1::__fs::filesystem::path::has_filename() const in fstest-049dac.o
.. and more ...

[kencu (Ken)]

But, this does work:

$ /usr/bin/clang++ -std=c++17 -D_LIBCPP_DISABLE_AVAILABILITY /opt/local/libexec/llvm-11/lib/libc++.a /opt/local/libexec/llvm-11/lib/libc++abi.a fstest.cpp

and gives us:

$ ls -la a.out
-rwxr-xr-x  1 kencu  staff  838712 10 Mar 18:06 a.out

$ ./a.out
Displaying path info for: "/Users/kencu"
path part: 0 = "/"
path part: 1 = "Users"
path part: 2 = "kencu"
exists() = 1
root_name() = ""
root_path() = "/"
relative_path() = "Users/kencu"
parent_path() = "/Users"
filename() = "kencu"
stem() = "kencu"
extension() = ""
canonical() = "/Users/kencu"
path concat/append:
"C:\\temp/user/data"
"C:\\temp\\userdata"

[kencu (Ken)]

So -- at least in this minimal proof-of-concept, it is possible to build software that uses new libc++ features on older systems (Mojave in this case), without replacing the system libc++.dylib, and this software can work.

[kencu (Ken)]

Next will be to sort out how to do this using the dylibs instead of static linking libc++, and then to see if we can build entire ports against this newer libc++ and have them actually work in the wild.

See <​https://libcxx.llvm.org/docs/UsingLibcxx.html#getting-started> for more.

The CPP filessystem example used above comes from here <​https://www.bfilipek.com/2017/08/cpp17-details-filesystem.html>

[kencu (Ken)]

There is an important topic in this arena called "ODR violations". The One Definition Rule says that the same object can't be defined differently in different places.

It is not unlikely that we might find out that using two different versions of libc++ could run into this rule -- something similar has happened in the libgcc world on the systems that default to that. There are workarounds for this that might, or might not, be found to be effective and/or palatable.

[kencu (Ken)]

It might not be too hard to patch clang to allow another new stdlib option, something like -stdlib=macports-libcxx say.

Then we could use that to force linking to a new libc++.dylib that we put in some consistent location instead of the one in /usr/lib/libc++.dylib.

We already patch it for macports-libstdc++, and the mechanism is pretty similar.

[kencu (Ken)]

to use dylib linking using the current @rpath setting, you do it like this:

/usr/bin/clang++ -std=c++17 -D_LIBCPP_DISABLE_AVAILABILITY -L/opt/local/libexec/llvm-11/lib/ -Wl,-rpath,/opt/local/libexec/llvm-11/lib/ fstest.cpp

Then the binary comes with this:

$ ls -la a.out
-rwxr-xr-x  1 kencu  staff  55372 11 Mar 18:30 a.out

Linkages are set properly:

$ otool -L a.out
a.out:
	@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)

The @rpath search path is set correctly:

Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libc++.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1.0.0
compatibility version 1.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1252.250.1
compatibility version 1.0.0
Load command 14
          cmd LC_RPATH
      cmdsize 48
         path /opt/local/libexec/llvm-11/lib/ (offset 12)

and the binary works:

$ ./a.out
Displaying path info for: "/Users/kencu"
path part: 0 = "/"
path part: 1 = "Users"
path part: 2 = "kencu"
exists() = 1
root_name() = ""
root_path() = "/"
relative_path() = "Users/kencu"
parent_path() = "/Users"
filename() = "kencu"
stem() = "kencu"
extension() = ""
canonical() = "/Users/kencu"
path concat/append:
"C:\\temp/user/data"
"C:\\temp\\userdata"

[kencu (Ken)]

So the next step would appear to be to try building some (hopefully simpler) ports, and try adding flags like this:

configure.cppflags-append -D_LIBCPP_DISABLE_AVAILABILITY 
configure.ldflags-append -L/opt/local/libexec/llvm-11/lib/ -Wl,-rpath,/opt/local/libexec/llvm-11/lib/

as of some system vintage, we might find we need to also use some newer clang such as clang-11 as the build compiler to get headers that have all we need in them. I think clang-9.0 was the first one that added these features to the main headers instead of the the experimental headers. I'm not sure when Xcode started allowing them.

[kencu (Ken)]

Don't get too sidetracked by the @rpath thing. Using @rpath is not a standard MacPorts linkage plan. We can change to a full path spec, and install the new libc++ somewhere other than the llvm-11 tree.

But for now, training wheels --- we're just trying to get this going at all at this point.

[kencu (Ken)]

and this:

 clang++-mp-11 -std=c++17 -D_LIBCPP_DISABLE_AVAILABILITY -L/opt/local/libexec/llvm-11/lib/ -Wl,-rpath,/opt/local/libexec/llvm-11/lib/ fstest.cpp

also works on 10.6.8, I'm happy to say.

$ otool -L a.out
a.out:
	@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

$ ./a.out
Displaying path info for: "/Users/kencu/Desktop"
path part: 0 = "/"
path part: 1 = "Users"
path part: 2 = "kencu"
path part: 3 = "Desktop"
exists() = 1
...

[kencu (Ken)]

well, that's nice. works with mkvtoolnix on Mojave:

$ port -v installed mkvtoolnix
The following ports are currently installed:
  mkvtoolnix @55.0.0_0+qtgui (active) platform='darwin 18' archs='x86_64' date='2021-03-12T18:01:49-0800'

and so far runs, without extensive testing of all features.

[kencu (Ken)]

I will be creating a new port, so as not to collide with or confuse the existing libcxx port, which has it's own purposes at present and is deeply rooted in base right now. We can call this new port anything we like, eg. macports-libcxx (that has a nice ring).

It copies the c++ headers and libc++ dylibs from the clang-11 port, and gives them proper full path linkage names in the /opt/local/include and /opt/local/lib directories.

Then we use something like this in the Portfile, eg mkvtoolnix:

if {${os.platform} eq "darwin" && ${os.major} < 19} {
    PortGroup compiler_blacklist_versions 1.0

    # use a recent compiler that will understand the new features
    compiler.blacklist-append {clang < 1100} {macports-clang-3.[3-9]} {macports-clang-[5-8].0}

    # requires a newer libc++ than the system can provide
    depends_lib-append         port:macports-libcxx
    configure.cxxflags-append  -nostdinc++ -I${prefix}/include/libcxx/v1
    configure.ldflags-append   -L${prefix}/lib/libcxx
 }

and we get a binary that works, and is linked as we expect it to be linked:

$ otool -L /Applications/MacPorts/MKVToolNix.app/Contents/MacOS/MKVToolNix
/Applications/MacPorts/MKVToolNix.app/Contents/MacOS/MKVToolNix:
    /opt/local/lib/libcxx/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    ... etc ...

On reasonably recent systems, there is not likely to be much ODR trouble, as the libc++ is very similar. As the ages between the system libc++ and the one we are using diverge, probability of trouble increases, I would think.

As before, it would be possible to make a new -stdlib=macports-libc++ for clang, but that (at present) sounds like more trouble than it's worth, with all the changes to base etc that might be needed.

We also have the option of using DYLD_LIBRARY_PATH to force everything to use the new libc++ (like we do in the legacysupport 1.1 PG for libstdc++) but that has it's own chances of troubles.

[kencu (Ken)]

mkvtoolnix v55.0.0 running on 10.14 Mojave using libc++.dylib from clang-11 tree

[kencu (Ken)]

see <​https://github.com/macports/macports-ports/pull/10238>

[Wowfunhappy (Jonathan)]

I understand approximately 15% of what's going on here, so please excuse the possibly-stupid question: If we've swapped out the system's libc++ for our special macports-libc++.dylib, for a particular piece of software, why is ODR a problem? Shouldn't that software only ever use macports-libc++, and never the system's libc++? Does it really matter that other software might be using the system's libc++?

On a completely separate track, I wonder if there's some sort of macho-o linking trick (presumably one of ​http://blog.darlinghq.org/2018/07/mach-o-linking-and-loading-tricks.html) that would allow the system's libc++ to fall back to macports-libc++ if and only if it encountered an undefined symbol. I'm thinking primarily about how I was able to get Unity games working on Mavericks, by replacing and then re-exporting libSystem.B.dylib in order to add one missing function: ​https://apple.stackexchange.com/questions/414688/how-can-i-run-newer-unity-games-on-os-x-10-9-mavericks/414689.

Again, my sincere apologies if this is all very dumb. Cheers!

[kencu (Ken)]

I will start by saying I'm still learning too.

The issue that can happen is that a certain structure is defined, malloc'd, and initialized by software linked against /usr/lib/libc++.dylib, let's say a file structure.

But then over time as libc++ evolved, this structure has changed, and in the new libc++.dylib in /opt/local/lib/libcxx/libc++.dylib, that structure is now defined differently. And software linked against that new libc++ has a different idea what it looks like.

Think maybe VHS vs. Beta.

So if these two pieces of software try to interact, eg. an application using a library, one built against /usr/lib/libc++.dylib and the other built against /opt/local/lib/libcxx/libc++.dylib, then Unexpected Things Can Happen.

In real life, we knew about this but we never saw anything like this happen in the pre-10.6 libstdc++.dylib world until one day libgcc evolved to 7.5, and then it started happening quite frequently.

[kencu (Ken)]

Indeed there is a nice trick where you can reexport libraries, possibly adding functions in the process. I think we might indeed find nice ways to use this, possibly making a new libSystem.dylib that adds in most or all the symbols in libMacportsLegacySupport.dyib.

Here's another couple of links I found interesting:

<​http://mirror.informatimago.com/next/developer.apple.com/documentation/mac/runtimehtml/RTArch-52.html>

<​https://stackoverflow.com/questions/20020715/re-export-shared-library-symbols-from-other-library-os-x-posix>

[kencu (Ken)]

OK, the game is on. A whole new type of compatibility begins.

Here's hoping it all works out as well as I hope it will work out [bc823dc429bf6c3fda983b0d7b78aac4225a0da2/macports-ports].

[kencu (Ken)]

For any early users, I have not as yet added the thread_local_storage enhancements that were made in the libcxx port's libc++/libc++abi for use on system < 10.7 to macports-libcxx, so (most likely) the thread_local stuff on 10.6 will need some tweaking.

[Wowfunhappy (Jonathan)]

Replying to kencu:

I will start by saying I'm still learning too.

The issue that can happen is that a certain structure is defined, malloc'd, and initialized by software linked against /usr/lib/libc++.dylib, let's say a file structure.

But then over time as libc++ evolved, this structure has changed, and in the new libc++.dylib in /opt/local/lib/libcxx/libc++.dylib, that structure is now defined differently. And software linked against that new libc++ has a different idea what it looks like.

Think maybe VHS vs. Beta.

So if these two pieces of software try to interact, eg. an application using a library, one built against /usr/lib/libc++.dylib and the other built against /opt/local/lib/libcxx/libc++.dylib, then Unexpected Things Can Happen.

In real life, we knew about this but we never saw anything like this happen in the pre-10.6 libstdc++.dylib world until one day libgcc evolved to 7.5, and then it started happening quite frequently.

Thanks for this, I guess the problem is software is never really separate, since the libraries are linked together.

Is this something two-level name-spacing can help with, or would that be too large a change?

[kencu (Ken)]

Two-level namespace won't differentiate between the libc++ dylibs, I believe.

If we run into trouble with this issue (not clear that we will, but we might) the only thing I know of that could work is to force everything to use one libc++.dylib (the new one) by setting DYLD_LIBRARY_PATH to the new libc++.dylib.

The legacysupport 1.1 PG does that now by generating a wrapper for software linked against libstdc++.dylib that shows this problem. Check out the cmake wrappers used on Leopard or Tiger for an example.

If we run into this issue then we could do a wrapper like that for the new libc++.dylib too. To a large extent, though, it has to be low-maintenance or no-maintenace, and as transparent as possible to current devs with the current systems, otherwise it will collapse as unsustainable for the main MacPorts repo.

[kencu (Ken)]

Replying to Wowfunhappy:

Replying to kencu:

I will start by saying I'm still learning too.

Is this something two-level name-spacing can help with, or would that be too large a change?

As an example of the "I'm still learning too." I have recently come across information that suggests that two libraries with the same name but with different install names (paths) might indeed be separated by the system via two-level namespaces.

If so, that could turn out to be helpful for us here, and make the chances of confusion (crashes) a bit lower.

We still see crashes in the libstdc++.dylib world mixing dylibs, so it's still open for experimentation/validation just where it might be an issue.

[mascguy (Christopher Nielsen)]

This is a very exciting prospect. Looking forward to it being available via PortGroup, and/or enhancements to core MacPorts.

[kencu (Ken)]

Oh, it's ready to use now. It's so simple I was not planning on making a PortGroup... people need to specifically test it anyway.

See the following two examples for how to make it work:

<​https://github.com/macports/macports-ports/pull/10238>

and

<​https://github.com/macports/macports-ports/pull/10382>

Basically, for a standard c++ port, you do this:

if {${os.platform} eq "darwin" && ${os.major} < 19} {
    PortGroup compiler_blacklist_versions 1.0

    # compiler floor currently set at clang-9.0 and Xcode 11 from Mojave, as those are tested and work
    compiler.blacklist-append {clang < 1100} {macports-clang-3.[3-9]} {macports-clang-[5-8].0}

    # requires a newer libc++ than the system can provide
    depends_lib-append        port:macports-libcxx
    configure.cxxflags-append -nostdinc++ -I${prefix}/include/libcxx/v1
    configure.ldflags-append  -L${prefix}/lib/libcxx
}

I have come across a more elegant flag to set the C++ standard include path -stdlib++-isystem that is accepted on newer clangs (macports-clang-10 and probably clang-1200+), so it could be used instead, but in reality it's not much simpler so probably not worth the extra hassle and compiler restrictions.

[mascguy (Christopher Nielsen)]

Just catching up on the various details, particularly the discussion surrounding re-exporting symbols, proxy dylibs, etc.

In the interim, to test this more broadly, would it be feasible to simply tweak all MacPorts binaries to reference the replacement CXX library, rather than /usr/lib/libc++.dylib? (Via install_name_tool, or whatever.)

Obviously this would be done in a cloned throwaway VM. But is this something we would generally expect to "just work?"

[kencu (Ken)]

you should not need to use any reexporting symbols tricks with this, and no proxy libraries. Just change the clang++ includes and default libc++ with the minor flag tweaks, and have at 'er. Your mame port is probably an early candidate, perhaps.

you can think of this as being similar (identical) to the way that MacOS has /usr/lib/libstdc++.dylib for gcc to use, and then years ago we installed /opt/local/lib/libgcc/libstdc++.dylib for newer gcc versions to use instead. When built, the software is linked against the one specified, and unless you force it, it will always use that.

You can force it with DYLD_LIBRARY_PATH or install_name_tool -- I have to check exactly how DYLD_LIBRARY_PATH does or does not function on newer OS versions, as it changed at a certain point for security reasons -- (it was not supposed to work any more, but still seems to? -- but I digress).

Could we make ALL of MacPorts use the new libc++/libc++abi instead of the one in /usr/local, eg on 10.7 through 10.11? You know, we probably could do that, if we were suitably motivated to do so and worked out the bootstrapping involved to get it there. But I don't think we are near there yet, as so far, AFAIK, no software actually uses this new libc++.dylib yet (other than the several ports I have built with it as demonstrators).

If we get a number of ports using this newer libc++, then we could put in the work to make a new target, like -stdlib=macports-libc++ or something, and alter our clang builds to respect that like Marcus did with -stdlib=macports-libstdc++, and make a portgroup, complicate base further, etc etc, but we're a long way from putting in that work, IMHO.

[mascguy (Christopher Nielsen)]

Replying to kencu:

you should not need to use any reexporting symbols tricks with this, and no proxy libraries. Just change the clang++ includes and default libc++ with the minor flag tweaks, and have at 'er. Your mame port is probably an early candidate, perhaps.

Well, my concern relates to ensuring that every library a port depends on, all utilize the same libc++. Memory allocation, for example, would be at the top of the list. (If everything ultimately utilizes malloc() and free() provided by /usr/lib/libSystem.dylib, then no problem. But if library A allocates memory one way, passes a structure to library B, and the latter frees it... oye! Perhaps not a very likely scenario, but you get the idea.)

And as you mentioned in one of your earlier comments, if certain classes/structures aren't the same between the system and replacement libc++... that would also be a bit... problematic.

So the idea of mixing-and-matching libc++ implementations between libraries, scares the daylights out of me. But perhaps I'm simply getting old, and with it, risk-averse. :-)

[kencu (Ken)]

That is called an ODR violation, and it is the primary concern with this process, for sure.

But -- we cannot hope to force all of MacPorts to use a new libc++.dylib, and even if we did, the issue generally arises when working against system software built against /usr/lib/libc++.dylib anyway.

So -- that is exactly what we are waiting to see if it will become an issue, as above 62426#comment:5.

We shall see :>

[kencu (Ken)]

As an example, it was more than a decade before ODR violations showed up when mixing libstdc++.dylib from gcc, although they eventually did (Iain has figured out how to fix that for libstdc++.dylib he tells me, by the way).

[mascguy (Christopher Nielsen)]

Great to hear Ken, we're definitely on the same page.

Perhaps I'll try this out with Mame today or tomorrow. I'm not 100% certain what the minimum required MacOS version is, in terms of APIs/Frameworks used... so that will ultimately be a limiting factor. But even if it only allows us to support one or two earlier MacOS releases with the latest version, that's still a win for our users.

Regardless, I'll certainly keep you in-the-loop with my findings. Thanks for the help/info!

[mascguy (Christopher Nielsen)]

Ken, I just tried compiling Mame using this library, on MacOS 10.12. But encountered the same issue from a few months ago, whereby aligned allocation support isn't available.

Per the ticket, it sounded like you may need to patch LLVM to fix this? Or is there something else I'm missing?

​https://trac.macports.org/ticket/62049#comment:3

[kencu (Ken)]

The headers are patched, and work properly in other ports (all the other ones tried):

<​https://github.com/macports/macports-ports/blob/master/lang/macports-libcxx/files/patch-disable-availabilty.diff>

please show me exactly what you did in mame. In the end you will have to get this onto the cxx build line to force the compiler to use the new headers instead of the old headers:

configure.cxxflags-append -nostdinc++ -I${prefix}/include/libcxx/v1

and the linker has to find the new libc++.dylib first:

configure.ldflags-append  -L${prefix}/lib/libcxx

that is the essence of it. (and perhaps why someday -stdlib=macports-libc++ might prove attractive, if enough use for this comes along, as that would make it "automagical")

[mascguy (Christopher Nielsen)]

Ken, I used the recipe you provided, essentially verbatim. However, since Mame doesn't use CMake, things aren't always as straightforward as they otherwise would be.

I'm verifying that the new compiler and linker flags are truly being used everywhere during the Mame build. Stay tuned.

[kencu (Ken)]

by macports-libcxx port

[RJVB (René Bertin)]

Replying to kencu:

That is called an ODR violation, and it is the primary concern with this process, for sure.

Somehow I missed this discussion (and new port, which is a more evolved version of something I've been doing for years now; installing a newer libc++.1.dylib in $prefix/lib).

I don't think we have to worry about ODR violations as long as

• shared libraries are used
• libcxx provides the v1 ABI and we make sure to use it (there's a CMake option to select the ABI version)
• users don't activate an older version of libc++

Under those conditions, a binary that is linked against our libc++ will almost certainly load it before the system libc++, and that *should* prevent the system version from being loaded, just as when you used DYLD_INSERT_LIBRARY. Thus, binaries like system frameworks that were linked against the system version will end up using the newer version, and that should work just like with any other library. Despite its crucial role, libc++ is not different in this, there's nothing magical about it.

If this backwards ABI compatibility were not guaranteed Apple couldn't update the system libc++ without obliging users to update or rebuild all their applications.

[kencu (Ken)]

Installing macports libcxx port to replace the system /usr/lib/libc++.dylib and libc++abi.dylib on Lion causes some systems apps to crash (eg iCal), so I had to revert that on my Lion system. I didn't try it on every system.

it does happen that linking apps against the new libcxx in macports-libcxx (in ${prefix}) will cause some applications to crash if they interact with other software built against /usr/lib/libc++.dylib on many systems, eg Mojave. The crashes are sadly almost completely unpredictable.

[kencu (Ken)]

I asked (and others asked) upstream two years ago if they would consider making sure that newer versions of libc++ would be compatible with the structures from older versions of libc++.

The unabashed answer was certainly not.

[kencu (Ken)]

So -- what does this mean?

Software linked against libc++ that asks for, say, a certain structure to be instantiated into memory will get that. The accessor functions will work, on various versions of libc++.

However, it is NOT guaranteed that two different versions of libc++ will have the exact same structure in memory. They can be different, as upstream decides they might be.

So my understanding is that is what leads to the crashes when sharing these objects between different versions of libc++ at the same time. BOOM.

[RJVB (René Bertin)]

A while back ago already I found this, which I copied into my port:libcxx version as a comment:

# FUTURE WARNING:
# Starting with LLVM 8.0.0, users that wish to link together translation units built
# with different versions of libc++'s headers into the same final linked image MUST
# define the _LIBCPP_HIDE_FROM_ABI_PER_TU macro to 1 when building those translation
# units. Not defining _LIBCPP_HIDE_FROM_ABI_PER_TU to 1 and linking translation units
# built with different versions of libc++'s headers together may lead to ODR violations
# and ABI issues. On the flipside, code size improvements should be expected for
# everyone not defining the macro.

Looks like it could be a good idea to set that macro too in the __config file where you disable the availability conditional. I'll be doing that in my implementations from now on. EDIT: the proper way would be to define _LIBCPP_HIDE_FROM_ABI_PER_TU_BY_DEFAULT, which still allows users to override the setting.

Sadly I can't say I'm surprised that any project in which Apple are a big player doesn't bother with backwards compatibility... (just like I can't be bothered to memorise which fancy name goes with what OS version? ;) )

And yeah, if that's their mindset I can imagine that the likelihood of issues is going to increase when you "override" a v8.0.0+ libc++ with a newer version. You'd hope that Apple build their libraries with the above macro defined, at least.

But again, I've been injecting libc++ 8.0 to override OS 10.9.5's libc++ for a few years now, in highly complex applications like KDevelop5. Never an issue that I do not also see on Linux.

BTW, I do patch the libc++* CMake files so that the dylibs have a current_version set that corresponds to the actual LLVM version.

[kencu (Ken)]

That was a good find I thought, and I asked about that upstream. They felt that was not related to the two libc++-versions-at-once issue, although I’m not so sure, myself. That is one of the reasons I never considered pushing libcxx past 7.0 though, at least not without significant testing. My ical crashes were on 10.7 w libcxx-5 and 7, so not affected by this.

You can find these interactions in yhe libcxx-devel mailing list in 2019 and 2020.

There was a big push to modernize libcxx and lbcxxabi two or three years ago, and a lot of older system workarounds, etc, were being cut. It now only builds with a very current clang compiler, for example, only past two versions supported. I forget what the gcc promise is, something similar.

They want to be able to move ahead without too much trailing baggage to support, which is fair enough.

[RJVB (René Bertin)]

I'm not too keen on digging through libcxx-devel ML archives, curiously ;)

What version was LLVM at when they started that modernisation push? I noticed that libcxx 12 no longer builds with the same (cmake) approach of organising the source tree as works at least up to 9.0.1, but 9.0.1 is new enough to get the std::filesystem implementation.

BTW, there is no "two libc++-versions-at-once" issue. I checked multiple times. All versions that are linked to the various components do indeed show up when you set DYLD_PRINT_LIBRARIES, but if you check via the GetInfo/OpenFiles feature of the Activity Monitor they are not all actually loaded. At least not if current_version is set and the latest version does not lack symbols from older versions, I suppose.

It's really weird though. Qt have shown that you can develop very complex code while guaranteeing that binaries built against an older library version will run against the new one. Their approach isn't even trailing tons of baggage, it relies mostly on strict observation of a few relatively simple rules and careful planning. In comparison libc++ is tiny (it's actually really quite small; the cmake configure step takes several times longer than the actual build).

I think that GCC's libstdc++ guarantees that you can update the library as long as the ABI version remains constant. Of course Linux also has versioned symbols which I don't think exist on Darwin, so that could help.

[kencu (Ken)]

using macports-libcxx, you have a “two libc++-versions-at-once" issue.

If you replace the system libc++ instead, you don’t. Too bad things crashed doing that… someone else on Lion should try it…maybe they can fix it somehow.

Yeah, I’m not digging through mailing list archives any time soon either.

[kencu (Ken)]

I believe libcxx-5.0 (our current) has filesystem, and I believe it works on 10.6.8.

Let me verify.

[kencu (Ken)]

I thought our libcxx-5.0 provided filesystem on 10.6.8, but it doesn't; although filesystem is there in experimental in libcxx-5.0, and it is possible to have it move into the std namespace, we don't currently do that with the buildit script approach.

[RJVB (René Bertin)]

Yep, it was moved from experimental to in 9.0.0 .

using macports-libcxx, you have a “two libc++-versions-at-once" issue.

Either you missed what I wrote about what is actually loaded into memory (and the fact of setting the dylib current_version info), or you have proof that I don't. Do you?

FWIW, libcxx 5 also builds with cmake. I've modified my version of the port so it uses that method when possible, which makes it a lot easier to patch the build. It complexifies the Portfile but IMHO that's worth it.

[kencu (Ken)]

Replying to RJVB:

Either you missed what I wrote

I did have trouble following it, probably my fault.

You are saying if you have executable A linked against ${prefix}/lib/newlibc++.dylib and ${prefix}/lib/libMyLib.dylib, and

${prefix}/lib/libMyLib.dylib is linked against /usr/lib/libc++.dylib

then you see that ${prefix}/lib/libMyLib.dylib never brings /usr/lib/libc++.dylib into play.... ?

I just can't follow how that would be, so would have to verify that myself. Kinda doubt it, though, without proving it wrong. Goes against the laws of physics.

[RJVB (René Bertin)]

Well, not exactly; in my scenario executable A is linked against $prefix/lib/libc++.dylib (not newlibc++.dylib). If these were frameworks then dyld might indeed use the bundle ID or some similar metadata to recognise two libraries as identical (except for version), REGARDLESS of their filename.

The whole idea behind shared libraries that I know of is that the dynamic loader (dyld on Darwin) will consider each linked library but only load it if it resolves as-yet-unresolved symbols. And even then it will only load the symbols that were not yet resolved.

On Darwin with its compatibility_version and current_version metadata embedded in the dylibs the linker could also decide to prioritise multiple copies by version - this I don't know.

As far as I can see this is really no different than forcing the loader to use a different library by using DYLD_INSERT_LIBRARIES (or LD_PRELOAD on Linux). You must be familiar with libraries that override malloc() and family; that's the same principle even if this involves an add-on library. Jemalloc for instance (port:jemalloc) can be used 2 ways: either you link your executable with it, or you insert/preload/inject it.

There is one scenario where the above could go wrong. If dyld first encounters a shared library that depends on /usr/lib/libc++ (or indeed if the executable itself depends on that), the scenario painted above may be reversed, and then code built with newer libc++ headers (in some other dependency) would be executed against the older libc++ binary implementation. I *think* there are 2 protections against that:

• setting the current_version info (in which case at worst dyld will print an error and abort execution)
• setting DYLD_INSERT_LIBRARIES=$prefix/lib/libc++.dylib

That latter suggestion might also be interesting to try in known cases where having a newer libc++ causes problems.

I do realise that there would be a number of practical issues to sort out if MacPorts were ever to provide libc++.dylib (and I do mean not just libc++.1.dylib!) directly in $prefix/lib. A priori you'd want to rebuild every installed port. Or relink. This is a bit thinking ahead, but I know "base" has at least one mechanism to alter installed files (renaming conflicting files). The rev-upgrade scanner can be used to find all binaries that depend on /usr/lib/libc++.1.dylib, which can then be modified to depend on $prefix/lib/libc++.1.dylib . Advanced magic, but it should work.

A last consideration: on Linux a lot of this would be automatic, provided $prefix/lib corresponds to one of the standard (or configured/customised!) library search paths, because it (mostly) lacks the possibility to hardcode absolute library paths in binaries.

[RJVB (René Bertin)]

An example. KDevelop5 entirely rebuilt against libc++ 9.0.1 installed in /opt/local/lib, including libc++.dylib . None of KDevelop's dependencies are built against this libc++ version, and I don't have DYLD_INSERT_LIBRARIES set in the environment.

> otool -L /opt/local/lib/libc++.dylib | fgrep libc++
/opt/local/lib/libc++.dylib:
        /opt/local/lib/libc++.1.dylib (compatibility version 1.0.0, current version 9.0.1)
        /opt/local/lib/libc++abi.1.dylib (compatibility version 1.0.0, current version 9.0.1)

> otool -L /Applications/MacPorts/KF5/kdevelop.app/Contents/MacOS/kdevelop.bin  | fgrep libc++
        /opt/local/lib/libc++.1.dylib (compatibility version 1.0.0, current version 9.0.1)

> otool -L /opt/local/libexec/qt5/Library/Frameworks/QtCore.framework/QtCore | fgrep libc++
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

> env DYLD_PRINT_LIBRARIES=1 /Applications/MacPorts/KF5/kdevelop.app/Contents/MacOS/kdevelop.bin --ps |& fgrep libc++ &
dyld: loaded: /opt/local/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libc++abi.dylib
dyld: loaded: /opt/local/lib/libc++abi.1.dylib
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libc++abi.dylib
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libc++abi.dylib
<snip>

> lsof +c 0 | fgrep kdevelop | fgrep libc++
kdevelop.bin          69337 bertin  txt      REG                1,1    2223932  9538392 /opt/local/lib/libc++.1.dylib
kdevelop.bin          69337 bertin  txt      REG                1,1     617328  9538394 /opt/local/lib/libc++abi.1.dylib
kdevelop.bin          76697 bertin  txt      REG                1,1    2223932  9538392 /opt/local/lib/libc++.1.dylib
kdevelop.bin          76697 bertin  txt      REG                1,1     617328  9538394 /opt/local/lib/libc++abi.1.dylib

Here I used lsof to obtain the files open in/by KDevelop but the open files list given by the Activity Monitor shows the same result: only a single version of both libc++ components is kept opened (the newest version), despite multiple load attempts of the system versions (one for each module/library being loaded).

[RJVB (René Bertin)]

Checking an older Qt5 application:

> otool -L /Applications/MacPorts/Qt5/QtAssistant.app/Contents/MacOS/QtAssistant  | fgrep libc++
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

> env DYLD_PRINT_LIBRARIES=1 /Applications/MacPorts/Qt5/QtAssistant.app/Contents/MacOS/QtAssistant |& fgrep libc++ 
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libc++abi.dylib

> lsof +c 0 | fgrep -i QtAssistant | fgrep libc++
Exit 1

I see this same result with another "pure Qt" application?! But:

> env DYLD_PRINT_LIBRARIES=1 DYLD_INSERT_LIBRARIES=/opt/local/lib/libc++.dylib /Applications/MacPorts/Qt5/QtAssistant.app/Contents/MacOS/QtAssistant | & fgrep libc++
dyld: loaded: /opt/local/lib/libc++.dylib
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libc++abi.dylib
dyld: loaded: /opt/local/lib/libc++abi.1.dylib

> lsof +c 0 | fgrep -i QtAssistant | fgrep libc++
QtAssistant           77231 bertin  txt      REG                1,1    2223932  9538392 /opt/local/lib/libc++.1.dylib
QtAssistant           77231 bertin  txt      REG                1,1     617328  9538394 /opt/local/lib/libc++abi.1.dylib

A bit of hackery to see if I can make something go wrong:

> lsof +c 0 | fgrep kdevelop | fgrep libc++
kdevelop.bin          69337 bertin  txt      REG                1,1    2223932  9538392 /opt/local/lib/libc++.1.dylib
kdevelop.bin          69337 bertin  txt      REG                1,1     617328  9538394 /opt/local/lib/libc++abi.1.dylib
kdevelop.bin          77002 bertin  txt      REG                1,1    2223932  9538392 /opt/local/lib/libc++.1.dylib
kdevelop.bin          77002 bertin  txt      REG                1,1     617328  9538394 /opt/local/lib/libc++abi.1.dylib

env DYLD_PRINT_LIBRARIES=1 DYLD_INSERT_LIBRARIES=/usr/lib/libc++.1.dylib /Applications/MacPorts/KF5/kdevelop.app/Contents/MacOS/kdevelop.bin --ps | & fgrep libc++

dyld: loaded: /usr/lib/libc++.1.dylib dyld: loaded: /opt/local/lib/libc++.1.dylib dyld: loaded: /usr/lib/libc++abi.dylib dyld: loaded: /opt/local/lib/libc++abi.1.dylib

}}}

> sudo /opt/local/bin/install_name_tool -change /opt/local/lib/libc++.1.dylib /usr/lib/libc++.1.dylib  /Applications/MacPorts/KF5/kdevelop.app/Contents/MacOS/kdevelop.bin

> otool -L /Applications/MacPorts/KF5/kdevelop.app/Contents/MacOS/kdevelop.bin | fgrep libc++
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 9.0.1)

> lsof +c 0 | fgrep kdevelop | fgrep libc++
kdevelop.bin          69337 bertin  txt      REG                1,1    2223932  9538392 /opt/local/lib/libc++.1.dylib
kdevelop.bin          69337 bertin  txt      REG                1,1     617328  9538394 /opt/local/lib/libc++abi.1.dylib
kdevelop.bin          76908 bertin  txt      REG                1,1    2223932  9538392 /opt/local/lib/libc++.1.dylib
kdevelop.bin          76908 bertin  txt      REG                1,1     617328  9538394 /opt/local/lib/libc++abi.1.dylib

Note how install_name_tool did NOT change the current_version info stored in KDevelop!!

[kencu (Ken)]

yes, so there you go, both versions are loaded, and you’re mixing:

dyld: loaded: /opt/local/lib/libc++.1.dylib
dyld: loaded: /usr/lib/libc++.1.dylib

I am not putting myself forward as the expert, most of what I know comes from Jeremy and Iain.

When you mix, bad things can happen, but don’t always happen. The objects passed between software linked against different libc++ versions can be different, but are not always different — and even if they are different, you might not crash.

You have to test carefully. But it can work, which is why I pushed macports-libcxx a while back.

[kencu (Ken)]

is there something you are trying to tell us/me here I’m not getting?

Perhaps a one-sentence line might put my nose right on it. I’m sorry, you’re so clever about all this and I feel slow to catch on.

[kencu (Ken)]

one thing.. static linking the new libc++.a might be better in some cases. Then less chance of mixing troubles.

That is what Chrome does for MacOS, and what upstream libcxx suggested for users who wanted this to work.

​https://lists.llvm.org/pipermail/libcxx-dev/2021-July/001179.html

[RJVB (René Bertin)]

Replying to kencu:

Perhaps a one-sentence line might put my nose right on it. I’m sorry, you’re so clever about all this and I feel slow to catch on.

Look at the output from lsof above to see what libraries actually *remain* loaded into memory while the application is running, and consider that there is a distinction between loading a library into memory (dlopen'ing it) and importing symbols from it. (That was a single though long sentence ;) )

This must also be why we see a single DYLD_PRINT_LIBRARIES line for /opt/local/lib/libc++.1.dylib and /opt/local/lib/libc++abi.1.dylib : they're loaded and symbols are imported from it. Whereas I see multiple cases of /usr/lib/libc++*.dylib being loaded; this has to mean that each time the library is unloaded again because it is found to be superfluous.

I think of it this way. During the build process the linker (link editor, ld) will have built a table of symbols that have to be loaded from shared libraries, and in which library they are expected. When you launch the program, dyld will read that table, starts loading the libraries registered as dependencies (probably using dlopen()) and tries to obtain the address of each of the listed symbols from the expected library (importing; probably using dlsym()). Once an address is found, the symbol is removed from the list. Now maybe I'm wrong, maybe the application and its shared libraries aren't all read into a single, global memory space. It's true that this process works a bit differently on Linux, so I may be mixing observations from the two systems and arriving at inappropriate conclusions.

If libc++ were made of a single source file then linking with the static library would presumably be equivalent to using DYLD_INSERT_LIBRARIES. But it's not, so ld will link only the required modules needed by the application. I do not know to what extent ld will also consider the symbols required by the shared libraries on which the application depends, if those shared libraries are already linked to the shared libc++ from the system.

If it does, then linking to a static libc++ should indeed guarantee that the executable contains all the required libc++ functions and no longer needs to import anything from the shared libc++ system library.

But it looks like this is also what happens with my newer shared libc++ in /opt/local/lib: dyld can apparently see it's the newest version of the 2 libc++ versions in the list of libraries to load, and is clever enough to import from that newest version first.

This is quite a bit more than a single sentence and I hope I've not managed to make things even less clear.

[RJVB (René Bertin)]

Anyway, as suggested in an earlier post:

if you have a reproducable case where an application linked to $prefix/lib/libcxx/libc++.dylib crashes, try launching it with DYLD_INSERT_LIBRARIES=$prefix/lib/libcxx/libc++.dylib set.

[RJVB (René Bertin)]

BTW, I have indeed found that some 3rd party (= non-MacPorts) applications do not like it when the system libc++ is overriden with DYLD_INSERT_LIBRARIES; IIRC Google Chrome is one of those. They either become instable, or crash. That could be explained by the fact that GChrome links statically to their own libc++, if that was indeed already the case with the version I can still run.

[RJVB (René Bertin)]

Replying to RJVB:

BTW, I have indeed found that some 3rd party (= non-MacPorts) applications do not like it when the system libc++ is overriden with DYLD_INSERT_LIBRARIES; IIRC Google Chrome is one of those.

Belay that: it isn't true anymore. I can only conclude that this caused a problem because I was overriding/injecting a version of libc++ that was *older* than the static version those applications were linked with. And that is evidently a recipe for disaster...

[kencu (Ken)]

So for anyone still reading this who has trouble following:

MacPorts offers a newer libc++ for older systems (< 10.15) to try. It is very simply used, available using the legacysupport Portgroup.

legacysupport.use_mp_libcxx     yes

Software may not function properly when it is used, however, so please test if you can, and run the test suite if there is one.

[RJVB (René Bertin)]

I've been trying to get things to crash, without luck. At some point while checking out the benchmarks, I discovered that the reference builds which are supposed to be linked against the host/old libc++ (or libstdc++ on Linux) were actually linked to both /usr/lib/libc++.dylib and /opt/local/lib/libc++.dylib (because MacPorts sets LIBRARY_PATH). They ran (using the newer version AFAICT).

---

About those benchmarks: irrelevant here but they suggest that libcxx 9 lost any advantage it may have had over libstdc++ in terms of performance. The "algorithms" benchmark collection (of very common operations) terminates twice faster with the latter, and one of the vector benchmarks runs a whopping 10x faster.

[kencu (Ken)]

OK, I have an example for you where mixing libc++ versions causes unexpected issues, I believe. This comes from an example I was looking over on another package manager.

Take this small test.cxx program, that looks for a nonexistent file and should generate an error if it is not found:

#include "llvm/Support/Errc.h"
#include "clang/Basic/FileManager.h"

#include <iostream>

int main() {
  clang::FileManager fileMgr((clang::FileSystemOptions()));

  auto file = fileMgr.getFileRef("./nonexistant.h", /*OpenFile=*/true);

  std::error_code EC = llvm::errorToErrorCode(file.takeError());

  std::cout << "EC.message is " << EC.message() << "\n";

  bool not_no_such = (EC != llvm::errc::no_such_file_or_directory);
  bool is_no_such = !not_no_such;

  std::cout << "EC == no_such_file_or_directory is " << is_no_such << "\n";

  return 0;
}

compile it like this (I used clang-14 for this, on Monterey):

clang++-mp-14  -c test.cxx -I/opt/local/libexec/llvm-14/include -std=c++17 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

and now link it like this:

clang++-mp-14 test.o -o test1 -Wl,-rpath,/opt/local/libexec/llvm-14/lib -L/opt/local/libexec/llvm-14/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -lLLVM

check the link, and you will see it is linked against the new libc++.1.dylib from llvm14:

% otool -L test1
test1:
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 14.0.6)
	/opt/local/libexec/llvm-14/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

now run it, and you get this unexpected output:

% ls *.h
zsh: no matches found: *.h
% ./test1
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

% touch nonexistant.h
cunningh@MacBookPro-2012 ~ % ./test1
EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0

so you can see that is is not working properly.

Now, let's build it again, this time linking against /usr/lib/libc++.1.dylib. We'll just manually change it to keep things simple:

clang++-mp-14 test.o -o test2 -Wl,-rpath,/opt/local/libexec/llvm-14/lib -L/opt/local/libexec/llvm-14/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -lLLVM

install_name_tool -change /opt/local/libexec/llvm-14/lib/libc++.1.dylib /usr/lib/libc++.1.dylib test2

and the linkage is as we expect:

 % otool -L test2
test2:
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 14.0.6)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

and it works properly:

% ls *.h
zsh: no matches found: *.h
% ./test2
EC.message is No such file or directory
EC == no_such_file_or_directory is 1
% touch nonexistant.h
% ./test2
EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0

The reason this happens is because /opt/local/libexec/llvm-14/lib/libLLVM.dylib is linked against /usr/lib/libc++.dylib:

% otool -L /opt/local/libexec/llvm-14/lib/libLLVM.dylib
/opt/local/libexec/llvm-14/lib/libLLVM.dylib:
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 14.0.6)
	/opt/local/lib/libffi.8.dylib (compatibility version 10.0.0, current version 10.0.0)
	/opt/local/lib/libedit.0.dylib (compatibility version 1.0.0, current version 1.68.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
	/opt/local/lib/libncurses.6.dylib (compatibility version 6.0.0, current version 6.0.0)
	/opt/local/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.14.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

and if test is linked against a different libc++.dylib, then bad things can happen.

Just to really prove it, let's leave test1 linked against the new libc++, but we'll change /opt/local/libexec/llvm-14/lib/libLLVM.dylib to link against the new one too. If this theory is right, test1 should now work properly, and test2 will be broken:

sudo install_name_tool -change /usr/lib/libc++.1.dylib /opt/local/libexec/llvm-14/lib/libc++.1.dylib /opt/local/libexec/llvm-14/lib/libLLVM.dylib

% ls *.h
zsh: no matches found: *.h
% ./test1
EC.message is No such file or directory
EC == no_such_file_or_directory is 1
% ./test2
EC.message is No such file or directory
EC == no_such_file_or_directory is 0
% touch nonexistant.h
% ./test1            
EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0
% ./test2            
EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0

so the message here is that when test.cxx is linked against the same libc++.1.dylib that /opt/local/libexec/llvm-14/lib/libLLVM.dylib is linked against, all is well.

If they are linked against different versions of libc++.1.dylib ===> BOOM. You get garbage.

Now I have to change the linkage in libLLVM.dylib back to how it was originally, before I forget :>

% sudo install_name_tool -change  /opt/local/libexec/llvm-14/lib/libc++.1.dylib /usr/lib/libc++.1.dylib /opt/local/libexec/llvm-14/lib/libLLVM.dylib

[kencu (Ken)]

Homebrew found a list of about 20 formula where this issue was potentially causing troubles.

In the end, to prevent inadvertent linking to an unexpected libc++.1.dylib, they moved libc++.1.dylib out of the directory where all the other LLVM/clang libs are stored, and into a subdirectory, so it would not be automatically found during the link phase if -L/opt/local/libexec/llvm-14/lib was being used to find the LLVM libs.

[RJVB (René Bertin)]

I'll play around with this, but using the newest llvm versions I have installed (9 and 12). Do you prevent the availability check in these tests, and have you tried with the PER_TU macro discussed above?

Why is libc++ being built and installed with the newer clang versions? From what I understand it shouldn't be used and will not under normal circumstances provide new functionality anyway?

BTW, did you notice that you have both

/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

and

/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

(look at the current_version!) which seems a bit odd?

I realise that Apple's stubborn use of unrelated version numbers could wreak havoc on appropriate use of self-built library versioning; do we have a table mapping their version numbers to/from the stock LLVM/libc++ versions?

[kencu (Ken)]

Replying to RJVB:

have you tried with the PER_TU macro discussed above?

I just used the stock installs, and (after getting my fingers thoroughly rapped by upstream for suggesting it) have not used the PER_TU macro as yet.

Why is libc++ being built and installed with the newer clang versions? From what I understand it shouldn't be used and will not under normal circumstances provide new functionality anyway?

It was added to the default clang build to simplify creating the macports-libcxx port for one, and to allow testing by interested folks such as yourself :> You're welcome!

BTW, did you notice that you have both

/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

and

/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

(look at the current_version!) which seems a bit odd?

that seems to have happened after I used install_name_tool to switch the libc++ reference. I guess it doesn't pick up the correct current version. These tools do kind weirder things since dyld_shared_library_cache, I find, as the files don't exist at all any more.

I realise that Apple's stubborn use of unrelated version numbers could wreak havoc on appropriate use of self-built library versioning; do we have a table mapping their version numbers to/from the stock LLVM/libc++ versions?

It's not a clean mapping. At last count, Apple was carrying something like 12,000 patches for xcode on top of the LLVM tree, or some similar number. So their versions are not quite llvm's versions.

[RJVB (René Bertin)]

Replying to kencu:

It's not a clean mapping. At last count, Apple was carrying something like 12,000 patches for xcode on top of the LLVM tree, or some similar number. So their versions are not quite llvm's versions.

This could explain things, if they introduce change that cause ABI compatibility problems. But if they do that it's a miracle that we never ran into issues using the libc++ headers from clang++-mp-XY with /usr/lib/libc++.dylib! I suppose you can see why?!

Even if the mapping isn't clean we can still make it based on the LLVM version Apple apply their patches to. For instance, suppose OS Foo includes libc++ 1200.0.1 which is a patched version of LLVM 12.0.1 that mapping could say 1200.0.1 corresponds to stock 12.0.2 (which AFAIK would never exist). Then port:libcxx_macports could set the current_version of libc++ 13.0.x to 1200.13.x, or something comparable that sits between 2 Apple libc++ version numbers.

---

I've been tinkering with your example.

libcxx-test.cpp:

#include "llvm/Support/Errc.h"
#include "clang/Basic/FileManager.h"

#include <iostream>

int main(int argc, const char *argv[])
{
  clang::FileManager fileMgr((clang::FileSystemOptions()));

  for (int i = 0 ; i < argc ; ++i) { 
#if LLVM_VERSION_MAJOR > 9
    auto file = fileMgr.getFileRef(argv[i], /*OpenFile=*/true);
    std::error_code EC = llvm::errorToErrorCode(file.takeError());
#else
    auto file = fileMgr.getFile(argv[i], /*OpenFile=*/true);
    std::error_code EC = file ? llvm::errorToErrorCode(llvm::Error::success()) : llvm::errorToErrorCode(llvm::createStringError(llvm::errc::no_such_file_or_directory, "%s"));
#endif
  
    std::cout << argv[i] << " : EC.message is " << EC.message() << "\n";
  
    bool not_no_such = (EC != llvm::errc::no_such_file_or_directory);
    bool is_no_such = !not_no_such;
  
    std::cout << "EC == no_such_file_or_directory is " << is_no_such << "\n";
  } 
  
  return 0;
} 

build-libcxx-test.sh:

#!/bin/sh

CC="$1"
shift

# -I/opt/local/libexec/llvm-14/include -Wl,-rpath,/opt/local/libexec/llvm-14/lib -L/opt/local/libexec/llvm-14/lib 

${CC} -c libcxx-test.cpp -std=c++17 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS \
     -D__STDC_LIMIT_MACROS "$@"
${CC} libcxx-test.o -o libcxx-test -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lclangFrontend \
     -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit \
     -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -lLLVM "$@"

With port:clang-12 I can reproduce your symptom, and if I link explictly to /usr/lib/libc++.dylib I get correct behaviour

> build-libcxx-test.sh clang++-mp-12 -I/opt/local/libexec/llvm-12/include/ -Wl,-rpath,/opt/local/libexec/llvm-12/lib -L/opt/local/libexec/llvm-12/lib -Wl,-rpath,/opt/local/lib -L/opt/local/lib /usr/lib/libc++.dylib && libcxx-test ./nonexistant.h libcxx-test.cpp 
clang: warning: -Wl,-rpath,/opt/local/libexec/llvm-12/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: -Wl,-rpath,/opt/local/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: /usr/lib/libc++.dylib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/libexec/llvm-12/lib' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/lib' [-Wunused-command-line-argument]
libcxx-test : EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0
./nonexistant.h : EC.message is No such file or directory
EC == no_such_file_or_directory is 1
libcxx-test.cpp : EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0

This DOES NOT apply to using /opt/local/lib/libc++.dylib from my own macstrop:libcxx port, confirming your suspicions

> build-libcxx-test.sh clang++-mp-12 -I/opt/local/libexec/llvm-12/include/ -Wl,-rpath,/opt/local/libexec/llvm-12/lib -L/opt/local/libexec/llvm-12/lib -Wl,-rpath,/opt/local/lib -L/opt/local/lib /opt/local/lib/libc++.dylib && libcxx-test ./nonexistant.h libcxx-test.cpp && ldd libcxx-test
clang: warning: -Wl,-rpath,/opt/local/libexec/llvm-12/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: -Wl,-rpath,/opt/local/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: /opt/local/lib/libc++.dylib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/libexec/llvm-12/lib' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/lib' [-Wunused-command-line-argument]
libcxx-test : EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0
./nonexistant.h : EC.message is No such file or directory
EC == no_such_file_or_directory is 0
libcxx-test.cpp : EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0
libcxx-test:
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 12.0.1)
	/opt/local/lib/libc++.1.dylib (compatibility version 1.0.0, current version 9.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

I would have to build my macstrop:libcxx@12.0.1 to verify if something changed between libc++ 9 and libc++ 12 . That is not impossible of course, but then again *AFAIK* libc++ 9 is new enough to provide support for current C++ standards, on systems that ship an earlier version.

Now, the weird thing I see is when I remove /opt/local/libexec/llvm-12/lib/libc++.dylib so the linker/editor doesn't find that libc++ any more.

In that case, ld refuses to acknowledge both libc++.dylib versions that are in its path. I can add either explicitly (= with the absolute path) as in the commands shown above but without that I get missing symbol errors for everything provided by libc++ . Can you confirm this? I have no explanation for it, but it would be interesting to rebuild clang-12 WITHOUT libc++ .

I can do that, but it takes hours on my system so if that is not the case for you I'd appreciate if you could check first if that makes a difference.

[RJVB (René Bertin)]

Now, using clang-12 with the libraries from port:clang-9.0:

> build-libcxx-test.sh clang++-mp-12 -I/opt/local/libexec/llvm-9.0/include/ -Wl,-rpath,/opt/local/libexec/llvm-9.0/lib -L/opt/local/libexec/llvm-9.0/lib -Wl,-rpath,/opt/local/lib -L/opt/local/lib /opt/local/lib/libc++.dylib && libcxx-test ./nonexistant.h libcxx-test.cpp && ldd libcxx-test
clang: warning: -Wl,-rpath,/opt/local/libexec/llvm-9.0/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: -Wl,-rpath,/opt/local/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: /opt/local/lib/libc++.dylib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/libexec/llvm-9.0/lib' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/lib' [-Wunused-command-line-argument]
libcxx-test : EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0
./nonexistant.h : EC.message is No such file or directory
EC == no_such_file_or_directory is 1
libcxx-test.cpp : EC.message is Undefined error: 0
EC == no_such_file_or_directory is 0
libcxx-test:
	/opt/local/libexec/llvm-9.0/lib/libLLVM.dylib (compatibility version 1.0.0, current version 9.0.1)
	/opt/local/lib/libc++.1.dylib (compatibility version 1.0.0, current version 9.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

Note that there is also the fact that the llvm-9 builds do not use @rpath, which may make a big difference here.

[RJVB (René Bertin)]

Adding -D_LIBCPP_HIDE_FROM_ABI_PER_TU_BY_DEFAULT to the test build doesn't make a difference, but I think it stands to reason that libLLVM.dylib should also have been built with it.

BTW, here's an interesting manifestation of what could be the same problem:

> sudo install_name_tool -change {/usr,/opt/local}/lib/libc++.1.dylib /opt/local/libexec/llvm-12/lib/libLLVM.dylib ; build-libcxx-test.sh clang++-mp-12 -I/opt/local/libexec/llvm-12/include/ -Wl,-rpath,/opt/local/libexec/llvm-12/lib -L/opt/local/libexec/llvm-12/lib -v
clang version 12.0.1
Target: x86_64-apple-darwin13.4.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-12/bin
clang: warning: -Wl,-rpath,/opt/local/libexec/llvm-12/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/libexec/llvm-12/lib' [-Wunused-command-line-argument]
 (in-process)
 "/Volumes/Debian/MP9/libexec/llvm-12/bin/clang" -cc1 -triple x86_64-apple-macosx10.9.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name libcxx-test.cpp -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-rounding-math -munwind-tables -faligned-alloc-unavailable -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -target-cpu core2 -tune-cpu generic -debugger-tuning=lldb -target-linker-version 450.3 -v -resource-dir /Volumes/Debian/MP9/libexec/llvm-12/lib/clang/12.0.1 -D __STDC_CONSTANT_MACROS -D __STDC_FORMAT_MACROS -D __STDC_LIMIT_MACROS -I /opt/local/libexec/llvm-12/include/ -stdlib=libc++ -internal-isystem /opt/local/libexec/llvm-12/bin/../include/c++/v1 -internal-isystem /usr/local/include -internal-isystem /Volumes/Debian/MP9/libexec/llvm-12/lib/clang/12.0.1/include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /Volumes/Debian/Users/bertin/work/src/new -ferror-limit 19 -fmessage-length=132 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcolor-diagnostics -o libcxx-test.o -x c++ libcxx-test.cpp
clang -cc1 version 12.0.1 based upon LLVM 12.0.1 default target x86_64-apple-darwin13.4.0
#include "..." search starts here:
#include <...> search starts here:
 /opt/local/libexec/llvm-12/include
 /opt/local/libexec/llvm-12/bin/../include/c++/v1
 /usr/local/include
 /Volumes/Debian/MP9/libexec/llvm-12/lib/clang/12.0.1/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
libcxx-test.cpp:1:10: fatal error: cannot open file './llvm/Support/Errc.h': No such file or directory
#include "llvm/Support/Errc.h"
         ^
1 error generated.
clang version 12.0.1
Target: x86_64-apple-darwin13.4.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-12/bin
clang: error: no such file or directory: 'libcxx-test.o'
Exit 1

> sudo install_name_tool -change {/opt/local,/usr}/lib/libc++.1.dylib /opt/local/libexec/llvm-12/lib/libLLVM.dylib ; build-libcxx-test.sh clang++-mp-12 -I/opt/local/libexec/llvm-12/include/ -Wl,-rpath,/opt/local/libexec/llvm-12/lib -L/opt/local/libexec/llvm-12/lib 
clang: warning: -Wl,-rpath,/opt/local/libexec/llvm-12/lib: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/opt/local/libexec/llvm-12/lib' [-Wunused-command-line-argument]

[RJVB (René Bertin)]

update.

I have rebuilt llvm-12 and clang-12 twice (the former more out of conscience but with a small config tweak) so that the install corresponds to what was done up till at least LLVM 9. The libc++ and libc++abi headers are prepared, but libc++ itself isn't built and neither of these libraries gets installed (via a +no_libcxx variant).

Thus, clang++-mp-12 will built code against the v12 libc++ headers but link against whichever libc++ binary it will find.

I did this twice, in fact (clang-12 takes almost 6h to build on my system, but fortunately I used ccache):

1) against the system's libc++ (current_version 120.x.y). With this I can still reproduce what you see with clang-14.

2) against my own libc++ (current_version 9.0.1). Same conclusion though the situation is now reversed: behaviour of the test app is only correct when linked against my own libc++.

It would seem thus that code that depends on libLLVM needs to be linked against the same libc++ as libLLVM is linked, which confirms what you have been saying. The difference must be in libLLVM; remember that my builds all use the libc++ *headers* that come with the compiler and that building with clang-12 but against the libraries of llvm-9 does create the issue. So to be really exhaustive someone would need to check with clang-10 and clang-11 to know at which major version this change was introduced, in case there are older systems where that information could be useful to keep things simpler.

What annoys me is that injecting libc++ via DYLD_INSERT_LIBRARIES does not seem to have an effect on this ... and that *apparently* /usr/lib/libc++.1.dylib is being filtered out from the list of open files (blame the dyld cache??). Injecting libc++ (and libc++abi) via DYLD_INSERT_LIBRARIES *used* to have an effect; in the past it allowed me to prevent warnings from appearing in the system.log and/or on the calling terminal when running KDE4 applications. It's possible that was on OS X 10.6 though.

All this does indeed not speak for installing a full libc++ library in $prefix/lib

So is this an ODR issue? Probably, but for now it's one that appears to be limited to llvm/clang libraries. I have some applications that use libClang, I will have to check if that library is concerned too.

I've pushed the changes to my macstrop:libcxx and macstrop:llvm-12 ports so you can check what I did exactly. It is my intention to check what happens when I 1) generate a local libc++ with current_version higher than the system lib's (will DYLD_INSERT_LIBRARIES have more effect?) and 2) update my libc++ to v12.0.1 .

[kencu (Ken)]

Although this example with the llvm libraries demonstrates the issue, probably by using the newest/greatest, I believe a similar thing can happen with any software.

BTW, I'm sure you know, up until I changed it, the clang-N ports just ditto copied the new headers from libc++. I removed that when I started installing libc++ instead. So all macports-clang-N installations always use the headers that match that libc++ version:

​https://github.com/macports/macports-ports/blob/d08617c395900282d3034185baf81987a9954e59/lang/llvm-9.0/Portfile#L597

[kencu (Ken)]

For the needs of the newer libc++ versions on older systems, ie the support for filesystem and the landing pads for the exceptions in the three files that call them, eg:

​https://github.com/llvm/llvm-project/blob/e7f133191008b137aa9ef611b676ce2ec3c116a7/libcxx/src/variant.cpp#L13

IMHO the simplest thing to do is copy those needed objects into compiler_rt for that clang version.

The landing pads are trivial to do -- filesystem might take a few hours to sort out.

[RJVB (René Bertin)]

Replying to kencu:

Although this example with the llvm libraries demonstrates the issue, probably by using the newest/greatest, I believe a similar thing can happen with any software.

I think what's happening is that the failing = operation/or is being executed in the "influence domain" of the wrong libc++ (or more likely, libc++abi). I have just done some checking with KDevelop, which has a code-parser *plugin* that is based on libClang (and thus libLLMV). All LLVM operations are internal to the plugin, which of course also links to a libc++. I haven't been able to notice any off behaviour by forcing the wrong libc++ to load. Yet.

BTW, I'm sure you know, up until I changed it, the clang-N ports just ditto copied the new headers from libc++. I removed that when I started installing libc++ instead.

I saw, yes. You'll see in my clang-12 port changes that there is a proper way to achieve the same thing, which includes the generated libc++ headers (the libc++abi headers will still have to be installed by hand).

So all macports-clang-N installations always use the headers that match that libc++ version

Yes, and that could at some point become a API problem ... since LLVM do not appear to care about that.

Remember that my testing above seems to suggest that all should be fine as long as you use the same libc++ that was used for building the compiler(s). There's a bootstrap problem here, but also the question how LLVM imagine their default build should be used. Because your demo issue should bite on every platform that uses libc++ as the system C++ runtime.

EDIT: I never understood what compiler-rt stands for, apparently some kind of runtime?

[kencu (Ken)]

I don’t interpret the testing like that, at least so far.

So long as the executable and the library are using the same libc++, whatever version, all is well.

So the system libc++ is fine, if consistently used.

[kencu (Ken)]

compiler_rt is a storehouse of code to be statically linked in if needed. A kitchen-sink of stuff that the compiler expects but not all systems provide in libc or libc++.

[RJVB (René Bertin)]

Right, so putting std::filesystem in there would not be, erm, upstreamable ;)

I checked the effect of setting the current version to something definitely newer than I have on my system (900.99.1 instead of 9.0.1). It seems dyld ignores this information for libc++; building an app against that self-proclaimed newer version and then activating an older port:libcxx version does not cripple the app. There is also no impact on the effect of using DYLD_INSERT_LIBRARIES. Really weird.

Though I can imagine Apple did this on purpose. Otherwise it would probably be impossible to target earlier OS versions when building software (libc++ not being included in the platform SDK, AFAIK).