ld64-latest upgrade time?

[RJVB (René Bertin)]

Reading up on LLVM's lld vs. system linkers (after noticing that it's a *lot* faster) I saw that Apple's own linker has had a number of recent upgrades that improve its performance,

Port:ld64-latest hasn't been upgraded though, and is now somewhat inaptly named as upstream seem to be at v690 (if I haven't overlooked newer versions).

Is there a reason why we should stay at 450.3 ?

Bonus question: how about a ld64-zld subport - see ​https://github.com/michaeleisel/zld ?

[kencu (Ken)]

upgrading ld64 is always a bit of an adventure, like upgrading cctools ... there are going to be lots of workarounds needed to get the older systems to build it.

I guess the simple reason it hasn't been updated is that the current ld64-450 can link pretty much anything we currently make.

And we still haven't found a way to bootstrap to using even that (ld64-450) as the default llinker on older systems, so the actual linker being used is usually much older.

[kencu (Ken)]

zld isn't being actively maintained any more.

IMHO adding more variables into the MacPort's older systems build toolchain is just asking for trouble, but here I defer to others.

[RJVB (René Bertin)]

Replying to kencu:

zld isn't being actively maintained any more.

No, but so is ld64 450.3 and in both cases it's because there's a newer version/alternative of the tool ;)

In case of zld that is ld64.lld, but without explicit indication of which lld version is on feature parity with ld64.

Now, if that new enough lld version can be built on a good selection of supported OS versions that would give another option to provide a more performant linker on the systems where it's known to build (and pass the tests?). But in that case it'd probably also be a good idea to ship lld in its own subport because the latest clang compilers are becoming too unwieldy even for my 16Gb i7(2nd gen) system. IOW, yes to using lld-mp-16 as far as I'm concerned, but I'll even avoid clang-mp-12 if I can... Were it not for that I'd just configure the port:clang in question with -DCLANG_DEFAULT_LINKER=lld. (FWIW, I haven't tried with lld but lldb can be built "standalone" and thus a lot faster with -DLLVM_CONFIG=${sub_prefix}/bin/llvm-config -DLLVM_DIR=${sub_prefix}/lib/cmake/llvm.)

[kencu (Ken)]

Replying to RJVB:

Replying to kencu:

zld isn't being actively maintained any more.

No, but so is ld64 450.3 and in both cases it's because there's a newer version/alternative of the tool ;)

Yeah -- but adding another out-of-date never to be updated dinosaur isn't going to update ld64 ;>

zld says to use the lld in the LLVM tree instead -- we can look at doing that (IMHO, I am not in charge of MacPorts, YMMV, etc, etc ).

[RJVB (René Bertin)]

Given how recently zld was archived it is more than likely (a lot) newer than 450.3 but you're not wrong.

I think no one person is "in charge of MacPorts", and there are 2 people who're "in charge" of the ld64 port who also used to take care of the llvm ports ;)

I'm currently checking out if there's any build time gain to be had by rolling the llvm port into the clang port but once I've satisfied or lost my curiosity at that point I can definitely have a look at` how easily one can put lld in its own subport.

[RJVB (René Bertin)]

So, I've been tinkering with lld and subsequently going down a rabbit hole with LLVM-17... Here's a rough overview about using lld before the details escape me completely...

• lld won't generate 32bit code, at least not for Intel. Not the biggest problem probably, but a bummer since a (much) faster linker would be more beneficial for older systems that need to do (lots of) universal builds.
• lld isn't a perfect drop-in replacement for ld64 (see below for the list of as-yet-unsupported arguments). I haven't found any adverse effects though. In this aspect lld-17 isn't better than lld-16.
• somewhere between LLVM-12 and LLVM-15 a new -platform_version argument was introduced that apparently deprecates -macosx_version_min which is still accepted but apparently ignored at least to the extent that it doesn't provide the info expected via -platform_version. A wrapper script is thus required which must live at the place of the actual ld64.lld *hardlink*, and invoke lld as ld64.lld (so I moved the link to ${llvm_prefix}/bin/wrapped/ld64.lld).
• -fuse-ld=/path/to/l*d is deprecated (because it doesn't allow to determine the linker variant ... easily); there is a new argument to specify /path/to/linker (which is incompatible with gcc). With the deprecated form the linker is apparently assumed to be compatible with ld64 (which is true via the aforementioned wrapper).
• lld can be built as a standalone project, so against the already installed port:llvm-xy . I had hoped to build it as a monolithic standalone *port* with a minimal build of the required llvm bits linked statically. Sadly it does not appear to be possible to build libLLVM as a static library, so ultimately I opted to write a subport that installs to its own subprefix ${prefix}/libexec/lld-17, with only the libLLVM dylib (and currently libLTO which is probably NOT needed).
• GCC's -fuse-ld=lld appears not to have any effect on Darwin and if you force it an error is raised:

  ld64.lld: error: No LC_DYLD_INFO_ONLY or LC_DYLD_EXPORTS_TRIE found in /opt/local/bin/../lib/gcc13/gcc/x86_64-apple-darwin13/13.2.0/../../../libgcc_s.1.1.dylib
  

Missing:

ld64.lld: warning: Option `-reexported_symbols_list' is not yet implemented. Stay tuned...
ld64.lld: warning: Option `-force_symbols_not_weak_list' is undocumented. Should lld implement it?
ld64.lld: warning: Option `-force_symbols_weak_list' is undocumented. Should lld implement it?
ld64.lld: warning: Option `-reexported_symbols_list' is not yet implemented. Stay tuned...
ld64.lld: warning: Option `-no_compact_unwind' is undocumented. Should lld implement it?

"Standalone build": My goal was to have access to the most recent/capable lld without necessarily having to have at least the full corresponding llvm port active. Given what a resource hog LLVM has become there is a good reason not to waste possibly precious disk space to versions that you're not going to be using regularly. Plus, clang++-17 has proven to fail to link a lot of code (including port:llvm-17 itself) because of symbols like (IIRC) std::verbose_abort missing from libc++ (and I have libc++ 13 in my personal $prefix!).

Here's my current implementation: ​https://github.com/RJVB/macstrop/blob/master/lang/llvm-17/Portfile#L415 configured to install to $prefix/libexec/lld-17. Note that I fixed a number of other "issues" with the port, in particular

• re-allow building with the MacPorts CMAKE_BUILD_TYPE (the build system no longer rejects unknown types)
• use Ninja (or in my case, the lighter Samurai). All I needed to do was figure out the install targets that the cumbersome way of invoking make install from subdirectories evidently also use. FWIW, CMake makes this relatively easy because it generates a help target that will list all known targets (in the current dir, which means for the entire project when using Ninja).
• fix the use of ccache (a complete rebuild after a clean or changing only a single C++ file takes about 35min instead of over 6 hours on my machine).