llvm-17 fails to build agains port:legacy-support=1.1.1 when the latter is build with -flto

[RJVB (René Bertin)]

Mostly a FYI:

Trying to build llvm-17 fails after upgrading my port:legacy-support from 1.0.9 to 1.1.1, still built with -flto in configure.optflags for optimal performance and clang-12 .

:info:build ninja: job failed: : && /opt/local/bin/clang++-mp-16 -Os -march=native -fvectorize -DNDEBUG -D__DISABLE_MP_LEGACY_SUPPORT_SYSCONF_WRAP__=1 -D__DISABLE_MP_LEGACY_SUPPORT_REALPATH_WRAP__=1 -isystem/opt/local/include/LegacySupport -isystem/opt/local/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -arch x86_64 -mmacosx-version-min=10.9 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L/opt/local/lib -Wl,-headerpad_max_install_names /opt/local/lib/libMacportsLegacySupport.a    -Wl,-dead_strip utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/Attributes.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/CodeGenIntrinsics.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/DirectiveEmitter.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/IntrinsicEmitter.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/RISCVTargetDefEmitter.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/SDNodeProperties.cpp.o utils/TableGen/CMakeFiles/obj.LLVMTableGenCommon.dir/VTEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-min-tblgen.dir/TableGen.cpp.o -o bin/llvm-min-tblgen  -Wl,-rpath,/opt/local/lib -Wl,-rpath,/opt/local/libexec/lld-17/lib  lib/libLLVMSupport.a  lib/libLLVMTableGen.a  lib/libLLVMSupport.a  -lm  /opt/local/lib/libz.dylib  /opt/local/lib/libzstd.dylib  /opt/local/lib/libcurses.dylib  lib/libLLVMDemangle.a && :
:info:build Undefined symbols for architecture x86_64:
:info:build   "___mpls_closedir", referenced from:
:info:build       llvm::sys::fs::detail::directory_iterator_destruct(llvm::sys::fs::detail::DirIterState&) in libLLVMSupport.a(Path.cpp.o)
:info:build   "___mpls_readdir", referenced from:
:info:build       llvm::sys::fs::detail::directory_iterator_increment(llvm::sys::fs::detail::DirIterState&) in libLLVMSupport.a(Path.cpp.o)
:info:build   "___mpls_opendir", referenced from:
:info:build       llvm::sys::fs::detail::directory_iterator_construct(llvm::sys::fs::detail::DirIterState&, llvm::StringRef, bool) in libLLVMSupport.a(Path.cpp.o)
:info:build ld: symbol(s) not found for architecture x86_64

For some reason -flto does not have this effect on the shared library (tested by replacing the .a archive above with the corresponding .dylib).

I've been able to work around the issue by adding this to the legacy-support Portfile:

# build static library without LTO
build.post_args-append \
                    SLIBCFLAGS=-fno-lto

but I suspect that a better solution should be possible (generate the static library from the same sources as the dylib, as so many projects do?).

[cjones051073 (Chris Jones)]

I am no longer active on legacy-support, please assign to the current maintainers.

[RJVB (René Bertin)]

Replying to cjones051073:

I am no longer active on legacy-support, please assign to the current maintainers.

Apologies, and tried that but not sure if it worked.

[fhgwright (Fred Wright)]

This is probably a duplicate of #68314. I'm not sure about the -flto part, but why don't you try again (and comment here) once legacy-support 1.2.1 is published (​https://github.com/macports/macports-ports/pull/23011).

[fhgwright (Fred Wright)]

Replying to fhgwright:

This is probably a duplicate of #68314. I'm not sure about the -flto part, but why don't you try again (and comment here) once legacy-support 1.2.1 is published (​https://github.com/macports/macports-ports/pull/23011).

Or if you don't want to wait, the current legacy-support-devel is the same code as the upcoming release.

[fhgwright (Fred Wright)]

Replying to fhgwright:

Replying to fhgwright:

This is probably a duplicate of #68314. I'm not sure about the -flto part, but why don't you try again (and comment here) once legacy-support 1.2.1 is published (​https://github.com/macports/macports-ports/pull/23011).

Or if you don't want to wait, the current legacy-support-devel is the same code as the upcoming release.

I should warn you, though, that the new version is upward- but not downward-compatible with the old version, so anything built against the old version should work with the new, but not necessarily vice-versa. Thus, it's probably simpler just to wait for the new release version.

[RJVB (René Bertin)]

Replying to fhgwright:

This is probably a duplicate of #68314. I'm not sure about the -flto part, but why don't you try again (and comment here) once legacy-support 1.2.1 is published (​https://github.com/macports/macports-ports/pull/23011).

The symptoms are the same, the question is why I'm getting them when compiling with -flto but without -fPIC (which AFAICT is the only difference between the generated *.o and *.dl.o object files).

I can wait, of course, but I could also see if adding -fPIC instead of -fno-lto addresses the problem without adding any others.

Do you happen to have an idea which ports link statically? Without clear indication of significant benefits of LTO builds it's probably not worth going down a rabbithole to fix this more properly...

BTW, -fno-lto overrides and disables any request to use LTO; given how the Makefile is set up that includes any attempt of the user to obtain it via configure.optflags.

(PS: my session apparently timed out while typing this reply. Bad trac! Thank goodness it wasn't any longer and that the browser cache didn't get invalidated...'')

[fhgwright (Fred Wright)]

Since legacy-support 1.2.1 is now available, you can just try it. If it fixes the problem, it's up to you how much time you want to spend chasing a non-problem with -flto. :-)

You mentioned that you use ccache. I know little about ccache, but if it's allergic to "system" includes that aren't immutable, that could cause trouble with anything that uses legacy-support.

[fhgwright (Fred Wright)]

Did you ever determine whether this is still an issue?

[RJVB (René Bertin)]

Sorry, no, I moved on to other things since I had a workaround (and llvm-17 turns out to be barely useable for me anyway). You must have sensed though that I was planning to update my legacy-support install (with the tiny hope that'll fix go from crashing systematically).

Do you have any interest in knowing whether building the components of the static library with -fPIC fixes the issue (supposing it persists otherwise)? As I implied above, that should allow you to simplify the buildsystem and reduce the build time; compile everything once, $(AR) the static library and then invoke $(CC) (or $(CXX)) to link that archive into a shared library.

[fhgwright (Fred Wright)]

Replying to RJVB:

Sorry, no, I moved on to other things since I had a workaround (and llvm-17 turns out to be barely useable for me anyway). You must have sensed though that I was planning to update my legacy-support install (with the tiny hope that'll fix go from crashing systematically).

At the time of my previous response, the symptom you were seeing looked very much like a compatibility issue that I'd already fixed. At the time, the fix was only in legacy-support-devel, but that was a few releases ago. So the question is, if you do exactly the same thing as you were doing before, but with the current legacy-support, do you still see the problem? If not, then this ticket should be closed.

BTW, was my comment about ccache relevant?

Do you have any interest in knowing whether building the components of the static library with -fPIC fixes the issue (supposing it persists otherwise)? As I implied above, that should allow you to simplify the buildsystem and reduce the build time; compile everything once, $(AR) the static library and then invoke $(CC) (or $(CXX)) to link that archive into a shared library.

The reason one normally doesn't use -fPIC unconditionally is because there's usually some performance cost, albeit not necessarily a large one. Granted, most uses are for the shared library, where -fPIC is mandatory, but I don't think it makes sense to inflict it on the static library without good reason. Build time isn't really an issue, given that it typically builds in well under a minute, even without parallel builds. It's not gcc or llvm. :-)

I wouldn't be inclined to apply -fPIC unconditionally just as a potshot at a nonunderstood problem.

[RJVB (René Bertin)]

Replying to fhgwright:

So the question is, if you do exactly the same thing as you were doing before, but with the current legacy-support, do you still see the problem? If not, then this ticket should be closed.

OK, so the only problem is that the "it's not^HH^{H llvm". I hope I've kept my build directory...}

BTW, was my comment about ccache relevant?

No, I doubt that. Ccache is quite clever and usually errs on the side of caution but above all I saw the problem from the start, i.e. when ccache's only action is to create a cached version of the compilation result. We're not talking about replacement headers like the ones compiler like to include or headers specified on the commandline. Last time I checked the PG also used a simple -I rather than -isystem so ccache has no reason to treat the legacy-support headers like system ones.