clang-14: crashes occurring for multiple ports

[mascguy (Christopher Nielsen)]

When compiling darktable 3.8.1, the following crash occurs, for source file src/common/iop_profile.c. Build log attached.

1.  <eof> parser at end of file
2.  Code generation
3.  Running pass 'Function Pass Manager' on module 'work/darktable-3.8.1/src/common/iop_profile.c'.
4.  Running pass 'Debug Variable Analysis' on function '@dt_ioppr_transform_image_colorspace'
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libLLVM.dylib            0x000000010a0eea8b llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 43
1  libLLVM.dylib            0x000000010a0ed928 llvm::sys::RunSignalHandlers() + 248
2  libLLVM.dylib            0x000000010a0ee040 llvm::sys::CleanupOnSignal(unsigned long) + 208
3  libLLVM.dylib            0x000000010a00a06f CrashRecoverySignalHandler(int) + 191
4  libsystem_platform.dylib 0x00007fff95c38b3a _sigtramp + 26
5  libsystem_platform.dylib 0x00007fff58ebd930 _sigtramp + 18446744072688782864
6  libLLVM.dylib            0x000000010a3e9371 (anonymous namespace)::LDVImpl::runOnMachineFunction(llvm::MachineFunction&, bool) + 5457
7  libLLVM.dylib            0x000000010a4ac0b4 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 308
8  libLLVM.dylib            0x000000010a24ae2a llvm::FPPassManager::runOnFunction(llvm::Function&) + 922
9  libLLVM.dylib            0x000000010a251383 llvm::FPPassManager::runOnModule(llvm::Module&) + 67
10 libLLVM.dylib            0x000000010a24b449 llvm::legacy::PassManagerImpl::run(llvm::Module&) + 937
11 libclang-cpp.dylib       0x0000000107db85db clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::__1::unique_ptr<llvm::raw_pwrite_stream, std::__1::default_delete<llvm::raw_pwrite_stream> >) + 2395
12 libclang-cpp.dylib       0x00000001080ecb98 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) + 1928
13 libclang-cpp.dylib       0x0000000106ebfe63 clang::ParseAST(clang::Sema&, bool, bool) + 643
14 libclang-cpp.dylib       0x0000000108988dfa clang::FrontendAction::Execute() + 90
15 libclang-cpp.dylib       0x00000001088f1616 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) + 902
16 libclang-cpp.dylib       0x00000001089ff5f8 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) + 680
17 clang                    0x0000000106d42e53 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) + 2147
18 clang                    0x0000000106d41036 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) + 310
19 libclang-cpp.dylib       0x000000010853c707 void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, bool*) const::$_1>(long) + 23
20 libLLVM.dylib            0x000000010a009dc2 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) + 226
21 libclang-cpp.dylib       0x000000010853c295 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optional<llvm::StringRef> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, bool*) const + 389
22 libclang-cpp.dylib       0x000000010850314f clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const + 415
23 libclang-cpp.dylib       0x000000010850364c clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::__1::pair<int, clang::driver::Command const*> >&) const + 124
24 libclang-cpp.dylib       0x000000010851f11c clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::__1::pair<int, clang::driver::Command const*> >&) + 396
25 clang                    0x0000000106d409b4 main + 10820
26 libdyld.dylib            0x00007fff95a29235 start + 1
27 libdyld.dylib            0x0000000000000088 start + 18446603338005704276
clang: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 14.0.3
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-14/bin

Similarly, a link-time crash occurs with libopenraw. Build log attached.

  = note: 0  0x104026b51  __assert_rtn + 144
          1  0x104057063  ld::Fixup::Fixup(unsigned int, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 107
          2  0x10402f4e9  mach_o::relocatable::Parser<x86_64>::addFixup(mach_o::relocatable::Parser<x86_64>::SourceLocation const&, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 33
          3  0x10402a751  mach_o::relocatable::Section<x86_64>::addRelocFixup(mach_o::relocatable::Parser<x86_64>&, macho_relocation_info<Pointer64<LittleEndian> > const*) + 1471
          4  0x104052bb7  mach_o::relocatable::Section<x86_64>::makeFixups(mach_o::relocatable::Parser<x86_64>&, mach_o::relocatable::Parser<x86_64>::CFI_CU_InfoArrays const&) + 91
          5  0x10404e7c6  mach_o::relocatable::Parser<x86_64>::parse(mach_o::relocatable::ParserOptions const&) + 2034
          6  0x1040320fd  mach_o::relocatable::Parser<x86_64>::parse(unsigned char const*, unsigned long long, char const*, long, ld::File::Ordinal, mach_o::relocatable::ParserOptions const&) + 375
          7  0x1040629cc  archive::File<x86_64>::makeObjectFileForMember(archive::File<x86_64>::Entry const*) const + 758
          8  0x104062518  archive::File<x86_64>::justInTimeforEachAtom(char const*, ld::File::AtomHandler&) const + 122
          9  0x10407819b  ld::tool::InputFiles::searchLibraries(char const*, bool, bool, bool, ld::File::AtomHandler&) const + 215
          10  0x104080c90  ld::tool::Resolver::resolveUndefines() + 160
          11  0x104082f63  ld::tool::Resolver::resolve() + 79
          12  0x1040276c7  main + 689
          A linker snapshot was created at:
          /tmp/build_script_build-0fd30e212c66cd59-2022-05-30-185930.ld-snapshot
          ld: Assertion failed: (name != NULL), function Fixup, file /SourceCache/ld64/ld64-241.9/src/ld/ld.hpp, line 510.
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

[mascguy (Christopher Nielsen)]

Of note, this occurs on multiple macOS releases, and doesn't appear to be affected by the macOS version.

[mascguy (Christopher Nielsen)]

Also, compiling with Clang 13 works just fine.

[Christopher Nielsen <mascguy@…>]

In 642a988efbf587db10145e7ed95133a513f9b78a/macports-ports (master):

darktable/darktable-devel: blacklist macports clang-14 for now, due to compilation crash
See: #65246

[mascguy (Christopher Nielsen)]

PR created, for llvm-14 update to 14.0.4:

​https://github.com/macports/macports-ports/pull/14996

[mascguy (Christopher Nielsen)]

Unfortunately this crash still occurs, even with LLVM 14.0.4.

Are there any additional upstream patches that we also need to apply...?

[kencu (Ken)]

Usually what you would now is see if it is fixed in clang-devel.

If it is, then --> figure out why/which commit / which file (and ideally report it upstream for possible backports).

If it is not, then --> report it upstream with the files the error says to include with the report.

[kencu (Ken)]

at least it is simpler now, as I believe it is all just github issues, and no longer the more complex bug tracker they've had all the previous years.

[Christopher Nielsen <mascguy@…>]

In d9e7e218242f34605d9fc13f590e52d35c43e192/macports-ports (master):

llvm-14: update to 14.0.4
See: #65246

[Christopher Nielsen <mascguy@…>]

In d8b7dea4e5ba84f2c2d941cecaa17b22bdb45519/macports-ports (master):

libopenraw: blacklist macports clang-14 for now, due to link crash
See: #65246

[cjones051073 (Chris Jones)]

LLVM 14 is proving to be a bit of a pain release. It has seen way more updates than LLVM devs normally put out, its up to 14.0.6, and if you read the release notes you can see the problems.

I think it might make sense to remove clang-14 as the ;default fallback, and go back to using clang-13 which seems more stable...

[cjones051073 (Chris Jones)]

​https://github.com/macports/macports-ports/commit/d44abda1ebf6cd623c7b6ee3deb2e754ff53e17b

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

LLVM 14 is proving to be a bit of a pain release. It has seen way more updates than LLVM devs normally put out, its up to 14.0.6, and if you read the release notes you can see the problems.

I think it might make sense to remove clang-14 as the default fallback, and go back to using clang-13 which seems more stable...

Sounds good, though it looks like clang-13 may be missing one or more patches too. See attachment libopenraw-build-10.9-clang-13-crash-link.log​.

[cjones051073 (Chris Jones)]

Can we have a different ticket for that, otherwise its gets messy...

[cjones051073 (Chris Jones)]

b.t.w. its not obvious the issue is clang-13 there. The seg. faults there appear to come from rust, which internally builds against its own LLVM version...

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Can we have a different ticket for that, otherwise its gets messy...

See issue:65434

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

b.t.w. its not obvious the issue is clang-13 there. The seg. faults there appear to come from rust, which internally builds against its own LLVM version...

Ah, the plot thickens! Suggestions as to how we proceed?

[cjones051073 (Chris Jones)]

debugging rust internals is a rabbit hole I am not going down... I suggest reporting to upstream devs for the port in question and see what they say...

[catap (Kirill A. Korinsky)]

Just for keep things in one place. Building of llvm-devel on 10.6 by clang-13 and clang-14 fails with the same reason like:

Assertion failed: (name != NULL), function Fixup, file src/ld/ld.hpp, line 394.
0  0x100001fa0  __assert_rtn + 79
1  0x10007564e  ld::Fixup::Fixup(unsigned int, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 70
2  0x10009015c  mach_o::relocatable::Parser<x86_64>::FixupInAtom::FixupInAtom(mach_o::relocatable::Parser<x86_64>::SourceLocation const&, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 30
3  0x10009baff  mach_o::relocatable::Parser<x86_64>::addFixup(mach_o::relocatable::Parser<x86_64>::SourceLocation const&, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 31
4  0x10008882d  mach_o::relocatable::Section<x86_64>::addRelocFixup(mach_o::relocatable::Parser<x86_64>&, macho_relocation_info<Pointer64<LittleEndian> > const*) + 1977
5  0x1000a4b2d  mach_o::relocatable::Section<x86_64>::makeFixups(mach_o::relocatable::Parser<x86_64>&, mach_o::relocatable::Parser<x86_64>::CFI_CU_InfoArrays const&) + 75
6  0x1000b8a0f  mach_o::relocatable::Parser<x86_64>::parse(mach_o::relocatable::ParserOptions const&) + 1251
7  0x1000b8e5f  mach_o::relocatable::Parser<x86_64>::parse(unsigned char const*, unsigned long long, char const*, long, unsigned int, mach_o::relocatable::ParserOptions const&) + 66
8  0x100089643  mach_o::relocatable::parse(unsigned char const*, unsigned long long, char const*, long, unsigned int, mach_o::relocatable::ParserOptions const&) + 319
9  0x10006ff07  archive::File<x86_64>::makeObjectFileForMember(archive::File<x86_64>::Entry const*) const + 407
10  0x1000709de  archive::File<x86_64>::justInTimeforEachAtom(char const*, ld::File::AtomHandler&) const + 86
11  0x10000de13  ld::tool::InputFiles::searchLibraries(char const*, bool, bool, bool, ld::File::AtomHandler&) const + 251
12  0x100064785  ld::tool::Resolver::resolveUndefines() + 171
13  0x100066579  ld::tool::Resolver::resolve() + 113
14  0x10000b896  main + 220
15  0x1000039b4  start + 52

[mascguy (Christopher Nielsen)]

Replying to catap:

Just for keep things in one place. Building of llvm-devel on 10.6 by clang-13 and clang-14 fails with the same reason

So the $100,000 question I'd like to throw out to the group, is this: Why are we introducing another upstream LLVM release - 15 - when LLVM 13 and 14 aren't rock-solid?

That makes zero sense to me...

[cjones051073 (Chris Jones)]

Who knows, Maybe its better...

Just introducing the new port does not mean it will be used anywhere yet. It will only be used as a fallback once it is added to the _resources, and this hasn't happened yet.

[kencu (Ken)]

The above linker issues might also be related to the absolutely ancient linker being used. Certainly upstream never tests something this old.

I use ld64-450 on 10.6.8, and I've never seen errors like that with clang-13 or clang-14, but it could just be coincidence.

Perhaps the next line of effort might most effectively be to get that bootstrapping sorted out to have all the Intel systems 10.6+ use at least ld64-450 as a linker (or newer if possible).

[kencu (Ken)]

Actually - no. clang-14 errors out building darktable on the most up-to-date Monterey, with:

% ld -v
@(#)PROGRAM:ld  PROJECT:ld64-819.6
BUILD 14:58:37 Aug  5 2022
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 14.0.0, (clang-1400.0.29.102) (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 14.0.0 (tapi-1400.0.11)

[mascguy (Christopher Nielsen)]

Just witnessed another crash on our buildbots, using clang-15, for port coreutils:

/opt/local/bin/clang-mp-15   -pipe -Os -arch x86_64  -L/opt/local/lib -Wl,-headerpad_max_install_names -arch x86_64 -o src/df src/df.o src/find-mount-point.o src/libver.a lib/libcoreutils.a -lintl -Wl,-framework -Wl,CoreFoundation  lib/libcoreutils.a

0  0x100001fa0  __assert_rtn + 79
1  0x10007564e  ld::Fixup::Fixup(unsigned int, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 70
2  0x10009015c  mach_o::relocatable::Parser<x86_64>::FixupInAtom::FixupInAtom(mach_o::relocatable::Parser<x86_64>::SourceLocation const&, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 30
3  0x10009baff  mach_o::relocatable::Parser<x86_64>::addFixup(mach_o::relocatable::Parser<x86_64>::SourceLocation const&, ld::Fixup::Cluster, ld::Fixup::Kind, bool, char const*) + 31
4  0x10008882d  mach_o::relocatable::Section<x86_64>::addRelocFixup(mach_o::relocatable::Parser<x86_64>&, macho_relocation_info<Pointer64<LittleEndian> > const*) + 1977
5  0x1000a4b2d  mach_o::relocatable::Section<x86_64>::makeFixups(mach_o::relocatable::Parser<x86_64>&, mach_o::relocatable::Parser<x86_64>::CFI_CU_InfoArrays const&) + 75
6  0x1000b8a0f  mach_o::relocatable::Parser<x86_64>::parse(mach_o::relocatable::ParserOptions const&) + 1251
7  0x1000b8e5f  mach_o::relocatable::Parser<x86_64>::parse(unsigned char const*, unsigned long long, char const*, long, unsigned int, mach_o::relocatable::ParserOptions const&) + 66
8  0x100089643  mach_o::relocatable::parse(unsigned char const*, unsigned long long, char const*, long, unsigned int, mach_o::relocatable::ParserOptions const&) + 319
9  0x10006ff07  archive::File<x86_64>::makeObjectFileForMember(archive::File<x86_64>::Entry const*) const + 407
10  0x1000709de  archive::File<x86_64>::justInTimeforEachAtom(char const*, ld::File::AtomHandler&) const + 86
11  0x10000de13  ld::tool::InputFiles::searchLibraries(char const*, bool, bool, bool, ld::File::AtomHandler&) const + 251
12  0x100064785  ld::tool::Resolver::resolveUndefines() + 171
13  0x100066579  ld::tool::Resolver::resolve() + 113
14  0x10000b896  main + 220
make[2]: *** [src/df] Error 1

Logfile attached; filename: coreutils-clang-15-crash-10.6_x64.log.gz

[mascguy (Christopher Nielsen)]

Replying to mascguy:

Just witnessed another crash on our buildbots, using clang-15, for port coreutils

Logfile attached; filename: coreutils-clang-15-crash-10.6_x64.log.gz

Queued another build attempt for 10.6_x64, hopefully it will succeed this time. Stay Tuned...

[kencu (Ken)]

Oh, interesting. I suppose it could be that we'll have to force 10.6.8 to always use a newer linker when it's using a new clang.

Well, that could prove awkward, but needs to happen anyway. That is what clang11-bootstrap was supposed to allow.

[kencu (Ken)]

I had to revert the new defaulting to newer clangs. I believe the issue is related to the old ld64-127 that is the buildbot default for SL.

We have been meaning to sort out how to default the linker on SL to something newer; I guess it's time we did.

Chris once-upon-a-time suggested the ${prefix}/bin/ld could be a shell script that looks for the newest to oldest ld64 versions we offer. That would work, but is a bit non-deterministic as you never really nail which linker you're getting ahead of time... but that could be worked around to a degree.

Alternatively we could sort out how to make clang-11-bootstrap build the newest ld64. The issue there is we need an LLVM for ld64 to link against, and libtapi too, so there are several parts to be sorted out.

SIgh.