clang12: please re-enable the +analyzer as a default variant

[mouse07410 (Mouse)]

iMac (Intel-based), MacOS BigSur 11.4, Xcode-12.5. Note: I do not want to build universal binaries, though the ability to cross-compile on this iMac for M1 platform (again, not universal!) would be beneficial.

First, to my surprise, the pre-built clang-12 is not built with +analyzer, only with +libstdcxx. Is this an omission, or a "strategic" change? If an omission - could you please remedy it. If an intentional change - please reconsider and return to providing binaries pre-built with analyzer.

Second - my attempt to install clang-12 with analyzer fails:

$ sudo port install llvm-12
$ sudo port install clang-12 +analyzer
.  .  .  .  .
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build [ 18%] Built target clang_rt.builtins_arm64_osx
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build [ 18%] Built target obj.llvm-tblgen
:info:build make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build make: *** [all] Error 2
:info:build make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build" && /usr/bin/make -j20 -w all VERBOSE=ON 
:info:build Exit code: 2
:error:build Failed to build clang-12: command execution failed
:debug:build Error code: CHILDSTATUS 77307 2
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$callback {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec -callback portprogress::target_progress_callback build"
:debug:build     (procedure "portbuild::build_main" line 8)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/main.log for details.

Complete log attached.

I suspect it could do something with the number of arguments to the last shell command...???

[mouse07410 (Mouse)]

Main build log

[kencu (Ken)]

You actually use the +analyzer in macports-clang-*? Or you just noticed the change?

I left it off the default list during the refactoring of the clang-12 port because it (possibly needlessly) calls in a full installation of perl for everyone who just wants to install clang-12. I've been trying, with little success, to cut the dep and install footprint down a bit for the clang ports.

[kencu (Ken)]

your compiler-rt build failure is something different, by the way:

:info:build CMake Error: failed to create symbolic link '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldclr1_3.S': file already exists
:info:build cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build/projects/compiler-rt/lib/builtins && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build/projects/compiler-rt/lib/builtins -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/compiler-rt/lib/builtins -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build/include -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/include -O3 -DNDEBUG -arch armv7em -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -Oz -Wall -fomit-frame-pointer -ffreestanding -arch armv7em -fPIC -mfloat-abi=soft -o CMakeFiles/clang_rt.soft_pic_armv7em_macho_embedded.dir/arm/sync_fetch_and_sub_4.S.o -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/compiler-rt/lib/builtins/arm/sync_fetch_and_sub_4.S
:info:build make[2]: *** [projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldclr1_3.S] Error 1

I believe this occurs due to a race condition in some build attempts for clang. Marcus disabled parallel_building one time to get past something similar to this -- but as it only happens to me 1 / 100 times I build clang, and because building clang-12 without parallel building takes an extremely long time, I re-enabled it again.

If you know how to properly fix the race condition, that would be appreciated.

If it happens a lot, on the buildbots and elsewhere, and there is no choice left but to disable parallel_buildling, so be it, we'll disable it and live with the outcome.

[kencu (Ken)]

I'll look at adding a default for +analyzer again in 12.1.x which is coming out pretty soon, if it seems that there is enough interest.

on a similar note, I am thinking of dumping the default building of the 10 targets other than X86 and ARM that we use on macOS. They can be a non-default variant.

You would be upset about that?

[kencu (Ken)]

see also #63026 where another user noted the analyzer variant had a typo, and also wanted it.

perhaps having the world install perl5 (which is perl5.28 at present) is not a big deal for everyone.

[mouse07410 (Mouse)]

You actually use the +analyzer in macports-clang-*? Or you just noticed the change?

Yes and yes. ;-)

I believe this occurs due to a race condition in some build attempts for clang. . . .

Unlike your experience, where the failure is observed 1/100 times or so, in my case (I tried four times) it is consistent. :-(

If you know how to properly fix the race condition, that would be appreciated.

I wish I did. Sorry!

I'll look at adding a default for +analyzer again in 12.1.x which is coming out pretty soon, if it seems that there is enough interest.

Yes please!! I'm really looking to the pre-built clang-12.1 binaries that I won't need to rebuild locally from sources.

on a similar note, I am thinking of dumping the default building of the 10 targets other than X86 and ARM that we use on macOS. They can be a non-default variant. You would be upset about that?

I don't think I understand what build targets you're planning to drop. As long as x86_64 and M1 with +analyzer +libstdcxx are supported - I see no reason for being upset. ;-)

perhaps having the world install perl5 (which is perl5.28 at present) is not a big deal for everyone.

For me - not a big deal at all. Is sure port install perl5 sufficient? The machine in question had the older Perl5 (5.26). Proven not to be a factor - upgraded Perl5 to 5.28, got the same result. Build fails on libeling_rt.builtins_arm64_osx.a

.  .  .  .  .
:info:build [ 16%] Linking C static library ../../../../lib/libclang_rt.builtins_arm64_osx.a
:info:build cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm
-12/clang-12/work/build/projects/compiler-rt/lib/builtins && /opt/local/bin/cmake -P CMakeFiles/clang_rt.builtins_arm64_osx.dir/
cmake_clean_target.cmake
:info:build cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm
-12/clang-12/work/build/projects/compiler-rt/lib/builtins && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/clang_rt.built
ins_arm64_osx.dir/link.txt --verbose=ON
:info:build "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool" -static -no_warning
_for_no_symbols -o ../../../../lib/libclang_rt.builtins_arm64_osx.a CMakeFiles/clang_rt.builtins_arm64_osx.dir/comparetf2.c.o CM
akeFiles/clang_rt.builtins_arm64_osx.dir/divtc3.c.o CMakeFiles/clang_rt.builtins_arm64_osx.dir/extenddftf2.c.o CMakeFiles/clang_
rt.builtins_arm64_osx.dir/extendhftf2.c.o .  .  .  .  .
.  .  .  .  .
CMakeFiles/clang_rt.builtins_arm64_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldset8_3.S.o CMakeFiles/clang_rt.builtins_arm64_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldset8_4.S.o
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build [ 16%] Built target clang_rt.builtins_arm64_osx
:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build [ 16%] Built target obj.llvm-tblgen
:info:build make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build make: *** [all] Error 2
:info:build make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build'
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build" && /usr/bin/make -j20 -w all VERBOSE=ON 
:info:build Exit code: 2
:error:build Failed to build clang-12: command execution failed
:debug:build Error code: CHILDSTATUS 64187 2
:debug:build Backtrace: command execution failed
:debug:build     while executing
:debug:build "system {*}$notty {*}$callback {*}$nice $fullcmdstring"
:debug:build     invoked from within
:debug:build "command_exec -callback portprogress::target_progress_callback build"
:debug:build     (procedure "portbuild::build_main" line 8)
:debug:build     invoked from within
:debug:build "$procedure $targetname"
:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/main.log for details.

Thanks!

[kencu (Ken)]

Replying to mouse07410:

I believe this occurs due to a race condition in some build attempts for clang. . . .

Unlike your experience, where the failure is observed 1/100 times or so, in my case (I tried four times) it is consistent. :-(

Unfortunate

I'll look at adding a default for +analyzer again in 12.1.x which is coming out pretty soon, if it seems that there is enough interest.

Yes please!! I'm really looking to the pre-built clang-12.1 binaries that I won't need to rebuild locally from sources.

Just stay with clang-11 for now then, until 12.1.x rolls out. You're no worse off than you were three days ago. Or clang-devel, which is a similar vintage to clang-12, if you feel that something was added to clang-12 you can't live without.

Build fails on libeling_rt.builtins_arm64_osx.a

add: use_parallel_build no

to the Portfile and report back please.

You will notice every buildbot system (and every system of mine) had no trouble building this as it is, so I assumed that upstream had fixed the issue. It takes several hours with parallel building enabled. With only one processor, you can guess it might take four times that long, or more, not sure.

[mouse07410 (Mouse)]

Just stay with clang-11 for now then, until 12.1.x rolls out. You're no worse off than you were three days ago

Yes, that's what I will do. I'm not aware of any clang-12 feature (compared to clang-11) that I absolutely myst have,

add: use_parallel_build no to the Portfile and report back please.

Apologies - can't promise a quick report here. Playing with Portfiles is not my forte. :-(

It takes several hours with parallel building enabled. With only one processor, you can guess it might take four times that long, or more, not sure.

This is a big concern, and one reason I (and probably many others) would like to see analyzer in the default variant, so it doesn't need local re-build.

[kencu (Ken)]

I hear you, I hear you -- you like the analyzer. I was under the impression it never even worked properly.

#62308

I have never used it, nor did I think most anyone ever did.

Just trying to cut out cruft.

[szhorvat (Szabolcs Horvát)]

I will add my "+1" to this. I would also like to have the +anayzer by default. Building all of clang-12 takes a very, very long time on my laptop (didn't measure but I think it was well over an hour). I can't do much else while it's compiling, and can't unplug the laptop.

I also wanted to note that while set-xcode-analyzer is broken in Clang 12 (for no fault of MacPorts), the analyzer is probably not used through Xcode by most people. It is used through the scan-build script, which works fine (mostly, see #63047). So the fact that set-xcode-analyzer is broken should not be an obstacle to enabling +analyzer by default. Noting this just in case :-)

[kencu (Ken)]

OK. Seems popular demand. For clang-12 v12.1.x, which is already tagged upstream, I will re-enable the analyzer, and everyone can just eat the PERL installation.

TBH I thought this was more of a "You moved my cheese!" thing, but I guess some actually use this.

I'm still trying to get clang-12 working on all our needed systems -- it does not yet -- and that takes precedence.

If you have a suggested fix to make the analyzer actually work correctly -- I am getting the impression from your reports that it sorta does and sorta doesn't, but I am not sure what your suggested fix is to make it work -- that would be nice, as shipping something that just doesn't work anyway is rather irritating.

[kencu (Ken)]

And yes indeed, it was exactly your bug report about the analyzer that got this whole idea going that it was a fairly useless, rather burdensome variant, that didn't work anyway and may have not worked in years.

[kencu (Ken)]

Replying to mouse07410:

Apologies - can't promise a quick report here. Playing with Portfiles is not my forte. :-(

Mouse -- I admit surprise that you would find a use for the static analyzer in a macport-clang-* installation, yet adding one line to a Portfile would be daunting.

As I want to make sure we have your use case covered, can you point me to your example of how you use it, and I'll run some checks to make sure it works as promised? Thanks.

[mouse07410 (Mouse)]

I hear you, I hear you -- you like the analyzer. I was under the impression it never even worked properly

It's not that static analyzer does not work - it occasionally produces false results. Sad, but still much better than nothing.

OK. Seems popular demand. For clang-12 v12.1.x, which is already tagged upstream, I will re-enable the analyzer, and everyone can just eat the PERL installation.

Thank you! I' say, (a) the PERL cost is very easy to eat, and (b) PERL is already required for several ports, as far as I remember. So, no problem.

TBH I thought this was more of a "You moved my cheese!" thing, but I guess some actually use this.

:-) :-)

Mouse -- I admit surprise that you would find a use for the static analyzer in a macport-clang-* installation, yet adding one line to a Portfile would be daunting

Well, I'm fairly decent (and long-tooth experienced) with C/C++. I'm not experienced at all with the bowels of Macports. For example, I can't even locate the Portfile in the existing failed installation of clang-12 (appears that it's not even there in the extracted source?!). I'm not saying that I can't be bothered to learn Macports, but it would take time I'd rather not invest right now, and probably useless in the long term - I'm not a maintainer of or contributor to any port. And I'm not itching to have a manually-maintained clone of Macports ports - one of the main reasons (for me) to use Macports instead of just compiling everything form sources is exactly the fact that Macports allows me to avoid messing with that stuff.

As I want to make sure we have your use case covered, can you point me to your example of how you use it, and I'll run some checks to make sure it works as promised?

I'm a co-maintainer of Crypto++ library ​https://github.com/weidai11/cryptopp . Among other validation tools, It uses ASAN and UBSAN, like make asan and make ubsan. It would be nice if it worked with clang-12.

[kencu (Ken)]

super-secret one-liner one off Portfile tweaker command:

bbedit `port file clang-12`

make your edit, save your file, install your build.

It all goes back to normal next time you selfupdate (unless you are using a git checkout of the ports tree, in which case your edits are kept).

[mouse07410 (Mouse)]

super-secret one-liner one of Portfile tweaker command . . .

OK, that's not very difficult, thanks.

Is there any particular place in that file where I should add this separate line:

use_parallel_build no

Again, I've no clue how Portfile is structured.

Thanks

[kencu (Ken)]

Most often it does not matter where you put a line, but sometimes, it does, if things below that in the Portfile depend on the value.

In this case, anywhere will be fine, but I usually stick it in the top 1/3, near the patchfiles, etc.

[mouse07410 (Mouse)]

On an iMac Pro (configured "heavy"), with use_parallel_build no added to clang-12 Portfile:

$ time sudo port install clang-12 +analyzer
Portfile changed since last build; discarding previous state.
--->  Computing dependencies for clang-12
--->  Fetching archive for clang-12
--->  Attempting to fetch clang-12-12.0.0_0+analyzer+libstdcxx.darwin_20.x86_64.tbz2 from https://packages.macports.org/clang-12
--->  Attempting to fetch clang-12-12.0.0_0+analyzer+libstdcxx.darwin_20.x86_64.tbz2 from https://nue.de.packages.macports.org/clang-12
--->  Attempting to fetch clang-12-12.0.0_0+analyzer+libstdcxx.darwin_20.x86_64.tbz2 from http://atl.us.packages.macports.org/clang-12
--->  Fetching distfiles for clang-12
--->  Verifying checksums for clang-12
--->  Extracting clang-12
--->  Applying patches to clang-12
--->  Configuring clang-12
--->  Building clang-12                                  
--->  Staging clang-12 into destroot                     
--->  Installing clang-12 @12.0.0_0+analyzer+libstdcxx
--->  Activating clang-12 @12.0.0_0+analyzer+libstdcxx
--->  Cleaning clang-12
--->  Scanning binaries for linking errors
--->  No broken files found.                             
--->  No broken ports found.

real	287m3.014s
user	270m2.013s
sys	13m7.159s
$ 

But that won't be practical on a "normal" (not "heavy") machine.

[kencu (Ken)]

None of upstream uses the Makefiles method of building the llvm tree. They all use ninja, and presumably ninja does not show this issue.

We can't use ninja, because we install llvm separately from the clang and other components, and the ninja build does not allow going into each directory and running destroot separately like we have to do.

What I can look into is adding a Makefile command .notparallel into the compiler-rt build top Makefile, so that only that part builds with one process.

To do that using cmake as a generator turns out to be non-trivial. It's all made worse because I almost never see this issue on my 16 core machines, so I can't really test it properly.

[mouse07410 (Mouse)]

If clang-12 seems to parallel-build fine on your machines and on the build-bot - perhaps all you need to do is replace the default with the +analizer variant?

I, for one, would much rather download clang binaries than build it locally from source.

[kencu (Ken)]

Replying to mouse07410:

I'm a co-maintainer of Crypto++ library ​https://github.com/weidai11/cryptopp . Among other validation tools, It uses ASAN and UBSAN, like make asan and make ubsan. It would be nice if it worked with clang-12.

The analyzer has nothing to do with asan or ubsan or any other sanitizer.

​https://clang-analyzer.llvm.org/scan-build.html

[mouse07410 (Mouse)]

The analyzer has nothing to do with asan or ubsan or any other sanitizer

I admit that I haven't used scan-build so far. Thank you for providing the URL.

I still think it would be a good thing to add, and will try it with my big project ​https://github.com/weidai11/cryptopp.git

Thanks again!

[mouse07410 (Mouse)]

Well, trying build-scan make for Crypto++ failed (in a way I don't quite understand):

$ cat ~/src/cryptopp/scan-report.txt 
/opt/local/libexec/llvm-12/bin/scan-view:9: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Traceback (most recent call last):
  File "/opt/local/libexec/llvm-12/bin/scan-view", line 150, in <module>
    main()
  File "/opt/local/libexec/llvm-12/bin/scan-view", line 147, in main
    run(port, args, args.root)
  File "/opt/local/libexec/llvm-12/bin/scan-view", line 74, in run
    import ScanView
  File "/opt/local/libexec/llvm-12/bin/../share/scan-view/ScanView.py", line 29, in <module>
    import Reporter
ModuleNotFoundError: No module named 'Reporter'
WARNING: Unable to detect that server started.
$ 

Here's how that report was generated:

$ sudo port select --list clang
Enter PIN for 'Certificate For PIV Authentication (Blumenthal, Uri (UR20980))': 
Available versions for clang:
	apple-clang (active)
	mp-clang-10
	mp-clang-11
	mp-clang-9.0
	none
$ scan-build-mp-12 make
scan-build: Using '/opt/local/libexec/llvm-12/bin/clang' for static analysis
Using testing flags: -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
Running make again to see what failed
TestPrograms/test_x86_sse2.cpp:5:5: warning: Value stored to 'x' is never read [deadcode.DeadStores]
    x=_mm_add_epi64(x,x);
    ^ ~~~~~~~~~~~~~~~~~~
1 warning generated.

/opt/local/libexec/llvm-12/bin/../libexec/c++-analyzer -DCRYPTOPP_DISABLE_ASM -fPIC -fno-common -pipe -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c cryptlib.cpp
/opt/local/libexec/llvm-12/bin/../libexec/c++-analyzer -DCRYPTOPP_DISABLE_ASM -fPIC -fno-common -pipe -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c cpu.cpp
/opt/local/libexec/llvm-12/bin/../libexec/c++-analyzer -DCRYPTOPP_DISABLE_ASM -fPIC -fno-common -pipe -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c integer.cpp
/opt/local/libexec/llvm-12/bin/../libexec/c++-analyzer -DCRYPTOPP_DISABLE_ASM -fPIC -fno-common -pipe -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c 3way.cpp
/opt/local/libexec/llvm-12/bin/../libexec/c++-analyzer -DCRYPTOPP_DISABLE_ASM -fPIC -fno-common -pipe -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c adler32.cpp
/opt/local/libexec/llvm-12/bin/../libexec/c++-analyzer -DCRYPTOPP_DISABLE_ASM -fPIC -fno-common -pipe -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c algebra.cpp
/opt/local/libexec/llvm-12/bin/../libexec/c++-analyzer -DCRYPTOPP_DISABLE_ASM -fPIC -fno-common -pipe -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -c algparam.cpp
In file included from algparam.cpp:3:
In file included from ./pch.h:14:
In file included from ./config.h:22:
In file included from ./config_align.h:27:
In file included from ./config_cxx.h:37:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/string:506:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/string_view:175:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__string:58:10: fatal error: 'cstdio' file not found
#include <cstdio>     // For EOF.
         ^~~~~~~~
1 error generated.
make: *** [GNUmakefile:1790: algparam.o] Error 1
scan-build: Analysis run complete.
scan-build: 2 bugs found.
scan-build: Run 'scan-view /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/scan-build-2021-06-13-211957-91810-1' to examine bug reports.
$ scan-view-mp-12 /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/scan-build-2021-06-13-211957-91810-1 > scan-report.txt 2>&1

As far as I can tell, Xcode-12.5 does not include build-scan or scan-view.

[kencu (Ken)]

you're doing a good job proving my point.

1. nobody uses the analyzer on our MacPorts builds
2. it may not even work
3. so why bother with it?

[mouse07410 (Mouse)]

you're doing a good job proving my point

:-)

nobody uses the analyzer on our MacPorts builds

Based on your interpretation - correct. Though I still don't quite grasp why ASAN and UBSAN are not parts of the analyzer.

nobody uses the analyzer on our MacPorts builds

If "analyzer" is "scan-build" only - then you're probably right.

so why bother with it?

The URL you kindly provided, says:

scan-build: running the analyzer from the command line

It does not imply or explicitly say that "analyzer" is only the "scan-build" (and "scan-view") part. And while scan-build does not seem to work, ASAN and UBSAN (among other sanitizers) work OK.

[kencu (Ken)]

Yeah, I think that is what got you (and probably everyone else) a bit confused.

The sanitizers are useful, we always include them (on systems that can build them).

The analyzer is niche.

The two things have absolutely nothing to do with each other.

[mouse07410 (Mouse)]

OK, you're probably right. I can only say that I have never used scan-build, and the one time I tried - it did not work properly.

P.S. Though this message from it is correct:

scan-build: Using '/opt/local/libexec/llvm-12/bin/clang' for static analysis
Using testing flags: -std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
Running make again to see what failed
TestPrograms/test_x86_sse2.cpp:5:5: warning: Value stored to 'x' is never read [deadcode.DeadStores]
    x=_mm_add_epi64(x,x);
    ^ ~~~~~~~~~~~~~~~~~~
1 warning generated.

[kencu (Ken)]

You see the issue with adding the analyzer is that means you need a full perl installation. Which in and of itself is not so terrible, but that means you need to have the entire ports tree set up to build perl without using the current clang versions (as that makes a circular dependency). And so then you are forced to consider how you are going to safely build perl (on some systems) with all it's dependencies without relying on clang to build it, blah blah blah.

Just adds to the mess of trying to keep this whole ship afloat.

Likewise libxml2. clang actually has no need at all for our libxml2 port, other than it will opportunistically link to it if it is found, which causes no end of trouble. libxml2 pulls in -- get ready -- ICU! -- and ICU is a major PITA to build these days as it needs a modern compiler, with all its now circular deps. So -- the solution is to purge libxml2 out of the clang builds, and perl, and anything else that is not really needed for clang to function.

ERGO this push to cut the cruft out of the clang builds, like python (where we can) and perl and what-have-you.

[mouse07410 (Mouse)]

You see the issue with adding the analyzer is that means you need a full perl installation. Which in and of itself is not so terrible, . . .

Yes, that by itself would be nothing from my point of view.

but that means you need to have the entire ports tree set up to build perl without using the current clang versions (as that makes a circular dependency)

I see the problem. Can only sympathize...

But doesn't Macports already require Xcode (or Xcode CLT) installed in order to work? Therefore, couldn't Macports require that Perl is only built with Xcode clang, thus alleviating all the clang version issues?

[kencu (Ken)]

Replying to mouse07410:

But doesn't Macports already require Xcode (or Xcode CLT) installed in order to work? Therefore, couldn't Macports require that Perl is only built with Xcode clang, thus alleviating all the clang version issues?

Not on all systems...

Just all extra needless stuff to worry about.

[mouse07410 (Mouse)]

Just all extra needless stuff to worry about

Understood. So, Analyzer is not completely useless (in my example above, out of two messages one was correct; the other one wasn't) - but a lot less useful than I thought prior to this conversation.

I'll let you decide the best course of action here.

[kencu (Ken)]

In e5dd44494ea579bb4bc6b5b97ab3b8aa82087680/macports-ports (master):

clang-12: default analyzer variant

tried to trim the deps but a few people
seemed to feel strongly they wanted the
analyzer left in the default build

closes: #63038

no revbump as too trivial a change
will come out with next update

[jmroot (Joshua Root)]

Replying to kencu:

1. nobody uses the analyzer on our MacPorts builds
2. it may not even work

I've run scan-build-mp-11 on base. It seemed to work fine and found some memory leaks. (The results for the vendor subdir are rather worrisome in fact…)

[mouse07410 (Mouse)]

Not sure what's going on, but for some reason Macports insists on rebuilding clang-12 from the source?!

I updated via port selfupdate, removed the old clang-12, and did this:

$ sudo port clean clang-12
--->  Cleaning clang-12
$ sudo port install clang-12
--->  Computing dependencies for clang-12
--->  Fetching archive for clang-12
--->  Attempting to fetch clang-12-12.0.0_1+analyzer+libstdcxx.darwin_20.x86_64.tbz2 from https://packages.macports.org/clang-12
--->  Attempting to fetch clang-12-12.0.0_1+analyzer+libstdcxx.darwin_20.x86_64.tbz2 from https://nue.de.packages.macports.org/clang-12
--->  Attempting to fetch clang-12-12.0.0_1+analyzer+libstdcxx.darwin_20.x86_64.tbz2 from http://atl.us.packages.macports.org/clang-12
--->  Fetching distfiles for clang-12
--->  Verifying checksums for clang-12
--->  Extracting clang-12
--->  Applying patches to clang-12
--->  Configuring clang-12
--->  Building clang-12                                  
Error: Failed to build clang-12: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_llvm-12/clang-12/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port clang-12 failed

[mouse07410 (Mouse)]

New build failure

[mouse07410 (Mouse)]

$ env | grep MacOSX
CXXFLAGS=-std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
CFLAGS=-O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
$ 

:info:build [ 11%] Building CXX object lib/Target/AArch64/CMakeFiles/LLVMAArch64CodeGen.dir/AArch64TargetMachine.cpp.o
:info:build cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/c
lang-12/work/build/lib/Target/AArch64 && /usr/bin/clang++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/op
t/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build
/lib/Target/AArch64 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_ll
vm-12/clang-12/work/llvm-project-12.0.0.src/llvm/lib/Target/AArch64 -I/opt/local/var/macports/build/_opt_local_var_macports_sources_r
sync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/build/include -I/opt/local/var/macports/build/_opt_local_var_macp
orts_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/include -isystem /opt/
local/include -pipe -Os -DNDEBUG -I/opt/local/include -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/Ma
cOSX.platform/Developer/SDKs/MacOSX11.3.sdk -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -W
all -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallth
rough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion
 -O3 -DNDEBUG -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.s
dk -mmacosx-version-min=11.0  -fno-exceptions -std=c++14 -o CMakeFiles/LLVMAArch64CodeGen.dir/AArch64TargetMachine.cpp.o -c /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp
:info:build In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/lib/Target/AArch64/AArch64TargetMachine.cpp:12:
:info:build In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/lib/Target/AArch64/AArch64TargetMachine.h:16:
:info:build In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/lib/Target/AArch64/AArch64InstrInfo.h:16:
:info:build In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/lib/Target/AArch64/AArch64.h:17:
:info:build In file included from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_lang_llvm-12/clang-12/work/llvm-project-12.0.0.src/llvm/lib/Target/AArch64/MCTargetDesc/AArch64MCTargetDesc.h:18:
:info:build In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/memory:688:
:info:build /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/cassert:20:10: fatal error: 'assert.h' file not found
:info:build #include <assert.h>
:info:build          ^~~~~~~~~~
:info:build 1 error generated.
:info:build make[2]: *** [lib/Target/AArch64/CMakeFiles/LLVMAArch64CodeGen.dir/AArch64TargetMachine.cpp.o] Error 1

You can see that for some reason, the build process appears to look for the SDK location using xcrun --sdk macosx. In theory, that should work - but in practice it fails, as I observed with my own projects. And that command points at the MacOSX11.3.sdk that (as said above), symlinks to MacOSX.sdk (which somehow doesn't seem to work/help.

$ xcrun --sdk macosx --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk

What one should do instead is drop the --sdk macosx argument - building on Mac won't use anything else:

$ xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

This points at a usable (and current!) MacOS SDK.

[kencu (Ken)]

The buildbot has finished building all the new clang-12's except for the one for 10.8, so if you try to install it now, you should get the prebuilt buildbot version, which has the analyzer enabled.

[mouse07410 (Mouse)]

The buildbot has finished building all the new clang-12's except for the one for 10.8, so if you try to install it now, you should get the prebuilt buildbot version, which has the analyzer enabled.

Thank you!

I still think that Macports maintainers should consider removing --sdk macosx from the place where sysroot gets determined.

[kencu (Ken)]

I know you do.

(BTW - I am the macports maintainer for this port).

Building compilers requires a very cleanly set up system.

[mouse07410 (Mouse)]

Building compilers requires a very cleanly set up system

I understand. But if a setting is unnecessary for a clean build in general, and is harmful to some machines - why not remove it?

If you know any negative consequences of ditching that flag, could you please share?

[kencu (Ken)]

It is an upstream setup, and is set by them. We carry it along on systems that support it to do our best to match what they are trying to do.

Your build error is not due to that, specifically, as all the buildbots, me, and most other people can build it without trouble.

So -- what is wrong on your system? I don't know. Maybe something to do with those environmennt variables you are setting? Mismatched Xcode and Command Line Tools? Something out of date?

If you feel that llvm should do things differently, feel free to take to the compiler-rt mailing list upstream, and let them know. This code is all written by Apple-employed engineers, and they have every bit as strong an idea how things should be done as you have seen various people around here have. But you might sell it.

If they change, I'll change!

[mouse07410 (Mouse)]

It is an upstream setup, and is set by them. We carry it along on systems that support it to do our best to match what they are trying to do.

Understood.

. . . what is wrong on your system? I don't know. Maybe something to do with those environmennt variables you are setting? Mismatched Xcode and Command Line Tools? Something out of date?

Env variables? Can't tell, maybe. Will explore that.

Mismatch between Xcode and CLT? Impossible, for two reasons: first, when I install CLT I make sure they match the Xcode version and update as appropriate; second - as I've discovered that CLT breaks Haskell platform (​https://gitlab.haskell.org/ghc/ghc/-/issues/19968 and ​https://github.com/haskell/haskell-language-server/issues/1913), I removed CLT from all of my machines (frankly, no loss).

Something out of date? While possible, it's highly unlikely - usually I'm meticulous about keeping things patched and upgraded...

If you feel that llvm should do things differently, feel free to take to the compiler-rt mailing list upstream and let them know

I do, and I will. Would you be kind enough to point me at the right mailing list, please?

This code is all written by Apple-employed engineers, and they have every bit as strong an idea how things should be done as you have seen various people around here have. But you might sell it.

I've dealt with Apple engineers in the past. I'll only say that some of them are better than others, and vs. versa. ;-) Let me find out. ;-)

[kencu (Ken)]

​https://compiler-rt.llvm.org

looks like they want you use the llvm-dev mailing list, according to the bottom of that page.

Good luck!

[mouse07410 (Mouse)]

I submitted my report and request. I got one answer, saying that this exact problem was reported to Apple a few years ago, and has been assigned reference number FB7253366.

Not sure how to proceed...

[kencu (Ken)]

Well done. We'll keep the current setup for the time being, but consider changing.

BTW, nice question framing I thought... and you got a quick response from someone who thinks similarly. Perhaps they will address this!