cctools: using clang < 5.0 as default assembler generates assembly errors in some cases

[kencu (Ken)]

To fix errors in newer ports that require gcc (fortran) to assemble newer assembly, the as in cctools was modified to send the assembly to a specific version of clang for processing, if present. For SnowLeopard, that default clang version is clang-3.4.

It turns out there is (at least one) bug in the clang i386 assembler that was not fixed until after clang-3.7. It is fixed in clang-5.0. This error shows up during the build or bootstrapping of gcc / libgcc, and perhaps will turn up elsewhere:

/opt/local/i386-apple-darwin10/bin/ -nostdinc++ -B/opt/local/var/macports/build/_opt_macports-ports_lang_gcc9/libgcc9/work/build/prev-i386-apple-darwin10/libstdc++-v3/src/.libs -B/opt/local/var/macports/build/_opt_macports-ports_lang_gcc9/libgcc9/work/build/prev-i386-apple-darwin10/libstdc++-v3/libsupc++/.libs  -isystem /opt/local/var/macports/build/_opt_macports-ports_lang_gcc9/libgcc9/work/build/prev-i386-apple-darwin10/libstdc++-v3/include/i386-apple-darwin10  -isystem /opt/local/var/macports/build/_opt_macports-ports_lang_gcc9/libgcc9/work/build/prev-i386-apple-darwin10/libstdc++-v3/include  -isystem /opt/local/var/macports/build/_opt_macports-ports_lang_gcc9/libgcc9/work/gcc-9.1.0/libstdc++-v3/libsupc++ -L/opt/local/var/macports/build/_opt_macports-ports_lang_gcc9/libgcc9/work/build/prev-i386-apple-darwin10/libstdc++-v3/src/.libs -L/opt/local/var/macports/build/_opt_macports-ports_lang_gcc9/libgcc9/work/build/prev-i386-apple-darwin10/libstdc++-v3/libsupc++/.libs    -g -O2  -fno-checking -mdynamic-no-pic -gtoggle -DIN_GCC     -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H -static-libstdc++ -static-libgcc  -o cc1 c/c-lang.o c-family/stub-objc.o attribs.o c/c-errors.o c/c-decl.o c/c-typeck.o c/c-convert.o c/c-aux-info.o c/c-objc-common.o c/c-parser.o c/c-fold.o c/gimple-parser.o c-family/c-common.o c-family/c-cppbuiltin.o c-family/c-dump.o c-family/c-format.o c-family/c-gimplify.o c-family/c-indentation.o c-family/c-lex.o c-family/c-omp.o c-family/c-opts.o c-family/c-pch.o c-family/c-ppoutput.o c-family/c-pragma.o c-family/c-pretty-print.o c-family/c-semantics.o c-family/c-ada-spec.o c-family/c-ubsan.o c-family/known-headers.o c-family/c-attribs.o c-family/c-warn.o c-family/c-spellcheck.o i386-c.o darwin-c.o \
	  cc1-checksum.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a  -liconv ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a  -L/opt/local/lib -lisl -L/opt/local/lib -L/opt/local/lib -L/opt/local/lib -lmpc -lmpfr -lgmp   -lz
ld: in libbackend.a(i386.o), in section __TEXT,__text reloc 5479: unsupported r_length=0 for scattered vanilla reloc for architecture i386

This is a known clang assembler error <​https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63773>.

There is a reduced test case for it:

	.text
	.align 4,0x90
	.globl _choose_tmpdir
_choose_tmpdir:
	sbbl	$3+_vartmp, %esi
	ret
	.space 254

	.const
	.align 2
_vartmp:
	.byte	47

and I confirm that clang-3.4 and clang-3.7 error out on this assembly, but clang-5.0 can compile it successfully:

$ clang-mp-3.4 -arch i386  testi386asm.S
ld: in /var/folders/If/IfyiELboGT08qyd+9nD3DU+++TI/-Tmp-/testi386asm-fc6237.o, in section __TEXT,__text reloc 0: unsupported r_length=0 for scattered vanilla reloc for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ clang-mp-3.7 -arch i386  testi386asm.S
ld: in /var/folders/If/IfyiELboGT08qyd+9nD3DU+++TI/-Tmp-/testi386asm-c2fa0b.o, in section __TEXT,__text reloc 0: unsupported r_length=0 for scattered vanilla reloc for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ clang-mp-5.0 -arch i386  testi386asm.S
Undefined symbols for architecture i386:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Likewise, the same assembly generates errors when compiling in x86_64 mode.

$ clang-mp-3.4 -arch x86_64  testi386asm.S
ld: in /var/folders/If/IfyiELboGT08qyd+9nD3DU+++TI/-Tmp-/testi386asm-07fc5b.o, in section __TEXT,__text reloc 0: length < 2 and X86_64_RELOC_UNSIGNED not supported for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ clang-mp-3.7 -arch x86_64  testi386asm.S
ld: in /var/folders/If/IfyiELboGT08qyd+9nD3DU+++TI/-Tmp-/testi386asm-52d45e.o, in section __TEXT,__text reloc 0: length < 2 and X86_64_RELOC_UNSIGNED not supported for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ clang-mp-5.0 -arch x86_64  testi386asm.S
Undefined symbols for architecture x86_64:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found for architecture x86_64

So - it appears that clang should not be used as the default assembler unless it is clang-5.0 or greater, and for the system clang should not be used (according to the gcc bug report) unless the Xcode version is at least 7.3.1.

This probably requires only a rather minor adjustment to the cctools port.

[cjones051073 (Chris Jones)]

Could we just update the default llvm variant used in cctools to be at least 5.0 ?

[kencu (Ken)]

I think we'd have bootstrap loop.

We can't get to llvm-5.0 without clang/llvm-3.4, and we can't get to clang/llvm-3.4 without cctools being upgraded.

I am thinking the best idea would be to allow mp-clang as assembler if llvm >= 5.0, and use the system clang only if xcode >= 7.3.1.

I'm hoping to come up with an easy tweak to the current cctools Portfile that would enable that idea...

[kencu (Ken)]

well, defaulting to llvm 5 would be way better. Let me see if I can figure out how we might do that...

[kencu (Ken)]

during bootstrap of 10.6.8, llvm-3.4 builds using the stock toolchain. Then clang-3.4 can be built, having a build dependency on cctools at that point. So the only llvm variant available for cctools during bootstrap is llvm-3.4.

Once clang-3.4 is built, it can then build clang-3.7, and then that can build llvm/clang-5.0.

So there does not appear to be a way to build cctools with llvm-5.0 from the outset using the stock toolchain. You would have to go through a cctools +llvm34 stage to get to cctools +llvm50.

[jmroot (Joshua Root)]

This all seems very strange. /usr/bin/as will be running /usr/bin/clang internally for quite a few OS and Xcode versions. Is it really impossible to build gcc with every Xcode version prior to 7.3.1? How did anyone manage when that was all that existed?

If we stop using /usr/bin/clang in cctools as on old versions, we're back to square one in terms of providing a more modern assembler by default.

[kencu (Ken)]

Replying to jmroot:

Is it really impossible to build gcc with every Xcode version prior to 7.3.1? How did anyone manage when that was all that existed?

Prior to that point in time, /usr/bin/as defaulted to using gas.

The errors started showing up when they flipped to defaulting to clang as assembler, and then only in i386 code generation.

Here is what appears to be the bug report in the llvm tree, but unfortunately there is no reference to any single commit that fixed it <​https://bugs.llvm.org/show_bug.cgi?id=7635>.

[cjones051073 (Chris Jones)]

Currently the logic in the cctools port that picks the clang version to use with as is tied to the llvm version used to build the port, via the variant. This is a reasonable enough way to do it, but technically is not required. Instead of hardcoding into the as binary the MP clang version matching the llvm variant, we *could* pass in a list of all possible reasonable versions, starting with the most recent and working back. i.e. the list of all possible llvm variant versions. as would then start at the beginning of this list and use which either it first finds.... This would allow the port to still be built using the llvm 3.x version, but still when as is used at runtime pick a more recent clang version.

Note that this would actually allow the somewhat convoluted logic the gcc ports now need to make sure the correct clang version cctools needs is installed to be removed, and instead just declare a dependency on a recent clang version, say 8.0, safe in the knowledge it will be used as the assembler if needed.

[kencu (Ken)]

Here's an idea I had about it <​https://github.com/macports/macports-ports/pull/4397>

but - it is not ideal as it would require a manual step for systems < 13 to upgrade their cctools to a newer llvm version to get the fully-functional clang working as assembler.

sudo port -v -n upgrade --enforce-variants cctools -llvm34 +llvm50 configure.compiler=macports-clang-3.4

[cjones051073 (Chris Jones)]

The more I think about the more I feel decoupling the macports clang version(s) that as uses internally from the llvm version that is used to build the cctools port is the way to go. There really is no technical reason these have to be the same. Then, as I describe above, instead of a single version, pass in a list of alternatives, in order of preference, and use the first that is found installed.

[kencu (Ken)]

From that point of view, we should just use mp-clang-5.0 for the assembler on all older systems, even if not installed when cctools is built/installed, as in the end, every older system will wind up having that installed, via the cxx11 1.1 PortGroup.

That is very easy to do.

[cjones051073 (Chris Jones)]

maybe, but please note any changes to the logic in cctools as to how the version of clang is picked needs to be mirrored in the logic in the gcc9 and gcc10 ports. See

​https://github.com/macports/macports-ports/blob/8d134cf384d3b9bd856511540dde9f44b641bc12/lang/gcc9/Portfile#L104

The change you suggest will require you to also update the logic above to match.

What I describe, whilst requiring a bit more work in cctools would allow the logic above to more or less be completely removed, and instead the gcc ports would just need to make sure a new enough clang was available, safe in the knowledge that the logic in cctools would pick it up.

[kencu (Ken)]

OR - to ponder - have we reached the stage where people who still want to run older OS versions will just have to do a few simple-enough steps manually?

We could leave the cctools port as it is, in that case.

[jmroot (Joshua Root)]

Replying to kencu:

Replying to jmroot:

Is it really impossible to build gcc with every Xcode version prior to 7.3.1? How did anyone manage when that was all that existed?

Prior to that point in time, /usr/bin/as defaulted to using gas.

I don't think that's correct; clang seems to be the default in Xcode 7.0.

[jmroot (Joshua Root)]

Replying to cjones051073:

The more I think about the more I feel decoupling the macports clang version(s) that as uses internally from the llvm version that is used to build the cctools port is the way to go. There really is no technical reason these have to be the same. Then, as I describe above, instead of a single version, pass in a list of alternatives, in order of preference, and use the first that is found installed.

That's a bit of a kludge but then so is all of this. Not an unreasonable solution.

[cjones051073 (Chris Jones)]

Replying to kencu:

OR - to ponder - have we reached the stage where people who still want to run older OS versions will just have to do a few simple-enough steps manually?

We could leave the cctools port as it is, in that case.

If it where just for old 32 bit builds I would agree. However, to be honest, I am much more interested in making the changes I suggested to cctools to simplify life downstream, such as removing the tom foolery needed in the gcc ports just to make sure the clang version cctools needs is available. Changing to a more predictable scheme that dos not depend on the llvm variant that happens to have been used will make things much easier.

[kencu (Ken)]

My worry with your plan is there will be no way to know which assembler was used in a given build. If that's true, triaging tickets will be difficult.

[kencu (Ken)]

Replying to jmroot:

Replying to kencu:

Replying to jmroot:

Is it really impossible to build gcc with every Xcode version prior to 7.3.1? How did anyone manage when that was all that existed?

Prior to that point in time, /usr/bin/as defaulted to using gas.

I don't think that's correct; clang seems to be the default in Xcode 7.0.

My apologies if my notation was not more clear.

Every Xcode version < 7.0 defaulted to gas as assembler.

This bug apparently showed up building gcc in 32bit mode with Xcode 7, and was fixed in xcode 7.3.1.

[cjones051073 (Chris Jones)]

Replying to kencu:

My worry with your plan is there will be no way to know which assembler was used in a given build. If that's true, triaging tickets will be difficult.

Is it that worse to what happens now ? The selected clang version is only used *if* the build happens to have it installed.

Given the fact cctools cannot have any sort of dependency on any MacPorts clang, what ever version(s) is involved, there is always going to be a degree of non-determinism here...

[mouse07410 (Mouse)]

Yes it probably would be worse than now.

Having to install a whole c...load of unnecessary old LLVM and possibly Clang (and spending time for that monster to compile/build, as usually happens on my installations) is extremely annoying.

I would prefer dependency on Xcode assembler instead. Especially since in most cases you have to have Xcode installed for Macports to be able to build ports anyway.

[cjones051073 (Chris Jones)]

Replying to mouse07410:

Yes it probably would be worse than now.

Having to install a whole c...load of unnecessary old LLVM and possibly Clang (and spending time for that monster to compile/build, as usually happens on my installations) is extremely annoying.

I would prefer dependency on Xcode assembler instead. Especially since in most cases you have to have Xcode installed for Macports to be able to build ports anyway.

None of what I propose would have any impact on what you describe above. In fact it would make it better.

Just to be clear, cctools does *not* have any dependency on any macports clang version for the as implementation it has. By construction, as otherwise there would be a circular dep. The only reason you have to 'install a whole c...load of unnecessary old LLVM and possibly Clang' is because of the dep in gcc on clang, which is needed because of the current logic in the cctools port. What I am proposing is for cctools to be made a bit more liberal in allowing more macports clang compilers to be use *if* they happen to be installed, and to make the list of versions something indepedent to the llvm version that cctools happens to be built using. If this is done, the logic in gcc can be much reduced.

It would also be possible to check the Xcode version, and if new enough have gcc skip adding any macports clang dep, and thus just using the system (Xcode) /usr/bin/clang. This could of course be done now, if you want. I just don't personally consider the clang dep enough of an issue myself to address it. PRs welcome of course.

[kencu (Ken)]

so long as the oldest clang we use as assembler is clang-5.0 / Xcode 7.3.1, I'm fine with Chris' plan.

[mouse07410 (Mouse)]

What I am proposing is for cctools to be made a bit more liberal in allowing more macports clang compilers to be use *if* they happen to be installed, and to make the list of versions something indepedent to the llvm version that cctools happens to be built using.

That makes perfect sense. Thanks for explaining.

It would also be possible to check the Xcode version, and if new enough have gcc skip adding any macports clang dep, and thus just using the system (Xcode) /usr/bin/clang. This could of course be done now, if you want

That would be perfect. If you could add this, it would be outstanding.

THANK YOU!

[kencu (Ken)]

We do want to continue having a method for people using MacPorts to test out the new assembler in a new version of clang, though.

[jmroot (Joshua Root)]

I'd like to confirm that the clang assembler bug does exist all the way back to Xcode 4.

[kencu (Ken)]

Here are a series of systems ( I skipped 10.8):

$ cat testi386asm.S
	.text
	.align 4,0x90
	.globl _choose_tmpdir
_choose_tmpdir:
	sbbl	$3+_vartmp, %esi
	ret
	.space 254

	.const
	.align 2
_vartmp:
	.byte	47

macOS 10.7.5 11G63
Xcode 4.6.3 4H1503 

$ /usr/bin/clang -v
Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn)
Target: x86_64-apple-darwin11.4.2
Thread model: posix

$  /usr/bin/clang -arch i386 testi386asm.S
ld: in section __TEXT,__text reloc 0: unsupported r_length=0 for scattered vanilla reloc for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

macOS 10.9.5 13F1911
Xcode 6.2 6C131e

$ /usr/bin/clang -v
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix

$ /usr/bin/clang -arch i386 testi386asm.S
ld: in section __TEXT,__text reloc 0: unsupported r_length=0 for scattered vanilla reloc file '/var/folders/7s/6wzny8vx3h1_nt4rpw2911200000gn/T/testi386asm-c1569b.o' for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

macOS 10.10.5 14F2511
Xcode 7.2.1 7C1002

$ /usr/bin/clang -v
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin14.5.0
Thread model: posix

$ /usr/bin/clang -arch i386 testi386asm.S
ld: in section __TEXT,__text reloc 0: unsupported r_length=0 for scattered vanilla reloc file '/var/folders/dc/565wd82x4v790v4xb1hv_f940000gn/T/testi386asm-7d404d.o' for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

and -- 7.3.1 works:

macOS 10.11.6 15G22010
Xcode 7.3.1 7D1014 

$ /usr/bin/clang -v
Apple LLVM version 7.3.0 (clang-703.0.31)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ /usr/bin/clang -arch i386 testi386asm.S
Undefined symbols for architecture i386:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[mouse07410 (Mouse)]

We do want to continue having a method for people using MacPorts to test out the new assembler in a new version of clang, though

Understood and agreed.

However, we're discussing what should be the default, not what should be possible. And in my opinion, it should be possible to configure cctools to use whatever installed version of clang you want to point it at - but it should not install another version of clang unless I explicitly ask for it.

[cjones051073 (Chris Jones)]

Replying to mouse07410:

We do want to continue having a method for people using MacPorts to test out the new assembler in a new version of clang, though

Understood and agreed.

However, we're discussing what should be the default, not what should be possible. And in my opinion, it should be possible to configure cctools to use whatever installed version of clang you want to point it at - but it should not install another version of clang unless I explicitly ask for it.

As I have already stated, cctools *never* declares any dependency on any clang version, for reasons of doing so would create a circular dependency issue. What it does, and will continue to do in the future, is at runtime check to see what is installed at that time, and decide what to use based on the outcome of this check. The proposal is simply to instead of only looking for the specific clang version corresponding to the llvm version used to build the port, allow a range of 'good' clang versions. based on what I have seen, that list I think would be '[8,7,6,5]' in that order of preference. So use the most recent 5+ clang version that is available.

The devel clang port is an interesting question. My first thought was to not include this in the list above (where would you put it ? first ? last ?) but perhaps it should be a special case, so if the user decides they want cctools installed with the devel llvm variant, then force this to be the (only) clang version considered, on the basis they are doing this for some specific non-standard reason.... Thoughts ?

[mouse07410 (Mouse)]

What it does, and will continue to do in the future, is at runtime check to see what is installed at that time, and decide what to use based on the outcome of this check...

What I'm asking for is including the Xcode clang in that check, and making it possible for the user to request that the Xcode clang is selected if it's installed. Better yet, do this by default - use Xcode clang if it's installed, unless the user explicitly requested something else.

The devel clang port . . . if the user decides they want cctools installed with the devel llvm variant, then force this to be the (only) clang version considered, on the basis they are doing this for some specific non-standard reason

Yes, that makes sense. Allow the user to specify this non-standard variant, even if it's a devel version.

[cjones051073 (Chris Jones)]

Replying to mouse07410:

What it does, and will continue to do in the future, is at runtime check to see what is installed at that time, and decide what to use based on the outcome of this check...

What I'm asking for is including the Xcode clang in that check, and making it possible for the user to request that the Xcode clang is selected if it's installed. Better yet, do this by default - use Xcode clang if it's installed, unless the user explicitly requested something else.

Xcode’s clang is already part of the check. Nothing needs to change in this regard.

The devel clang port . . . if the user decides they want cctools installed with the devel llvm variant, then force this to be the (only) clang version considered, on the basis they are doing this for some specific non-standard reason

Yes, that makes sense. Allow the user to specify this non-standard variant, even if it's a devel version.

[mouse07410 (Mouse)]

Xcode’s clang is already part of the check. Nothing needs to change in this regard

It was my impression that by default it would pick a Macports clang, or even install it (I'm not sure which Macports component does it though). So if by default Xcode clang would be used when present - I'm happy, and nothing needs to change in this regard.

[cjones051073 (Chris Jones)]

Replying to mouse07410:

Xcode’s clang is already part of the check. Nothing needs to change in this regard

It was my impression that by default it would pick a Macports clang, or even install it (I'm not sure which Macports component does it though). So if by default Xcode clang would be used when present - I'm happy, and nothing needs to change in this regard.

As i have said, a few times now, cctools will never trigger the installation of any macports clang port. If it fails to find a suitable one at runtime, xcode clang is used instead. The logic Is this way around, as we know the macports clang work on all platforms, clang 5.0 is the same on all, whereas the systems, whereas Xcode clang varies in capabilities. So the logic is in my view correct.

[cjones051073 (Chris Jones)]

​https://github.com/macports/macports-ports/pull/4472

[kencu (Ken)]

clang-3.7 fails:

$ clang-mp-3.7 -arch i386 testasm.S
ld: in section __TEXT,__text reloc 0: unsupported r_length=0 for scattered vanilla reloc file '/var/folders/I0/I0fOotq3ErGZHYxrmmWiqU+++TI/-Tmp-/testasm-dad071.o' for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

but clang-3.8 and newer are fine:

$ clang-mp-3.8 -arch i386 testasm.S
Undefined symbols for architecture i386:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ clang-mp-5.0 -arch i386 testasm.S
Undefined symbols for architecture i386:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

this is as expected, as clang-3.8 is about Apple's clang 800.

I will update the clang blacklisting in the gcc portfiles to match, because clang-9.0 can't build gcc (a different error, to be in a different ticket) and we need some proper fallbacks available.

[kencu (Ken)]

In 3412c6c9c2f8162a4d82d6bb1c26e63bef8ce068/macports-ports (master):

gcc9/libgcc9: liberalize build compilers

macports-clangs >= 3.8 can properly build gcc9
see: #58493#comment:34

[kencu (Ken)]

In e29edf4b96db25bb99c40d6092c6a6bf64b5fe2c/macports-ports (master):

libgcc9/gcc9, libgcc-devel, gcc10: harmonize blacklisting

these ports are very similar wrt blacklisting
they don't build properly with macports-clang < 3.8
they don't build at all with clang-9.0 at present

see: #59253
see: #58493

[kencu (Ken)]

The issues in this ticket have been effectively resolved by the new compiler defaulting that is done with MacPorts 2.6.x, together with the changes to cctools to use the newest clang available that is installed as the assembler.