OpenBLAS @0.3.20_0+gcc11+lapack+native build failure on intel mac 11.6.5: gcc is putting out x86-pad-for-align=false but assembler is not accepting it

[dsavransky (Dmitry Savransky)]

Here's a 'fun' one: went to upgrade openblas today and got a build failure (log attached). Basic error appears to be bad llvm argument ('-x86-pad-for-align=false') but I can't find much info on that anywhere. Most likely some llvm/clang clash. However, if I do: sudo port upgrade --enforce-variants openblas +lapack +native +gcc10 -gcc11 it builds just fine.

I'm on macos 11.6.5, with xcode 13.2.1 (and 13.2 command line tools installed). I previously had OpenBLAS @0.3.19_0+gcc11+lapack+native installed with no apparent issues.

Thanks.

[dsavransky (Dmitry Savransky)]

OpenBLAS @0.3.20_0+gcc11+lapack+native failing build log

[cooljeanius (Eric Gallager)]

This seems to be blocking a lot of my upgrades; I didn't realize so many ports depended on OpenBLAS...

[dsavransky (Dmitry Savransky)]

Replying to cooljeanius:

This seems to be blocking a lot of my upgrades; I didn't realize so many ports depended on OpenBLAS...

Have you tried building with gcc10 instead of 11? That previously worked and allowed me to update everything else.

[kencu (Ken)]

that argument x86-pad-for-align=false was added to gcc to get around a bug:

​https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100340

but here it appears that the Xcode clang assembler on the failing system doesn't know what to do with it, and errors...

This is a strange turn of events; the option is meant to be added only when needed I believe, so exactly why it's now unaccepted is a bit confusing. There are a lot of variables at play here now.

[kencu (Ken)]

probably more gccs involved than just gcc11 I assume, but gcc11 is what we know (knew) then

[cjones051073 (Chris Jones)]

For those blocked by this, you can work around the issues by removing the native variant which is forcing you to build from source.

Also note the default gcc used now is gcc12, so if you move to using that as well you should just oick up the binary tarball, rather than building from source yourself.

[cooljeanius (Eric Gallager)]

Replying to dsavransky:

Replying to cooljeanius:

This seems to be blocking a lot of my upgrades; I didn't realize so many ports depended on OpenBLAS...

Have you tried building with gcc10 instead of 11? That previously worked and allowed me to update everything else.

Oh yeah I was actually already trying with gcc10 in the first place; I didn't notice that this bug was originally reported against gcc11... anyways, using the gcc12 variant lets me get a binary from the server; let me try actually building with it next...

[cjones051073 (Chris Jones)]

i cannot reproduce. OpenBLAS +native built fine for me on Intel macOS12.

[cooljeanius (Eric Gallager)]

Replying to cooljeanius:

Replying to dsavransky:

Replying to cooljeanius:

This seems to be blocking a lot of my upgrades; I didn't realize so many ports depended on OpenBLAS...

Have you tried building with gcc10 instead of 11? That previously worked and allowed me to update everything else.

Oh yeah I was actually already trying with gcc10 in the first place; I didn't notice that this bug was originally reported against gcc11... anyways, using the gcc12 variant lets me get a binary from the server; let me try actually building with it next...

OK so building with +gcc12+lapack+native fails too; I dunno how the buildbot managed it...

[cjones051073 (Chris Jones)]

Buildbot does not use native variant, as its not enabled by default.

However, I used the variant and was also unable to reproduce.

[cjones051073 (Chris Jones)]

Please ensure your ports are up to date before building, just to be sure..

sudo port -d sync
sudo port upgrade outdated

Then try again.

[dsavransky (Dmitry Savransky)]

OpenBLAS @0.3.20_1+gcc12+lapack+native failing build log

[dsavransky (Dmitry Savransky)]

I just did a full selfupdate and then attempted to build with gcc12. Same exact error as before. I think this is very much a macOS 11.6/Xcode 12.4 -specific issue. new log attached.

[cjones051073 (Chris Jones)]

self update does not necessarily update outdated ports. Can you please confirm running the commands i sent above does nothing ?

[cjones051073 (Chris Jones)]

Buildbot macOS 11 build used Xcode 13

​https://build.macports.org/builders/ports-11_x86_64-builder/builds/79119/steps/install-port/logs/stdio

Could you try updating your Xcode and try again ?

[dsavransky (Dmitry Savransky)]

sorry, typo on the xcode version. I have Xcode 13.2.1 on that system (as per my orig bug report). port -d sync reports:

Total number of ports parsed: 0 Ports successfully parsed: 0 Ports failed: 0 Up-to-date ports skipped: 29607

port upgrade outdated fails on OpenBLAS build, as before.

[cjones051073 (Chris Jones)]

Please attempt a build from source without the native variant, using only the defaults

sudo port uninstall OpenBLAS
sudo port -s install OpenBLAS

[dsavransky (Dmitry Savransky)]

Same error building without native.

[cjones051073 (Chris Jones)]

Then I am afraid you are going to have to play spot-the-difference with the buildbot log I posted above, as that uses the same variants and Xcode 13, so on the face of it is the same and worked just fine. Its looking like something specific to your system i am afraid….

[cjones051073 (Chris Jones)]

… or just use the binary install and forget about the problem..

[kencu (Ken)]

Eric, exactly what cctools do you have installed, and do you have any macports-clang-* compilers installed?

I have a suspicion different assemblers may be at work here...

[iains]

as noted in the GCC PR, I cannot reproduce the failure - the option seems to be accepted by clang -cc1as for all the versions I've tried from CLT 12.x onwards (on darwin20 and 21). I'd go with @kencu's suspicion about a bogus assembler in the mix.

[cjones051073 (Chris Jones)]

ken’s suggestion of the cctools version you are using is a good shout. Fwiw I have the version using the xcode variant installed , which basically just installs wrapper scripts around whatever xcrun provides, and is the default on the new OS versions.

cctools @949.0.1_2+xcode (active)

Oberon ~ > cat /opt/local/bin/as
#!/bin/bash
if [[ -x /usr/bin/xcrun ]] ; then
  exec /usr/bin/xcrun as "${@}"
elif [[ -x /usr/bin/as ]] ; then
  exec /usr/bin/as "${@}"
else
  exec as "${@}"
fi
Oberon ~ > 

[dsavransky (Dmitry Savransky)]

oh wow. I feel pretty silly now. I had an active clang-11 (not quite sure why that was getting used in this build). I got rid of that (and everything other than clang-14) and am now able to build OpenBLAS from source, with and without -native. Thank you all for your help and patience.

[cjones051073 (Chris Jones)]

Just having macports clang 11 installed would not on its own affect the build, there must be some-other piece to the puzzle.

To repeat, please confirm what cctools version you have installed ?

Had you also by chance port selected a specific version clang to be the default ?

[dsavransky (Dmitry Savransky)]

you're right. my active cctools was @949.0.1_2+llvm10. i had not selected a specific clang.

[kencu (Ken)]

Aha -- any non-Xcode variant of cctools will spin off assembly to a relatively new macports-clang-N if it is found. So that is likely to be part of it.

If the build failed when clang-11 was installed but succeeded when it was deactivated, then possibly clang-11 doesn't accept that argument.

But clang-14 (if installed, and it appears it was) should have been selected *first* anyway, unless the order these clangs are chosen is wrong (backwards).

[kencu (Ken)]

lots of moving parts, indeed.

[cooljeanius (Eric Gallager)]

Replying to kencu:

Eric, exactly what cctools do you have installed, and do you have any macports-clang-* compilers installed?

I have a suspicion different assemblers may be at work here...

$ port installed cctools
The following ports are currently installed:
  cctools @949.0.1_2+llvm10 (active)
$ port installed clang*
The following ports are currently installed:
  clang-9.0 @9.0.1_6+analyzer+emulated_tls+libstdcxx (active)
  clang-10 @10.0.1_7+analyzer+emulated_tls+libstdcxx (active)
  clang-11 @11.1.0_6+analyzer+debug+emulated_tls+libstdcxx (active)
  clang-12 @12.0.1_0+analyzer+debug+libstdcxx+tests (active)
  clang-13 @13.0.1_2+analyzer+libstdcxx (active)
  clang-14 @14.0.6_0+analyzer+debug+libstdcxx+tests (active)
  clang_select @2.2_0 (active)
$ port select clang
Available versions for clang:
	mp-clang-10
	mp-clang-11
	mp-clang-12
	mp-clang-13
	mp-clang-14
	mp-clang-9.0 (active)
	none

[kencu (Ken)]

So Eric I'm not 100% sure what OS version you have going there, but anything fairly recent should be running cctools +xcode to pick up the tools (including the assembler) that Xcode supplies. (Also the ld64 xcode variant, for the same reason). Here is what I have, on Monterey:

% port -v installed | grep cctools
  cctools @949.0.1_2+xcode (active) requested_variants='' platform='darwin 21' archs='noarch' date='2022-07-28T06:58:15-0700'

% port -v installed | grep ld64
  ld64 @3_4+ld64_xcode (active) requested_variants='' platform='darwin 21' archs='x86_64' date='2022-07-28T06:58:16-0700'
  ld64-xcode @2_4 (active) requested_variants='' platform='darwin 21' archs='x86_64' date='2022-07-28T06:58:16-0700'

With what you have installed there, your cctools "as" will look at the macports-clangs that you have installed, and choose one of them to be used as the assembler instead.

Now that *should* be the newest one (clang-14) which should accept that assembler command that is failing -- but because you have "selected" clang-9.0 as your active clang, maybe that one is somehow being chosen above clang-14. Or maybe clang-14 doesn't like that directive, I would have to check to see exactly what likes what when and where.

for sure, clang-9.0 is likely to be too old to understand the x86-pad-for-align=false directive, I would think.