OpenBLAS @0.3.5: error: instruction requires: AVX-512 DQ ISA

[EJFielding (Eric Fielding)]

Tried to install OpenBLAS port on my iMac Pro under MacOS 10.13 with

port install OpenBLAS +gcc7 +lapack

After a while, it fails with an error about a missing instruction:

:info:build /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/.tmp/ccY92TYq.s:70:2: error: instruction requires: AVX-512 DQ ISA
:info:build         vxorps  %zmm0, %zmm0, %zmm0
:info:build         ^

I also tried with the default gcc8, and there were more missing instructions.

Full log is attached.

[EJFielding (Eric Fielding)]

main log

[michaelld (Michael Dickens)]

I'm guessing the issue is that OpenBLAS failed to properly detect CPU features, and enabled AVX512 even though it's not actually available. You can get an idea of what OSX thinks the processor's capabilities are via a command such as sysctl -A | grep -i avx .

Sometimes the "devel" port will work, since it contains fixes since the latest release. Thus, can you try installing the "openblas-devel" port and see if that works?

[michaelld (Michael Dickens)]

Probably has duplicate #57913 , but we're treating them separately at the moment since it's not 100% clear.

[NicosPavlov]

This also seems related to some issues I found, where any "-march" flag will cause the fortran compiler to fail. For example, even "-march=native" causes comparable errors.

[datajerk]

$ sysctl -A | grep -i avx
hw.optional.avx1_0: 1
hw.optional.avx2_0: 1
hw.optional.avx512f: 1
hw.optional.avx512cd: 1
hw.optional.avx512dq: 1
hw.optional.avx512bw: 1
hw.optional.avx512vl: 1
hw.optional.avx512ifma: 0
hw.optional.avx512vbmi: 0

some AVX512 features available, but not all of them.

[smeingast (Stefan Meingast)]

Same problem here on iMac Pro (18-core)

hw.optional.avx1_0: 1
hw.optional.avx2_0: 1
hw.optional.avx512f: 0
hw.optional.avx512cd: 0
hw.optional.avx512dq: 0
hw.optional.avx512bw: 0
hw.optional.avx512vl: 0
hw.optional.avx512ifma: 0
hw.optional.avx512vbmi: 0

[EJFielding (Eric Fielding)]

On my iMac Pro (8 cores), I get this output from the sysctl -A:

hw.optional.avx1_0: 1
hw.optional.avx2_0: 1
hw.optional.avx512f: 1
hw.optional.avx512cd: 1
hw.optional.avx512dq: 1
hw.optional.avx512bw: 1
hw.optional.avx512vl: 1
hw.optional.avx512ifma: 0
hw.optional.avx512vbmi: 0
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 HLE AVX2 BMI2 INVPCID RTM SMAP RDSEED ADX IPT AVX512F AVX512CD AVX512DQ AVX512BW AVX512VL PQM FPU_CSDS MPX PQE CLFSOPT

I see in the log file that it is using this call to the gfortran compiler:

/opt/local/bin/gfortran-mp-7 -O3 -m128bit-long-double -Wall -frecursive -m64 -fPIC -march=skylake-avx512 -c -o sgebrd.o sgebrd.f

[NicosPavlov]

As mentioned by Michael, the *-devel port might be worth trying, as there have been some commits these past few days in OpenBLAS addressing issues in configuring the code to compile AVX512 instructions.

If that does not work, I attached a patch that would provide a quick fix, by disabling the -march flag when calling the gfortran compiler. There is already a patch with this name that addresses a similar issue, so it should be sufficient to just replace the old patch with the new one of the same name.

I do not have the possibility of checking that issue, so I would be grateful if someone could confirm the fix.

[EJFielding (Eric Fielding)]

I tried to install openblas-devel with the same errors. I am not sure how to apply the patch. Can someone give a description?

[NicosPavlov]

The simplest would most likely be to create a ​local repository and make a copy of the port OpenBLAS in it (for a standard installation, it should be found at /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/math). Then it should be enough to replace the patch (in the files folder) with the new patch attached to this ticket.

[smeingast (Stefan Meingast)]

Replying to NicosPavlov:

As mentioned by Michael, the *-devel port might be worth trying, as there have been some commits these past few days in OpenBLAS addressing issues in configuring the code to compile AVX512 instructions.

If that does not work, I attached a patch that would provide a quick fix, by disabling the -march flag when calling the gfortran compiler. There is already a patch with this name that addresses a similar issue, so it should be sufficient to just replace the old patch with the new one of the same name.

I do not have the possibility of checking that issue, so I would be grateful if someone could confirm the fix.

I just tried the *-devel port and it unfortunately did not work. I currently do not have time to try your patch on a local repository, but will do so at a later time. Thanks for your effort!

[NicosPavlov]

Another thing to try, which would also be simpler than the patch, would be to try compiling while ensuring that cctools with the +xcode variant is installed. Based on comments I received on the ​mailing list, it seems that the current default (cctools +llvm70) is causing such issues.

I tried myself, and using cctools+xcode solved similar (but not identical) issues.

[smeingast (Stefan Meingast)]

Replying to NicosPavlov:

Another thing to try, which would also be simpler than the patch, would be to try compiling while ensuring that cctools with the +xcode variant is installed. Based on comments I received on the ​mailing list, it seems that the current default (cctools +llvm70) is causing such issues.

I tried myself, and using cctools+xcode solved similar (but not identical) issues.

I have cctools + xcode installed (cctools @921_0+xcode (active)) and still no luck. ;)

[EJFielding (Eric Fielding)]

Same here. I already had "cctools @921_0+xcode (active)" and it fails to compile OpenBLAS or OpenBLAS-devel.

[cjones051073 (Chris Jones)]

I was able to reproduce a very similar issue (with AVX2 not AVX512) by removing the patch currently in the Portfile that forcibly disables the use of AVX2. This is with the default cctools installed.

If I installed reinstall cctools with +xcode variable, that at least for AVX2 this does fix the problem. The port then builds fine with -mavx2 used for both clang and fortran.

I could be wrong, but I would guess that if this does not work for you, with AVX512, then this is a slightly different (but similar) issue. Perhaps even the Xcode provided toolkit is not up to handling this latest SIMD instruction set. Unfortunately I have no way to test this as I do not have access to an AVX512 capable machine.

Probably though, the fix, at least until the cctools toolkit issues are resolved is to also disable AVX512 in the OpenBLAS portfile.

[kencu (Ken)]

we might need to let openblas run it's tests and enable/disable features internally. why do we override that, again?

[NicosPavlov]

Replying to kencu:

we might need to let openblas run it's tests and enable/disable features internally. why do we override that, again?

As far as I know, we are letting openblas do its test and set features most of the time. The thing is that openblas sets features based on the CPU architecture, and the capability of the compiler might be different from that, hence limiting some features in the Portfile.

[NicosPavlov]

Replying to cjones051073:

I could be wrong, but I would guess that if this does not work for you, with AVX512, then this is a slightly different (but similar) issue. Perhaps even the Xcode provided toolkit is not up to handling this latest SIMD instruction set. Unfortunately I have no way to test this as I do not have access to an AVX512 capable machine.

Probably though, the fix, at least until the cctools toolkit issues are resolved is to also disable AVX512 in the OpenBLAS portfile.

It indeed seems that the issue with AVX2 could be handled by having the right set of tools, and that currently cctools +xcode does that. There is quite some discussion currently to see how to make that more consistent.

In the case of AVX512, it seems that there is still quite some commits upstream to fix bugs related to these instruction sets, so the issue is indeed different.

For the time being, it seems to me that a good (albeit temporary) fix would be to deactivate AVX512 in openblas to ensure compilation, but let it in the *-devel port where there is active development on that matter.

[EJFielding (Eric Fielding)]

For the time being, it seems to me that a good (albeit temporary) fix would be to deactivate AVX512 in openblas to ensure compilation, but let it in the *-devel port where there is active development on that matter.

Yes, I would be happy if the main OpenBLAS port would deactivate the use of the AVX512 enhanced performance option so that people like me can get a working version on the iMac Pro, even if it is somewhat slower. I agree that this may be more a problem in the gfortran compiler than in the OpenBLAS code.

[kencu (Ken)]

Openblas does compile tests in detail for avx512 features in c_check, but I believe we don't let it.

instead, the compilers PG does this simplistic test:

proc avx_compiler_isset {} {
    global configure.cc

    set cc ${configure.cc}

    # check for mpi
    if {[string match *mpi* $cc]} {
        set cc [exec ${configure.cc} -show]
    }

    if {[string match *clang* $cc]} {
        return 1
    }

    return 0
}

It looks to me that the real issue here at the end of it appears to be that Jeremy switched the default variant on newer systems for cctools away from +xcode and back to +llvm70 for some reason, and all this fallout has happened due to that.

[NicosPavlov]

You are then referring to an issue which is possibly relevant, but not directly related to the issue here. The gcc variants in this port are only to select the fortran compiler, and clang is always used by default. The avx_compiler_test you mention is here because as far as I know, clang is the only AVX-capable compiler on OS X. This has at least been true in the past.

The issue happens here because we are letting OpenBLAS make its test. The cctools issue is relevant, and is clearly the issue for instance for the -mavx2 problem mentioned above. However, the avx512 code has been added to Openblas very recently, and the code is very active to fix issues on that, so it seems to me there are also issues upstream.

[cjones051073 (Chris Jones)]

Sorry, but I disagree. I do not see anything wrong with what OpenBLAS is doing w.r.t. enabling AVX512. As far as I can see, from a afar, it is correctly enabling it on systems where it *is* available.

The issue is the fact the assembler, from cctools, does not then know what to do with AVX2/AVX512 instructions.

[NicosPavlov]

I agree that there is indeed nothing wrong with the fact of enabling AVX512 in general.

However, there is a practical difference in the sense that AVX2 can be fixed when cctools +xcode is enabled, while AVX512 cannot.

Furthermore, it is a fact that there have been multiple commits upstream in the past few days to fix and refine AVX512 use and detection.

[kencu (Ken)]

What I see is test compiles for avx512 assembly in the cmake files, with switches toggled if they pass or not. And I don't see this port running any of the cmake tests.

But hey, I don't use this. Just trying to be useful. I'll leave it to you guys who obviously do see the avx512 assembly tests being run somewhere. Might just be my system that isn't running them.

[NicosPavlov]

Thanks, that is in fact useful, as I missed your point. I did not consider the cmake files, as compiling with cmake is still considered experimental based on the comments in the code, but it might be a good thing to look at.

[Nicolas Pavillon <pavillon.nicolas@…>]

In ecf9dcd9de1674db831e23210153ca7b8ed42f60/macports-ports (master):

OpenBLAS: deactivating AVX512 instructions that break build (see #57912)

[NicosPavlov]

I temporarily deactivated AVX512 in OpenBLAS, to allow for a version to build and not block other ports.

I however left AVX512 instructions in "openblas-devel" to enable testing, and also as changes in the code have been made on that matter upstream. I just updated the Portfile to the latest commit.

[EJFielding (Eric Fielding)]

Thanks so much for temporarily disabling the AVX512 option for the main OpenBLAS port. I was able to successfully build it on my iMac Pro, along with the SciPy port that depends on it.

Installing OpenBLAS @0.3.5_0+gcc8+lapack

[kencu (Ken)]

can we close this ticket?

[EJFielding (Eric Fielding)]

The present version of OpenBLAS is working for me. I even migrated to 10.14 and reinstalled everything with no problems. I guess there could be some people who would want the AVX512 instructions used to speed their OpenBLAS calculations. At the moment, they need to use the OpenBLAS-devel version.

[cjones051073 (Chris Jones)]

Maybe its time to try again reenabling the possibility to allow avx512 in the main port, as its been updated a few times since this ticket was first submitted....

[NicosPavlov]

I don't have ways to check directly if avx512 works with the *-devel port, but it is true that we did not get tickets about this. Also, I checked the commit history of OpenBLAS, and the main commits about avx512 are now integrated since version 0.3.6.

[michaelld (Michael Dickens)]

I can't verify whether avx512 or not either. I am planning on updating my "old" 2017 MBP retina/touch (4 cores) to a 2019 model with 8 cores ... thinking it has avx512 ... but I'm not sure ... but it'll be a little time until I commit here. Maybe others with an avx512-capable OSX computer can test the OpenBLAS-devel port to see if it works for them? ... and if so then we try enabling it in the main port && see if anyone complains?

[cjones051073 (Chris Jones)]

EJFielding - Does the OpenBLAS-devel port work correctly for you or not ?

[michaelld (Michael Dickens)]

OpenBLAS is at 0.3.15 and about to be upgraded to 0.3.16. This issue is about 0.3.5, which is pretty old now. I'm closing this as "fixed" since it probably has been in the interim. If anybody has this issue still, please update us with the new build log and we'll see what we can do.