OpenBLAS @0.3.5_1 does not build on PPC Leopard, Mac OS X 10.5.8, because Makefile. cannot be found

[ballapete (Peter "Pete" Dyballa)]

--->  Building OpenBLAS
DEBUG: Executing proc-pre-org.macports.build-build-0
Warning: the chosen compiler cannot handle advanced optimisation instructions.
         AVX instructions are disabled.
DEBUG: Executing org.macports.build (OpenBLAS)
DEBUG: Environment: 
CC_PRINT_OPTIONS='YES'
CC_PRINT_OPTIONS_FILE='/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/.CC_PRINT_OPTIONS'
CPATH='/opt/local/include'
LIBRARY_PATH='/opt/local/lib'
MACOSX_DEPLOYMENT_TARGET='10.5'
Executing:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.5" && /usr/bin/make -w all AR=/opt/local/bin/ar RANLIB=/opt/local/bin/ranlib 
DEBUG: system:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.5" && /usr/bin/make -w all AR=/opt/local/bin/ar RANLIB=/opt/local/bin/ranlib 
make: Entering directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.5'
Makefile.system:1069: Makefile.: No such file or directory
make: *** No rule to make target `Makefile.'.  Stop.
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.5'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_nue.de.rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.5" && /usr/bin/make -w all AR=/opt/local/bin/ar RANLIB=/opt/local/bin/ranlib 

This directory has:

-rw-r--r--   1 macports  admin   10538 31 Dez 23:09 Makefile
-rw-r--r--   1 macports  admin    1110 31 Dez 23:09 Makefile.alpha
-rw-r--r--   1 macports  admin     460 31 Dez 23:09 Makefile.arm
-rw-r--r--   1 macports  admin     987 31 Dez 23:09 Makefile.arm64
-rw-r--r--   1 macports  admin    1184  6 Mär 23:05 Makefile.conf
-rw-r--r--   1 macports  admin      30 31 Dez 23:09 Makefile.generic
-rw-r--r--   1 macports  admin     863 31 Dez 23:09 Makefile.ia64
-rw-r--r--   1 macports  admin    8763  6 Mär 23:04 Makefile.install
-rw-r--r--   1 macports  admin      26 31 Dez 23:09 Makefile.mips
-rw-r--r--   1 macports  admin      26 31 Dez 23:09 Makefile.mips64
-rw-r--r--   1 macports  admin    2876 31 Dez 23:09 Makefile.power
-rw-r--r--   1 macports  admin    1601 31 Dez 23:09 Makefile.prebuild
-rw-r--r--   1 macports  admin     176  6 Mär 23:04 Makefile.rule
-rw-r--r--   1 macports  admin     819 31 Dez 23:09 Makefile.sparc
-rw-r--r--   1 macports  admin   24868  6 Mär 23:04 Makefile.system
-rw-r--r--   1 macports  admin   21482 31 Dez 23:09 Makefile.tail
-rw-r--r--   1 macports  admin    1836 31 Dez 23:09 Makefile.x86
-rw-r--r--   1 macports  admin    3764 31 Dez 23:09 Makefile.x86_64
-rw-r--r--   1 macports  admin     100 31 Dez 23:09 Makefile.zarch

Line #1069 has:

include $(TOPDIR)/Makefile.$(ARCH)

[ballapete (Peter "Pete" Dyballa)]

Main.log from PPC Leopard

[kencu (Ken)]

partial fix

[kencu (Ken)]

the above patch fixes the architecture recognition issues, and gets the build on PowerPC Leopard (at least -- not tested on Tiger yet) quite a long way.

Then there is an issue with something else that is not so easy to fix, at least for me at this moment:

/opt/local/bin/gcc-mp-6 -c -O3 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m32 -DF_INTERFACE_GFORT -fPIC -DNO_AVX -DNO_AVX2 -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DMAX_CPU_NUMBER=2 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"1\" -force_cpusubtype_ALL -DASMNAME=_sgemm -DASMFNAME=_sgemm_ -DNAME=sgemm_ -DCNAME=sgemm -DCHAR_NAME=\"sgemm_\" -DCHAR_CNAME=\"sgemm\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX gemm.c -o sgemm.o
/opt/local/bin/gcc-mp-6 -c -O3 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -m32 -DF_INTERFACE_GFORT -fPIC -DNO_AVX -DNO_AVX2 -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DMAX_CPU_NUMBER=2 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"1\" -force_cpusubtype_ALL -DASMNAME=_ssymm -DASMFNAME=_ssymm_ -DNAME=ssymm_ -DCNAME=ssymm -DCHAR_NAME=\"ssymm_\" -DCHAR_CNAME=\"ssymm\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX symm.c -o ssymm.o
In file included from ../common.h:536:0,
                 from gemm.c:41:
gemm.c: In function 'sgemm_':
../common_param.h:981:18: error: 'SGEMM_DEFAULT_P' undeclared (first use in this function)
 #define SGEMM_P  SGEMM_DEFAULT_P
                  ^
../common_param.h:1117:18: note: in expansion of macro 'SGEMM_P'
 #define GEMM_P   SGEMM_P
                  ^~~~~~~
gemm.c:414:37: note: in expansion of macro 'GEMM_P'
   sb = (XFLOAT *)(((BLASLONG)sa + ((GEMM_P * GEMM_Q * COMPSIZE * SIZE + GEMM_ALIGN) & ~GEMM_ALIGN)) + GEMM_OFFSET_B);
                                     ^~~~~~
../common_param.h:981:18: note: each undeclared identifier is reported only once for each function it appears in
 #define SGEMM_P  SGEMM_DEFAULT_P
                  ^
../common_param.h:1117:18: note: in expansion of macro 'SGEMM_P'
 #define GEMM_P   SGEMM_P
                  ^~~~~~~
gemm.c:414:37: note: in expansion of macro 'GEMM_P'
   sb = (XFLOAT *)(((BLASLONG)sa + ((GEMM_P * GEMM_Q * COMPSIZE * SIZE + GEMM_ALIGN) & ~GEMM_ALIGN)) + GEMM_OFFSET_B);
                                     ^~~~~~
make[1]: *** [sgemm.o] Error 1

[kencu (Ken)]

See <​https://github.com/xianyi/OpenBLAS/pull/2046>.

Now to fix the missing definitions for the GEMM defines in param.h. Help would be appreciated!

[NicosPavlov]

I unfortunately don't have the ability of testing compilation with a PPC CPU, but it seems from other reports that the problem comes from the fact that the architecture is not fully defined.

This is just a guess at this point, but I wonder if there is not a need to set for instance the TARGET on top of ARCH to allow definitions to be set.

See (as an example, it can't be just ported directly):
​https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=218547
and the corresponding commit:
​https://svnweb.freebsd.org/ports?view=revision&revision=444695

[kencu (Ken)]

Yes, that helped -- thanks for those links.

I had tried adding TARGET=PPC970 last night on my DualG5s, but it was not working for some reason. But (for the moment) disabling the Makefile.rule generation and building like this did work:

build.args          "AR=${prefix}/bin/ar RANLIB=${prefix}/bin/ranlib CC=${configure.cc} FC=${configure.fc} TARGET=PPCG4"

I guess the automated CPU/Core detection routines in the cpuid-* C files must not be picking up the CPU correctly on PPC Darwin. Maybe that can be fixed. I'll check.

If I can't fix that, then I'll see if I can come up with a way to tweak the Portfile for PPC.

[michaelld (Michael Dickens)]

@kencu: How do get the compiler selection to work? I'm trying to build on my PPC G5 box & port is making openblas-devel depend on clang-7.0 regardless of variant selection!

[michaelld (Michael Dickens)]

I just needed to specify the configure.compiler=macports-gcc-6 on the CLI. Moving along now ...

[michaelld (Michael Dickens)]

Yes it looks like CPU detection isn't working for PPC. If I tweak the Portfile in the pre-build to do puts $makeINC "TARGET = PPCG4", along with the __POWERPC__ patch & specifying the configure.compiler, then it seems to be building successfully. Not ideal, but viable until better fixes come along.

[kencu (Ken)]

For me, gcc6 came right up without specifying.

Destroot needs something; lib has full name, not *-r1 name...

But builds through.

[kencu (Ken)]

I tweaked the destrooting in the Portfile to match conditions on the ground, and got this:

$ port -v installed openblas
The following ports are currently installed:
  OpenBLAS @0.3.5_1+gcc6+lapack (active) platform='darwin 9' archs='ppc' date='2019-03-07T07:47:02-0800'

$ port contents openblas
Port OpenBLAS contains:
  /opt/local/include/cblas_openblas.h
  /opt/local/include/f77blas.h
  /opt/local/include/lapacke.h
  /opt/local/include/lapacke_config.h
  /opt/local/include/lapacke_mangling.h
  /opt/local/include/lapacke_utils.h
  /opt/local/include/openblas_config.h
  /opt/local/lib/libopenblas-r0.3.5.a
  /opt/local/lib/libopenblas-r0.3.5.dylib
  /opt/local/lib/libopenblas.a
  /opt/local/lib/libopenblas.dylib
  /opt/local/lib/pkgconfig/openblas.pc

I assume you guys changed the r0.3.5 to r1 for some reason, but that didnt' make it into this build...probably in the patchfiles?

[kencu (Ken)]

FYI - __POWERPC__ patch has been committed upstream already, so we can disappear that eventually. Good to hear that forcing the Target in the $makeINC does it. Maybe this will be easier than I thought...

[kencu (Ken)]

Oh I see. In $makeINC you guys changed the VERSION to 1, but I deleted that for my tests. my bad.

[kencu (Ken)]

Looks like TARGET=PPC970 is not working. Only PPCG4 is getting past the gauntlet at present for me.

        if { ${build_arch} eq "ppc" || ${build_arch} eq "ppc64"} {
#            puts $makeINC "TARGET = PPCG4"
            puts $makeINC "TARGET = PPC970"
        }

[michaelld (Michael Dickens)]

My build is still going ... single core G5 ... takes forever to build much of anything useful, but it still works after ~20 years!

[michaelld (Michael Dickens)]

Guessing if you grep for PPCG4 in the source you'll find some code where we might be able to add in for PPC970 and/or others ... just a guess since PPCG4 works.

[ballapete (Peter "Pete" Dyballa)]

Replying to michaelld:

My build is still going ... single core G5 ... takes forever to build much of anything useful, but it still works after ~20 years!

Cannot be! The first PowerMac G5 came out in June 2003.

[ballapete (Peter "Pete" Dyballa)]

Replying to kencu:

Looks like TARGET=PPC970 is not working. Only PPCG4 is getting past the gauntlet at present for me.

        if { ${build_arch} eq "ppc" || ${build_arch} eq "ppc64"} {
#            puts $makeINC "TARGET = PPCG4"
            puts $makeINC "TARGET = PPC970"
        }

I think it's not G4, because G4 is not 64 bit, but the "G" is a "6"… So G5 seems to be supported but not G4. (Or G3.)

[kencu (Ken)]

So, it works, with the __POWERPC__ patch, and that little force in $makeINC. That was surprisingly easy, in the end.

$ port -v installed openblas
The following ports are currently installed:
  OpenBLAS @0.3.5_1+gcc6+lapack (active) platform='darwin 9' archs='ppc' date='2019-03-07T09:03:49-0800'

The CPU autodetection for PPC is not working, but forcing PPCG4 does work. Forcing PPC970 even on a ppc970 machine does not work, for reasons unclear at present.

Anyone know how to run the test-suite? It builds without a hiccup, so presumably it would pass, but always nice to see.

[michaelld (Michael Dickens)]

I just updated OpenBLAS-devel to the latest [db3dc9e2 (20190307)], which includes your __POWERPC__ fix, Ken. We now need to patch just the release. And I think the puts command will be common to both subports.

I don't know how to run tests ... still playing around to see how to do it!

[kencu (Ken)]

I think this might be the 970 issue, in param.h some #defines seem to be missing, or may be guarded by a Linux blocker:

#ifdef PPC970

#define SNUMOPT		4
#define DNUMOPT		4

#define GEMM_DEFAULT_OFFSET_A 2688
#define GEMM_DEFAULT_OFFSET_B 3072
#define GEMM_DEFAULT_ALIGN 0x03fffUL

#define SGEMM_DEFAULT_UNROLL_M 16
#define SGEMM_DEFAULT_UNROLL_N 4
#define DGEMM_DEFAULT_UNROLL_M 4
#define DGEMM_DEFAULT_UNROLL_N 4
#define CGEMM_DEFAULT_UNROLL_M 8
#define CGEMM_DEFAULT_UNROLL_N 2
#define ZGEMM_DEFAULT_UNROLL_M 2
#define ZGEMM_DEFAULT_UNROLL_N 2

#ifdef OS_LINUX
#if L2_SIZE == 1024976
#define SGEMM_DEFAULT_P 320
#define DGEMM_DEFAULT_P 256
#define CGEMM_DEFAULT_P 256
#define ZGEMM_DEFAULT_P 256
#else
#define SGEMM_DEFAULT_P 176
#define DGEMM_DEFAULT_P 176
#define CGEMM_DEFAULT_P 176
#define ZGEMM_DEFAULT_P 176
#endif
#endif

#define SGEMM_DEFAULT_Q 512
#define DGEMM_DEFAULT_Q 256
#define CGEMM_DEFAULT_Q 256
#define ZGEMM_DEFAULT_Q 128

#define SYMV_P	 4

#endif

[michaelld (Michael Dickens)]

Ah: make tests looks like it's a starting point.

[kencu (Ken)]

I think that's it. If I change param.h and do this on line 2002:

- #ifdef OS_LINUX
+ #if defined(OS_LINUX) || defined(OS_DARWIN)

then the build seems to pick up the PPC970 without forcing the Target, and the build is proceeding past the previous areas of wreckage.

If wonder if the #ifdef OS_LINUX guard is needed at all?

[kencu (Ken)]

So the way I'm putting this together is that the build was correctly detecting the PPC970, but the PPC970 build was broken, so it only worked for us if we forced a PPCG4 build.

The above patch (comment:22) fixes the PPC970 build, so -- no more forcing needed.

Edit: spoke too soon: WAAAAYYYY into the build, there is an assembly error. TBA.

[kencu (Ken)]

This 970 patch seems to do the right things to properly build with the 970. I am finding that with this patch, I don't have to force the TARGET to PPCG4 in the Portfile any longer -- the openblas logic handles it correctly now.

I'll try it on a G4 machine I have as well. Builds on Tiger too, if you use gmake instead of /usr/bin/make -- that's quite nice to find.

Just getting ready to send this patch upstream as well, as they seem receptive.

[kencu (Ken)]

confirmed it selects the right configuration automatically on a G4 7450 as well. There is an upstream PR up for the second patch as well. With these no Target forcing is needed.

[michaelld (Michael Dickens)]

See also: ​https://github.com/macports/macports-ports/pull/3815

[kencu (Ken)]

In cd5c8b25d01e5c82e8379d49261f518f696765ad/macports-ports (master):

OpenBLAS: add PowerPC support

closes: #58179