OpenBLAS @0.3.12-universal: Dependencies fail to install on ARM64

[jpanetta (Julian Panetta)]

Installing OpenBLAS on an Apple Silicon Mac fails during installation of the icu dependency's universal variant due to #45268--even when requesting the -universal OpenBLAS variant.

Is it possible to force the -universal variant to propagate to all dependencies?

[jpanetta (Julian Panetta)]

I tried adding -universal to /opt/local/etc/macports/variants.conf as a workaround, but icu+universal is still attempted. I also tried changing universal_archs in macports.conf to only arm64, but this caused other problems...

[kencu (Ken)]

what do you already have installed as +universal?

port -v installed | grep universal

I think, for today, trying to install +universal on an Apple Silicon Mac will be difficult.

[jpanetta (Julian Panetta)]

$ port -v installed | grep universal
  bzip2 @1.0.8_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-22T18:58:15-0800'
  gettext @0.19.8.1_2+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-23T14:02:37-0800'
  gperf @3.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-23T13:56:40-0800'
  libedit @20191231-3.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-22T18:57:29-0800'
  libffi @3.3_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-22T18:58:11-0800'
  libiconv @1.16_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-23T13:57:23-0800'
  libtapi @1000.10.8_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-22T18:56:21-0800'
  libtool @2.4.6_10+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-22T18:57:43-0800'
  ncurses @6.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-22T18:57:09-0800'
  xz @5.2.5_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-23T14:03:01-0800'
  zlib @1.2.11_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2020-11-23T14:03:05-0800'

[jpanetta (Julian Panetta)]

I would be happy to install a non-universal builds of everything, but I've not managed to do that. Are you suggesting it'll help to uninstall everything that's built with +universal so far?

[jpanetta (Julian Panetta)]

I reinstalled all of those +universal ports so that now they're listed as -universal. However, when I try to install OpenBLAS now, the +universal variants are still getting installed:

$ sudo port install OpenBLAS
--->  Computing dependencies for OpenBLAS
The following dependencies will be installed:
 bzip2
 gcc10
 gettext
 icu
 ld64
 ld64-latest
 libedit
 libffi
 libgcc
 libgcc10
 libiconv
 libmpc
 libtapi
 libtool
 libxml2
 llvm-9.0
 ncurses
 openssl
 xar
 xz
 zlib
Continue? [Y/n]:
--->  Fetching archive for libtapi
--->  Attempting to fetch libtapi-1000.10.8_1+universal.darwin_20.arm64-x86_64.tbz2 from https://packages.macports.org/libtapi
...

[kencu (Ken)]

well -- yeah, I would think so.

Eventually, almost everything should build universal -- but on 23 Nov 2020, I think you will run into lots of potential headaches building universal. In three months, the situation will no doubt look better.

For example, I have not yet enabled meson to build anything universal with arm64 when using the muniversal portgroup, so until I (or someone) does enable that, lots of universal builds might fail.

We need to figure out what you are trying to install that is starting this cascade. What are you after just now?

I would deactivate everything

sudo port -f deactivate active

and then try to install whatever it is you are after

sudo port -v install MYPORT

and see if +universal gets called in. Hopefully not, but if so, we need to figure out what is doing that and stop it from doing that.

[jpanetta (Julian Panetta)]

Thanks, I will try this. Should I remove -universal from my variants.conf before doing this?

[kencu (Ken)]

If you leave -universal in your variants.conf your life will be easier because nothing should build universal.

However, we will not know what started the cascade, but that is our problem.

So -- up to you -- leave it in for easy life and get on with your work. Take it out to figure out what started the mess.

[jpanetta (Julian Panetta)]

If only it were so easy ;). I left -universal in my variants.conf but +universal ports are still being installed with a sudo port install OpenBLAS, the first of which appears to be bzip2-1.0.8_0+universal.darwin_20.arm64-x86_64.tbz2.

This happens after either sudo port -f deactivate active or sudo port uninstall all. Do you have any suggestions for how I can track down what's causing this?

[jpanetta (Julian Panetta)]

Ok, I think I've narrowed things down somewhat. Installing llvm-9.0 tries to bring in universal variants of its dependencies, while manually installing each dependency gets a non-universal variant. So I guess this means the problem originates in llvm-9.0?

[jpanetta (Julian Panetta)]

Aha! The culprit appears to be the supported_archs commands here:

​https://github.com/macports/macports-ports/blob/b2bc6da4d98143000e7534de4f7a931c60eca3a4/lang/llvm-9.0/Portfile#L271 ​https://github.com/macports/macports-ports/blob/b2bc6da4d98143000e7534de4f7a931c60eca3a4/lang/llvm-9.0/Portfile#L398

I commented out the first and added arm64 to the second, and now +universal is not propagated. I'm trying to build this now with fingers crossed.

[kencu (Ken)]

Yes -- we'll have to clean that all up. LLVM-9.0 might build arm64; it's pretty well written that way. clang-9.0 might also, but compiler_rt in clang-9.0 needs some cleanup to build for arm64.

What is asking for llvm-9.0? ld64 should want the +xcode variant, so not that.

I have llvm-devel just finishing an update now, and if it succeeds I'll push that tonight. That is the most likely thing that will build on arm64 right now, and I guess I'll pull out the supported_archs from that and let-'er-rip and see what happens.

I had llvm-11 / clang-11 all set to go but got into a huge fight (once again) with the GitHub PortGroup and the GitHub PortGroup won, so I gave up on that for now while I figure out how I'm going to defeat it.

That should probably build i386/x86_64/arm64 too.

[jpanetta (Julian Panetta)]

Changes to ld64 Makefile to support ld-530

[jpanetta (Julian Panetta)]

Changes to ld64 portfile to support arm64

[jpanetta (Julian Panetta)]

I think it actually was ld64 asking for llvm-9.0, and it's gcc10 asking for ld64.

llvm-9.0 actually built fine for me after those changes I mentioned above. But then ld64 also was missing arm64, and adding it was significantly trickier (I had to update to ld64-530, since backporting arm64 support to ld64-450 looked too difficult). I was able to get 530 working with the attached hacked-together patches. One surprise is the rebase binary seems to have been dropped.

Walking up the dependency tree, now I'm blocked by gcc10, which just says:

error:fetch gcc10 10.2.0 is not supported on Darwin 20 arm
error:fetch Failed to fetch libgcc10: incompatible OS X version
debug:fetch Error code: NONE
debug:fetch Backtrace: incompatible OS X version

:(

[kencu (Ken)]

Oh dear; you should not be using anything other than ld64 +ld64_xcode on newer systems.

ld64 +ld64_xcode doesn't build or need llvm -- it is just a few symlinks into xcode.

If you presently have ld64 installed with any variant other than ld64 +ld64_xcode please do this:

sudo port -f uninstall ld64
sudo port -v install ld64

and you hopefully will get ld64 +ld64_xcode

if you do not get that, -- oops, we have some work to do (which we might well have to do).

By the way, gcc10 will not build for arm64 at present. We do have a nearly-working fork of gcc-devel that is written by GCC's darwin lead that we are desperately trying to get working on Apple Silicon.

At this moment, it builds neat, on the system, but does not build inside MacPort's infrastructure, so that is where we are with that at present.

[kencu (Ken)]

when it comes to ld64-latest, I update that every once in a while -- the trick is it has to / should build all way back to SnowLeopard (it does at present) so I always spend some time working on that.

The only systems that really need ld64-latest are the ones where ld64_xcode is too old for the current SDKs we want to build against. That is not really happening on any systems right now, but if we want, for example, to build against the MacOSX11.0.sdk on Mojave, we might have an issue there. I'll likely have to puff up ld64 for something like that.

[kencu (Ken)]

I have pushed up a new build of llvm/clang/lldb-devel that -- I hope -- will properly handle the supported_archs issue with respect to Apple Silicon and every other system.

[jpanetta (Julian Panetta)]

Thanks, I wasn't aware of that. sudo port install ld64 does not select the +ld64_xcode variant for me, but rather tries to install ld64-latest, hence my efforts to get that building.

Thanks for the information on gcc10!

[kencu (Ken)]

well, one more thing I need to fix. On Apple Silicon, we need to get this to happen:

$ port info ld64
ld64 @3_3 (devel)
Sub-ports:            ld64-97, ld64-127, ld64-236, ld64-274, ld64-latest,
                      ld64-xcode
Variants:             ld64_127, ld64_236, ld64_274, ld64_97, [+]ld64_xcode,
                      universal

[kencu (Ken)]

This part in the ld64 Portfile is supposed to force the ld64_xcode variant for you:

    # Xcode 11 has a newer ld64 than MacPorts currently supplies, so we use it
    if {[vercmp $xcodeversion 11.0] >= 0} {
        default_variants +ld64_xcode
    }

I wonder how it could be that it is not doing that? I haven't seen that issue on Big Sur intel, which gives me this:

% port info ld64
ld64 @3_3 (devel)
Sub-ports:            ld64-97, ld64-127, ld64-236, ld64-274, ld64-latest,
                      ld64-xcode
Variants:             ld64_127, ld64_236, ld64_274, ld64_97, [+]ld64_xcode,
                      universal

[jpanetta (Julian Panetta)]

Probably because I didn't install XCode (just the command line tools)! $xcodeversion for me is none. Sorry for wasting your time with that--downloading the full XCode now. Interestingly, ld64_xcode does still work fine for me if I force it to install.

[kencu (Ken)]

Well -- that is my fault then, for not setting the portfile for ld64 such that it forces the need for Xcode, or otherwise considering that option.

That is worth knowing. We like those xcodeversion tests for certain things, but they are obviously painfully unreliable...

[kencu (Ken)]

In 9a05eaec189f863713e8f8257fe551c1e336fa98/macports-ports (master):

ld64: default to ld64_xcode with system test

we previously used an xcodeversion test to see how new the
system ld64 was, but this test fails when users have only
installed the CLTs, which we are trying to support.

So... we have to use a system version test instead here.

see: #61636#comment:20

[jpanetta (Julian Panetta)]

Cool, thanks! My Xcode installation finally finished, and I can confirm the original check works on Apple Silicon--but it's nice to know I can remove it if my drive fills up.

So I guess I'm stuck for now until gcc-devel is working. Thanks again for all the help!

[kencu (Ken)]

You seem agile enough: the working gcc is here <​https://github.com/iains/gcc-darwin-arm64> if you want to help out.

First build it from scratch, using MP gmp and mpfr or your own.

Once you get that working, you can use it if you like (couple of symlinks and you can hack your way in).

Then we have to get the gcc-devel Portfile to build it and install it into MacPorts prefixes properly... and that is where we are now.

[jpanetta (Julian Panetta)]

I was able to build that gcc branch using the attached Portfile with only a couple changes from the existing gcc-devel version. The main change was adding depends_build port:texinfo to work around a malloc bug in the system /usr/bin/makeinfo.

Unfortunately now it's failing at what I assume is the post-destroot phase:

:debug:destroot Executing proc-post-org.macports.destroot-destroot-0
:debug:destroot PortGroup select: Installing select files to destroot
:info:destroot xinstall: mkdir /opt/local/var/macports/build/_usr_local_ports_lang_gcc-develarm/gcc-develarm/work/destroot/opt/local/etc/select
:info:destroot xinstall: mkdir /opt/local/var/macports/build/_usr_local_ports_lang_gcc-develarm/gcc-develarm/work/destroot/opt/local/etc/select/gcc
:error:destroot Failed to destroot gcc-develarm: xinstall: Cannot stat: /usr/local/ports/lang/gcc-develarm/files/mp-gcc-develarm, No such file or directory
:debug:destroot Error code: NONE
:debug:destroot Backtrace: xinstall: Cannot stat: /usr/local/ports/lang/gcc-develarm/files/mp-gcc-develarm, No such file or directory
:debug:destroot     while executing
:debug:destroot "$post $targetname"
:error:destroot See /opt/local/var/macports/logs/_usr_local_ports_lang_gcc-develarm/gcc-develarm/main.log for details.

though I can't figure out what could be looking for mp-gcc-develarm (all my generated binaries look like gcc-mp-develarm). Also, I tried inserting print statements at the top of post-destroot that do not seem to run. Is this roughly where you guys were stuck?

[jpanetta (Julian Panetta)]

Portfile for building gcc11 on arm

[kencu (Ken)]

you did well!

Michael -- some progress here.

the file you're missing is a macports created file for our "select" process. It has to have a specific name, and sit in the "files" dir under the Portfile.

See this one for the template <​https://github.com/macports/macports-ports/blob/master/lang/gcc-devel/files/mp-gcc-devel>

looks like you're nearly there!

btw, when tweaking destrooting, you don't have to rebuild all of gcc; just delete the destroot folder in the work directory and run the destroot again until you get it right.

[kencu (Ken)]

Michael has the Apple Silicon system and has been running point on getting that gcc into MP; I suspect you've just given some wind into his sails as he's been building it outside but has not been able to get it to build inside MacPorts up to now...

[kencu (Ken)]

I'm thinking that we might just leave this port as being named gcc-devel, but have it use that github repo when building on arm64. That way we won't wind up with a slight mess to clean up later --we'll see what Michael thinks about that too....

[jpanetta (Julian Panetta)]

Great, thanks, with this file it installs! I should have read the missing file path more carefully...

I'm trying to build and install it again after uninstalling libgcc and gcc and cleaning everything. Hopefully it all works now!

[kencu (Ken)]

the issue you will possibly run into is getting MP to actually use it for builds...there is no logic in the portfiles to spec a port named gcc-develarm so the portfiles won't know how to ask for it, exactly, or default to it.

doing this will help a bit to get you using it

sudo port select gcc gcc-develarm

but that will not be enough for the portfiles to find it.

We'll ask Chris.

[kencu (Ken)]

Chris, our intrepid explorer here has done the deed and installed the arm version of gcc on MacPorts.

Can you help us sort out how we might most easily use it, until gcc-11 comes out?

Thanks!

[cjones051073 (Chris Jones)]

I am not sure creating a new port, 'develarm' is necessary here. Why not just update the existing port gcc-devel to (temporarily) use the new branch ? Once the changes are merged with upstream, and/or gcc11 merged, then gcc-devel can just be switched back to the normal snapshots.

[cjones051073 (Chris Jones)]

I presume the Arm branch builds on both Intel and Arm, right ?