python38 +universal fails to build on Big Sur (Apple Silicon Mac)

[Gregory-Gelfond (Gregory Gelfond)]

sudo port install chez-scheme yields the following output:

The following dependencies will be installed: 
 icu
 libxml2
 python38
 xorg-libX11
 xorg-libpthread-stubs
 xorg-libxcb
 xorg-util-macros
 xorg-xcb-proto
 xorg-xtrans
Continue? [Y/n]: 
--->  Fetching archive for python38
--->  Attempting to fetch python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2 from https://packages.macports.org/python38
--->  Attempting to fetch python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2 from https://ywg.ca.packages.macports.org/mirror/macports/packages/python38/python38
--->  Attempting to fetch python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2 from https://lil.fr.packages.macports.org/python38
--->  Fetching distfiles for python38
--->  Attempting to fetch Python-3.8.6.tar.xz from https://distfiles.macports.org/python38
--->  Verifying checksums for python38                                          
--->  Extracting python38
--->  Applying patches to python38
--->  Configuring python38
--->  Building python38
Error: Failed to build python38: command execution failed
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python38/python38/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port chez-scheme failed

The error appears to be with building python38.

This is strange since installing python38 with the +optimizations variant worked fine. Chez-scheme seems to be calling for a universal build variant. I am attaching the main.log for reference.

[jmroot (Joshua Root)]

​https://bugs.python.org/issue41100

[jmroot (Joshua Root)]

BTW, chez-scheme's dependencies are being installed with +universal because it's marked as only supporting i386 and x86_64 architectures. So it can only be built for x86_64 on an Apple Silicon machine and has to run with Rosetta 2.

[julesberry]

I have the same problem with installing dependencies for R. I also can install python38 +optimizations, but R is requiring python38 +universal (python38-3.8.6_0+universal.darwin_20.arm64-x86_64.tbz2).

Similar problem (in the sense that +universal wont install) with port icu, which is discussed in ticket #45268 and #61704.

Is there a way around this or are we waiting on the developers of R to update?

[kencu (Ken)]

somebody is going to have to fix icu to build universal properly.

this has been a problem for years, but we got away without fixing it when universal was two flavors of Intel.

Now it is not.

[julesberry]

I feel so stupid, but there’s a simple workaround for users ... compile everything under Rosetta 2 ! You can either run your Terminal from Rosetta 2. Or do commands as:

arch -x86_64 sudo port install R

It’s good too - code runs about twice as fast as 2 or 3 year old 13 inch equivalent MacBook Pro!

[arthurcnorman]

I have just got a new macbook and am also hit by this. "port install python38 +universal" fails and on inspection most of its concept of "universal" goes back to the transition from ppc to intel and 32 bit to 64-bit. The reason this hits me is that I wish to build my own code as univrsal so want arm64 & x86_64 versions of libraries installeds, so I try eg the xorg libraries in universal mode and that picks up pything38 as a dependency and tries for a universal build there even though what architecture tools run on does not matter a lot to me. I can edit the python configure files and sources to get a version that builds, but not having dug that deep into macports before I do not know how to get that instralled so that macports believes it has a universal python that satisfies the depoendencies of other things... Eg when I go "port destroot" that seems to repeat earlier steps and wipes out my edits. Is there wither an explanation of how to make a local version of a package so I can patch it, or how I can get libraries that are to be build for universal to accept a non-universal version of python38?. If there is a sectio in TFM that I can R then a pointer towards it will be duly appreciated! Arthur

[kencu (Ken)]

python38 is not working very well on arm64. python39 is better.

building anything +universal x86_64 and arm64 is very rudimentary just now. You can expect to help us fix many issues in so doing.

[arthurcnorman]

It is very reasonable that the arm64+x86_64 +universal is still new and fragile. I am happy to try to identify issues and suggest fixes in the ones I try - but it would help to understand the protocol for testing and installing a modified version of a port without (eg) "port install" tending to revert it back to the official version that I wish to change. I am certain that those used to macports internals will view this as a dire newby question, but while I have a a decent number of years hacking autoconf scripts etc I have not looked at how they are driven my macports before, and when I have had troiuble with individual ports I have tended to fetch source and edit/configure/make as a separate activirt (eg alpine JUST need "#include <time.h>" stuck in at the front of half a dozen files to be able to build, but I do not know how to merge that in with the port system and pass it back as a proposed fix). Thanks and I am really rather impressed with how much you do have working and appearing effortless on Big Sur aarch64.

[kencu (Ken)]

I decided to get started, with Michael, on seeing how this goes. So far:

% /opt/universal/bin/port -v installed
The following ports are currently installed:
  autoconf @2.69_5 (active) platform='darwin 20' archs='noarch' date='2021-01-09T21:44:39-0500'
  automake @1.16.3_0 (active) platform='darwin 20' archs='noarch' date='2021-01-09T21:44:46-0500'
  curl-ca-bundle @7.74.0_0 (active) platform='darwin 20' archs='noarch' date='2021-01-09T21:44:33-0500'
  gettext @0.19.8.1_2+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:35:04-0500'
  gperf @3.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:26:47-0500'
  libiconv @1.16_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:27:47-0500'
  libtool @2.4.6_11+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:45:02-0500'
  ncurses @6.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:26:16-0500'
  openssl @1.1.1i_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:43:05-0500'
  xz @5.2.5_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:36:56-0500'
  zlib @1.2.11_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-09T21:37:01-0500'

and we're just getting started.

BTW -- MacPorts has wonderful universal support in base and most ports. Homebrew has nil, for example, and as far as I know, no other pkg manager for MacOS has any either.

[arthurcnorman]

Quite a lot of the ports survice +universal already. I had trouble with icu (but I am not quite certain if that was real or an accident and of the ports I tried that blocks

    at-spi2-atk                     @2.38.0_0
    at-spi2-core                    @2.38.0_0
    atk                             @2.36.0_1
    brotli                          @1.0.9_0
    cairo                           @1.16.0_0+quartz+x11
    gdk-pixbuf2                     @2.42.2_0+x11
    glib2                           @2.58.3_1+x11
    gnutls                          @3.6.15_0
    gobject-introspection           @1.60.2_4
    libepoxy                        @1.5.4_1+python38
    harfbuzz                        @2.7.4_0
    harfbuzz-icu                    @2.7.4_0
    graphite2                       @1.3.13_1
    gtk3                            @3.24.23_0+x11
    libarchive                      @3.5.1_0
    libcerf                         @1.13_0
    libxml2                         @2.9.10_1
    libxslt                         @1.1.34_4
    p11-kit                         @0.23.21_0
    pango                           @1.42.4_2+quartz+x11

that are tagged as depending on it. pythin38 is the other big blocker and the "38" is because that is what other ports seemsd to say the depended on, sof for instance

  python38:
    cyrus-sasl2                     @2.1.27_2+kerberos
    fontconfig                      @2.13.1_1
    freetype                        @2.10.4_0
    gd2                             @2.3.0_0+x11
    xorg-libXt                      @1.2.0_0
    xpm                             @3.5.13_0
    kerberos5                       @1.18.3_0
    mesa                            @17.1.6_2+osmesa+python27
    woff2                           @1.0.2_0
    Xft2                            @2.3.3_0
    xorg-libX11                     @1.7.0_0

and a p[ile more X11 things clog there. jpeg also did not work first time and blocked tiff, webp etc. The only other one my scan showed up was osso-uuid and I do not know what tried to pull that in.

That is a short list compatred with the list of all things that worked. And my bet is that if building say xorg-libX11 knew it only depended on an executable version of python on whatev4er the current architecture and did not need to ask for a universal one then many more things woold sail through very happily. and harfbuzz-icu is maybe an issue and may block texlive-bin

It is great that on the m1 compilation is respactably fast! And let me repeat how pleased I am about all ths things that work, so this is not a complaint it is a report.

[kencu (Ken)]

the system python on BigSur is universal, so it can be done...MacPorts does not make it easy for ports to just use that, however.

Some ports may use python to just process scripts; there is a way to avoid a universal python there by telling the Portfile to skip the architecture check in that case for that port.

But others will need a real universal python, so that indeed seems to be the next hurdle.

icu has been difficult to build universal for a while...and then we will tackle gcc...

[kencu (Ken)]

I hacked together a universal build of python39 here #62023 but it needs to be automated and then tested, as Ryan has found some messup with universal pythons that he referenced in another ticket #61282

[kencu (Ken)]

Now all the way up to curl, and python39 as well:

% port -v installed | grep active
  autoconf @2.69_5 (active) platform='darwin 20' archs='noarch' date='2021-01-10T12:19:44-0800'
  automake @1.16.3_0 (active) platform='darwin 20' archs='noarch' date='2021-01-10T12:19:44-0800'
  bzip2 @1.0.8_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:38-0800'
  curl @7.74.0_0+ssl+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T13:24:31-0800'
  curl-ca-bundle @7.74.0_0 (active) platform='darwin 20' archs='noarch' date='2020-12-27T18:45:29-0800'
  db48 @4.8.30_4+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:20:23-0800'
  expat @2.2.10_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:39-0800'
  gdbm @1.18.1_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:21:16-0800'
  gettext @0.19.8.1_2+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:47-0800'
  glib2 @2.58.3_1+universal+x11 (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:52:45-0800'
  help2man @1.47.16_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:29:53-0800'
  libedit @20191231-3.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:51-0800'
  libffi @3.3_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:51-0800'
  libiconv @1.16_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:40-0800'
  libidn2 @2.3.0_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:48:14-0800'
  libpsl @0.21.1-20200817_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:53:17-0800'
  libtool @2.4.6_11+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:20:16-0800'
  libunistring @0.9.10_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:46:49-0800'
  lz4 @1.9.3_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T13:13:03-0800'
  ncurses @6.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:41-0800'
  openssl @1.1.1i_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:51:51-0800'
  p5.28-locale-gettext @1.70.0_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:29:47-0800'
  pcre @8.44_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:49:15-0800'
  perl5.28 @5.28.3_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:29:29-0800'
  perl5.30 @5.30.3_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:38:57-0800'
  pkgconfig @0.29.2_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:11-0800'
  python3_select @0.0_2 (active) platform='darwin 20' archs='noarch' date='2020-12-29T11:43:37-0800'
  python39 @3.9.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:09:42-0800'
  python_select @0.3_9 (active) platform='darwin 20' archs='noarch' date='2020-12-29T11:43:37-0800'
  readline @8.1.000_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:20:52-0800'
  sqlite3 @3.34.0_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:16-0800'
  texinfo @6.7_1+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T12:40:39-0800'
  the_silver_searcher @2.2.0_0 (active) platform='darwin 20' archs='x86_64' date='2021-01-10T09:58:02-0800'
  xz @5.2.5_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:52:20-0800'
  zlib @1.2.11_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T09:41:52-0800'
  zstd @1.4.8_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T13:21:26-0800'

MacPorts has settled on python39 as our current standard, so now that python39 can be built +universal, then we should just change anything that calls for python38 without a good reason (like libpsl) to switch to python39.

[arthurcnorman]

With icu and python39 building in universal mode thinsg indeed go a lot further. So this is a report on what I have got going locally and what I seemed toi need to change. None of the changes I have made are in any respect deep so I hope that just comments here will help if bnobody has prodded these cases otherwise.

1. jpag showed checkdum failure on the download for me. When I just made a private Portfile wtaht wuoted the checksuims of what did get fetched it appeared to build. Note I have not tried USING anything yet - just having "port install X +universl" has felt like a triumph.
2. kerberos5 - just change to depend on python39 not 38
3. create py39-libxml2 as a copy of py38/libxml2. But then python/libxml2.c and python/types.c have a bunch of cases where the C code goes

      if PyXXX_Check(obj) {
   

   where parens around the condition seem to be missing. When I inserted parens in half a doxen places things compiled!
4. xorg-libxcb: change to use pythin39 not 38. Thej LOTS of bits of xorg stuff build in universal mode. Hoorah!
5. Then ossp-uuid is where I am currently stalled. The starting issue is that it uses autoconf does not use automake and that means that the autotools support files do not get refreshed very automatically - and its config.gues, .sub and various libtool files are there in versions from maybe 2008 such that they do not know about aarch64.

I could copy in config.guess, config.sub and libtool.m4 and could make some progress, but right now I have libtool version disagreement woes and am muddles about whether this is down to ossp-uuid private copies or native mac libtoo vs macports or what. But that package is then a precondition that blocks quite a lot more.

6. mesa and libexpoy (both wanted by other things) depend on py27-libxml2 has a build failure from within aetup-py atuff. I do not understand that.

YOu may be well ahead of me on all of these, so this is just encouragement and in case noting easy case and what seems the biggest blocker helps. Arthur

[jmroot (Joshua Root)]

Any improvement here with 3.8.10, which adds arm64 support upstream?

[jmroot (Joshua Root)]

Assuming the original problem here is fixed, please reopen if not.