Opened 4 years ago
Closed 4 years ago
#62023 closed defect (fixed)
python39 @3.9.1 does not build universal on BigSur
| Reported by: | kencu (Ken) | Owned by: | jmroot (Joshua Root) |
|---|---|---|---|
| Priority: | Normal | Milestone: | |
| Component: | ports | Version: | |
| Keywords: | BigSur universal | Cc: | |
| Port: | python39 |
Description
The lack of a universal python is blocking the installation of a number of universal ports on BigSur at present.
Building python39 as universal arm64/x86_64 fails on BigSur (both Intel and arm machines) with this error:
if test "x-extract i386" != "x" ; then \ rm -f /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/binpython3.9-32; \ lipo -extract i386 \ -output /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9-32 \ /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9; \ fi fatal error: /Library/Developer/CommandLineTools/usr/bin/lipo: -extract i386 specified but fat file: /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 does not contain that architecture
that can be overcome quite easily by overriding the default setting for LIPO_32BIT_FLAGS in the Makefile, and commenting it out (for a quick fix):
#LIPO_32BIT_FLAGS=-extract i386
and then python39 indeed finishes installing as +universal on BigSur.
However, there is an error in the installed files:
---> Cleaning python39 ---> Removing work directory for python39 ---> Updating database of binaries ---> Scanning binaries for linking errors Could not open /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/Python: Error opening or reading file (referenced from /opt/local/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python) ---> Found 1 broken file, matching files to ports ---> Found 1 broken port, determining rebuild order You can always run 'port rev-upgrade' again to fix errors. The following ports will be rebuilt: python39 @3.9.1+universal
and it looks like at least one of them has the wrong library reference embedded, for some reason:
$ otool -L /opt/local/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python /opt/local/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0) /opt/local/var/macports/build/_opt_macports-ports_lang_python39/python39/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/3.9/Python (compatibility version 3.9.0, current version 3.9.0) /opt/local/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.5.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
not quite sure why that is wrong just now. I will try to fix it manually first, and then if that works, perhaps we can sort out why this is happening.
Attachments (3)
Change History (15)
comment:1 Changed 4 years ago by kencu (Ken)
comment:2 Changed 4 years ago by kencu (Ken)
Here are the changes induced into python39 by the patch phase and post-patch reinplaces:
generated by this:
35 sudo port clean 36 sudo port -v extract subport=python39 +universal 37 cd work 38 sudo chmod -R a+rw ./* 39 cd P* 40 git init 41 git add . 42 git commit -m init > /dev/null 43 cd ../.. 44 sudo port -v patch subport=python39 +universal 45 cd work 46 cd P* 47 git status 48 git diff 49 git diff --no-prefix > ~/python39-changes-from-patch.txt
I'll attach the whole diff, but it looks like this part in configure is what needs fixing:
intel)
- UNIVERSAL_ARCH_FLAGS="-arch i386 -arch x86_64"
+ UNIVERSAL_ARCH_FLAGS="-arch arm64 -arch x86_64"
LIPO_32BIT_FLAGS="-extract i386"
ARCH_RUN_32BIT="/usr/bin/arch -i386"
;;
Changed 4 years ago by kencu (Ken)
| Attachment: | python39-changes-from-patch.txt added |
|---|
comment:3 Changed 4 years ago by kencu (Ken)
(sidebar) during work on this, I see this issue when building universal on an Intel system:
/usr/bin/clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -arch arm64 -arch x86_64 build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/_ctypes.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callbacks.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callproc.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/cfield.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/malloc_closure.o build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.o -L/opt/local/lib -lffi -ldl -o build/lib.macosx-11.1-x86_64-3.9/_ctypes.cpython-39-darwin.so ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callbacks.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64 ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/_ctypes.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64 ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callproc.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64 ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/cfield.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64 ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/malloc_closure.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64 ld: warning: ignoring file build/temp.macosx-11.1-x86_64-3.9/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
and a similar but inverse issue building universal on an arm64 system:
/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I_ctypes/darwin -I/opt/universal/include -I./Include -I. -I/opt/universal/include -I/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Include -I/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1 -c /opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.c -o build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.o -DPy_BUILD_CORE_MODULE -DUSING_MALLOC_CLOSURE_DOT_C=1 -DMACOSX -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1 /usr/bin/clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -L/opt/universal/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk -arch arm64 -arch x86_64 build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/_ctypes.o build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callbacks.o build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callproc.o build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/cfield.o build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/malloc_closure.o build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.o -L/opt/universal/lib -lffi -ldl -o build/lib.macosx-11.0-arm64-3.9/_ctypes.cpython-39-darwin.so ld: warning: ignoring file build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/_ctypes.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64 ld: warning: ignoring file build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callbacks.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64 ld: warning: ignoring file build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/callproc.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64 ld: warning: ignoring file build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/cfield.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64 ld: warning: ignoring file build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/malloc_closure.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64 ld: warning: ignoring file build/temp.macosx-11.0-arm64-3.9/opt/universal/var/macports/build/_Users_Shared_kencurepos_myports_lang_python39/python39/work/Python-3.9.1/Modules/_ctypes/stgdict.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
comment:4 Changed 4 years ago by kencu (Ken)
This patch to the portfile allows a universal arm/x86_64 build to finish. I tried to handle the 32bit situation, but I have not as yet tested it:
% diff -u `port file python39` Portfile
--- /opt/local/var/macports/sources/rsync.macports.org/macports/release/tarballs/ports/lang/python39/Portfile 2020-12-08 03:30:58.000000000 -0800
+++ Portfile 2021-01-10 11:52:59.000000000 -0800
@@ -174,12 +174,20 @@
variant universal {
post-patch {
+
+ # first null out any existing lipo extract archs
+ reinplace \
+ "s|LIPO_32BIT_FLAGS=\".*\"|LIPO_32BIT_FLAGS=\"\"|" \
+ ${worksrcpath}/configure
+
set universal_arch_flags {}
set arch_run_32bit {}
+ set lipo_32bit_flags {}
foreach arch ${universal_archs} {
lappend universal_arch_flags -arch ${arch}
if {${arch} eq "i386" || ${arch} eq "ppc"} {
lappend arch_run_32bit -${arch}
+ lappend lipo_32bit_flags -extract ${arch}
}
}
reinplace \
@@ -189,6 +197,9 @@
reinplace \
"s|ARCH_RUN_32BIT=\".*\"|ARCH_RUN_32BIT=\"arch ${arch_run_32bit}\"|" \
${worksrcpath}/configure
+ reinplace \
+ "s|LIPO_32BIT_FLAGS=\".*\"|LIPO_32BIT_FLAGS=\"${lipo_32bit_flags}\"|" \
+ ${worksrcpath}/configure
}
}
if {${configure.sdkroot} ne ""} {
I will attach the diff for the pre-patch and post-patch source tree.
Changed 4 years ago by kencu (Ken)
| Attachment: | python39-fixed-changes-from-patch.diff added |
|---|
Changed 4 years ago by kencu (Ken)
| Attachment: | patch-portfile-fix-lipo-archs.diff added |
|---|
comment:5 Changed 4 years ago by kencu (Ken)
And for some reason, I no longer see that issue that requires install_name_tool to fix, so that Portfile patch appears to be all that is needed. I will have to test the universal build on older systems to see if it actually works properly, however.
And I'm not sure what this means <https://trac.macports.org/ticket/62023#comment:3> if anything.
comment:6 Changed 4 years ago by kencu (Ken)
That same simple patch unfortunately is not enough to fix python38, which is still getting backport fixes for arm64 upstream I understand:
DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Traceback (most recent call last):
File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 712, in <module>
_main()
File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 700, in _main
_generate_posix_vars()
File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 402, in _generate_posix_vars
pybuilddir = 'build/lib.%s-%s' % (get_platform(), _PY_VERSION_SHORT)
File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/sysconfig.py", line 679, in get_platform
osname, release, machine = _osx_support.get_platform_osx(
File "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7/Lib/_osx_support.py", line 495, in get_platform_osx
raise ValueError(
ValueError: Don't know machine value for archs=('arm64', 'x86_64')
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7'
Command failed: cd "/opt/local/var/macports/build/_opt_myports_lang_python38/python38/work/Python-3.8.7" && /usr/bin/make -j8 -w all
comment:7 Changed 4 years ago by kencu (Ken)
There is perhaps another approach to be explored, here:
It is possible to build a number of flavours of the universal binary build, the default is a 32-bit only binary (i386 and ppc). The flavour can be specified using the option ``--with-universal-archs=VALUE``. The following values are available: * ``32-bit``: ``ppc``, ``i386`` * ``64-bit``: ``ppc64``, ``x86_64`` * ``all``: ``ppc``, ``ppc64``, ``i386``, ``x86_64`` * ``3-way``: ``ppc``, ``i386`` and ``x86_64`` * ``intel``: ``i386``, ``x86_64`` To build a universal binary that includes a 64-bit architecture, you must build on a system running OSX 10.5 or later. The ``all`` flavour can only be built on OSX 10.5. The makefile for a framework build will install ``python32`` and ``pythonw32`` binaries when the universal architecures includes at least one 32-bit architecture (that is, for all flavours but ``64-bit``).
comment:8 Changed 4 years ago by kencu (Ken)
building it universal on 10.6 i386/x86_64 seems to be fine. Patching:
Executing: cd "/opt/local/var/macports/build/_opt_myports_lang_python39/python39/work/Python-3.9.1" && /usr/bin/patch -p0 < '/opt/myports/lang/python39/files/implicit.patch' patching file configure ---> Patching cgi.py: s|@@PREFIX@@|/opt/local|g ---> Patching dyld.py: s|@@PREFIX@@|/opt/local|g ---> Patching Makefile.pre.in: s|/setup.py|/setup.py --no-user-cfg| ---> Patching configure: s|@@APPLICATIONS_DIR@@|/Applications/MacPorts| ---> Patching configure: s|LIPO_32BIT_FLAGS=".*"|LIPO_32BIT_FLAGS=""| ---> Patching configure: s|UNIVERSAL_ARCH_FLAGS=".*"|UNIVERSAL_ARCH_FLAGS="-arch x86_64 -arch i386"| ---> Patching configure: s|ARCH_RUN_32BIT=".*"|ARCH_RUN_32BIT="arch -i386"| ---> Patching configure: s|LIPO_32BIT_FLAGS=".*"|LIPO_32BIT_FLAGS="-extract i386"| ---> Configuring python39
Archs and running:
file "/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python" /Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python: Mach-O universal binary with 2 architectures /Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture x86_64): Mach-O 64-bit executable x86_64 /Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture i386): Mach-O executable i386 macbookpro21:python39 cunningh$ python3.9 Python 3.9.1 (default, Jan 10 2021, 18:21:38) [Clang 9.0.1 ] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import subprocess >>> ^D
So it appears that this Lipo_32bit_flags business works there.
comment:9 Changed 4 years ago by kencu (Ken)
JMR - would you like a PR, or would you rather not fix / fix differently?
comment:10 Changed 4 years ago by jmroot (Joshua Root)
TBH, I lost track of what you were actually proposing a few comments ago, so I just went with [4cfc3d6e2fa4d507771d17c8c347dfe6734b191a/macports-ports]. I don't have easy access to a system that can do universal builds, so please test.
comment:11 Changed 4 years ago by kencu (Ken)
This was what I proposed <https://trac.macports.org/attachment/ticket/62023/patch-portfile-fix-lipo-archs.diff> but I'll try out what you did. Thanks.
comment:12 Changed 4 years ago by kencu (Ken)
| Resolution: | → fixed |
|---|---|
| Status: | assigned → closed |
Note: See
TracTickets for help on using
tickets.
This manual fix:
repairs the port: