python39 @3.9.0, python38 @3.8.7, python37 @3.7.10, python36 @3.6.13: +universal does not work with longer than normal CFLAGS

[ryandesign (Ryan Carsten Schmidt)]

python39 installs with or without the universal variant, but when installed with the universal variant, it doesn't work. This is a regression from python38 and earlier versions which work fine universal or not.

For example when python39 is installed universal, py39-setuptools fails to build:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_python_py-setuptools/py39-setuptools/work/setuptools-50.3.0/setup.py", line 10, in <module>
    import setuptools
  File "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_python_py-setuptools/py39-setuptools/work/setuptools-50.3.0/setuptools/__init__.py", line 10, in <module>
    import distutils.core
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 16, in <module>
    from distutils.dist import Distribution
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 19, in <module>
    from distutils.util import check_environ, strtobool, rfc822_escape
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/util.py", line 14, in <module>
    from distutils.spawn import spawn
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 11, in <module>
    import subprocess
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

This can be reproduced easily at the command line:

$ python3.9
Python 3.9.0 (default, Oct  6 2020, 19:05:10)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
>>> ^D

Examining the contents of python39, I see that when it is installed non-universal, it contains the file:

/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_posixsubprocess.cpython-39-darwin.so

But when it is installed universal, that file name contains "failed":

/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_posixsubprocess.cpython-39-darwin_failed.so

There are many other "failed" files in the universal install.

[ryandesign (Ryan Carsten Schmidt)]

This was on macOS 10.13.6.

[ryandesign (Ryan Carsten Schmidt)]

The problem remains with python 3.9.1.

$ port -vq installed python39
  python39 @3.9.1_0+universal (active) platform='darwin 17' archs='i386 x86_64' date='2020-12-18T07:33:31-0600'
$ python3.9
Python 3.9.1 (default, Dec 18 2020, 07:27:59)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
>>> ^D

[ryandesign (Ryan Carsten Schmidt)]

The problem now also occurs with python38 @3.8.7. See duplicate #61915.

[kencu (Ken)]

A universal build of python39 with arm/x86_64, installed as per #62023, seems to work (on BigSur Intel at present):

% port -vq installed python39
  python39 @3.9.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T10:07:17-0800'

% python3.9                  
Python 3.9.1 (default, Jan 10 2021, 10:06:23) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> ^D

[kencu (Ken)]

Universal on i386/86_64 is OK on 10.6.8:

$ port -vq installed python39
  python39 @3.9.1_0+universal (active) platform='darwin 10' archs='i386 x86_64' date='2021-01-10T18:29:47-0800'

$ file "/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python"
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python: Mach-O universal binary with 2 architectures
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture x86_64):	Mach-O 64-bit executable x86_64
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture i386):	Mach-O executable i386

$ python3.9
Python 3.9.1 (default, Jan 10 2021, 18:21:38) 
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> ^D

[kencu (Ken)]

Ah -- but the i386 build is broken:

$ arch -i386 python3.9
Python 3.9.1 (default, Jan 10 2021, 18:21:38) 
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
    import _posixsubprocess
ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
>>> ^D

[hehongbo (HeHongbo)]

Same on macOS 11.2.1 when I try installing python39.

[ryandesign (Ryan Carsten Schmidt)]

python37 is affected now too after having been updated to 3.7.10.

We need to identify whatever change they've put in and backported to all these versions and get rid of it.

[ryandesign (Ryan Carsten Schmidt)]

python36 is also affected now after being updated to 3.6.13.

[jmroot (Joshua Root)]

Not that much changed between 3.7.9 and 3.7.10: ​https://github.com/python/cpython/compare/v3.7.9..v3.7.10
or between 3.6.12 and 3.6.13: ​https://github.com/python/cpython/compare/v3.6.12..v3.6.13

[ryandesign (Ryan Carsten Schmidt)]

Aha! Thank you for pointing that out. Then I think it is something I changed on my machine some months ago: I build with this change to base:

--- src/port1.0/portconfigure.tcl	(revision 155969)
+++ src/port1.0/portconfigure.tcl	(working copy)
@@ -235,8 +235,8 @@
         configure.classpath
 # compiler flags section
 default configure.optflags      -Os
-default configure.cflags        {${configure.optflags}}
-default configure.objcflags     {${configure.optflags}}
+default configure.cflags        {${configure.optflags} -Werror=implicit-function-declaration}
+default configure.objcflags     {${configure.optflags} -Werror=implicit-function-declaration}
 default configure.cppflags      {[portconfigure::configure_get_cppflags]}
 proc portconfigure::configure_get_cppflags {} {
     global prefix

If I rebuild python39 @3.9.2 +universal without that, then it works.

Let me get clean build logs both with and without that and let's see how they differ.

[ryandesign (Ryan Carsten Schmidt)]

It looks like python's build system must do some kind of word wrapping when it creates the file _sysconfigdata__darwin_darwin.py (perhaps to comply with ​PEP-8). The Portfile uses reinplace to remove -arch flags from _sysconfigdata__darwin_darwin.py (see #39669 where that was first added) but because my CFLAGS are longer, some of the -arch flags happen to end up broken across multiple lines due to the word wrapping, so reinplace doesn't remove all of them. (Even in the default build, it doesn't remove all of them—look at CONFIGURE_CFLAGS, PY_CORE_CFLAGS, and PY_STDMODULE_CFLAGS.) Here's the diff between how that file got generated in a standard "vanilla" build and how it got generated with -Werror=implicit-function-declaration added:

python39-3.9.2_0+universal.darwin_17.i386-x86_64/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/_sysconfigdata__darwin_darwin.py

@@ -25 +25 @@
  'CCSHARED': '',
  'CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common '
            '-dynamic -DNDEBUG -g -fwrapv -O3 -Wall   '
-           '-pipe -Os  ',
+           '-pipe -Os -Werror=implicit-function-declaration  -arch '
+           'i386',
  'CFLAGSFORSHARED': '',
  'CFLAGS_ALIASING': '-fno-strict-aliasing',
  'CFLAGS_NODIST': '',
  'CONFIGFILES': 'configure configure.ac acconfig.h pyconfig.h.in '
                 'Makefile.pre.in',
- 'CONFIGURE_CFLAGS': '  -pipe -Os  -arch '
+ 'CONFIGURE_CFLAGS': '  -pipe -Os '
+                     '-Werror=implicit-function-declaration  -arch '
                      'i386',
  'CONFIGURE_CFLAGS_NODIST': '-std=c99 -Wextra -Wno-unused-result '
                             '-Wno-unused-parameter '
@@ -50 +52 @@
                 "'--with-computed-gotos' '--with-ensurepip=no' "
                 "'--with-system-expat' '--with-system-ffi' "
                 "'--enable-universalsdk=/' 'CC=/usr/bin/clang' 'CFLAGS=-pipe "
-                "-Os  ' 'LDFLAGS=-L/opt/local/lib "
+                '-Os -Werror=implicit-function-declaration  -arch '
+                "i386' 'LDFLAGS=-L/opt/local/lib "
                 "-Wl,-headerpad_max_install_names  ' "
                 "'CPPFLAGS=-I/opt/local/include'",
  'CONFINCLUDEDIR': '/opt/local/Library/Frameworks/Python.framework/Versions/3.9/include',
@@ -649 +652 @@
  'PY_BUILTIN_MODULE_CFLAGS': '-Wno-unused-result -Wsign-compare '
                              '-Wunreachable-code -fno-common -dynamic -DNDEBUG '
                              '-g -fwrapv -O3 -Wall   '
-                             '-pipe -Os   -std=c99 '
-                             '-Wextra -Wno-unused-result -Wno-unused-parameter '
+                             '-pipe -Os -Werror=implicit-function-declaration '
+                             '  -std=c99 -Wextra '
+                             '-Wno-unused-result -Wno-unused-parameter '
                              '-Wno-missing-field-initializers '
                              '-Wstrict-prototypes '
                              '-Werror=implicit-function-declaration '
@@ -659 +663 @@
                              '-DPy_BUILD_CORE_BUILTIN',
  'PY_CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code '
               '-fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall  '
-              ' -pipe -Os  ',
+              ' -pipe -Os -Werror=implicit-function-declaration '
+              ' ',
  'PY_CFLAGS_NODIST': '-std=c99 -Wextra -Wno-unused-result '
                      '-Wno-unused-parameter -Wno-missing-field-initializers '
                      '-Wstrict-prototypes '
@@ -668 +673 @@
  'PY_COERCE_C_LOCALE': 1,
  'PY_CORE_CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code '
                    '-fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch '
-                   'x86_64  -pipe -Os   '
-                   '-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter '
-                   '-Wno-missing-field-initializers -Wstrict-prototypes '
-                   '-Werror=implicit-function-declaration -fvisibility=hidden  '
-                   '-I./Include/internal -I. -I./Include -I/opt/local/include '
-                   '-DPy_BUILD_CORE',
+                   'x86_64  -pipe -Os '
+                   '-Werror=implicit-function-declaration  -arch '
+                   'i386 -std=c99 -Wextra -Wno-unused-result '
+                   '-Wno-unused-parameter -Wno-missing-field-initializers '
+                   '-Wstrict-prototypes -Werror=implicit-function-declaration '
+                   '-fvisibility=hidden  -I./Include/internal -I. -I./Include '
+                   '-I/opt/local/include -DPy_BUILD_CORE',
  'PY_CORE_LDFLAGS': '  -L/opt/local/lib '
                     '-Wl,-headerpad_max_install_names  ',
  'PY_CPPFLAGS': '-I. -I./Include -I/opt/local/include',
@@ -685 +691 @@
  'PY_SSL_DEFAULT_CIPHER_STRING': 0,
  'PY_STDMODULE_CFLAGS': '-Wno-unused-result -Wsign-compare -Wunreachable-code '
                         '-fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall '
-                        '  -pipe -Os  -arch '
-                        'i386 -std=c99 -Wextra -Wno-unused-result '
+                        '  -pipe -Os '
+                        '-Werror=implicit-function-declaration  '
+                        ' -std=c99 -Wextra -Wno-unused-result '
                         '-Wno-unused-parameter -Wno-missing-field-initializers '
                         '-Wstrict-prototypes '
                         '-Werror=implicit-function-declaration '

I suspect that that's what's causing the problem. Or if not, it should be fixed anyway, perhaps by disabling the word wrapping or by doing the -arch flag removal before that file gets generated (perhaps in whatever script builds it—that might then be suitable to send upstream).

[ryandesign (Ryan Carsten Schmidt)]

main.log - vanilla

[ryandesign (Ryan Carsten Schmidt)]

config.log - vanilla

[ryandesign (Ryan Carsten Schmidt)]

main.log - -Werror=implicit-function-declaration

[ryandesign (Ryan Carsten Schmidt)]

config.log - -Werror=implicit-function-declaration

[ryandesign (Ryan Carsten Schmidt)]

Note that in both main.logs some stuff gets compiled in the destroot phase, with that modified sysconfig. In main.2.log, you can see some things being compiled with only -arch i386. The post-build block tries to save off the sysconfig mtime before changing it and restoring it afterward so that things don't get rebuilt, and yet...

[jmroot (Joshua Root)]

Try these.

[ryandesign (Ryan Carsten Schmidt)]

Ooh, thanks. I was just thinking about making this my project for the evening, but you beat me to it.

It looks good to me. Attaching my log with this change (plus a revbump to 1).

Changing the awk to a reinplace... is that just stylistic or is there a functional change?

Any idea why some stuff still gets built in the destroot phase?

I wonder if this will also fix #59466. (Edit: no, this issue only affects python34 and later but that issue was with python27.)

[ryandesign (Ryan Carsten Schmidt)]

successful build of python39 @3.9.2 with jmr's change

[ryandesign (Ryan Carsten Schmidt)]

Looks like this sysconf thing was ​new in python 2.7 so the fix wouldn't have to be brought farther back than python27.

[jmroot (Joshua Root)]

Replying to ryandesign:

I was just thinking about making this my project for the evening, but you beat me to it.

More time to spend on getting the 10.15 builder back up? :)

It looks good to me. Attaching my log with this change (plus a revbump to 1).

I'm not going to rev bump unless there's evidence that anyone not making changes to base was affected.

Changing the awk to a reinplace... is that just stylistic or is there a functional change?

Awk was only used in the first place so that the second line could be removed. Reinplace is simpler for a single line.

Any idea why some stuff still gets built in the destroot phase?

PythonLauncher gets built in the destroot phase normally (not sure why), but I don't know why you're seeing modules getting rebuilt. I don't see that with a non-universal build on 10.15.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmroot:

More time to spend on getting the 10.15 builder back up? :)

Just did :)

I'm not going to rev bump unless there's evidence that anyone not making changes to base was affected.

Well all users are affected, not just me. See comment:12:

Even in the default build, it doesn't remove all of them—look at CONFIGURE_CFLAGS, PY_CORE_CFLAGS, and PY_STDMODULE_CFLAGS.

I'm not sure what the significance of having wrong single -arch flags in those sysconfig values is, but why not revbump it to fix it? It's consistent with our guidance to increase the revision when installed files change.

Awk was only used in the first place so that the second line could be removed. Reinplace is simpler for a single line.

Ahh right.

[jmroot (Joshua Root)]

Replying to ryandesign:

Even in the default build, it doesn't remove all of them—look at CONFIGURE_CFLAGS, PY_CORE_CFLAGS, and PY_STDMODULE_CFLAGS.

I'm not sure what the significance of having wrong single -arch flags in those sysconfig values is, but why not revbump it to fix it? It's consistent with our guidance to increase the revision when installed files change.

Ah, I see what you mean now. But python27 at least appears to be unaffected.

[ryandesign (Ryan Carsten Schmidt)]

Yeah looks like python 3.4 is the first version in which _sysconfigdata.py gets word wrapped.

[ryandesign (Ryan Carsten Schmidt)]

It's mentioned in the 3.4 changes: ​https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Doc/whatsnew/3.4.rst#pprint

[ryandesign (Ryan Carsten Schmidt)]

​https://bugs.python.org/issue17150

[jmroot (Joshua Root)]

In 9fd97292e6a751db08d9fe5e9e59bec71b868494/macports-ports (master):

python3*: protect against issues with long CFLAGS

Prevent line wrapping in _sysconfigdata.py so that '-arch foo' can't be
split across multiple lines, preventing the reinplace from removing it
correctly. In practice this only seems to have happened with universal
builds.

All variables being on a single line also means that awk is no longer
needed to edit LINKFORSHARED.

Also fix implicit declaration errors in python35's configure script.

Closes: #61282