python*: Compiler path is baked into _sysconfigdata.py

[ryandesign (Ryan Carsten Schmidt)]

The paths to the compilers used to build python27 are baked into the installed file /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py:

$ grep /usr/bin/clang /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py
 'BLDSHARED': '/usr/bin/clang -bundle -undefined dynamic_lookup   -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48  ',
 'CC': '/usr/bin/clang',
 'CONFIG_ARGS': "'--prefix=/opt/local' '--disable-dependency-tracking' '--enable-framework=/opt/local/Library/Frameworks' '--enable-ipv6' '--with-system-expat' '--with-system-ffi' '--enable-universalsdk=/' 'CC=/usr/bin/clang' 'CFLAGS=-pipe -Os  ' 'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48  ' 'CPPFLAGS=-I/opt/local/include -I/opt/local/include/db48'",
 'CXX': '/usr/bin/clang++',
 'LDCXXSHARED': '/usr/bin/clang++ -bundle -undefined dynamic_lookup',
 'LDSHARED': '/usr/bin/clang -bundle -undefined dynamic_lookup   -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48  ',
 'LINKCC': '/usr/bin/clang',
 'MAINCC': '/usr/bin/clang',

Some ports that build with python use these compilers, rather than using the compiler MacPorts told them to use. This may be due to a bug in the python portgroup that should be fixed. But even if the portgroup is fixed, it doesn't help ports not using that portgroup, and it doesn't help software built outside of MacPorts but still using MacPorts python.

The problem occurs when the paths baked into that file at build time don't exist at runtime. For example, python built on our Snow Leopard buildbot build worker will use gcc-4.2, because the build worker has our recommended version of Xcode, 3.2.6, installed. But if a user has Xcode 4.2 installed, then gcc-4.2 will not exist, and the user will encounter build failures because of that. See for example #55831 and #55833.

The problem only affects Snow Leopard, because only on Snow Leopard do we default to a compiler (gcc-4.2) with one version of Xcode (3.2.6) which does not exist in later versions of Xcode (4.2). On all other macOS versions, we use the same default compiler regardless of Xcode version.

The problem affects python27 and later. python26 and earlier don't have a _sysconfigdata.py file so I don't think they are affected. In python36, the filename changed to _sysconfigdata_m_darwin_darwin.py.

Some possible solutions occur to me:

1. Force python* to build with llvm-gcc-4.2 on 10.6, since llvm-gcc-4.2 is available in all versions of Xcode on 10.6. At first this seemed workable, since llvm-gcc-4.2 is the default compiler for Xcode 4.0.x–4.2.x, until I realized that llvm-gcc-4.2 is blacklisted for python33 and later; see #45667. I confirmed just now that this blacklisting is still required for python33 and later; they fail to build with llvm-gcc-4.2.
2. Force python* to build with clang on 10.6, since clang is available in all versions of Xcode on 10.6. This may not be a good idea because it would make other python modules build with that very early version of clang, and that seems likely to result in build failures.
3. In post-destroot, replace the real compiler paths in _sysconfigdata.py with placeholders, and in post-activate, replace the placeholders with the current values from ${configure.cc} and ${configure.cxx}. I can't think of a reason why this wouldn't work. For Xcode 4.2, that would still end up being clang, which might be undesirable as in (2) above. We could special-case 10.6 and use llvm-gcc-4.2 there instead of clang. I am assuming that the build failure of python33 and later with llvm-gcc-4.2 is specific to the source code of python itself, and would not manifest itself when building most modules.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

3. In post-destroot, replace the real compiler paths in _sysconfigdata.py with placeholders, and in post-activate, replace the placeholders with the current values from ${configure.cc} and ${configure.cxx}. I can't think of a reason why this wouldn't work. For Xcode 4.2, that would still end up being clang, which might be undesirable as in (2) above. We could special-case 10.6 and use llvm-gcc-4.2 there instead of clang. I am assuming that the build failure of python33 and later with llvm-gcc-4.2 is specific to the source code of python itself, and would not manifest itself when building most modules.

I've implemented this here: ​https://github.com/macports/macports-ports/pull/1773

[ryandesign (Ryan Carsten Schmidt)]

You closed the PR, so I'll respond here:

Sysconfig is supposed to record the compiler that was used to build python. This would render it incorrect.

I understand that.

You'll notice it records a lot of other things like the configure args that were used.

Right. This PR doesn't change the CONFIG_ARGS line. It's possible there are other values the PR should not change, such as CC and CXX, and I'm happy to change the PR for whatever decision is reached in that regard. I do not know the distinction between the various variables.

Distutils does not blindly use that compiler to build things, it has code to choose a default compiler that exists (if one is not specified).

In some cases, yes, it definitely does use these compilers to build other things, resulting in the problems reported in the above-mentioned tickets.

[ryandesign (Ryan Carsten Schmidt)]

How would you like to address this problem, if not in the way that I suggested in the PR?

[olupton (Olli Lupton)]

I have come across what seems to be a variant of this problem: some of the sysconfig fields include an explicit path to an SDK:

$ python
Python 3.7.4 (default, Sep 22 2019, 09:12:40)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> print(sysconfig.get_config_var('CFLAGS'))
-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk

and this path is used when building some ports. In my case building the python part of the xrootd port ran into problems.

In my case the problem was apparently that the python37 I had installed was built on a system with a full XCode install, which assumes that the SDK can be found with -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk, but my own system just has the commandline tools installed (SDK path shown above). Building the xrootd port locally tried to compile the python bindings using the SDK path baked into the binary installation of python37, which failed. I could work around this by forcing the python37 to be built from source, which bakes in a valid path (this is the python output shown above).

[yan12125 (Chih-Hsuan Yen)]

Olli Lupton: FYI, your issue (mismatched sysroot) is discussed in ​https://trac.macports.org/ticket/59078 and [02eac73dfdc362a003adf7148df843b9a5023b31/macports-ports] is commited as a fix.