python27 @2.7.5_0: incorrect LINKFORSHARED from distutils.sysconfig

[skymoo (Adam Mercer)]

I've been having some problems build some software against python27 @2.7.5_0:

checking for a Python interpreter with version >= 2.6... python
checking for python... /opt/local/bin/python
checking for python version... 2.7
checking for python platform... darwin
checking for python script directory... ${prefix}/lib/python2.7/site-packages
checking for python extension module directory...
${exec_prefix}/lib/python2.7/site-packages
checking for python2.7... (cached) /opt/local/bin/python
checking for a version of Python >= '2.1.0'... yes
checking for the distutils Python package... yes
checking for Python include path...
-I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
checking for Python library path...
-L/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config
-lpython2.7
checking for Python site-packages path...
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
checking python extra libraries...  -ldl  -framework CoreFoundation
checking python extra linking flags... -u _PyMac_Error
Python.framework/Versions/2.7/Python
checking consistency of all components of python development environment... no
configure: error: in `/Users/ram/git/lalsuite/lalburst':
configure: error:
  Could not link test program to Python. Maybe the main Python library has been
  installed in some non-standard library path. If so, pass it to configure,
  via the LDFLAGS environment variable.
  Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
  ============================================================================
   ERROR!
   You probably have to install the development version of the Python package
   for your distribution.  The exact name of this package varies among them.
  ============================================================================

See `config.log' for more details

This seems to be due to distutils reporting incorrectly the flags required to link against python. With @2.7.5_0:

Python 2.7.5 (default, May 24 2013, 11:10:07)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.sysconfig
>>> conf = distutils.sysconfig.get_config_var
>>> print (conf('LINKFORSHARED'))
-u _PyMac_Error Python.framework/Versions/2.7/Python
>>>

i.e. it looks like the path is incomplete. Whereas if I revert back to @2.7.3_1 I get what I would expect:

Python 2.7.3 (default, Nov 26 2012, 15:06:55)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.sysconfig
>>> conf = distutils.sysconfig.get_config_var
>>> print (conf('LINKFORSHARED'))
-u _PyMac_Error  /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python
>>>

[mf2k (Frank Schima)]

I see the same result for python 2.7.5.

[cooljeanius (Eric Gallager)]

Cc Me!

[ned-deily (Ned Deily)]

LINKFORSHARED is a configuration variable used to build the Python interpreter itself in its build directory. It isn't intended to be used directly by other components. Unfortunately, Python builds export all configuration variables whether they are meaningful or not. To embed Python, the supported interface for finding build flags is to use the pythonx.y-config script:

$ /opt/local/bin/python2.7-config --cflags
-I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -pipe -O2 -fwrapv -arch x86_64 -arch i386 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
$ /opt/local/bin/python2.7-config --ldflags
-L/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -ldl -framework CoreFoundation -lpython2.7

This is explained in the "Extending and Embedding the Python Interpreter" manual of the Python docs ​http://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems Note that the Python 3 version of this doc is more up-to-date than the Python 2 version.

[jmroot (Joshua Root)]

There's a reinplace in post-destroot that looks like it's meant to "fix" this. It originated from #15099.

Upstream bug report: ​http://bugs.python.org/issue3588

[ned-deily (Ned Deily)]

I think the correct solution here is to patch and fix upstream whatever ports are referencing Python's LINKFORSHARED. They are likely causing problems for everyone. I know the Homebrew folks are currently reporting weird problems with /usr/bin/vim picking up the wrong python at runtime.

[skymoo (Adam Mercer)]

Replying to nad@…:

LINKFORSHARED is a configuration variable used to build the Python interpreter itself in its build directory. It isn't intended to be used directly by other components. Unfortunately, Python builds export all configuration variables whether they are meaningful or not. To embed Python, the supported interface for finding build flags is to use the pythonx.y-config script:

Thanks that's interesting, the software I was trying to build uses the ​`ax_python_devel.m4` macro from the autoconf archive but python-config looks like a better approach. I'll forward that to upstream.

[jmroot (Joshua Root)]

I guess someone should probably tell the maintainer of that autoconf macro that they're doing it wrong, too.

[cooljeanius (Eric Gallager)]

#39357 is related

[wichert@…]

python-config using autoconf macro

[wichert@…]

For what it's worth I whipped up a basic autoconf macro that will use python-config and should be a drop-in replacement for ax_python_devel.m4 . I've attached it to this ticket.

[skymoo (Adam Mercer)]

Nice, it's probably worth submitting something like this to the autoconf-archive.

[larryv (Lawrence Velázquez)]

Has duplicate #39363.

[wichert@…]

I'll try to submit it to the archive. I wonder if they'll take it though since the python document does mention python-config is not guaranteed to work everywhere, and if it does not work one should still us distutils.sysconfig. There is even an example using LINKFORSHARED.

[ned-deily (Ned Deily)]

I've opened ​http://bugs.python.org/issue18164 to fix the Python docs.

[t.m.isele@…]

Replying to wichert@…:

For what it's worth I whipped up a basic autoconf macro that will use python-config and should be a drop-in replacement for ax_python_devel.m4 . I've attached it to this ticket.

Thanx for cooking that thing together! Please excuse my stupid question: What do I exactly do with this file to fix the problem of installing a port linked to python 2.7.5 (graph-tool in my case..) Thank you!

[t.m.isele@…]

Cc Me!

[lpsinger (Leo Singer)]

Cc Me!

[cooljeanius (Eric Gallager)]

Replying to ram@…:

Replying to nad@…:

LINKFORSHARED is a configuration variable used to build the Python interpreter itself in its build directory. It isn't intended to be used directly by other components. Unfortunately, Python builds export all configuration variables whether they are meaningful or not. To embed Python, the supported interface for finding build flags is to use the pythonx.y-config script:

Thanks that's interesting, the software I was trying to build uses the ​`ax_python_devel.m4` macro from the autoconf archive but python-config looks like a better approach. I'll forward that to upstream.

#39804 would update the autoconf-archive port to the latest version, which includes ​changes to this macro. idk if that would fix this, but it's worth a shot...

[nonstop.server@…]

Cc Me!

[lpsinger (Leo Singer)]

Replying to egall@…:

Replying to ram@…:

Replying to nad@…:

LINKFORSHARED is a configuration variable used to build the Python interpreter itself in its build directory. It isn't intended to be used directly by other components. Unfortunately, Python builds export all configuration variables whether they are meaningful or not. To embed Python, the supported interface for finding build flags is to use the pythonx.y-config script:

Thanks that's interesting, the software I was trying to build uses the ​`ax_python_devel.m4` macro from the autoconf archive but python-config looks like a better approach. I'll forward that to upstream.

#39804 would update the autoconf-archive port to the latest version, which includes ​changes to this macro. idk if that would fix this, but it's worth a shot...

No, it still has LINKFORSHARED in it.

[jmroot (Joshua Root)]

Closing, since the problem is in the other software that tries to use this value.

[dcecchin@…]

Cc Me!

[larryv (Lawrence Velázquez)]

Cc Me!

[raimue (Rainer Müller)]

In b5b644ee4ae92eab09344a3f4f7e77aeee769acc/macports-ports:

python2*, python3*: fix flags in LINKFORSHARED

Other ports are unable to link against libpython as the value does not
properly reference Python.framework.

See: ​https://lists.macosforge.org/pipermail/macports-dev/2016-September/033712.html
See: #39223

[ryandesign (Ryan Carsten Schmidt)]

Replying to raimue:

In b5b644ee4ae92eab09344a3f4f7e77aeee769acc/macports-ports:

python2*, python3*: fix flags in LINKFORSHARED

Other ports are unable to link against libpython as the value does not
properly reference Python.framework.

See: ​https://lists.macosforge.org/pipermail/macports-dev/2016-September/033712.html
See: #39223

This change breaks 32-bit builds.

​https://build.macports.org/builders/ports-10.6_i386_legacy-builder/builds/31600

​https://build.macports.org/builders/ports-10.5_ppc_legacy-builder/builds/45981

--->  Patching Makefile: s|^\(LINKFORSHARED=\).*$|\1 -L$(LIBPL) -lpython$(VERSION) $(LIBS) $(SYSLIBS)|
DEBUG: Executing reinplace: /usr/bin/sed {s|^\(LINKFORSHARED=\).*$|\1 -L$(LIBPL) -lpython$(VERSION) $(LIBS) $(SYSLIBS)|} </opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_python24/python24/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/config/Makefile >@file10
DEBUG: couldn't read file "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_lang_python24/python24/work/destroot/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/config/Makefile": no such file or directory

[jmroot (Joshua Root)]

In dc4393493c6fab43759863194045de76cba71fc0/macports-ports:

python24: fix path to config/Makefile

Fixes: #39223