Opened 11 years ago

Closed 11 years ago

#39217 closed defect (worksforme)

py27-numpy @1.7.1_0: import fails due to missing symbol

Reported by: jdgleeson Owned by: jyrkiwahlstedt
Priority: Normal Milestone:
Component: ports Version: 2.1.3
Keywords: Cc: dh@…, michaelld (Michael Dickens), cooljeanius (Eric Gallager), ned-deily (Ned Deily), reid@…, petrrr
Port: python27 py27-numpy

Description

I've used py27-numpy @1.7.1_0+atlas+gcc47 since mid April with no problems. After I forced an upgrade a couple days ago I get this traceback when I import numpy:

$ python
Python 2.7.5 (default, May 20 2013, 21:30:50) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/__init__.py", line 137, in <module>
    import add_newdocs
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/__init__.py", line 4, in <module>
    from type_check import *
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: dlopen(/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so, 2): Symbol not found: restGPRx
  Referenced from: /opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so
  Expected in: dynamic lookup

This library has the missing symbol:

$ nm /opt47/local/lib/gcc47/libgcc_s.1.dylib | grep restGPRx
00005f98 t restGPRx

but multiarray.so does not link with it:

jdgleeson@Archimedes:~$ otool -L /opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so
/opt47/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so:
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.7)

I noted in the log file that gcc-4.0 is used to create the extension libraries (but the source is compiled with gcc-mp-4.7). I have the port -d output from the April build of numpy, and it confirms that the extension libraries were built with gcc-mp-4.7, not gcc-4.0, at that time.

I've attached the log file from the recent build of numpy.

Attachments (3)

main0521.log.zip (41.7 KB) - added by jdgleeson 11 years ago.
Main log of py27-numpy 1.7.1_0+atlas+gcc47 (sudo port -kdn upgrade --force) May 21
python27-Portfile.diff (481 bytes) - added by jdgleeson 11 years ago.
patch-issue18080_ldshared_2.7.diff (3.1 KB) - added by jdgleeson 11 years ago.

Download all attachments as: .zip

Change History (15)

Changed 11 years ago by jdgleeson

Attachment: main0521.log.zip added

Main log of py27-numpy 1.7.1_0+atlas+gcc47 (sudo port -kdn upgrade --force) May 21

comment:1 Changed 11 years ago by jdgleeson

Since numpy has not changed since it worked for me in April, I suspected a change in another port was to blame. Here is a list of numpy dependencies I've identified that I've upgraded since April 12 (when I first built 1.7.1_0):

atlas           3.10.1_3+gcc47  3.10.1_5+gcc47
db46            4.6.21_7        4.6.21_8
gcc47           4.7.2_2         4.7.3_0
glpk            4.47_0          4.48_0
libstdcxx       4.7.2_2         4.8.0_0
perl5.12        5.12.4_1        5.12.4_2
ppl             1.0_0           1.0_1
py27-distribute 0.6.36_0        0.6.39_0
python27        2.7.3_1         2.7.5_0
sqlite3         3.7.16.1_0      3.7.17_0
zlib            1.2.7_0         1.2.8_0

I thought changes in atlas and distribute were most likely to affect the way numpy is build, so I force-deactivated them and activated the versions that were active on April 12. When I rebuilt numpy the import still failed with the same error. But there is always the possibility that I did not do this correctly, since I've never tried it before.

comment:2 Changed 11 years ago by cooljeanius (Eric Gallager)

Cc: egall@… added

Cc Me!

comment:3 Changed 11 years ago by jdgleeson

I'm getting closer to figuring this out. When python 2.7.5 is active gcc-4.0 is used to created extension libraries for numpy. When python 2.7.3 is active gcc-mp-4.7 is used to created extension libraries for numpy.

$ sudo port activate python27 @2.7.5_0
$ sudo port clean py27-numpy
$ sudo port -kd build py27-numpy > termout16.txt 2>&1
$ grep -m 2 bundle termout16.txt
/usr/bin/gcc-4.0 -bundle -undefined dynamic_lookup -L/opt47/local/lib -L/opt47/local/lib/db46 -m32 -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/dummymodule.o -Lbuild/temp.macosx-10.5-ppc-2.7 -o build/lib.macosx-10.5-ppc-2.7/numpy/core/_dummy.so
/usr/bin/gcc-4.0 -bundle -undefined dynamic_lookup -L/opt47/local/lib -L/opt47/local/lib/db46 -m32 -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/multiarray/multiarraymodule_onefile.o -Lbuild/temp.macosx-10.5-ppc-2.7 -lnpymath -lnpysort -o build/lib.macosx-10.5-ppc-2.7/numpy/core/multiarray.so

$ sudo port activate python27 @2.7.3_1       
$ sudo port clean py27-numpy
$ sudo port -kd build py27-numpy > termout17.txt 2>&1
$ grep -m 2 bundle termout17.txt
/opt47/local/bin/gcc-mp-4.7 -bundle -undefined dynamic_lookup -L/opt47/local/lib -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/dummymodule.o -Lbuild/temp.macosx-10.5-ppc-2.7 -o build/lib.macosx-10.5-ppc-2.7/numpy/core/_dummy.so
/opt47/local/bin/gcc-mp-4.7 -bundle -undefined dynamic_lookup -L/opt47/local/lib -m32 -m32 -m32 build/temp.macosx-10.5-ppc-2.7/numpy/core/src/multiarray/multiarraymodule_onefile.o -Lbuild/temp.macosx-10.5-ppc-2.7 -lnpymath -lnpysort -o build/lib.macosx-10.5-ppc-2.7/numpy/core/multiarray.so

comment:4 Changed 11 years ago by jdgleeson

The error first occurs with this CPython revision: http://hg.python.org/cpython/rev/1d061f83a6bf (I am grateful for hg bisect). Now I might have better luck creating a unit test.

comment:5 Changed 11 years ago by ned-deily (Ned Deily)

Looking at the log, it appears that, for each C extension module, /opt47/local/bin/gcc-mp-4.7 is being used by Distutils for the compile step but /usr/bin/gcc-4.0 is used for the link step (-bundle). Normally, the same compiler should be used for both. By default, Distutils tries to use the same compiler and options as was used to build Python itself, unless overridden. In particular, the CC env variable is used to override the compiler in the CC config var and the LDSHARED env var is used to supply the compiler and link options from the LDSHARED config var. (You can inspect Python's configuration variables with sysconfig.get_config_var('CC') or ('LDSHARED').) With Python 2.7.3, setting a CC environment variable to override the compiler caused Distutils to update the compiler string in LDSHARED as well. Unfortunately, in Python 2.7.4 (and 2.7.5), the update of LDSHARED when CC is specified no longer happens; if you use CC env variable to supply a different compiler, you also need to inspect the LDSHARED config var and supply a complete modified string in an LDSHARED env variable. I've opened an upstream issue to restore the inadvertently changed behavior (http://bugs.python.org/issue18080) and attached a patch that should fix the problem for 2.7.5. I have not looked into the Numpy build process nor the MacPorts changes to see if this is the same problem but reading the log here rang a bell. You could try to apply the patch to the MacPorts Python 2.7.5 and see if it fixes the problem. There were other rather extensive changes in Distutils to better support Xcode 4 for Python 2.7.4 so it is possible that something else in them may have affected Numpy builds.

comment:6 Changed 11 years ago by ned-deily (Ned Deily)

Cc: nad@… added
Last edited 11 years ago by ned-deily (Ned Deily) (previous) (diff)

comment:7 Changed 11 years ago by jdgleeson

Thanks, Ned! Your patch does indeed fix the numpy build problem.

I've attached Ned's patch (---/+++ paths edited to match Macports patch convention) and the associated patch for the python27 Portfile.

Can this ticket be transferred from py27-numpy to the python27 port, or do I need to open a new ticket and link to this one?

Changed 11 years ago by jdgleeson

Attachment: python27-Portfile.diff added

Changed 11 years ago by jdgleeson

comment:8 Changed 11 years ago by ned-deily (Ned Deily)

Good news and I'm sorry for the trouble it caused you. FWIW, I'm in the process of applying the patch upstream. It should be official in a couple of hours.

comment:9 Changed 11 years ago by mf2k (Frank Schima)

Owner: changed from macports-tickets@… to jwa@…
Port: python27 added

comment:10 Changed 11 years ago by reid@…

Cc: reid@… added

Cc Me!

comment:11 Changed 11 years ago by petrrr

Cc: Peter.Danecek@… added

Cc Me!

comment:12 Changed 11 years ago by jyrkiwahlstedt

Resolution: worksforme
Status: newclosed

Currently seems to be ok (with 2.7.6, 1.8.0)

Note: See TracTickets for help on using tickets.