py27-scipy @1.2.0+gfortran: ImportError: dlopen: Symbol not found: _cbbcsd_

[renzresearch]

"from scipy import optimize" causes an ImportError. "import scipy.optimize" results in the same ImportError. "import scipy" is fine. But it does not import the sub-module optimize.

% python
Python 2.7.15 (default, Oct  1 2018, 15:59:56) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy import optimize
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/__init__.py", line 387, in <module>
    from ._minimize import *
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/_minimize.py", line 26, in <module>
    from ._trustregion_dogleg import _minimize_dogleg
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/_trustregion_dogleg.py", line 5, in <module>
    import scipy.linalg
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/__init__.py", line 207, in <module>
    from ._decomp_update import *
  File "_decomp_update.pyx", line 1, in init scipy.linalg._decomp_update
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/cython_lapack.so, 2): Symbol not found: _cbbcsd_
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/cython_lapack.so
  Expected in: flat namespace
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/cython_lapack.so

[ryandesign (Ryan Carsten Schmidt)]

I can confirm this problem on High Sierra with py27-scipy @1.2.0_0+gfortran. The problem was not present with py27-scipy @1.1.0_1+gfortran.

[lockhart (Thomas Lockhart)]

Can also confirm this issue on Sierra, affecting the scipy.linalg submodule. I've uninstalled and reinstalled every package that seems to be related but this did not help:

sudo port install py36-cython py36-numpy py36-matplotlib py36-bokeh py36-bottleneck py36-tables py36-numba py36-astropy py36-numexpr py36-scipy py36-pandas py36-scikit-learn

sudo port install opencv +python36

[kencu (Ken)]

See this thread <​https://mail.python.org/pipermail/scipy-dev/2018-June/022856.html> when this issue turned up after a commit to master that has now been incorporated into 1.2.0.

In that thread, it looked like the issue is picking up the wrong libraries.

> The build logs show that it's picking up /usr/lib/libblas.* and
> /usr/lib/liblapack.* and using those. Apparently, they are missing
> symbols.
>

[dershow]

This bug also breaks at least one other port, but probably others. I just upgraded to 10.14, and followed the migration instructions. Then, in python, I tried to do: import seaborn as sns and I got the same error as above. It seems that seaborn (py27-seaborn etc) uses scipy trying to use it shows the same issue.

[duncanmmacleod (Duncan Macleod)]

You can resolve things be rebuilding with the +openblas variant, I think. Should that now be the default operation?

[NeilBowles]

Yes, this does the trick and solves all the other problems with the updated scipy install - I was having problems with py37, spyder etc. Thanks for pointing this out, installing scipy with the +openblas variant solves the problem.

[EJFielding (Eric Fielding)]

I ran into this same bug when I upgraded my py36-scipy to @1.2.0_0+gfortran. It used the prebuilt binary "py36-scipy-1.2.0_0+gfortran.darwin_17.x86_64.tbz2" for the install. I got the missing symbol error when I tried to run a Python3 scipy script. The same script worked when I activated the older @1.1.0_1+gfortran port. Trying the workaround to install with +openblas variant and, of course, it has to rebuild the whole py36-scipy and takes quite a while. I can confirm that py36-scipy @1.2.0_0+gfortran+openblas works.

[mf2k (Frank Schima)]

Has duplicate #57880.

[seanfarley (Sean Farley)]

I just read the thread on the mailing list and it's unfortunate they dropped support for Apple's Accelerate framework (a big reason to use macs in academia). Well, if support has been dropped then it's dropped. But this is a much larger problem in general. So many things depend on scipy that we should decide if we want to revamp the port tree to use openblas by default. Having Apple's Accelerate mixed with openblas will lead to linking issues. And we can't depend on variants so I don't think +openblas should remain a variant for scipy.

I absolutely don't have the time to work on revamping this. Does anyone else have any ideas?

[mf2k (Frank Schima)]

Has duplicate #57907.

[DanyPPage]

How do I rebuild scipy with +openblast ?

[EJFielding (Eric Fielding)]

Replying to DanyPPage:

How do I rebuild scipy with +openblast ?

install scipy port again with the extra variant (not it is +openblas, not +openblast), of course you might have a different Python version, so adjust that:

port install py36-scipy +gfortran +openblas

[kencu (Ken)]

I don't use this but suggest we don't just leave it broken, so IMHO

1. revert

or

2. use openblas by default

Are there any other choices to be considered? If no, go with 2 and take it from there.

[DanyPPage]

It worked and solved my problem: scipy seems to work properly. THANX !

[ATL-Flaneur (Andreas Yankopolus)]

The present default config that fails to install seems less than optimal. Perhaps make openblas the default (option 2 above), with Apple's Accelerate being a configurable option.

[reneeotten (Renee Otten)]

Replying to kencu:

I don't use this but suggest we don't just leave it broken, so IMHO

1. revert

or

2. use openblas by default

Are there any other choices to be considered? If no, go with 2 and take it from there.

there are similiar +gfortran issues with py-numpy (see ​ticket-56954, which are not really figured out yet as far as I can tell. I think changing the default (compiler) variants, should be done in an consistent manner for at least these two ports (and perhaps others in the Python scientific stack).

[michaelld (Michael Dickens)]

I'll make (2) the default. Looking at the Portfile right now ...

[michaelld (Michael Dickens)]

In b10ed01c11fe815974c7d3acd766de3fcae93b8b/macports-ports (master):

py-scipy: Make +openblas a default variant

... at least temporarily, to overcome issues with Apple's /usr/lib/libblas.* and /usr/lib/liblapack.* missing a symbol.

Closes: #57829

[michaelld (Michael Dickens)]

I made +openblas a default variant. I don't know if this change will take when doing an update from a prior version without this variant being enabled. But it will take care of new/fresh installs, which is progress.

[seanfarley (Sean Farley)]

@michaelld it's a bit more difficult than that, though. I remember there was a package that added missing symbols to accelerate (veclibFort) that might be the way to quickly patch over this. Otherwise, as I mentioned above, they're be a lot of mismatched symbols. Perhaps Apple should just stop shipping /usr/lib/lib{blas,lapack}* if they're broken.

[michaelld (Michael Dickens)]

hmmm ... OK I'll check out "veclibFort" & see if it helps. I'd prefer to not use +openblas as a default, but it does work (I've verified; it's actually my default, which is why I committed this in the first place, since it "just worked" for me LOL).

[michaelld (Michael Dickens)]

I will note that when doing an "update", the new default variant is not selected. One has to do "install" to get it. Sigh.

[seanfarley (Sean Farley)]

I agree with your change by the way; just making sure you're aware of the potential problems users might have.

[mf2k (Frank Schima)]

Replying to michaelld:

I will note that when doing an "update", the new default variant is not selected. One has to do "install" to get it. Sigh.

Not entirely true. One can also do this:

sudo port upgrade --enforce-variants py37-scipy

[nwautomator (That Guy --^)]

This does not work on a clean Macports (i.e. no installed ports) on MacOS Mojave 10.14.2. I'm getting an error that the clang37 port is not supported on Mojave.

[michaelld (Michael Dickens)]

Well the clang37 port isn't supported on 10.14. No idea why it's trying to do that. port rdeps py27-scipy | grep -i clang returns empty, so SciPy doesn't by default depend on any Clang on 10.14 (which is what I'm running the query on). Maybe you have some default variants set in a config file?