gcc10: Symbol not found: _environ, Referenced from: libgfortran.5.dylib, Expected in: flat namespace

[ryandesign (Ryan Carsten Schmidt)]

There is something wrong with at least the py37-numpy subport on at least Mac OS X 10.7. Software that uses numpy ​fails to build like this:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/__init__.py", line 17, in <module>
    from . import multiarray
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Symbol not found: _environ
  Referenced from: /opt/local/lib/libgcc/libgfortran.5.dylib
  Expected in: flat namespace
 in /opt/local/lib/libgcc/libgfortran.5.dylib

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gsl_dist/gsl_Extension.py", line 25, in <module>
    from array_includes import array_include_dirs        
  File "gsl_dist/array_includes.py", line 7, in <module>
    from numpy.distutils.misc_util import get_numpy_include_dirs
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/__init__.py", line 47, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.7 from "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.17.3" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Symbol not found: _environ
  Referenced from: /opt/local/lib/libgcc/libgfortran.5.dylib
  Expected in: flat namespace
 in /opt/local/lib/libgcc/libgfortran.5.dylib


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    from gsl_Extension import gsl_Extension
  File "gsl_dist/gsl_Extension.py", line 27, in <module>
    from pygsl.gsl_dist.array_includes import array_include_dirs
  File "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_python_py-gsl/py37-gsl/work/pygsl-2.3.0/pygsl/__init__.py", line 47, in <module>
    import pygsl.errno
ModuleNotFoundError: No module named 'pygsl.errno'
Command failed:  cd "/opt/local/var/macports/build/_opt_bblocal_var_buildworker_ports_build_ports_python_py-gsl/py37-gsl/work/pygsl-2.3.0" && /opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 setup.py config 
Exit code: 1

[ryandesign (Ryan Carsten Schmidt)]

We did just do the libc++ switchover for 10.6-10.8. We did purge all the old archives that linked with /usr/lib/libstdc++.dylib so that they could be rebuilt with libc++. But it is possible that this is somehow related to that, and that we missed purging some archives that needed to be. (For example, we initially missed the ports that used the cxx11 1.1 portgroup, but have subsequently purged them as well.) If you can identify other archives that need purging let me know.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #59964.

[ryandesign (Ryan Carsten Schmidt)]

ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Symbol not found: _environ
  Referenced from: /opt/local/lib/libgcc/libgfortran.5.dylib
  Expected in: flat namespace

I think now that this is a gfortran problem.

environ used to be a thing that you weren't allowed to access from dylibs. Instead, you were supposed to use *_NSGetEnviron().

It looks like environ may have become available in 10.8, so software written by developers who don't test on older systems anymore may not know or remember that they were supposed to do it a different way before.

See ​https://gitlab.haskell.org/ghc/ghc/-/commit/43d12dc65e3bf974751486e7ec6be623831f705e for an example of how another project solved this. Something like that may need to be patched into the gfortran code somewhere.

If it's not that, then it could be an SDK-related issue. On 10.7 we use Xcode 4.6.3 which contains both the 10.7 and 10.8 SDKs. Maybe gfortran has found the 10.8 SDK and is trying to use symbols from it that are not available when running on 10.7. See #60637 for another instance where I suspect that may be the case.

[reneeotten (Renee Otten)]

ticket #62092 reports the same issue when building py-numpy on 10.7

[RobK88]

FYI -- I am able to build py39-numpy and py-numpy on Lion (10.7.5) without any issues.

bash-3.2$ sudo port clean  py39-numpy
--->  Cleaning py39-numpy

bash-3.2$ sudo port install  py39-numpy
--->  Computing dependencies for py39-numpy
--->  Fetching distfiles for py39-numpy
--->  Verifying checksums for py39-numpy
--->  Extracting py39-numpy
--->  Applying patches to py39-numpy
--->  Configuring py39-numpy
--->  Building py39-numpy
--->  Staging py39-numpy into destroot
--->  Installing py39-numpy @1.20.1_0+gfortran+openblas
--->  Activating py39-numpy @1.20.1_0+gfortran+openblas
--->  Cleaning py39-numpy
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.                             
--->  No broken ports found.
bash-3.2$ 
bash-3.2$ sudo port install  py-numpy
--->  Computing dependencies for py-numpy
--->  Fetching distfiles for py-numpy
--->  Verifying checksums for py-numpy
--->  Extracting py-numpy
--->  Configuring py-numpy
--->  Building py-numpy
--->  Staging py-numpy into destroot
--->  Installing py-numpy @1.20.1_0
--->  Activating py-numpy @1.20.1_0
--->  Cleaning py-numpy
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.
bash-3.2$ 

But I am getting a similar error when trying to build py39-matplotlib on Lion. Perhaps, it is Fortran related as well. See #62523

[RobK88]

FYI -- Even though I can build and install py39-numpy on my Mac running Lion, it looks like py39-numpy is not working properly.

Perhaps the problem with all the various numpy builds lies with the libgfortran.5.dylib library?

bash-3.2$ python3.9 -c "from numpy.core import multiarray"
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 2): Symbol not found: _environ
  Referenced from: /opt/local/lib/libgcc/libgfortran.5.dylib
  Expected in: flat namespace
 in /opt/local/lib/libgcc/libgfortran.5.dylib

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/__init__.py", line 145, in <module>
    from . import core
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/opt/local/bin/python3.9"
  * The NumPy version is: "1.20.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 2): Symbol not found: _environ
  Referenced from: /opt/local/lib/libgcc/libgfortran.5.dylib
  Expected in: flat namespace
 in /opt/local/lib/libgcc/libgfortran.5.dylib

bash-3.2$ 

[ryandesign (Ryan Carsten Schmidt)]

Replying to RobK88:

Perhaps the problem with all the various numpy builds lies with the libgfortran.5.dylib library?

Right. That's what I proposed in comment:3.

[reneeotten (Renee Otten)]

has duplicate: #62523

[kencu (Ken)]

No doubt this is a bug in libgfortran in gcc10 (at least gcc10).

In

libgfortran/intrinsics/execute_command_line.c

we have this bit:

#ifdef HAVE_POSIX_SPAWN
#include <spawn.h>
extern char **environ;
#endif

You will notice that this is just sitting loose like this:

extern char **environ;

In other files, gcc goes to great length to work around this issue, eg in ada/env.c:

char **
__gnat_environ (void)
{
#if defined (__MINGW32__)
  return _environ;
#elif defined (__sun__)
  extern char **_environ;
  return _environ;
#elif defined (__APPLE__) \
     && !(defined (__arm__) \
          || defined (__arm64__)                \
          || defined (__IOS_SIMULATOR__))
  return *_NSGetEnviron ();
#elif ! (defined (__vxworks))
  extern char **environ;
  return environ;
#else
  #if defined (__RTP__) || defined (VTHREADS) || (_WRS_VXWORKS_MAJOR <= 6)
    return environ;
  #elif (_WRS_VXWORKS_MAJOR >= 7)
    char **task_environ;

    task_environ = envGet (taskIdSelf ());

    if (task_environ == NULL)
       return ppGlobalEnviron;
    else
       return task_environ;
  #endif
#endif
}

I'll report this upstream if it's not already reported.

I'll see if I can come up with a workaround for us. Does anyone know with certainty what systems are affected by this? I can't actually see it being used in the 10.8 SDK, other than in PHP.

$ pwd
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include
$ ag -Q *environ
php/main/php.h
261:extern char **environ;

[kencu (Ken)]

It's already fixed upstream, in exactly the same way I was going to fix it:

<​https://github.com/gcc-mirror/gcc/blob/499fa254ae8c9752d8c2cf3130b13ffddfd83546/libgfortran/intrinsics/execute_command_line.c#L37>

#ifdef HAVE_POSIX_SPAWN
#include <spawn.h>
# ifdef __APPLE__
#  include <crt_externs.h>
#  define environ (*_NSGetEnviron ())
# else
extern char **environ;
# endif
#endif

[kencu (Ken)]

There is a new release for gcc coming out just around the corner. I'll see when that might be coming, rather than revbump a patch into ours only to have it be replaced in a few days.

[kencu (Ken)]

Looks like Iain fixed this ​nearly three months ago, and it's in the gcc10 branch, so it's sure to be fixed in the imminent gcc10 update that is coming this week.

[kencu (Ken)]

a new version of gcc10 with the fix in it is due within 10 days. If there is anyone really impatient, use this in the meantime:

<​https://github.com/kencu/macports-ports/tree/libgcc10fortranenvironfix/lang/gcc10>

It fixes this issue:

$ python3.9 -c "from numpy.core import multiarray"
 -- nothing --

[kencu (Ken)]

this should now be fixed by [87ad2e503c6c8ea81d4fbb4f3f1078b669967118/macports-ports]