py27-numpy @1.6.1_1 build fails with +atlas +gcc46

[mndavidoff (Monte Davidoff)]

The py27-numpy port fails to build on Mac OS X 10.6.8 with GCC 4.6:

$ sudo port -vs install py27-numpy +atlas +gcc46
...
C compiler: /opt/local/bin/gcc-mp-4.6 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -m64

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
gcc-mp-4.6: _configtest.c
/opt/local/bin/gcc-mp-4.6 -arch x86_64 _configtest.o -o _configtest
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
  File "setup.py", line 196, in <module>
    setup_package()
  File "setup.py", line 189, in setup_package
    configuration=configuration )
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.1/numpy/distutils/core.py", line 186, in setup
    return old_setup(**new_attr)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.1/numpy/distutils/command/build.py", line 37, in run
    old_build.run(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.1/numpy/distutils/command/build_src.py", line 152, in run
    self.build_sources()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.1/numpy/distutils/command/build_src.py", line 163, in build_sources
    self.build_library_sources(*libname_info)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.1/numpy/distutils/command/build_src.py", line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.1/numpy/distutils/command/build_src.py", line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 696, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program
shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.1" && /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build " returned error 1
Error: Target org.macports.build returned: shell command failed (see log for details)
Warning: the following items did not execute (for py27-numpy): org.macports.activate org.macports.build org.macports.destroot org.macports.install
Log for py27-numpy is at: /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/main.log
Error: Status 1 encountered during processing.
To report a bug, see <http://guide.macports.org/#project.tickets>

The complete build log is attached.

[skymoo (Adam Mercer)]

This seems specific to gcc46 and atlas as the other gcc variants are fine. Atlas support is notoriously flakey and is the reason why it's disabled by default.

miles: As you wrote the patch for the gcc46 variant any idea why gcc46 is passing the arch flags and the other complier variants aren't?

[miles@…]

I can't say for sure, but it looks like the numpy build script internally does a lot of version checking e.g. in numpy/distutils/fcompiler/gnu.py, so I wouldn't be surprised if the issue is hardcoded version numbers or a change in the format of the version string. Unfortunately it seems like it will take some debugging to work through, which I'm not able to do at the moment.

[skymoo (Adam Mercer)]

That does look like where this -arch flag is coming from, just need to figure out why now...

[skymoo (Adam Mercer)]

Seems to be coming from the customize method in the FCompiler class defined in numpy/distutils/fcompiler/__init.__.py. Not sure why this is only a problem for +atlas and +gcc46?

James: Any ideas if atlas could be doing something weird with the arch flags for gcc46?

[stromnov (Andrey Stromnov)]

Exactly the same problem with GCC 4.5 (py27-numpy +atlas +gcc45).

[stromnov (Andrey Stromnov)]

Cc Me!

[stromnov (Andrey Stromnov)]

Sorry, exactly the same problem with GCC 4.5 only with universal variant (py27-numpy +universal +atlas +gcc45).

Probably problem with GCC 4.6 compiler, which doesn't support -arch option at all.

~$ /opt/local/bin/gcc-mp-4.5 -arch x86_64
gcc-mp-4.5: no input files

~$ /opt/local/bin/gcc-mp-4.6 -arch x86_64
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'
gcc-mp-4.6: fatal error: no input files
compilation terminated.

[mf2k (Frank Schima)]

py27-numpy does not have a +gcc45 or +gcc46 variant.

[stromnov (Andrey Stromnov)]

Replying to macsforever2000@…:

py27-numpy does not have a +gcc45 or +gcc46 variant.

+gcc4x variants defined in if {[variant_isset atlas]} section. You can see it using: port variants py27-numpy +atlas.

[mf2k (Frank Schima)]

So variants can have variants? I learn something new every day.

[stromnov (Andrey Stromnov)]

Problem detected:

py-numpy includes special wrappers for handling -arch options (c-wrapper, cxx-wrapper and f-wrapper). So, build.env and destroot.env have special definitions for this case:

   build.env-append    CC="${worksrcpath}/c-wrapper" \
                            CXX="${worksrcpath}/cxx-wrapper" \
                            F77="${worksrcpath}/f-wrapper" \
                            F90="${worksrcpath}/f-wrapper"

        destroot.env-append CC="${worksrcpath}/c-wrapper" \
                            CXX="${worksrcpath}/cxx-wrapper" \
                            F77="${worksrcpath}/f-wrapper" \
                            F90="${worksrcpath}/f-wrapper"

But configure.compiler macports-gcc-4.x options overshadows that definitions and actual CC, CXX, F77 and F90 flags passed in environment are macports-gcc-4.x ones (for example CC: ${prefix}/bin/gcc-mp-4.5 instead of ${worksrcpath}/c-wrapper).

So, we must properly define compilers:

configure.cc    ${worksrcpath}/c-wrapper
configure.cxx   ${worksrcpath}/cxx-wrapper
configure.f77   ${worksrcpath}/f-wrapper
configure.f90   ${worksrcpath}/f-wrapper

[skymoo (Adam Mercer)]

So you mean that we should use configure.cc, and friends, in the build.env and destroot.env instead of setting the environment variables?

[stromnov (Andrey Stromnov)]

Replying to ram@…:

So you mean that we should use configure.cc, and friends, in the build.env and destroot.env instead of setting the environment variables?

I mean that for particular port we should use configure.cc and friends instead of setting the environment variables in the build.env and destroot.env.

Probably this is another, not py-numpy related, problem with environment variables. (You can add print os.environ in setup.py right after import os statement.)

[bonoba@…]

Cc Me!

[bonoba@…]

I am seeing this issue on Lion 10.7.3 with clang 3.1 (from XCode 4.3) and GCC 4.6.3

Does anybody have a patch for the Portfile?

[bonoba@…]

Never mind, I have figured it out.

[skymoo (Adam Mercer)]

Replying to bonoba@…:

Never mind, I have figured it out.

Patches are welcome.

[bonoba@…]

Well, I just added those configure.* strings suggested by stromnov@… at the end of post-configure function.

Basically you have two ways of compiling the thing with +universal:

1. install it with +universal -atlas -- in that case it builds with clang using Apple Accelerate Framework instead of ATLAS

2. apply the crude change I've described and install with +universal +atlas +gcc46 -gcc44 -gcc45

Both ways worked for me today.

[bonoba@…]

Correction: at the end of post-patch section

[skymoo (Adam Mercer)]

Can you attached a patch.

[bonoba@…]

Attached the patch.

Apply with: cd /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ patch < /tmp/py-numpy.correct-compiler.patch

[bonoba@…]

adds configure.* overrides to the end of the post-patch section to fix +atlas +universal +gcc46

[bonoba@…]

Correction:

Apply patch with (assumes the patch is in a file /tmp/py-numpy.correct-compiler.patch):

cd /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ patch -p1 < /tmp/py-numpy.correct-compiler.patch

[skymoo (Adam Mercer)]

thanks, I'll take a look at this after work.

[michelle.lynn.gill@…]

I am getting this same error trying to build py27-numpy @1.6.2_0+atlas+gcc46 (-universal) under MacPorts 2.1.1 with OS X 10.7.4. It looks like the above patch for configure variables patch has been incorporated, so I assume the issue must crop up somewhere else as well?

I was able to install py27-numpy @1.6.2_0+atlas+gcc45 just fine, though I don't recall which version of MacPorts was the latest at the time numpy version 1.6.2 was updated.

Here's the output of the install error. It looks like the offending -arch flag is still present. If I can test anything, please let me know.

Running from numpy source directory.non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
customize NAGFCompiler
Found executable /Volumes/Files/macports/bin/gfortran-mp-4.6
customize AbsoftFCompiler
customize IBMFCompiler
Could not locate executable xlf95
customize IntelFCompiler
Found executable /usr/bin/ifort
customize GnuFCompiler
gnu: no Fortran 90 compiler found
Could not locate executable g77
Could not locate executable f77
gnu: no Fortran 90 compiler found
customize Gnu95FCompiler
Found executable /Volumes/Files/macports/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}

C compiler: /Volumes/Files/macports/bin/gcc-mp-4.6 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -m64

compile options: '-c'
gcc-mp-4.6: _configtest.c
/Volumes/Files/macports/bin/gcc-mp-4.6 -arch x86_64 _configtest.o -L/Volumes/Files/macports/lib -lptf77blas -lptcblas -latlas -o _configtest
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'
failure.
removing: _configtest.c _configtest.o
Status: 255
Output: compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}

C compiler: /Volumes/Files/macports/bin/gcc-mp-4.6 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -m64

compile options: '-c'
gcc-mp-4.6: _configtest.c
/Volumes/Files/macports/bin/gcc-mp-4.6 -arch x86_64 _configtest.o -L/Volumes/Files/macports/lib -lptf77blas -lptcblas -latlas -o _configtest
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'

Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/Volumes/Files/macports/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

  FOUND:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/Volumes/Files/macports/lib']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /Volumes/Files/macports/lib
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/Volumes/Files/macports/lib']
    language = f77
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

  FOUND:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/Volumes/Files/macports/lib']
    language = f77
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]

running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building py_modules sources
creating build
creating build/src.macosx-10.7-x86_64-2.7
creating build/src.macosx-10.7-x86_64-2.7/numpy
creating build/src.macosx-10.7-x86_64-2.7/numpy/distutils
building library "npymath" sources
customize NAGFCompiler
customize AbsoftFCompiler
customize IBMFCompiler
customize IntelFCompiler
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize Gnu95FCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler using config
C compiler: /Volumes/Files/macports/bin/gcc-mp-4.6 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -m64

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
gcc-mp-4.6: _configtest.c
/Volumes/Files/macports/bin/gcc-mp-4.6 -arch x86_64 _configtest.o -o _configtest
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'
gcc-mp-4.6: error: x86_64: No such file or directory
gcc-mp-4.6: error: unrecognized option '-arch'
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
  File "setup.py", line 214, in <module>
    setup_package()
  File "setup.py", line 207, in setup_package
    configuration=configuration )
  File "/Volumes/Files/macports/var/macports/build/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.2/numpy/distutils/core.py", line 186, in setup
    return old_setup(**new_attr)
  File "/Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Volumes/Files/macports/var/macports/build/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.2/numpy/distutils/command/build.py", line 37, in run
    old_build.run(self)
  File "/Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Volumes/Files/macports/var/macports/build/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.2/numpy/distutils/command/build_src.py", line 152, in run
    self.build_sources()
  File "/Volumes/Files/macports/var/macports/build/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.2/numpy/distutils/command/build_src.py", line 163, in build_sources
    self.build_library_sources(*libname_info)
  File "/Volumes/Files/macports/var/macports/build/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.2/numpy/distutils/command/build_src.py", line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/Volumes/Files/macports/var/macports/build/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.2/numpy/distutils/command/build_src.py", line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 696, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program
Command failed:  cd "/Volumes/Files/macports/var/macports/build/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/work/numpy-1.6.2" && /Volumes/Files/macports/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 setup.py --no-user-cfg build 
Exit code: 1
Error: org.macports.build for port py27-numpy returned: command execution failed
Warning: targets not executed for py27-numpy: org.macports.activate org.macports.build org.macports.destroot org.macports.install
Please see the log file for port py27-numpy for details:
    /Volumes/Files/macports/var/macports/logs/_Volumes_Files_macports_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-numpy/py27-numpy/main.log
To report a bug, follow the instructions in the guide:
    http://guide.macports.org/#project.tickets

[sean@…]

Cc Me!

[jmroot (Joshua Root)]

r91459, r95318

[jmroot (Joshua Root)]

Also r95321