py-scipy: not completely universal

[daweonline@…]

I've added "i386 x86_64" as universal flags in macports.conf. I've built python2.6 enabling universal and I successfully run it in 64 bit mode. I've built p26-scipy +universal but:

>>> import scipy.signal
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/signal/__init__.py", line 9, in <module>
    from bsplines import *
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/signal/bsplines.py", line 3, in <module>
    import scipy.special
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/__init__.py", line 8, in <module>
    from basic import *
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/basic.py", line 8, in <module>
    from _cephes import *
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so, 2): no suitable image found.  Did find:
	/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so: mach-o, but wrong architecture

and also:

>>> import scipy.linalg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/__init__.py", line 8, in <module>
    from basic import *
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/basic.py", line 17, in <module>
    from lapack import get_lapack_funcs
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/lapack.py", line 17, in <module>
    from scipy.linalg import flapack
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so, 2): no suitable image found.  Did find:
	/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: mach-o, but wrong architecture

I've checked and there are a lot of others object only built for i386. Changing the universal flags to x86_64 only doesn't work. It appears that the build chain of scipy understands "universal" as "i386 ppc" only.

$ port version
Version: 1.710

$ uname -a
Darwin host032.2b11.ifom-ieo-campus.it 9.6.3 Darwin Kernel Version 9.6.3: Tue Jan 20 18:26:40 PST 2009; root:xnu-1228.10.33~1/RELEASE_I386 i386

[daweonline@…]

Cc Me!

[jmroot (Joshua Root)]

I assume this applies to py25 as well. One reason off the top of my head that scipy wouldn't build universal is that gfortran doesn't accept -arch.

[skymoo (Adam Mercer)]

There's discussion of this very issue on the scipy and numpy mailing lists, it seems that they changed something in numpy to try and fix issues with universal fortran compilers that has had adverse reactions for non-universal gfortran compilers

[daweonline@…]

I've done modifying this line

265 for arch in ["i686", "x86_64"]:

in

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/distutils/fcompiler/gnu.py

and installing without macports... It should be said this way used gfortran-4.2 downloaded here

​http://www.maths.otago.ac.nz/~fonnesbeck/gfortran-4.2.3.dmg

which has been patched for universal binaries support. I don't know if gfortran installed with macports supports '-arch' flag.

[(none)]

Milestone Port Bugs deleted

[Veence (Vincent)]

This version of gfortran is also available here : ​http://r.research.att.com/tools/

But the way it handles the command line is buggy. For example :

-> /usr/local/bin/gfortran -arch i386
i686-apple-darwin8-gfortran-4.2: no input files
-> /usr/local/bin/gfortran -arch i386 -arch x86_64
->

Strange, no? If two archs flags are present, there is no stderr output.

Here is a way to build a universal scipy with this compiler located in /usr/local/bin:

1. Add this file in the ${portpath} file directory, it wraps /usr/local/bin/gfortran-4.2, fixes the bug above and another one in the building process (automatic addition of -arch i686 and -arch ppc; why ???):

-> more files/gf.in
#!/bin/sh

NOF=yes
IGNORE=no
ARGS=

for i in $@ ; do
if [ $IGNORE == 'yes' ]; then
  IGNORE=no
else
  if [ $i == '-arch' ]; then
    IGNORE=yes
  else
    ARGS=`echo $ARGS " " $i`
    if [ $i == '-o' ]; then
      NOF=no
    fi
  fi
fi
done

if [ $NOF == 'yes' ]; then
/usr/local/bin/gfortran-4.2 $@
else
/usr/local/bin/gfortran-4.2 XXX $ARGS
fi

2. Add this variant to Portfile

variant gf42univ conflicts gcc42 gcc43 gcc44 description "Use gfortan-4.2 from http://r.research.att.com/tools to build universal binaries" {

if {! [file exists /usr/local/bin/gfortran-4.2]} then {
   puts "> Please install gfortran universal from http:////r.research.att.com//tools//"
   exit 1
}

set fc_options "build_clib --fcompiler=gnu95 build_ext --fcompiler=gnu95 config_fc --fcompiler=gnu95 \
		  --f77exec ${portpath}/files/gf --f90exec ${portpath}/files/gf"

set uarch ""
foreach arch ${universal_archs} {
	append uarch "-arch " ${arch} " "
}

delete ${portpath}/files/gf
copy ${portpath}/files/gf.in ${portpath}/files/gf
reinplace "s|XXX|${uarch}|" ${portpath}/files/gf

build.cmd-append      ${fc_options}
destroot.cmd-append   ${fc_options}
}

Maybe one should also add a system "chmod +x ${portpath}/files/gf" ?
Vincent

[jameskyle@…]

Though this ticket is fairly old, I'll chime in that the macports atlas/blas/lapack libraries are not universal. scipy and numpy are now building against the macports libs by default.

[Veence (Vincent)]

They easily could be. The only step to do that is to re-enable the --multilib option in gcc43 or gcc44 port. Cf. #22116, and then applying the same method as above, that is writing a small shell script wrapper that transforms gfortran-mp-4.x -arch XXX in gfortran-mp-4.x -m[32|64] according to arch type, and declare this shell script as gfortran default compiler.

[flip@…]

Cc Me!

[Veence (Vincent)]

Is anyone still interested by a universal (2-way: i386 + x86_64) build of scipy and numpy with atlas?

[skymoo (Adam Mercer)]

Replying to vince@…:

Is anyone still interested by a universal (2-way: i386 + x86_64) build of scipy and numpy with atlas?

Yep

[cpandar@…]

Replying to vince@…:

Is anyone still interested by a universal (2-way: i386 + x86_64) build of scipy and numpy with atlas?

Yes, very interested. Is it possible?

[cpandar@…]

Vincent,

this seems to be a really good way of building a universal scipy with the macports framework. I tried it, and with some minor modification, it seems like everything builds well, e.g.:

$ file /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/l
inalg/flapack.so 
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so: Mach-O universal binary with 2 architectures
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture x86_64):  Mach-O 64-bit bundle x86_64
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/flapack.so (for architecture i386):    Mach-O bundle i386

I am running into strange issues when trying to actually use the dynamic modules in python, though.

$ python -c "import scipy.linalg"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/__init__.py", line 8, in <module>
    from basic import *
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/basic.py", line 17, in <module>
    from lapack import get_lapack_funcs
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/linalg/lapack.py", line 17, in <module>
    from scipy.linalg import flapack
ImportError: dynamic module does not define init function (initflapack)

Did you run into this issue at all? Any ideas?

Thanks.

Replying to vince@…:

This version of gfortran is also available here : ​http://r.research.att.com/tools/

But the way it handles the command line is buggy. For example :

-> /usr/local/bin/gfortran -arch i386
i686-apple-darwin8-gfortran-4.2: no input files
-> /usr/local/bin/gfortran -arch i386 -arch x86_64
->

Strange, no? If two archs flags are present, there is no stderr output.

Here is a way to build a universal scipy with this compiler located in /usr/local/bin:

1. Add this file in the ${portpath} file directory, it wraps /usr/local/bin/gfortran-4.2, fixes the bug above and another one in the building process (automatic addition of -arch i686 and -arch ppc; why ???):

-> more files/gf.in
#!/bin/sh

NOF=yes
IGNORE=no
ARGS=

for i in $@ ; do
if [ $IGNORE == 'yes' ]; then
  IGNORE=no
else
  if [ $i == '-arch' ]; then
    IGNORE=yes
  else
    ARGS=`echo $ARGS " " $i`
    if [ $i == '-o' ]; then
      NOF=no
    fi
  fi
fi
done

if [ $NOF == 'yes' ]; then
/usr/local/bin/gfortran-4.2 $@
else
/usr/local/bin/gfortran-4.2 XXX $ARGS
fi

2. Add this variant to Portfile

variant gf42univ conflicts gcc42 gcc43 gcc44 description "Use gfortan-4.2 from http://r.research.att.com/tools to build universal binaries" {

if {! [file exists /usr/local/bin/gfortran-4.2]} then {
   puts "> Please install gfortran universal from http:////r.research.att.com//tools//"
   exit 1
}

set fc_options "build_clib --fcompiler=gnu95 build_ext --fcompiler=gnu95 config_fc --fcompiler=gnu95 \
		  --f77exec ${portpath}/files/gf --f90exec ${portpath}/files/gf"

set uarch ""
foreach arch ${universal_archs} {
	append uarch "-arch " ${arch} " "
}

delete ${portpath}/files/gf
copy ${portpath}/files/gf.in ${portpath}/files/gf
reinplace "s|XXX|${uarch}|" ${portpath}/files/gf

build.cmd-append      ${fc_options}
destroot.cmd-append   ${fc_options}
}

Maybe one should also add a system "chmod +x ${portpath}/files/gf" ?
Vincent

[Veence (Vincent)]

Okay. I must polish up the Portfile so that they look acceptable, and I'll post them (I hope today)

[Veence (Vincent)]

Portfile for universal numpy

[Veence (Vincent)]

Portfile for universal numpy

[Veence (Vincent)]

Wrapper for gcc,g++ and gfortran

[Veence (Vincent)]

Okay, there are three files to build 2-way universal i386/x86_64 or ppc/ppc64 (NOT tested) numpy and scipy (that's the Portfile called Portfile_scipy, sorry for the wrong comment).

To build universal numpy and scipy, you should first build a universal version of Atlas: I've posted a solution some time ago. Anyhow, you will need a multilib enabled version of gcc, preferably gcc44 or gcc45.

The 'wrapper' file shall go in the files directory of both ports. It is a Bourne shell script that analyses the various calls to compilers, transforms the -arch flags into suitable -mXX and then calls lipo if necessary. In a way, it reproduces what the Apple gcc does. It might be buggy, but at least it works for numpy or scipy. The 'wrapper' is a template that gets transformed into a C, a C++ and a Fortran wrapper by the Portfiles. It also corrects missing flags when invoked to link dynamic libraries (a.k.a .so, here).

Please test in a local repository if possible, and report any bug. It seems scipy.test fails test_iv_cephes_vs_amos_mass_test, but this seems to be a known bug (if I have figured out the few infos I found).

[anddam (Andrea D'Amore)]

Addedd exectuable permissions to the wrapper files

xinstall -m 755 ${filespath}/c-wrapper ${filespath}
xinstall -m 755 ${filespath}/c++-wrapper ${filespath}
xinstall -m 755 ${filespath}/f-wrapper ${filespath}

I'm attaching the edited portfiles.

[anddam (Andrea D'Amore)]

executable permission for wrappers

[anddam (Andrea D'Amore)]

executable permission for wrappers

[skymoo (Adam Mercer)]

any reports of success with this? as I personally don't use universal builds

[Veence (Vincent)]

Third version (hopefully last one)

[Veence (Vincent)]

Third version

[Veence (Vincent)]

I have attached a third version of the universal scipy and numpy Portfiles that should behave more correctly

• wrappers are not created unless the port is actually build (so lint or clean do not create them anymore);

• they are created in ${worksrcpath} rather than ${filespath} so they get clean automatically.

[clausenator@…]

Hi, I have the same problem:

sudo port install py26-numpy +no_gcc43

returns

--->  Building py26-numpy
Error: Target org.macports.build returned: shell command failed
Log for py26-numpy is at: /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py26-numpy/main.log
Error: Status 1 encountered during processing.
To report a bug, see <http://guide.macports.org/#project.tickets>

I tried to use the Portfile_numpy.3 from vince, following the steps outlined here​http://homepage.mac.com/simx/technonova/tips/using_experimental_macports_portfiles.html.

Unfortunately,

portindex ~/MacPorts/local-sources/

returns

Total number of ports parsed:	0 
Ports successfully parsed:	0 
Ports failed:			0 
Up-to-date ports skipped:	0

note: I renamed and placed the portfile to this:

~/MacPorts/local-sources/python/Portfile

Am I missing something? What am I supposed to do with the 'wrapper'?

[clausenator@…]

Cc Me!

[Veence (Vincent)]

Replying to clausenator@…:

I tried to use the Portfile_numpy.3 from vince, following the steps outlined here​http://homepage.mac.com/simx/technonova/tips/using_experimental_macports_portfiles.html.

Unfortunately,

portindex ~/MacPorts/local-sources/

returns

Total number of ports parsed:	0 
Ports successfully parsed:	0 
Ports failed:			0 
Up-to-date ports skipped:	0

What did you do exactly?

[clausenator@…]

Replying to vince@…:

Replying to clausenator@…: What did you do exactly?

Initially, I wanted to get py26-tables. So I first did

sudo port upgrade outdated

In that process, it wanted to update py26-numpy, which returned the above mentioned error message. This same error message was produced also with

sudo port install py26-tables

and with (from ​https://trac.macports.org/ticket/22360)

sudo port clean py26-numpy
sudo port install py26-numpy +no_gcc43

Then I found this ticket, including the portfiles provided. I had never dealt with portfiles, so I searched for some more info and found the above mentioned link. Here is what I did:

• I downloaded the Portfile_numpy.3, renamed it into portfile
  
• I created this folder structurer ~/MacPorts/local-sources/
  
• I looked in the portfile under "categories", and hence I created a new folder ~/MacPorts/local-sources/python where I put the portfile
  
• I added one line at the end of /opt/local/etc/macports/sources.conf to include ~/MacPorts/local-sources/
  
• I ran portindex ~/MacPorts/local-sources/

This returned zeros everywhere as mentioned above. I also renamed the portfile from "portfile" to "py26-numpy" because it seemed like this could make sense as well, but the results were also zeros.

One thing I am not sure about is what to do with the "wrapper" file. Also, again, I have never dealt with portfiles before, so essentially I have no clue what I am doing, I was just following the steps I found.

[hmgaudecker@…]

Cc Me!

[hans.ekkehard.plesser@…]

Replying to clausenator@…:

Then I found this ticket, including the portfiles provided. I had never dealt with portfiles, so I searched for some more info and found the above mentioned link. Here is what I did:

• I downloaded the Portfile_numpy.3, renamed it into portfile
  
• I created this folder structurer ~/MacPorts/local-sources/
  
• I looked in the portfile under "categories", and hence I created a new folder ~/MacPorts/local-sources/python where I put the portfile
  
• I added one line at the end of /opt/local/etc/macports/sources.conf to include ~/MacPorts/local-sources/
  
• I ran portindex ~/MacPorts/local-sources/

This returned zeros everywhere as mentioned above. I also renamed the portfile from "portfile" to "py26-numpy" because it seemed like this could make sense as well, but the results were also zeros.

One thing I am not sure about is what to do with the "wrapper" file. Also, again, I have never dealt with portfiles before, so essentially I have no clue what I am doing, I was just following the steps I found.

It took me quite a while to figure this out, so here is a description of what I (successfully) did in the end. For a number of reasons I placed my local ports in /opt/LocalDarwinPorts instead of ~/Macports, but that should not make a difference. So here is what I did:

• Created directory /opt/LocalDarwinPorts/local-sources
• Created directory /opt/LocalDarwinPorts/local-sources/python/py26-numpy
• Created directory /opt/LocalDarwinPorts/local-sources/python/py26-numpy/files
• Downloaded attachment:Portfile_numpy.3​ to /opt/LocalDarwinPorts/local-sources/python/py26-numpy and named it Portfile
• Downloaded attachment:wrapper​ to /opt/LocalDarwinPorts/local-sources/python/py26-numpy/files
• Copied all files in /opt/local/var/macports/sources/rsync.macports.org/release/ports/python/py26-numpy/files to /opt/LocalDarwinPorts/local-sources/python/py26-numpy/files
• Added file:///opt/LocalDarwinPorts/local-sources [nosync] to /opt/local/etc/macports/sources.conf just above rsync://rsync.macports.org/release/ports/ [default]
• Ran portindex /opt/LocalDarwinPorts/local-sources/
• Ran sudo port clean py26-numpy
• Ran sudo port upgrade py26-numpy

This worked fine and I now have

port -v installed py26-numpy
The following ports are currently installed:
  py26-numpy @1.4.0_0
  py26-numpy @1.4.1_1+gcc44 (active) platform='darwin 10' archs='x86_64'

Afterwards, I could upgrade py26-scipy straight from the standard portfile.

When will this wrapper solution be committed to the portfile repository, so one can avoid the manual workaround?

[michaelld (Michael Dickens)]

Cc Me!

[michaelld (Michael Dickens)]

The fix for the numpy issue, for me, is to remove all of the GCC variants (so lines 44-47) and thus change the command to "sudo port install py26-numpy [+universal]". This is one of possibly a few math-oriented ports that do not directly require linking to libgcc or libgfortran, so removing those variants should be safe.

I would encourage those on this ticket to try out this fix & see if it works for them, with and without +universal if possible.

[michaelld (Michael Dickens)]

There's a potential fix for the py26-numpy part of this issue on ticket #24942; please try the last diff & see if it works for you on this issue.

[hans.ekkehard.plesser@…]

Cc Me!

[michaelld (Michael Dickens)]

I've checked in the changes in r72220 for numpy; please do:

sudo port clean py26-numpy
sudo port selfupdate
sudo port install py26-numpy [+variants]

and see if it works. I don't know if fixing numpy will resolve this ticket or not. If so, then I hope the owner of this ticket please verify and close it out.

[michaelld (Michael Dickens)]

Even after the updated py26-numpy, py26-scipy is still not truly universal -- it does install correctly, but the *.so files are just one arch.

[jmroot (Joshua Root)]

What changes did you make to test this? It obviously doesn't work as-is because there's no universal variant, and if you enable it, we still make no attempt to tell the build system which archs we want it to compile for.

Out of curiosity, what do people need universal scipy for?

[michaelld (Michael Dickens)]

I was using the current svn Portfile of scipy, which I realized after the fact disables universal install. My interest was purely as part of upgrading numpy & addressing this ticket; I don't use scipy as universal myself, but I would also be curious as to who does need it to be universal.

That said, it looks like making scipy universal won't be too difficult -- assuming the SuiteSparse dependency isn't necessary. Anyone know why it's there? I see nothing in the debugging output to indicate that it is being found or used, but I certainly could be missing something.

Ignoring SuiteSparse for now: scipy follows closely the way numpy does things, which is to say that most of the setup.py arguments are ignored in favor of self-discovery of CC, CXX, F77, and F90, and then those are used internally. There's also an LDFLAGS overwriting issue again, which is just a matter of tracking down the correct setup.py script and patching it.

[michaelld (Michael Dickens)]

Just to follow up, one last time: I've given up trying to coerce scipy into compiling as universal. Removing the dependency on SuiteSparse (via a simple patch to a setup.py script) and adding in the LDFLAGS to various setup.py scripts, many of the fortran files are compiled just as native, not universal. I don't know if using the muniversal portgroup will help, since some of the fortran compiling is handled by imported modules that might ignore F##FLAGS and/or LDFLAGS. I think at least for the time being just leaving this port as "universal_variant no" make sense -- maybe the scipy / numpy folks will get around to it sometime.

[jmroot (Joshua Root)]

Closing. We can revisit the idea when upstream makes building universal more straightforward.

[Veence (Vincent)]

Yet, I still have my own personal universal copy

# port installed py26-scipy The following ports are currently installed:

py26-scipy @0.7.2_0+gcc45+universal (active)

I'm still unable to know why it was not imported…

[mojca (Mojca Miklavec)]

Replying to jmr@…:

Out of curiosity, what do people need universal scipy for?

Hardly relevant, but some ports depend(ed) on both scipy and 32-bit wxWidgets 2.8.

[larryv (Lawrence Velázquez)]

I have a local version of py27-scipy that correctly builds universal (i386/x86_64). I still have to clean it up a bit; hopefully I can get that done within the next week or two.

[larryv (Lawrence Velázquez)]

This is with SciPy 0.13.3, to be clear.

[cooljeanius (Eric Gallager)]

Cc Me!

[petrrr]

Cc Me!

[petrrr]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Replying to larryv@…:

I have a local version of py27-scipy that correctly builds universal (i386/x86_64). I still have to clean it up a bit; hopefully I can get that done within the next week or two.

Larry, have you worked on this further? If not maybe you could attach your work in progress.