py-scipy: work with updated SuiteSparse 4.0.0

[zan@…]

Patch to make Scipy's UMFPack work with updated SuiteSparse.

Related tickets: metis ​35018 and SuiteSparse ​35019

[zan@…]

Unified diff of Portfile

[zan@…]

Patchfile for umfpack

[skymoo (Adam Mercer)]

Patch didn't apply cleanly, there are small changes here so it's not difficult to manually apply the patch.

I do have a question however: Why remove the --no-user-cfg option? It builds cleanly without removing it?

I've pushed the patch minus the removal of the --no-user-cfg option in r94990. If it turns out it needs to be removed that can be addressed later...

[zan@…]

--no-user-cfg makes setup.py ignore the site.cfg local configuration file. I used site.cfg to make scipy find the SuiteSparse libraries needed for umfpack.

During compilation without the patch, libumfpack.a and libamd.a are found and so umfpack is built. There are no compilation errors. However, 'nm -u _ _umfpack.so' in scipy/sparse/linalg/dsolve/umfpack shows that several _cholmod functions are undefined. The only way to detect this is to do a complete installation of scipy, and then:

% python
>>> import scipy
>>> t = scipy.test('full')
>>> print t.skipped
(Test(test_umfpack.TestFactorization.test_complex_lu), Exception('Skipping test: 
test_complex_lu\nUMFPACK appears not to be compiled',)).....<more skipped>

This patch tells setup.py to look for SuiteSparse(.dylib) in stead of umfpack and amd.

[skymoo (Adam Mercer)]

I don't see that in the reason for the skipped tests, there are 10 failures though:

======================================================================
FAIL: test_asum (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 58, in test_asum
    assert_almost_equal(f([3,-4,5]),12)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals
 ACTUAL: 0.0
 DESIRED: 12

======================================================================
FAIL: test_dot (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 67, in test_dot
    assert_almost_equal(f([3,-4,5],[2,5,1]),-9)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals
 ACTUAL: 0.0
 DESIRED: -9

======================================================================
FAIL: test_nrm2 (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 78, in test_nrm2
    assert_almost_equal(f([3,-4,5]),math.sqrt(50))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals
 ACTUAL: 0.0
 DESIRED: 7.0710678118654755

======================================================================
FAIL: test_basic.TestNorm.test_overflow
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_basic.py", line 581, in test_overflow
    assert_almost_equal(norm(a), a)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 452, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 800, in assert_array_almost_equal
    header=('Arrays are not almost equal to %d decimals' % decimal))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals

(mismatch 100.0%)
 x: array(-0.0)
 y: array([  1.00000002e+20], dtype=float32)

======================================================================
FAIL: test_basic.TestNorm.test_stable
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_basic.py", line 586, in test_stable
    assert_almost_equal(norm(a) - 1e4, 0.5)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals
 ACTUAL: -10000.0
 DESIRED: 0.5

======================================================================
FAIL: test_basic.TestNorm.test_types
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_basic.py", line 568, in test_types
    assert_allclose(norm(x), np.sqrt(14), rtol=tol)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 1168, in assert_allclose
    verbose=verbose, header=header)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=2.38419e-06, atol=0

(mismatch 100.0%)
 x: array(1.0842021724855044e-19)
 y: array(3.7416573867739413)

======================================================================
FAIL: test_asum (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 99, in test_asum
    assert_almost_equal(f([3,-4,5]),12)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals
 ACTUAL: 0.0
 DESIRED: 12

======================================================================
FAIL: test_dot (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 109, in test_dot
    assert_almost_equal(f([3,-4,5],[2,5,1]),-9)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals
 ACTUAL: 0.0
 DESIRED: -9

======================================================================
FAIL: test_nrm2 (test_blas.TestFBLAS1Simple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 127, in test_nrm2
    assert_almost_equal(f([3,-4,5]),math.sqrt(50))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals
 ACTUAL: 0.0
 DESIRED: 7.0710678118654755

======================================================================
FAIL: test_iv_cephes_vs_amos_mass_test (test_basic.TestBessel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/tests/test_basic.py", line 1656, in test_iv_cephes_vs_amos_mass_test
    assert_(dc[k] < 1e-9, (v[k], x[k], special.iv(v[k], x[k]), special.iv(v[k], x[k]+0j)))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: (189.2947429454936, 3.0238805556481037, 4.089165443940765e-317, 0j)

----------------------------------------------------------------------
Ran 5834 tests in 643.557s

FAILED (KNOWNFAIL=14, SKIP=26, failures=10)

do you see these also?

[zan@…]

You were right, --no-user-cfg does not need to be deleted, it does not do what I thought it would. site.cfg from the patch file gets picked up and umfpack is built correctly.

After building I get 14 knownfail (single precision FFT, zero-order splines, fancy indexing, cephes, test_scxx_dict.TestDictGetItemOp), 18 skipped (all to do wih clapack/flapack), and 1 failure (test_iv_cephes_vs_amos_mass_test). The results are the same for the +atlas variant. Both are built with macports gcc 4.5 (variant +gcc45).

[skymoo (Adam Mercer)]

Not sure why we're getting different results from the testsuite. What version of OS X are you using? 10.7.4 here.

[zan@…]

I'm on 10.6.8, building for x86_64.

Your errors, except the last one which I also get, are mentioned ​in the scipy bug tracker. But that is unrelated to this ticket.

[skymoo (Adam Mercer)]

Looks like I filled that ticket, I'd forgotten about that...