atlas libraries improperly linked; prevents shogun installation

[ryandesign (Ryan Carsten Schmidt)]

When trying to install shogun (@0.7.3_0) I get:

/usr/bin/ld: warning can't open dynamic library: libptf77blas.dylib referenced from: /mp/lib/liblapack.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)
/usr/bin/ld: warning can't open dynamic library: libptcblas.dylib referenced from: /mp/lib/liblapack.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)
/usr/bin/ld: warning can't open dynamic library: libatlas.dylib referenced from: /mp/lib/liblapack.dylib (checking for undefined symbols may be affected) (No such file or directory, errno = 2)

Looking at this liblapack.dylib (provided by atlas @3.8.3_0) we see it's linking with these other libraries just by the name of the library, when on Mac OS X one must use the absolute path to the library (as is already being correctly done for e.g. libgfortran.3.dylib):

$ port provides /mp/lib/liblapack.dylib
/mp/lib/liblapack.dylib is provided by: atlas
$ otool -L /mp/lib/liblapack.dylib
/mp/lib/liblapack.dylib:
        liblapack.dylib (compatibility version 0.0.0, current version 0.0.0)
        libptf77blas.dylib (compatibility version 0.0.0, current version 0.0.0)
        /mp/lib/gcc43/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
        libptcblas.dylib (compatibility version 0.0.0, current version 0.0.0)
        libatlas.dylib (compatibility version 0.0.0, current version 0.0.0)
        /mp/lib/gcc43/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.3.11)

The author of the shogun port has noticed this in some way, and added this message to the post-install of the shogun port:

You will need to set:
DYLD_FALLBACK_LIBRARY_PATH=${prefix}/lib
Or shogun will not be able to find the macports atlas libs

However, the port never gets that far (fails during build). Ultimately, this is not shogun's responsibility to fix; it's atlas's.

I'm using Mac OS X 10.4.11 with Xcode 2.5 on Intel Core2 Duo with MacPorts 1.7.1.

[jameskyle@…]

I'm looking at how to resolve this. The lapack and atlas build process is tightly coupled and the linking to the atlas/blas libs in lapack is done before the libs are installed into the macports prefix. Thus the lack of absolute paths.

[ryandesign (Ryan Carsten Schmidt)]

I would just report the problem to the authors of the software, since it's not going to be a MacPorts-specific issue but a general issue when compiling their software on Mac OS X. They may not have been aware that on Mac OS X they should use the absolute path in their installname.

[jameskyle@…]

The atlas lib build scripts don't built dylib's. This is a step that the user on osx must take themselves and do manually. There's an errata on the topic that was used in the Portfile. However, the errata did not set the -install_name for when creating the dylib's and so my portfile also neglected this step. Toby pointed this out to me and I'm currently testing the revised build:

  set ld "ld -dynamic -arch i386 -dylib -single_module -dead_strip -x \
          -all_load -L. -L${prefix}/lib/gcc43 -ldylib1.o -dylib_install_name"
  
  system "cd ${workpath}/${name}-${version}/build/lib && \
          ${ld} libatlas.dylib libatlas.a -o libatlas.dylib -lSystem \
          -install_name ${prefix}/lib/libatlas.dylib"
  
  system "cd ${workpath}/${name}-${version}/build/lib && \
          ${ld} libptcblas.dylib libptcblas.a -o libptcblas.dylib -latlas \
           -lSystem -install_name ${prefix}/lib/libptcblas.dylib \
           -compatibility_version 1.0.0 -current_version 218.0.0"
  
  system "cd ${workpath}/${name}-${version}/build/lib && \
          ${ld} libptf77blas.dylib libptf77blas.a -o libptf77blas.dylib \
           -lgfortran -latlas -lSystem \
           -install_name ${prefix}/lib/libptf77blas.dylib \
           -compatibility_version 1.0.0 -current_version 218.0.0"

  system "cd ${workpath}/${name}-${version}/build/lib && \
          ${ld} liblapack.dylib liblapack.a -o liblapack.dylib -lptf77blas \
          -lgfortran -lptcblas -latlas -lgcc_s.1 -lSystem \
          -install_name ${prefix}/lib/liblapack.dylib \
          -compatibility_version 1.0.0 -current_version 218.0.0"

[jameskyle@…]

Thank you for your assistance is building a better atlas/lapack.

[ryandesign (Ryan Carsten Schmidt)]

jameskyle committed a fix in r53058. But now I get:

DEBUG: Executing proc-pre-org.macports.destroot-destroot-0
ld: -i argument: nstall_name must have a ':' between its symbol names
Error: Target org.macports.destroot returned: shell command "cd /mp/var/macports/build/_Users_rschmidt_macports_dports_math_atlas/work/atlas-3.8.3/build/lib &&  ld -dynamic -arch i386 -dylib -single_module -dead_strip -x  -all_load -L. -L/mp/lib/gcc43 -ldylib1.o -dylib_install_name libatlas.dylib libatlas.a -o libatlas.dylib -lSystem  -install_name /mp/lib/libatlas.dylib" returned error 1
Command output: ld: -i argument: nstall_name must have a ':' between its symbol names

I'm also concerned about the "-arch i386" part of the change. That would seem to make this port incompatible with PowerPC Macs. Is specifying the architecture necessary at all?

Once you do fix the issue, don't forget to increase the revision so everybody who already had the port installed gets informed of the update via "port outdated".

[jameskyle@…]

"I'm also concerned about the "-arch i386" part of the change. That would seem to make this port incompatible with PowerPC Macs. Is specifying the architecture necessary at all?"

I noticed that as well, was going to tackle it after fixing the dylib's. I was talking about appropriate changes in the #macports channel, first I'll try to omit the arch all together.

I'm not getting your above ld error at all though. I've compiled atlas about 3 or 4 times now while debugging the shogun port. The error looks like a type of ld -i nstall_name, but I have no such typo in my local port and svn stat shows no changes.

[jameskyle@…]

This is what I get if I don't specify the arch:

ld warning: -arch not specified

Though on my mac pro the resulting library is i386, I don't think this is the best option. My concern was how to appropriately determine the arch. I could do a shell out to determine the actual arch and assume all ppc are ppc (not ppc64) and all intel are i386 and not x86_64, but that feels kludgy and when Snow Leopard comes out would likely be incorrect.

I'm open to suggestions.

(also, on this most recent build I am still not getting the above ld errors)

[ryandesign (Ryan Carsten Schmidt)]

Are you on Leopard? Perhaps the issue only manifests on earlier systems. I see ​some hits for this error message on Google dating back to 2002.

[jameskyle@…]

Yes, leopard here. I don't have a tiger machine available. I'll look over those hits and see if I can find the root cause and work around it.

[jameskyle@…]

jmr looked up the ld man page for me on tiger. I think I have a fix. testing.

[jameskyle@…]

Please test, I think these changes will give you a error free build. I also changed i386 to ${os.arch}.

[jameskyle@…]

If you're on ppc, you might get a build error during linking. I just patched that.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jameskyle@…:

Please test, I think these changes will give you a error free build. I also changed i386 to ${os.arch}.

Getting closer! But not quite there yet:

DEBUG: Executing proc-pre-org.macports.destroot-destroot-0
ld: unknown flag: -compatibility_version
Error: Target org.macports.destroot returned: shell command "cd /mp/var/macports/build/_Users_rschmidt_macports_dports_math_atlas/work/atlas-3.8.3/build/lib &&  ld -arch i386 -dynamic -dylib -single_module -dead_strip -x  -all_load -L. -L/mp/lib/gcc43 -ldylib1.o -dylib_install_name /mp/lib/libptcblas.dylib libptcblas.a -o  libptcblas.dylib -latlas  -lSystem -compatibility_version 1.0.0 -current_version 218.0.0" returned error 1
Command output: ld: unknown flag: -compatibility_version

[jameskyle@…]

I can remove those, they're optional.

Removed.

[ryandesign (Ryan Carsten Schmidt)]

jameskyle removed that bit in r53082 and now it works! I increased the revision in r53135 so everyone gets this change.