kdelibs4 @4.6.3: Undefined symbols _uncompress _compress

[macports@…]

kdelibs4 (still) fails to build on 10.6.5 Snow Leopard with XCode 3.2.4 apparently due to unresolved symbols during linking. Zlib @1.2.5_0 is installed and active.

This seems to be the same problem, I reported in #26776. Unfortunatly the problem was not resolved in 4.5 and 4.6.

I just did a complete selfupdate and "upgrade outdated" (which took nearly the whole day) before trying to install kdelibs4.

:info:build ld: warning: Iex::InputExc::~InputExc()has different visibility (default) in /opt/local/lib/libIlmImf.a(ImfRleCompressor.o) and (hidden) in CMakeFiles/kimg_exr.dir/exr.o
:info:build ld: warning: Iex::InputExc::~InputExc()has different visibility (default) in /opt/local/lib/libIlmImf.a(ImfRleCompressor.o) and (hidden) in CMakeFiles/kimg_exr.dir/exr.o
:info:build ld: warning: typeinfo for Iex::InputExchas different visibility (default) in /opt/local/lib/libIlmImf.a(ImfRleCompressor.o) and (hidden) in CMakeFiles/kimg_exr.dir/exr.o
:info:build ld: warning: vtable for Iex::InputExchas different visibility (default) in /opt/local/lib/libIlmImf.a(ImfRleCompressor.o) and (hidden) in CMakeFiles/kimg_exr.dir/exr.o
:info:build ld: warning: typeinfo name for Iex::InputExchas different visibility (default) in /opt/local/lib/libIlmImf.a(ImfRleCompressor.o) and (hidden) in CMakeFiles/kimg_exr.dir/exr.o
:info:build   "_uncompress", referenced from:
:info:build       Imf::Pxr24Compressor::uncompress(char const*, int, Imath::Box<Imath::Vec2<int> >, char const*&)in libIlmImf.a(ImfPxr24Compressor.o)
:info:build       Imf::ZipCompressor::uncompress(char const*, int, int, char const*&)in libIlmImf.a(ImfZipCompressor.o)
:info:build   "_compress", referenced from:
:info:build       Imf::Pxr24Compressor::compress(char const*, int, Imath::Box<Imath::Vec2<int> >, char const*&)in libIlmImf.a(ImfPxr24Compressor.o)
:info:build       Imf::ZipCompressor::compress(char const*, int, int, char const*&)in libIlmImf.a(ImfZipCompressor.o)
:info:build shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdelibs4/work/build" && /usr/bin/make -j2 all " returned error 2
:error:build Target org.macports.build returned: shell command failed (see log for details)
:debug:build Backtrace: shell command failed (see log for details)

The whole log is attached.

[ryandesign (Ryan Carsten Schmidt)]

In addition to #26776, this had also been previously reported as #24427 and ​on the mailing list.

[ryandesign (Ryan Carsten Schmidt)]

Replying to macports@…:

10.6.5 Snow Leopard with XCode 3.2.4

Please consider upgrading to Mac OS X 10.6.7 and Xcode 3.2.6.

[macports@…]

Well considering past version upgrades which didnt help with this one, I'm kind of reluctant to believe that it will help this time. But I just started the downloads (just around 5 GB... should be finished in the morning).

[michaelld (Michael Dickens)]

Could you clean the install & then re-do it from scratch & then post the resulting -full- logfile? The current one includes just part of the build phase, and seeing what's happening in the configure phase would be useful. Another item you could attach is the file "port dir kdelibs4/work/build/CMakeCache.txt".

[macports@…]

@michaelld Yes - would you like this with my current setup or should I upgrade OSX and Xcode first?

[macports@…]

I did a "port clean kdelibs4" and "port install kdelibs4" with my current setup. The log is attached.

[macports@…]

build after clean

[michaelld (Michael Dickens)]

Probably doesn't matter if you did the clean & build before or after upgrading. If you have to choose one or the other, I'd choose XCode -- but, I also feel that keeping my OSX install up to date is "a good thing" unless there's a good reason why not (e.g., an application that works with 10.6.5 but not yet with 10.6.[6,7]).

[michaelld (Michael Dickens)]

Can you please also attach the file "port dir kdelibs4/work/build/CMakeCache.txt"?

[michaelld (Michael Dickens)]

And, what does "ls -lAF /opt/local/lib/libIlmImf*" return for you?

[macports@…]

Output of ls -lAF /opt/local/lib/libIlmImf*

perseus:~ eligs$ ls -lAF /opt/local/lib/libIlmImf*
-rw-r--r--  2 root  admin  1408040 16 Nov  2010 /opt/local/lib/libIlmImf.a
-rwxr-xr-x  2 root  admin     1000 16 Nov  2010 /opt/local/lib/libIlmImf.la*

CMakeCache.txt is appended

[michaelld (Michael Dickens)]

Strange; the issue now is:

:info:build In file included from /opt/local/include/QtCore/qstringlist.h:46,
:info:build                  from /opt/local/include/QtGui/qcolor.h:47,
:info:build                  from /opt/local/include/QtGui/qpixmap.h:46,
:info:build                  from /opt/local/include/QtGui/qpainter.h:49,
:info:build                  from /opt/local/include/QtGui/QPainter:1,
:info:build                  from /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_kde_kdelibs4/work/kdelibs-4.6.3/kdeui/plotting/kplotpoint.cpp:24:
:info:build /opt/local/include/QtCore/qdatastream.h: In member function 'QDataStream& QDataStream::operator<<(quint64)':
:info:build /opt/local/include/QtCore/qdatastream.h:238: internal compiler error: Segmentation fault

If you try to install again, does it come up with the same error? That might imply that upgrading XCode could help.

[michaelld (Michael Dickens)]

The issue for the first logfile is that OpenEXR somehow was installed just as static libraries, not dynamic ones, and its PKGCONFIG file (if any) was then ignored -- it states that LDFLAGS should include -lz, which would provide the _compress and _uncompress functions that were not found.

The primary difference between your 'configure' output and mine is:

:info:configure -- Found OPENEXR: /opt/local/lib/libImath.a;/opt/local/lib/libIlmImf.a;/opt/local/lib/libIex.a;/opt/local/lib/libHalf.a;/opt/local/lib/libIlmThread.a

Mine are all dylib, not .a . You might try reinstalling the ports ilmbase and openexr, just to see if doing that adds in .dylib versions of the above.

[macports@…]

Ok, I updated OS-X (to 10.6.7) then XCode (to 3.2.6). Then I uninstalled, cleand and installed ilmbase & openexr. The libs are still no .dylibs:

perseus:~ eligs$ ls -lAF /opt/local/lib/libIlm*
-rw-r--r--  2 root  admin  38336 24 Mai 12:31 /opt/local/lib/libIlmThread.a
-rwxr-xr-x  2 root  admin    921 24 Mai 12:31 /opt/local/lib/libIlmThread.la*

Then I cleaned kdelibs4 and tried another install run. It failed (due to missing symbols). The log is attached...

[michaelld (Michael Dickens)]

OK; 2 items now:

1) Does the file "/opt/local/lib/pkgconfig/OpenEXR.pc" exist? If so, please attach it.

2) Attach the output (the file openexr_build_out.txt) of "sudo port -d build openexr > openexr_build_out.txt 2>&1".

[macports@…]

done ...

[michaelld (Michael Dickens)]

OK; thanks. It is clear that OpenEXR is being build just as a static library, not as a dynamic one. Not sure why, but we'll keep testing & work on figuring this out.

[michaelld (Michael Dickens)]

Do "sudo port -d build ilmbase > ilmbase_build_out.txt 2>&1", and then attach the output file "ilmbase_build_out.txt" as well as the file "port dir ilmbase/work/ilmbase-1.0.2/config.log". OpenEXR depends on ilmbase's libraries, and since those are not dylibs I think libtool is correctly just building static libraries for OpenEXR. Thus, the question remains as to why ilmbase isn't creating dylibs. The above might (or not) shed some light.

[macports@…]

Attached ilmbase build output...

[michaelld (Michael Dickens)]

Do you know where "/bin/f77" came from? That seems to be messing up the 'configure' script, and it is not a standard Apple install. Can you try moving that file to some other name (e.g., "tmp_f77") and then re-running the build command for ilmbase (after cleaning)? You don't have to post the build log -- just check to see if the "getopt" error still happens, and if not, then if the libtool link command generates both dynamic and static libraries.

[macports@…]

That seems to be a link to "/usr/local/bin/fc" ... when I look at the creation date it seems, that I compiled & installed this when I played around with some old Fortran software last year. I removed the link and now there seems to be no getopt problem.

But there are still no dynamic libraries and the problem when building kdelibs persists.

[nerdling (Jeremy Lavergne)]

Any packages built after that fortran software was installed might just need to be rebuilt (to avoid passing on building static libraries).

[macports@…]

Urgs... that seems to be a lot. Just to get it right, would a "clean installed" and then "build installed" be sufficient?

[nerdling (Jeremy Lavergne)]

port clean installed is a good start. The second part I'd recommend the forced upgrade route (paraphrasing, please check the man pages): port upgrade --force installed. If you knew the lowest common denominator you could use something like this: port -R upgrade BOTTOM_PORT

[macports@…]

I did a forced upgrade of all installed ports (took only 12 hours ;) ). Now the dynamic libraries seem to be there:

perseus:~ eligs$ ls -lAF /opt/local/lib/libIlmImf*
-rwxr-xr-x  2 root  admin  1711296 27 Mai 11:04 /opt/local/lib/libIlmImf.6.dylib*
-rw-r--r--  2 root  admin  2776232 27 Mai 11:04 /opt/local/lib/libIlmImf.a
lrwxr-xr-x  1 root  admin       17 27 Mai 11:04 /opt/local/lib/libIlmImf.dylib@ -> libIlmImf.6.dylib
-rwxr-xr-x  2 root  admin     1050 27 Mai 11:04 /opt/local/lib/libIlmImf.la*

And kdelibs4 build and installation finally works. Thanks a lot !!

But I would like to understand why they were missing and why dylibs in the dependencies are required to build another dynamic library?

[michaelld (Michael Dickens)]

You're welcome; I'm glad we got it sorted out! Thanks for your prompt responses, btw; that really helps.

[michaelld (Michael Dickens)]

As to your other questions: It's sort of complicated, but I'll give it a go.

libtool (GNU's, not Apple's), when told to link a library or executable, is looking at the requested libraries & figuring out whether or not they exist as dynamic and/or static, as well as whether or not dynamic and/or static have been requested for linking. by the user (or, port in this case). When libtool cannot find all of the required dynamic ones, and the requested static ones all exist (as in your case), it tries to build a static library only. When it can find both static and dynamic, it builds both if it has been told to do so (which is usually the case). Now, at least with the OpenEXR and ilmbase ports, the build system it trying to build as both dynamic and static, but only static dependencies exist so that's what libtool builds. I would guess that some prior dependent of KDELIBS4 took issue with /bin/f77 not responding as it expected, and decided to not build dynamic libraries -- because f77 is erroring out when checking for dynamic linking capabilities as determined by some 'configure' script. Hence why you ended up with static libraries; just a guess, of course.

When linking libraries (whether static or dynamic) there are 2 basic ways of including external dependencies: the new library can be linked such that the dependency -must- be included after it, or it can be linked such that all of the dependency is included in it. In the distant past, the latter approach was taken since most libraries were static, and having a single dependent library was much easier than having multiple dependencies. In more recent years, the trend has moved towards the former and using dynamic libraries -- ones that can be loaded a runtime and then unloaded after they are no longer necessary. This method uses less in-RAM memory (since a given library can be shared across multiple applications), and also allows for developers to recompile a given library & test it out, without having to recompile an executable that depends on it. I think in the case of OpenEXR, which relies on 'libz' for compress and uncompress, it was being linked using the former method such that to create an executable one would need to specify "-lopenexr -lz" -- the 'libz' was not implied by the "libopenexr". For static libraries, there is no way to know this chain of dependencies without having some external file, such as that provided by PKGCONFIG, to define it -- or, just to know what the dependencies are beforehand.

Now, that leads us back to why KDELIBS4 failed: Most configure scripts these days are written to use PKGCONFIG when available. PKGCONFIG files provides info on how to include headers and link with install libraries for a given project via a specifically named file -- no matter if using static or dynamic linking. The configure for OpenEXR and ilmbase are very clever, and check for PKGCONFIG first & if not available, then it does the checking manually. KDELIBS4 uses CMake, which in general -does not- use PKGCONFIG, instead relying on its internal scripts for determining how to include headers and link with installed libraries. Time and again in my dealings with CMake, I've found very poorly written scripts that barely function and are easy to break -- some are installed by CMake itself, while others are found in the local project. I think what happens with KDELIBS4 is that the CMake script only checks to see if the openexr library is available, and not whether it has dependencies that need to be met.

Combining the various parts: KDELIBS4's configure script finds the static OpenEXR library, but does not know that it is dependent on 'libz'. Because the OpenEXR static library was linked such that libz was also required, and CMake doesn't know about this dependency, during linking in KDELIBS4 only the OpenEXR static library is used & thus linking fails due to undefined symbols (those found in libz).

Well, the above certainly sounds good; no idea how close I really am. And, as I said, it's a bit complex; I hope I did it justice :)

[macports@…]

Thanks for taking your time to answer my question and giving me some insigth in this whole configure/build script madness :)

[michaelld (Michael Dickens)]

You're welcome; and, you are quite correct that the configure / build stuff is madness. To name those build systems that I have used -- and there are more out there that I haven't -- QMake (part of Qt), CMake, GNU Make / Autotools, and BJam (was required by Boost a while back, but I think it's a separate project now). Each has benefits and drawbacks, depending on what you want to do. Some use PKGCONFIG or other external info (e.g., as installed by the project itself), while others do checking internally only; you basically have to pick your build system & go with it & hope it works well enough for anything you want to do. It's also impossible to test all possible system configurations, but one can use reasonable logic to work around most issues. Unfortunately, most developers do not care enough to follow reasonable logic, and so we end up with messes like this ticket's -- that said: it's the upstream developer's issue, not the end-user's. Anyway, I'm glad your MacPorts install is up and running!