ImageMagick @6.9.9-40_1: has broken libomp detection logic, and when built with a macports-clang compiler inserts -lgomp into pkgconfig files, which breaks builds as -lgomp can't be found

[kencu (Ken)]

In the process of trying to upgrade virtuoso-7 on my SnowLeopard system that is configured to LibcxxOnOlderSystems, the rebuild of virtuoso-7 with clang-3.9 fails due to:

  CCLD     im.la
ld: library not found for -lgomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [im.la] Error 1

it appears that this build is picking up this library from ImageMagick's pkgconfig files:

$ pwd
/opt/local/lib/pkgconfig

$ ag lgomp .
ImageMagick++-6.Q16.pc
13:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

ImageMagick++.pc
13:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

ImageMagick-6.Q16.pc
14:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

ImageMagick.pc
14:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

Magick++-6.Q16.pc
13:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

Magick++.pc
13:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

MagickCore-6.Q16.pc
15:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

MagickCore.pc
15:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

MagickWand-6.Q16.pc
15:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

MagickWand.pc
15:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

Wand-6.Q16.pc
15:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

Wand.pc
15:Libs.private: -L${libdir} -l${libname}  -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp     -lm

AFAIK, lgomp is only available when using gcc and as above, errors out when using clang. I am not certain why ImageMagick is inserting lgomp into it's recommended link libraries.

[kencu (Ken)]

I overrode the pkg-config files by adding this to the virtuoso-7 Portfile:

configure.env-append IM_LIBS="-L/opt/local/lib -lMagickWand-6.Q16 -llcms2 -lfreetype -lfftw3 -lxml2 -lfontconfig -lfreetype -lXext -lSM -lICE -lX11 -lXt -llzma -lbz2 -lz -lltdl -lm -lMagickCore-6.Q16 -llcms2 -lfreetype -lfftw3 -lxml2 -lfontconfig -lfreetype -lXext -lSM -lICE -lX11 -lXt -llzma -lbz2 -lz -lltdl -lm"

and the build of virtuoso-7 completed successfully, again pointing back to ImageMagick's pkg-config files as the culprit I believe.

[ryandesign (Ryan Carsten Schmidt)]

Hmm. It doesn't sound like a MacPorts-specific problem.

I see a ​bug report that looks related to this issue, but as far as I can tell, it was ​fixed 2 years ago in ImageMagick 6.9.5-6.

There's also a much older MacPorts ticket #27039 that seems related, but I never found out what was going on there.

libgomp is of course involved in OpenMP, and we have a zillion tickets open about ImageMagick OpenMP-related stuff.

[kencu (Ken)]

I think you found a good clue to the issue. That ​commit results in this block in configure.ac.

#
# Find OpenMP library
#
GOMP_LIBS=''
if test "$enable_openmp" != 'no'; then
  if test "${GCC}" = "yes"; then
    # Open64 (passes for GCC but uses different OpenMP implementation)
    if test "x$GOMP_LIBS" = x ; then
      if $CC --version 2>&1 | grep Open64 > /dev/null ; then
        AC_CHECK_LIB(openmp,omp_get_num_procs,GOMP_LIBS="-lopenmp",,)
      fi
    fi
    # Clang (passes for GCC but uses different OpenMP implementation)
    if test "x$LIB_OMP" = x ; then
      if $CC --version 2>&1 | grep clang > /dev/null ; then
        AC_CHECK_LIB(omp,GOMP_parallel_start,LIB_OMP="-lomp",,)
      fi
    fi
    # GCC
    if test "x$GOMP_LIBS" = x ; then
      AC_CHECK_LIB(gomp,GOMP_parallel_start,GOMP_LIBS="-lgomp",,)
    fi
  else
    # Sun CC
    if test "x$GOMP_LIBS" = x ; then
      AC_CHECK_LIB(mtsk,sunw_mp_register_warn,GOMP_LIBS="-lmtsk",,)
    fi
    # AIX xlc
    if test "x$GOMP_LIBS" = x ; then
    AC_CHECK_LIB(xlsmp,_xlsmpFlush,GOMP_LIBS="-lxlsmp",,)
    fi
    # SGI IRIX 6.5 MIPSpro C/C++
    if test "x$GOMP_LIBS" = x ; then
    AC_CHECK_LIB(mp,mp_destroy,GOMP_LIBS="-lmp",,)
    fi
  fi
  LIBS="$GOMP_LIBS $LIBS"
fi
AC_SUBST(GOMP_LIBS)

on my system, both tests pass

configure:30053: checking for GOMP_parallel_start in -lomp
configure:30078: /opt/local/bin/clang-mp-5.0 -o conftest  -fopenmp -pipe -Os -arch x86_64 -Wall -mtune=westmere -fexceptions -D_FORTIFY_SOURCE=0 -pthread -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16  -I/opt/local/include -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -L/opt/local/lib -Wl,-headerpad_max_install_names -arch x86_64 conftest.c -lomp  -lm    >&5
configure:30078: $? = 0
configure:30087: result: yes
configure:30097: checking for GOMP_parallel_start in -lgomp
configure:30122: /opt/local/bin/clang-mp-5.0 -o conftest  -fopenmp -pipe -Os -arch x86_64 -Wall -mtune=westmere -fexceptions -D_FORTIFY_SOURCE=0 -pthread -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16  -I/opt/local/include -DMAGICKCORE_HDRI_ENABLE=0 -DMAGICKCORE_QUANTUM_DEPTH=16 -L/opt/local/lib -Wl,-headerpad_max_install_names -arch x86_64 conftest.c -lgomp  -lm    >&5
configure:30122: $? = 0
configure:30131: result: yes

First of all I'm not at this moment sure how my clang-5.0 links with -lgomp during this test, but fails when I'm trying to build virtuoso-7 with that link flag. But that aside, it seems like the configure.ac bit completely ignores whether the -lomp test passes or not -- it just inserts the GOMP_LIBS into the link libs no matter what.

And so -lgomp gets passed along forever more as one of the required link libs, even though -lomp worked.

Let me check if they fixed this in the newer versions of ImageMagick.

[kencu (Ken)]

Yes -- just a minor typo. I see it's still in ImageMagick master. No wonder omp is such a headache in ImageMagick! It has never been correctly autodetected, it would appear.

This patch fixes it:

--- configure.ac.old	2019-09-11 21:50:21.000000000 -0700
+++ configure.ac	2019-09-11 21:50:52.000000000 -0700
@@ -1413,9 +1413,9 @@
       fi
     fi
     # Clang (passes for GCC but uses different OpenMP implementation)
-    if test "x$LIB_OMP" = x ; then
+    if test "x$GOMP_LIBS" = x ; then
       if $CC --version 2>&1 | grep clang > /dev/null ; then
-        AC_CHECK_LIB(omp,GOMP_parallel_start,LIB_OMP="-lomp",,)
+        AC_CHECK_LIB(omp,GOMP_parallel_start,GOMP_LIBS="-lomp",,)
       fi
     fi
     # GCC

And then clang passes the -lomp test (if capable), and -lomp gets inserted into the default link flags instead of -lgomp, as it should be for clang:

$ pkg-config ImageMagick --libs --static
-L/opt/local/lib -lMagickCore-6.Q16 -llcms2 -lfreetype -lfftw3 -lxml2 -lfontconfig -lfreetype -lXext -lSM -lICE -lX11 -lXt -llzma -lbz2 -lz -lltdl -lm -lomp -lm

See: <​https://github.com/macports/macports-ports/pull/5257>

However, all might not even yet be 100% perfect. During the build of virtuoso-7, now, for some reason, my mp-clang-5.0 doesn't find -lomp even though it should be plainly there....

../../../libsrc/Wi/wi.h:2160:9: note: previous definition is here
#define mutex_leave(m)  pthread_mutex_unlock (&((m)->mtx_mtx))
        ^
2 warnings generated.
2 warnings generated.
  CCLD     im.la
ld: library not found for -lomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[kencu (Ken)]

The libomp library directory is not in the default library search path

$ port contents libomp | grep libomp.dylib
  /opt/local/lib/libomp/libomp.dylib

so libtool doesn't find it.

'/opt/local/bin/clang-mp-5.0 -bundle  -arch x86_64 -arch x86_64  -o .libs/im.so  .libs/im_la-im.o .libs/im_la-import_gate_virtuoso.o  -L/opt/local/lib/openssl-1.0 -L/opt/local/lib /opt/local/lib/libMagickWand-6.Q16.dylib /opt/local/lib/libMagickCore-6.Q16.dylib /opt/local/lib/liblcms2.dylib /opt/local/lib/libfftw3.dylib /opt/local/lib/libxml2.dylib /opt/local/lib/libfontconfig.dylib /opt/local/lib/libfreetype.dylib /opt/local/lib/libXext.dylib /opt/local/lib/libSM.dylib /opt/local/lib/libICE.dylib /opt/local/lib/libX11.dylib /opt/local/lib/libXt.dylib /opt/local/lib/liblzma.dylib -lbz2 -lz /opt/local/lib/libltdl.dylib -lomp -lm -Wl,-headerpad_max_install_names'
+ eval '/opt/local/bin/clang-mp-5.0 -bundle  -arch x86_64 -arch x86_64  -o .libs/im.so  .libs/im_la-im.o .libs/im_la-import_gate_virtuoso.o  -L/opt/local/lib/openssl-1.0 -L/opt/local/lib /opt/local/lib/libMagickWand-6.Q16.dylib /opt/local/lib/libMagickCore-6.Q16.dylib /opt/local/lib/liblcms2.dylib /opt/local/lib/libfftw3.dylib /opt/local/lib/libxml2.dylib /opt/local/lib/libfontconfig.dylib /opt/local/lib/libfreetype.dylib /opt/local/lib/libXext.dylib /opt/local/lib/libSM.dylib /opt/local/lib/libICE.dylib /opt/local/lib/libX11.dylib /opt/local/lib/libXt.dylib /opt/local/lib/liblzma.dylib -lbz2 -lz /opt/local/lib/libltdl.dylib -lomp -lm -Wl,-headerpad_max_install_names'
++ /opt/local/bin/clang-mp-5.0 -bundle -arch x86_64 -arch x86_64 -o .libs/im.so .libs/im_la-im.o .libs/im_la-import_gate_virtuoso.o -L/opt/local/lib/openssl-1.0 -L/opt/local/lib /opt/local/lib/libMagickWand-6.Q16.dylib /opt/local/lib/libMagickCore-6.Q16.dylib /opt/local/lib/liblcms2.dylib /opt/local/lib/libfftw3.dylib /opt/local/lib/libxml2.dylib /opt/local/lib/libfontconfig.dylib /opt/local/lib/libfreetype.dylib /opt/local/lib/libXext.dylib /opt/local/lib/libSM.dylib /opt/local/lib/libICE.dylib /opt/local/lib/libX11.dylib /opt/local/lib/libXt.dylib /opt/local/lib/liblzma.dylib -lbz2 -lz /opt/local/lib/libltdl.dylib -lomp -lm -Wl,-headerpad_max_install_names
ld: library not found for -lomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)

explicitly adding it to the LDFLAGS works.

[kencu (Ken)]

The libomp notes say that -fopenmp needs to be added to the link flags. Perhaps that lets the software find the link libraries automatically, but it doesn't find -lomp with that flag. This is going to take some sorting out.

Newer versions of MacPorts' clang, for some reason, pass the -lgomp test, and so stick that into the link flags, and this breaks builds when software uses the full pkg-config flags like virtuoso-7 does.

I notice that if you build ImageMagick on Mojave using clang-8.0 or similar, the exact same error happens, and you wind up with a broken setup, with -lgomp in the ImageMagick link flags:

MAGICK_LIBS=' -L/opt/local/lib -llcms2 -L/opt/local/lib -lfreetype   -L/opt/local/lib -lfftw3  -L/opt/local/lib -lxml2 -L/opt/local/lib -lfontconfig -lfreetype -lXext   -lSM -lICE -lX11  -lXt -L/opt/local/lib -llzma -lbz2 -L/opt/local/lib -lz -lltdl  -lm -lgomp    '

So ImageMagick would be just as broken on Mojave, if you built it with one of macports newer clang compilers (anything newer than 4.0, I would think).

I think that in the end, enabling openmp right now by default would be a mess. Some compilers support it, others don't. The flags in ImageMagick need to match the ones the compiler that is building software against ImageMagick might be able to support. At the very least, we'd need to fix the -lomp detection with the patch above, and also sort out how to pass in the libomp library location if we're going to leave it with -lomp (but I doubt that is the right way to do it).

For now, I'm voting we just turn openmp explicitly off...then nobody gets it (nobody does now anyway) but when ImageMagick is built with a newer clang compiler, at least it won't be serving up broken pkg-config files.

That way we at least get the same behaviour on all systems. Then, over time, we can sort out the right way of invoking it in pkg-config. That'll be more important once Apple's clang starts supporting it.

ImageMagick's detection right now is just broken (although you can force it to work by passing the libomp link libraries explicitly yourself, if the compiler you're using can support openmp.

[kencu (Ken)]

In d86e7e1adc21d0aa50370306f404253350dd93fb/macports-ports (master):

ImageMagick: explicity disable openmp

ImageMagick's openmp detection is not working correctly with clang,
particularly with llvm clang versions newer than 4.0.

see: #57009

Although Apple's clang just indicates openmp is not supported,
llvm's newer clang versions pass the openmp test, but in the end
are misconfigured by inserting -gomp into the pkg-config link flags.

Even when this is corrected to -lomp, the implementation is flawed, and the
libomp libraries cannot be found by the default ImageMagick pkg-config libs.

Finally, if ImageMagick were built with an openmp-supporting clang, such as
macports-clang-8.0, and if the link issues were properly sorted out, the
implementation and support is compiler dependent and the pkg-config files
may not match the capabilities of the compiler that attempts to compile against
openmp in ImageMagick if it is not the same or a similar compiler.

Until this is properly sorted out, explicitly disable openmp on ImageMagick,
which prevents incorrect pkg-config link libraries from being specified
when the compiler is a mp-clang compiler.

Once a workable plan is in place, a variant to enable openmp can be considered.

see: #24944