octave @4.2.0_1: requires C++11, does not build correctly on default setups on < 10.9

[mojca (Mojca Miklavec)]

Octave fails to build on pre-10.9 platforms with libstdc++:

:info:build In file included from libinterp/corefcn/Cell.cc:25:
:info:build In file included from ./config.h:3485:
:info:build ./oct-conf-post.h:186:12: fatal error: 'cstdint' file not found
:info:build #  include <cstdint>
:info:build            ^
:info:build preserving existing HG-ID file
:info:build make[3]: Entering directory `/path/to/octave/work/octave-4.2.0'
:info:build make[3]: Leaving directory `/path/to/octave/work/octave-4.2.0'
:info:build   CXX      libinterp/corefcn/libinterp_corefcn_libcorefcn_la-__contourc__.lo
:info:build In file included from libinterp/corefcn/__contourc__.cc:36:
:info:build In file included from ./config.h:3485:
:info:build ./oct-conf-post.h:186:12: fatal error: 'cstdint' file not found
:info:build #  include <cstdint>
:info:build            ^
:info:build 1 error generated.

Some buildbot links:

• ​10.6 ​10.6 i386
• ​10.7
• ​10.8

I didn't even check the PPC builds and 10.9 fails in a different way:

• ​10.9

See:

• ​http://stackoverflow.com/questions/10116724/clang-os-x-lion-cannot-find-cstdint

[mojca (Mojca Miklavec)]

Just for completeness, 4.0.3 installed without any problems.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In 410bebe/macports-ports:

octave: use compiler that works for OS and cxx_stdlib values

For a full build, octave requires a C++11 compatible compiler
with the same standard library (libc++ or libstdc++) as its
dependencies.

See #53044

[mojca (Mojca Miklavec)]

Didn't you accidentally delete the # character?

[mojca (Mojca Miklavec)]

I didn't test the new commit yet, but in my case GraphicsMagick has been compiled with the standard/default clang compiler against libstdc++ rather than with gcc (against gcc's own & incompatible libstdc++).

If the dependency indeed has to be compiled with the same compiler, you are calling for bigger troubles by switching to macports-gcc-6.

What role does GraphicsMagick play for Octave? Could it be disabled on older platforms if that is the likely source of troubles? Or is it absolutely necessary to do any plotting?

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Replying to mojca:

Didn't you accidentally delete the # character?

Not only did I accidentally delete the # character, it took two tries to fix it.
This is one of the times that I wish git didn't keep a full history of commits.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Replying to mojca:

I didn't test the new commit yet, but in my case GraphicsMagick has been compiled with the standard/default clang compiler against libstdc++ rather than with gcc (against gcc's own & incompatible libstdc++).

If the dependency indeed has to be compiled with the same compiler, you are calling for bigger troubles by switching to macports-gcc-6.

Octave only silently defaults to macports-gcc-6 if ${configure.cxx_stdlib} eq "libstdc++".
On 10.12, 10.11, and 10.10, the system clang is used.
On 10.9, macports-clang is used because the buildbot failure seems to indicate the system clang doesn't work.
On 10.8, macports-gcc is used since C++11 is required.
This assumes, of course, that the user hasn't intentionally set configure.cxx_stdlib to something else.

If the user specifically requests a gcc* variant, then the graphicsmagick variant is not the default and is marked in conflict.

Won't this keep the C++ standard library consistent?
I am a little bit new to compiler.blacklist and compiler.whitelist.
Please feel free to correct my assessment.

[mojca (Mojca Miklavec)]

As predicted the build is ​still broken:

Undefined symbols for architecture x86_64:
  "Magick::Image::ping(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      F__magick_ping__(octave_value_list const&, int) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
  "Magick::Image::attribute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)", referenced from:
      is_indexed(Magick::Image const&)  in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      fill_exif(octave_scalar_map&, Magick::Image&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      fill_exif_ints(octave_scalar_map&, Magick::Image&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      fill_exif_floats(octave_scalar_map&, Magick::Image&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      F__magick_finfo__(octave_value_list const&, int) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      octave_value_list read_images<boolNDArray>(std::vector<Magick::Image, std::allocator<Magick::Image> >&, Array<int> const&, int const&, octave_scalar_map const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      octave_value_list read_images<intNDArray<octave_int<unsigned char> > >(std::vector<Magick::Image, std::allocator<Magick::Image> >&, Array<int> const&, int const&, octave_scalar_map const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      ...
  "Magick::CoderInfo::CoderInfo(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      F__magick_formats__(octave_value_list const&, int) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
  "Magick::Image::magick[abi:cxx11]() const", referenced from:
      is_indexed(Magick::Image const&)  in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      F__magick_ping__(octave_value_list const&, int) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      F__magick_finfo__(octave_value_list const&, int) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      octave_value_list read_images<boolNDArray>(std::vector<Magick::Image, std::allocator<Magick::Image> >&, Array<int> const&, int const&, octave_scalar_map const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      octave_value_list read_images<intNDArray<octave_int<unsigned char> > >(std::vector<Magick::Image, std::allocator<Magick::Image> >&, Array<int> const&, int const&, octave_scalar_map const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      octave_value_list read_images<intNDArray<octave_int<unsigned short> > >(std::vector<Magick::Image, std::allocator<Magick::Image> >&, Array<int> const&, int const&, octave_scalar_map const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      octave_value_list read_images<FloatNDArray>(std::vector<Magick::Image, std::allocator<Magick::Image> >&, Array<int> const&, int const&, octave_scalar_map const&) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
      ...
  "Magick::Image::comment[abi:cxx11]() const", referenced from:
      F__magick_finfo__(octave_value_list const&, int) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
  "Magick::CoderInfo::description[abi:cxx11]() const", referenced from:
      F__magick_formats__(octave_value_list const&, int) in libcorefcn.a(libinterp_corefcn_libcorefcn_la-__magick_read__.o)
ld: symbol(s) not found for architecture x86_64

You cannot simply fall back to macports-gcc-6 just because it would be nice to have C++11.

[mojca (Mojca Miklavec)]

Replying to MarcusCalhoun-Lopez:

Replying to mojca:

Didn't you accidentally delete the # character?

Not only did I accidentally delete the # character, it took two tries to fix it.
This is one of the times that I wish git didn't keep a full history of commits.

As long as you don't push the changes to the main server, you can edit the commits however you want. But once you push to the server, you cannot do it any longer (you can do it on your own fork of course, it has only been disabled on the main server on purpose). Of course you have to test your changes before pushing :)

SVN is actually worse in that respect.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Replying to mojca:

As predicted the build is ​still broken:
You cannot simply fall back to macports-gcc-6 just because it would be nice to have C++11.

As of version 4.2, C++11 is a requirement of Octave, so we must find a way of using a C++11 compliant compiler.
I apparently do not understand libc++ vs libstdc++ as well as I though I did.
Another wrinkle is that I have very limited ability to test my code on older systems.
I will keep trying to find a solution, but any suggestions are welcome.
Thank you for helping to keep an eye on this.

[mojca (Mojca Miklavec)]

Replying to MarcusCalhoun-Lopez:

Octave only silently defaults to macports-gcc-6 if ${configure.cxx_stdlib} eq "libstdc++".

OK, but how exactly doe that help?

On 10.12, 10.11, and 10.10, the system clang is used.
On 10.9, macports-clang is used because the buildbot failure seems to indicate the system clang doesn't work.

That's both fine.

On 10.8, macports-gcc is used since C++11 is required.

That's generally bad. Unless you make sure that all dependencies were built with macports-gcc as well, but even then it's better to switch to libc++ for all.

If the user specifically requests a gcc* variant, then the graphicsmagick variant is not the default and is marked in conflict.

This doesn't seem to be the case with your current code:

if { [variant_isset graphicsmagick] && ${configure.cxx_stdlib} eq "libstdc++"} {
    compiler.blacklist-append   {macports-clang-*}
    compiler.whitelist-append   \
        macports-gcc-6          \
        macports-gcc-5          \
        macports-gcc-4.9        \
        macports-gcc-4.8        \
        macports-gcc-4.7
}

You don't set any variant anywhere and you actually switch to macports-gcc when you specifically know that graphicsmagic is being used.

If you set a whitelist, you don't need a blacklist.

I also noticed that you blacklist clang at various places in a redundant way:

compiler.blacklist-append   {*gcc-3*} {*gcc-4.[0-6]} {clang < 700} cc
compiler.blacklist-append { clang <= 318.0.61 }

Won't this keep the C++ standard library consistent?

No. The macports-gcc compiler is specifically not compatible with the default compiler. And in a very bad way.

Replying to MarcusCalhoun-Lopez:

As of version 4.2, C++11 is a requirement of Octave, so we must find a way of using a C++11 compliant compiler.

My suggestion would be to switch to libc++ and disable graphicsmagic unless libc++ is default already (in that case graphicsmagic can stay). The only other alternative is to provide an alternative build of graphicsmagic against libc++ (I do that with a subport wxWidgets-3.0-libcxx for example, but it might not always be always possible or practical or reasonable to do so.)

What exactly does graphicsmagic bring?

[mojca (Mojca Miklavec)]

So two questions in fact:

• What exactly is the role of graphicsmagic, what do users loose if it's missing
• Why would users want to use macports-gcc-6? For the reasons of performance perhaps?

[mojca (Mojca Miklavec)]

I'm adding Chris to this ticket. Chris, given your broad experience with the nasty ROOT6 dependencies, would you be willing to give a helping hand also to octave?

The following patch works for me (that is: octave compiles, but the patch is far from being ready to commit):

@@ -13,6 +13,7 @@ PortGroup           linear_algebra 1.0
 # See https://trac.macports.org/ticket/53044 for restricting clang versions further
 PortGroup compiler_blacklist_versions 1.0
 compiler.blacklist-append   {*gcc-3*} {*gcc-4.[0-6]} {clang < 700} cc
+configure.cxx_stdlib libc++
 
 name                octave
 version             4.2.0
@@ -515,6 +516,9 @@ default_variants-append +docs
 #    "Magick::Image::ping(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)"
 set magickConflict {}
 set magickDefault yes
+if {${cxx_stdlib} ne "libc++"} {
+    set magickDefault no
+}
 if {${configure.cxx_stdlib} ne "libstdc++"} {
     foreach gccVar ${compilers.gcc_variants} {
         if {[variant_exists ${gccVar}] } {
@@ -534,16 +538,6 @@ if {${magickDefault}} {
     default_variants-append +graphicsmagick
 }
 
-if { [variant_isset graphicsmagick] && ${configure.cxx_stdlib} eq "libstdc++"} {
-    compiler.blacklist-append   {macports-clang-*}
-    compiler.whitelist-append   \
-        macports-gcc-6          \
-        macports-gcc-5          \
-        macports-gcc-4.9        \
-        macports-gcc-4.8        \
-        macports-gcc-4.7
-}
-
 post-destroot {
     # put any startup commands in ${destroot}${prefix}/share/octave/${version}/m/startup/octaverc
     # see https://www.gnu.org/software/octave/doc/interpreter/Startup-Files.html

Note that configure.cxx_stdlib libc++ should only be added when building with clang, maybe something like

if {[string match *clang* ${configure.cxx}]} {
    configure.cxx_stdlib libc++
}

even though I'm not absolutely sure how this interacts with the rest of the variants and I'm not sure about the order of execution.

I'm not 100% sure about the following, but I hope that

if {${cxx_stdlib} ne "libc++"}

checks for global setting of cxx_stdlib. If it is set to libc++, then it should be safe to build with graphicsmagick as graphicsmagic should be built against libc++ as well and should be safe to use.

If default is libstdc++, then the user could build either with gcc or with clang, but not against graphicsmagic in any case. I would probably do it in an even more drastic way:

if {${cxx_stdlib} eq "libc++"} {
    eval "variant graphicsmagick description {use GraphicsMagick for image I/O} conflicts ${magickConflict} {
        depends_lib-append port:GraphicsMagick
        configure.args-replace --without-magick --with-magick=GraphicsMagick
    }"
}

That is: don't even add the variant if there is no way for the user to build it.

Note that the following part of code will be slightly problematic:

if {${configure.cxx_stdlib} ne "libstdc++"} {

because the port will always set libc++ for clang.

Another problem is that whenever a compiler is picked up automatically (in my case that would be macports-clang-3.8), you don't activate a variant for it.

Oh, wait ... I just tried to run octave now to test it and I get a GUI instead of the command line? Is this something new from version 4.2 or is that because I disabled graphicsmagick? I like the GUI of course, but is there any trick to force it to stay in the Terminal? And the GUI is stalled at exit. I have to force quit it.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Replying to mojca:

Oh, wait ... I just tried to run octave now to test it and I get a GUI instead of the command line? Is this something new from version 4.2 or is that because I disabled graphicsmagick? I like the GUI of course, but is there any trick to force it to stay in the Terminal? And the GUI is stalled at exit. I have to force quit it.

sudo port install octave -qt4 -qt5 should install octave without the GUI.
To force octave to run without the GUI, either run octave-cli instead of octave or run octave --no-gui-libs.

Replying to mojca:

What role does GraphicsMagick play for Octave? Could it be disabled on older platforms if that is the likely source of troubles? Or is it absolutely necessary to do any plotting?

It is not necessary for plotting.
To quote the configure script:

configure: WARNING: --without-magick specified.  The imread, imwrite, and imfinfo functions for reading and writing image files will not be fully functional.

I use them quite a bit, so I was just trying to get it to work for others.
I will make one more attempt to get everything working before I turn off graphicsmagick on older systems.

Thanks for your patience and help.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In 35f9174/macports-ports:

octave: use older libstdc++ ABI on older system

See #53044

[mojca (Mojca Miklavec)]

I don't understand how this was supposed to fix the problem. (And the builds on the buildbot still fail.)

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In 7ea8644/macports-ports:

octave: use compiler that works for OS and cxx_stdlib values

For a full build, octave requires a C++11 compatible compiler
with the same standard library (libc++ or libstdc++) as its
dependencies.

See #53044

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

In f9ab7cac/macports-ports:

octave: use older libstdc++ ABI on older system

See #53044

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

Replying to mojca:

I don't understand how this was supposed to fix the problem. (And the builds on the buildbot still fail.)

As I understand the situation, not only is libc++ incompatible with with libstdc++, but libstdc++ is incompatible with older versions.
According to ​this, -D_GLIBCXX_USE_CXX11_ABI=0 allows the current libstdc++ to be compatible with the (older) system libstdc++.

From what I can tell, octave now builds ​back to Mac OS 10.7.
Can this ticket be closed?

[mojca (Mojca Miklavec)]

Replying to MarcusCalhoun-Lopez:

As I understand the situation, not only is libc++ incompatible with with libstdc++, but libstdc++ is incompatible with older versions.

Yes, I know.

According to ​this, -D_GLIBCXX_USE_CXX11_ABI=0 allows the current libstdc++ to be compatible with the (older) system libstdc++.

If one can indeed use gcc 5/6/7 to build C++11 and keep it linked against the system libstdc++, we need to start discussing that on the mailing list. I seriously mean that because that would solve most of our problems.

From what I can tell, octave now builds ​back to Mac OS 10.7.
Can this ticket be closed?

Sure. I noticed that, but did not manage to figure out how you got it working.

[kencu (Ken)]

Replying to mojca:

If one can indeed use gcc 5/6/7 to build C++11 and keep it linked against the system libstdc++, we need to start discussing that on the mailing list. I seriously mean that because that would solve most of our problems.

As you know from our previous posts comment:ticket:52468:45 I've been doing this for months on 10.4 and 10.5 PPC with good success. Occasionally you need to build something with apple's compilers (due to complex ObjC or apple-specific flags), but usually things build and work well. A list of what I have currently installed on 10.4 PPC is here if you're curious <​https://github.com/kencu/TigerPorts/blob/master/installed_Tiger_Ports_list.txt>. 10.5 builds more, but I don't have that list up right now.

I have llvm-3.7, libc++, libc++abi, libunwind, libmacho, ld64, and cctools installed and apparently working on 10.5 PPC. Just can't quite get clang-3.7 to build through to completion at present -- some kind of out-of-memory error in the final link...

[kencu (Ken)]

Replying to MarcusCalhoun-Lopez:

From what I can tell, octave now builds ​back to Mac OS 10.7.
Can this ticket be closed?

For a data point, just built and runs successfully on 10.6 as well (upgraded to LibcxxOnOlderSystems)

Thanks!

[mojca (Mojca Miklavec)]

Replying to kencu:

As you know from our previous posts comment:ticket:52468:45 I've been doing this for months on 10.4 and 10.5 PPC with good success.

Isn't that different? My understand has been that you are linking all binaries against libstdc++ shipped by the latest version of GCC rather than linking against the system libstdc++. On PPC it's a problem because clang doesn't quite work correctly yet due to a small number of remaining bugs that need to be fixed first.

This is about 10.6-10.8 (10.5 would probably be too optimistic) where one would try to keep using the shipped libstdc++.

[kencu (Ken)]

I'm surprised anyone with 10.6+ would consider anything other than clang-3.7-8-9 and libc++ at this point -- Jeremy has fixed that up very nicely and 10.6 + seems to work great with libc++ for any software I have tried; I have no concerns about using that going forward for all software, and would suggest (IMHO) that macports just default to that forever and be done with it. But that's not up to me.

A few last hiccups with some last few build systems like scons persist -- I have personally fixed all those, as you know, by patching clang, but sooner or later macports will get all the portfiles in line for that.

10.5 (and 10.4) are the only issue I have left with cxx11 -- although just tonight I have clang-3.6 installed and compiling c++ software on 10.5 PPC linking against libc++, so I still hold out hope there. In the meantime, tho, 10.5 and 10.4 are where I'm using gcc6.

[mojca (Mojca Miklavec)]

Replying to kencu:

I'm surprised anyone with 10.6+ would consider anything other than clang-3.7-8-9 and libc++ at this point

There's a single reason for that: #50448 that's blocking new build slaves from being started. Building everything from source is a bit annoying.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

As far as I can tell, this is fixed.
Please feel free to reopen if new problems crop up.