qscintilla: requires C++11, not using the right compiler

[mojca (Mojca Miklavec)]

qscintilla no longer compiles without C++11:

../src/Document.cpp:22:10: fatal error: 'regex' file not found
#include <regex>
         ^

[michaelld (Michael Dickens)]

In 01168f73/macports-ports:

qscintilla: require c++11; rev-bump just in case; addresses ticket #53625 .

[michaelld (Michael Dickens)]

Thanks for the pointer!

[mojca (Mojca Miklavec)]

This doesn't work yet because the port is not using the right compiler.

Neither the compiler nor the flags setting the stdlib are respected during the build.

[michaelld (Michael Dickens)]

In b9de20c6/macports-ports:

qscintilla: use qmake PortGroup instead of qt4, to "use the right compiler" and c++11 flags per ticket #53625 .

[michaelld (Michael Dickens)]

OK. How about now?

[mojca (Mojca Miklavec)]

Thanks, that looks better. Or at least the compiler is properly chosen. It fails to compile though:

In file included from ../src/Document.cpp:22:
In file included from /opt/local/include/gcc6/c++/regex:35:
/opt/local/include/gcc6/c++/bits/c++0x_warning.h:32:2: error: This file requires compiler and library support for the ISO C++ 2011 standard. This support must be
      enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support \
 ^
../src/Document.cpp:2581:41: error: expected ';' at end of declaration list
        ByteIterator(const ByteIterator &other) NOEXCEPT {
                                               ^
                                               ;
../src/Document.cpp:2756:41: error: expected ';' at end of declaration list
        UTF8Iterator(const UTF8Iterator &other) NOEXCEPT {
                                               ^
                                               ;

Adding -std=c++11 helps to go over this error, but let me try to check the full build first. I guess that this flag should be added by the PortGroup, not by the port itself.

[mojca (Mojca Miklavec)]

Adding -std=c++11 in the PortGroup helped. The compilation almost finished, except that the build is still not UsingTheRightCompiler for the final linking step.

As well as some weird "library not found" error.

/usr/bin/clang++ -headerpad_max_install_names -arch x86_64 -single_module -dynamiclib -Xarch_x86_64 -mmacosx-version-min=10.7 -o libqscintillaplugin.dylib qscintillaplugin.o moc_qscintillaplugin.o  -F/opt/local/libexec/qt4/Library/Frameworks -F/opt/local/libexec/qt4/lib  -L../Qt4Qt5 -lqscintilla2_qt4 -F/opt/local/libexec/qt4/Library/Frameworks -F/opt/local/libexec/qt4/lib -L/opt/local/libexec/qt4/lib -framework QtScript -F/opt/local/Library/Frameworks -framework QtCore -framework QtXml -framework QtGui -framework QtDesigner  
ld: library not found for -lqscintilla2_qt4

The build folder does contain Qt4Qt5/libqscintilla2_qt4.13.0.0.dylib, but probably a symlink from Qt4Qt5/libqscintilla2_qt4.dylib is missing.

The library name is also super strange and looks like "someone forgot to run install_name_tool":

> otool -L /opt/local/libexec/qt4/lib/libqscintilla2_qt4.13.0.0.dylib 
/opt/local/libexec/qt4/lib/libqscintilla2_qt4.13.0.0.dylib:
	libqscintilla2_qt4.13.0.0.dylib (compatibility version 13.0.0, current version 13.0.0)
	/opt/local/libexec/qt4/Library/Frameworks/QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.7)
	/opt/local/libexec/qt4/Library/Frameworks/QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.7)
	/opt/local/lib/libgcc/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.22.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

> otool -L /opt/local/libexec/qt4/share/plugins/designer/libqscintillaplugin.dylib
/opt/local/libexec/qt4/share/plugins/designer/libqscintillaplugin.dylib:
	libqscintillaplugin.dylib (compatibility version 0.0.0, current version 0.0.0)
	libqscintilla2_qt4.13.0.0.dylib (compatibility version 13.0.0, current version 13.0.0)
	/opt/local/libexec/qt4/Library/Frameworks/QtScript.framework/Versions/4/QtScript (compatibility version 4.8.0, current version 4.8.7)
	/opt/local/libexec/qt4/Library/Frameworks/QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.7)
	/opt/local/libexec/qt4/Library/Frameworks/QtXml.framework/Versions/4/QtXml (compatibility version 4.8.0, current version 4.8.7)
	/opt/local/libexec/qt4/Library/Frameworks/QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.7)
	/opt/local/libexec/qt4/Library/Frameworks/QtDesigner.framework/Versions/4/QtDesigner (compatibility version 4.8.0, current version 4.8.7)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

[mojca (Mojca Miklavec)]

See also #53599 and a64f1147 in particular.

[mojca (Mojca Miklavec)]

I managed to compile the library after applying the following patch:

_resources/port1.0/group/cxx11-1.1.tcl

@@ -63 +63 @@
         # for OS X Maverick and above, users must select libstdc++, so
         #    assume they want default ABI compatibility
         # see https://gcc.gnu.org/onlinedocs/gcc-5.2.0/libstdc++/manual/manual/using_dual_abi.html
-        configure.cxxflags-append -D_GLIBCXX_USE_CXX11_ABI=0
+        configure.cxxflags-append -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=0
     }
 } else {
     # GCC compilers can not use libc++

_resources/port1.0/group/qmake-1.0.tcl

@@ -44 +44 @@
                                     "QMAKE_CXX=${configure.cxx}" \
                                     "QMAKE_OBJC=${configure.objc}" \
                                     "QMAKE_CFLAGS=\"${configure.cflags} [get_canonical_archflags cc]\"" \
-                                    "QMAKE_CXXFLAGS=\"${configure.cxxflags} [get_canonical_archflags cxx]\""
+                                    "QMAKE_CXXFLAGS=\"${configure.cxxflags} [get_canonical_archflags cxx]\"" \
+                                    "QMAKE_LINK_C=${configure.cc}" \
+                                    "QMAKE_LINK_C_SHLIB=${configure.cc}" \
+                                    "QMAKE_LINK=${configure.cxx}" \
+                                    "QMAKE_LINK_SHLIB=${configure.cxx}"
+
     configure.universal_args-delete --disable-dependency-tracking
 
     if {[variant_exists universal] && [variant_isset universal]} {

(Perhaps -std=c++11 should be added for all ports that include the cxx11 PortGroup?)

I believe that cxxflags for the linker are still missing above since I had to manually add -D_GLIBCXX_USE_CXX11_ABI=0 -stdlib=macports-libstdc++ when calling the linker for libqscintilla2_qt4, else the linker would fail (I added both flags just in case, maybe just fixing the stdlib would suffice). This wasn't necessary for the other library libqscintillaplugin, but I guess that's just because it doesn't use C++11?

The qmake5 PortGroup uses some pretty "complex" code to decide which linker flags to use and I wasn't sure which one to apply here, but QMAKE_LFLAGS should be set in one way or another.

I also had to do the following in order to finish compilation:

• in Qt4Qt5:

  sudo -u macports install_name_tool -id /opt/local/libexec/qt4/lib/libqscintilla2_qt4.13.0.0.dylib libqscintilla2_qt4.13.0.0.dylib 
  sudo -u macports ln -s libqscintilla2_qt4.13.0.0.dylib libqscintilla2_qt4.dylib 
  

• in designer-Qt4Qt5:

  sudo -u macports install_name_tool -id /opt/local/libexec/qt4/share/plugins/designer/libqscintillaplugin.dylib libqscintillaplugin.dylib 
  

Let me check whether Octave works after this trickery.

[mojca (Mojca Miklavec)]

Octave compiles and works fine, the built-in editor doesn't start (once I try to start it, it hangs forever). But also keep in mind that I didn't use the right linker flags when compiling qscintilla.

Some errors/warnings shown:

octave-gui[91413:3803] unlockFocus called too many time.
objc[91413]: ALT HANDLERS: *** over 1000 alt handlers installed; this is probably a bug

[michaelld (Michael Dickens)]

Interesting. On what OS version are you compiling Qsci? I'm using latest stock Sierra & Xcode, and Qsci compiles fine with or without -std=c++11 added. It is certainly true that <regex> is a C++11 file, so I'm guessing there's a warning in the log file somewhere about it.

What would be ideal is if the default CXXFLAGS were "-std=c++98" or whatever. Then if C++11 is desired, we swap that to "-std=c++11 -Wc++11-extension". Likewise for C++14 or C++17, assuming the updated -W is valid for those. Basically: Try to force the compiler to a certain standard & warn if not abiding by it.

Sort of like checking for whether the port can "use the right compiler".

[michaelld (Michael Dickens)]

Back to your hacking: The Qsci build script has an install_name_tool tweak that's supposed to rename the libqsci library to be just "*.13.dylib". Not sure why it's not being executed for you. Maybe I'll give your qmake PG tweaks a whirl & see what comes of it. There has got to be a reasonable & cross-port way to coerce qmake into doing the right thing for both compilers and linkers & C++XX support for the vast majority of ports.

[mojca (Mojca Miklavec)]

Compare the following:

> clang++-mp-4.0 -stdlib=libc++ test.cpp -o test
# OK

> clang++-mp-4.0 -stdlib=libstdc++ test.cpp -o test
test.cpp:1:10: fatal error: 'regex' file not found
#include <regex>
         ^~~~~~~
1 error generated.

> clang++-mp-4.0 -stdlib=macports-libstdc++ test.cpp -o test
In file included from test.cpp:1:
In file included from /opt/local/include/gcc6/c++/regex:35:
/opt/local/include/gcc6/c++/bits/c++0x_warning.h:32:2: error: This file requires compiler and library support for the ISO C++ 2011 standard. This support must be
      enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support \
 ^
1 error generated.

Maybe Marcus just needs to improve his patch for macports-libstdc++.

What would be ideal is if the default CXXFLAGS were "-std=c++98" or whatever. Then if C++11 is desired, we swap that to "-std=c++11 -Wc++11-extension". Likewise for C++14 or C++17, assuming the updated -W is valid for those. Basically: Try to force the compiler to a certain standard & warn if not abiding by it.

I'm not particularly happy about this idea. Clang already works painlessly in most cases, C++98 soucres compile just fine under C++11 compiler, it's just the recent change with -stdlib=macports-libstdc++ that could be improved and use C++11 by default. Note that this macports-libstdc++ is pretty experimental.

If we force the C++98 flag, we are just calling for a huge number of broken port.

PS: playing on 10.7.

[michaelld (Michael Dickens)]

1) So on my 10.7 box, macports-clang-4.0 is unstable & I have to specify using 3.9 instead. As an example from watching debug output, when cmake hits "checking for log" (as in the function 'log'), it hangs & free file storage ("hard drive" or SSD) quickly decreases until the system hangs. No idea why. I'd much prefer to not be bleeding edge for C++11 support. 3.9 seems like a stable release, yes?

2) Using "-stdlib=macports-libstdc++" should definitely use C++11 by default: Agreed! Note that C++11 is not a strict superset of C++03 (or C++98), so there might be some idiosyncratic programming that fails to build using this change.

3) On my 10.7 box, I couldn't get the necessary dependencies (re)installed to test out how Qsci works with the C++11 change. I'll try again this week. The issue that I found was the qt5-* won't use the right compiler; it always chooses clang-4.0 even if I specify 3.9, and then (1) happens. I haven't looked into that issue yet.

[mojca (Mojca Miklavec)]

One important note. clang-x.y needs a special patch to support -stdlib=macport-libstdc++ which has only been enabled by default on clang-4.0.

> port info clang-3.9

clang-3.9 @3.9.1_3 (lang)
Variants:             [+]analyzer, assertions, debug, libstdcxx, universal

If you want to test all this with clang 3.9, you need to make sure that you install it with the +libstdcxx variant (and thus have to recompile it since the version compiled by the buildbot doesn't have that option enabled). If you then want to use clang 3.9 with the cxx11-1.1 PortGroup, you need to add macports-clang-3.9 to compiler.whitelist.

But I find it strange that clang 4.0 misbehaves. It works just fine for me.

[michaelld (Michael Dickens)]

BTW> Since we last wrote here, clang-4.0 has been updated & now works on my 10.8 install.

Is this ticket's issue still relevant?

[chrpepper]

Yes, at least for my installation... Just tried on 10.6.8 after installing clang-4.0; -stdlib=macports-libstdc++ is used automatically, error is "/opt/local/include/gcc6/c++/bits/c++0x_warning.h:32:2: error: This file requires compiler and library support for the ISO C++ 2011 standard. "

[michaelld (Michael Dickens)]

OK. Can you do the following so that we an see what's going on?

sudo port clean qscintilla
sudo port install qscintilla

and, when that errors out, do:

cp $(port logfile qscintilla) ~/Desktop/qscintilla.log.txt
bzip2 ~/Desktop/qscintilla.log.txt

and then attach the compressed log file here.

I'll also see if I see this issue on my 10.8 box ... not the same as 10.6, but the compiler issue should be similar.

[chrpepper]

10.6.8 build (error c++11 compiler option)

[chrpepper]

Sorry for the delay, did not watch this ticket the last days. Here is the log file, hope it helps.

[kencu (Ken)]

FYI, I had no issue installing this on 10.6.8, without modifications to the portfile. I have the default compiler set to clang-3.9, as per the LibcxxOnOlderSystems instructions, and I'm using libc++.

$ port -v installed qscintilla
The following ports are currently installed:
  qscintilla @2.10_1 (active) platform='darwin 10' archs='x86_64' date='2017-03-01T23:19:28-0800'

[michaelld (Michael Dickens)]

OK. I'm following the issue now (again). I've seen this with other ports that are using the clang-4.0 / macports-libstdc++ build. We have to add "-std=c++11" to the build somehow, just for these, and probably won't hurt for newer compilers either. I'll see what I can do.

[kencu (Ken)]

I worked on this a bit regarding adding into the clang code directly <​https://trac.macports.org/ticket/53634#comment:6>. It's possible, but it's a bit hard as there needs to be a test you can call in the clang code to see if macports-libstdc++ is the selected stdlib, and I don't know how to do that from that point in the code.

MUCH easier would be to add -std=c++11 in the cxx11 1.1 portgroup code, but that runs the risk of trashing some other specified standard in the port itself.

[michaelld (Michael Dickens)]

It's almost trivial to add in the std=c++11 to CXXFLAGS in each port that uses the cxx11 1.1 PG. That avoids the trashing you mention. Maybe a little clumsy, but since different ports use different build systems (automake, cmake, bmake, etc), this is probably the right way to go for now instead of trying a blanket approach that is likely to cause headaches. In my opinion.

So at least for this port I'll just add in the -std argument, to the port itself as Mojca describes. That will at least fix this specific issue, I think.

[chrpepper]

I've added a line "configure.cxxflags-append -std=c++11" to the Portfile. This lets it compile for my 10.6.8 box (it could cause problems for newer OS X versions), but at linking there are undefined symbols like "std::throw_out_of_range_fmt(char const*, ...)" , "std::throw_bad_function_call()" etc. So, some linker flag is needed, too...

[chrpepper]

I am afraid I'll have to go the way Ken (comment 23) described, as this seems to be the cleanest solution but it will take a very long time to re-build all the ports. So I have some hope I can get qscintilla (and later octave) running without rebuilding all ports. Do I have a chance?

[kencu (Ken)]

I would say so:

$ port -v installed octave
The following ports are currently installed:
  octave @4.2.1_1+accelerate+app+docs+fltk+gfortran+graphicsmagick+qt4+sound (active) platform='darwin 10' archs='x86_64' date='2017-04-25T11:21:43-0700'

at least that has been my experience.

[michaelld (Michael Dickens)]

I'm almost there with a fix for libstdc++ users. As Mojca described above, it's multi-part: (1) fixing the qmake PG to make sure qmake has the right settings for linking (generically); (2) getting the right settings to qmake to do c++11 compiling; and (3) making sure the correct libstdc++ setting is provided for both compiling and linking, so that the correct C++ framework is used. Fun fun fun!

[michaelld (Michael Dickens)]

Part 1: ​https://github.com/macports/macports-ports/commit/bf6ff46c8e33a31bf06096ff15d28450159f9f18

[michaelld (Michael Dickens)]

Part 2: ​https://github.com/macports/macports-ports/commit/a10f3c4aaa3429d9a17d273b771df8982dd0c685

[michaelld (Michael Dickens)]

In 94da98fbe2b661d134e69715b52cca0c012e4901/macports-ports:

qscintilla: set the correct stdlib for linking for older compilers.

Addresses ticket #53625 .

[michaelld (Michael Dickens)]

OK. With these 3 changes in place, QSci should now build correctly when using libstdc++ or libc++. I haven't tried -using- it yet, but that can be a different ticket ;)

[chrpepper]

awesome! Thank you very much for taking care of this ticket, Michael, and thanks to all the contributors! Now the last 10.6.8 users will be happy using QSci again. I've just had success in "port upgrade qscintilla" and will now upgrade the remaining ports. Great!

[michaelld (Michael Dickens)]

You're welcome! Good luck!

[cer54 (Carl Edward Rasmussen)]

also compiles on my 10.7.5.

[chrpepper]

And I could finally build the actual octave 4.2.0, it even seems to work. Thanks to you, Michael, I now have an up-to-date Macports on my old 10.6.8 MBP late 2006. Maintaining a port using C++ for different OS X versions must be lots of fun...

[michaelld (Michael Dickens)]

Yay! I'm so glad this is working for folks!