proposed improvements to port:qt5-mac

[RJVB (René Bertin)]

Following discussion on the -devel ML, I'm opening this ticket as an upbeat to improving the qt5-mac port with the changes I have been making to it over the past half year.

First up: changes to the Portgroup file targeting the install locations, presented as a diff against mcalhoun's current PortGroup and as the raw file (because a number of changes are simply expressions moved around for what I find better readability). The effect is that Qt's binary things are installed under ${prefix}/libexec/qt5 (instead of ditto/qt5-mac), except for the plugins which go with the other shared things in a directory under ${prefix}/share/qt5. Headerfiles go under ${prefix}/include/qt5. This scheme is based on the well-tested scheme used by Linux distros (Ubuntu in my case) and allows Qt to present QT_INSTALL_PREFIX and QT_HOST_PREFIX as ${prefix} which I think is preferable. I did try to install everything into ${prefix}/libexec/qt4 when I started making the Qt4 port concurrent, but that didn't work out very well, possibly because of the different QT_INSTALL_PREFIX and QT_HOST_PREFIX so I went with the layout shown in the attached PortGroup. That also minimises changes w.r.t. the previous, exclusive installation layout which I think preferable; undoubtedly an argument carrying more weight for Qt4 than for Qt5, but I also think it would be far preferable if both ports share the same layout principle.

[RJVB (René Bertin)]

Next up: the qt5-mac Portfile, again as a diff against the current mainstream version, and the raw file. The changeset here is more or less the minimal one required to build Qt5 with my PortGroup *and* bring the port on par with qt4-macconcerning the headerfiles, pkgconfig files and the principle of providing standard shared libraries (.dylibs) in addition to the framework bundles. Michaelld has found good reason to do this; without proof to the contrary I think it's wise to do the same thing (it certainly shouldn't hurt). I also rolled the sqlite3 plugin back into the main port; as explained elsewhere this was a suggestion from Bradley (pixilla) inspired by the fact that some of the applications installed by the main ports require this plugin. This "fix" is easier than moving the applications in question to their own port.

The changes have inline comments, and there are a number of additional remarks throughout the file; all carry my 4CC (RJVB) as a (temporary!) label.

[RJVB (René Bertin)]

This is the reference Portfile against which I'll publish further changes

[RJVB (René Bertin)]

FWIW, the full files and of course all patches are also available in my git repository:

​https://github.com/RJVB/macstrop/tree/master/aqua/qt5-devel

This port is simply called port:qt5-devel instead of qt5-mac-devel; that's no more than a detail for now.

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

There are allot of things to discuss, so here goes:

Easy Fixes and Changes (if necessary)

Whitespace changes, rewording comments, and changing code order for ''better readability'':
We can probably worry about these later. Right now, they just makes finding important changes harder.

Renaming port to qt5:
The name qt5-mac may not have been the optimal choice, but is it really a problem?
What is wrong with support qt5-x11 {...} in the Portfile if one so desires?
Like the whitespace changes, I am trying to discern high priority problems from low priority.

"[T]he docs subport should be renamed such that it comes last during an upgrade outdated":
Is this a personal preference, or is there a problem?

"use the code below ">>>..." instead of this block, which fails to populate qt_includes_dir completely.":
Which header code is not properly installed?

"what's the point in not optimising qmake?":
-no-optimized-qmake is the default behavior.
My default position is to use the default unless there is a compelling reason to change (see e.g. this (fixed) ​bug).
Of course, reasonable people can easily disagree on what constitutes a "compelling reason."

Do not explicitly set default options (e.g. -qml-debug and -c++11):
This is a resonable point.

Bugs:
Changing qt-project.org to qt.io should definitely be done.
Putting reinplace ... into a post-patch should be done (definitely a bug, although not one that has surfaced yet).

"[T]he sqlite3 plugin ("sqlite") should be re-absorbed into the main port":
Forgive me, but I am having a little difficulty understanding the concern.
What is wrong with depends_lib-append port:qt5-mac-sqlite3-plugin if the plugin is necessary?
Why would "moving the applications in question to their own port" be required?
The reasons are "explained elsewhere."
Is there a reference available?

Remove pulseaudio dependency:
This is a reasonable point.
If few people or ports need pulseaudio support, it is probably best to create a variant.

Fundamental Issue(s)

I have not gone through every line of code in detail, but the rest of the proposed changes seem to revolve around moving header, library, and configuration files to different locations.
I strongly believe installing Qt in one place is the best way to go.

• It is the default behavior
  • If you download and install Qt, it all goes in one place, which means that it is probably better supported
  • Qt is a big piece of software and does not always like surprises (i.e. non-default behavior)
    • A simple example: without a patch, part of Qt assumes the library is ${qt_dir}/lib no matter what the configure option -libdir is (see patch-shared.diff).
    • Another simple example: we have to clear the MacPorts variables (e.g. configure.cflags) to maintain consistency throughout the build.
    • From my own experience: Some time ago, QtCreator was regularly crashing. When I went back to the defaults, it worked correctly. I probably should have tracked down the error, but for the most part, I just want software installation to work smoothly so I can use it.
• It is much easier to maintain.
  • Making significant changes to a port requires significant testing for every major (or not so major) upgrade.
    • This takes maintainer time (a precious commodity if ever there was one).
    • This delays upgrades.
  • The fewer changes that are made, the fewer things can go wrong.
• It makes installing multiple versions of Qt trivial (#44193).

There are, of course, a few disadvantages:

• pkgconfig files are not found automatically.
• Library and header files are not found automatically.
• qmake is not found automatically
• As noted earlier, it is not the same "scheme used by Linux distros."
• Other problems of which I am unaware.

I think the advantages outweigh the disadvantages.

Conclusion

Please forgive me for the long post, but many changes have been proposed, and I wanted to make sure all work was recognized.
If I have missed or misunderstood any concerns, please feel free to correct me.
I think Qt is a very useful piece of software, and it would be nice if we had a fully functioning version in MacPorts.

[RJVB (René Bertin)]

Replying to mcalhoun@…:

Easy Fixes and Changes (if necessary)

Whitespace changes, rewording comments, and changing code order for ''better readability'':
We can probably worry about these later. Right now, they just makes finding important changes harder.

Yes we can and should - I've just left them in because they evolved over time, in large part because of other changes that I have not yet discussed here (but that are visible on the repo I linked to).

Renaming port to qt5:
The name qt5-mac may not have been the optimal choice, but is it really a problem?
What is wrong with support qt5-x11 {...} in the Portfile if one so desires?
Like the whitespace changes, I am trying to discern high priority problems from low priority.

I think I more or less said the same thing. I'm not even sure I find it extremely necessary myself.

"[T]he docs subport should be renamed such that it comes last during an upgrade outdated":
Is this a personal preference, or is there a problem?

No, it's not a preference, because it makes the port name less obvious. I just noticed an issue if you don't do this when you have multiple Qt5 subports installed, with the generated docs picking up parts of the previous version. That sounds weird and I'm not sure anymore that I understood the situation correctly, but I think that the doc generation picks up things from the install locations rather than using only the work source/build directories.

"use the code below ">>>..." instead of this block, which fails to populate qt_includes_dir completely.":
Which header code is not properly installed?

When I looked in the include directory after doing a destroot of the current published qt5-mac port (v. 5.4.1), I saw only a subset of what I find in my include directory. I didn't write down the exact list, but here's what I have in ${prefix}/include/qt5:

%> lc /opt/local/include/qt5
total 320
0 ./                    8 QtDesignerComponents@ 8 QtOpenGL@             8 QtQuickWidgets@       8 QtWebKit@
0 ../                   8 QtGui@                0 QtOpenGLExtensions/   8 QtScript@             8 QtWebKitWidgets@
8 Enginio@              8 QtHelp@               0 QtPlatformHeaders/    8 QtScriptTools@        8 QtWebSockets@
8 QtBluetooth@          8 QtLocation@           0 QtPlatformSupport/    8 QtSensors@            8 QtWidgets@
8 QtCLucene@            8 QtMacExtras@          8 QtPositioning@        8 QtSerialPort@         8 QtXml@
8 QtConcurrent@         8 QtMultimedia@         8 QtPrintSupport@       8 QtSql@                8 QtXmlPatterns@
8 QtCore@               8 QtMultimediaQuick_p@  8 QtQml@                8 QtSvg@
8 QtDBus@               8 QtMultimediaWidgets@  8 QtQuick@              8 QtTest@
8 QtDeclarative@        8 QtNetwork@            8 QtQuickParticles@     0 QtUiTools/
8 QtDesigner@           8 QtNfc@                8 QtQuickTest@          8 QtWebChannel@

"what's the point in not optimising qmake?":
-no-optimized-qmake is the default behavior.

True, and I never understood that; qmake is not an application users are likely to need to debug for instance. I remember from my PPC machines that qmake was slow as molasses unless built with optimisation (C++ tends to behave like that).

Changing qt-project.org to qt.io should definitely be done.
Putting reinplace ... into a post-patch should be done (definitely a bug, although not one that has surfaced yet).

It probably wasn't triggered by your choices for the install locations.

"[T]he sqlite3 plugin ("sqlite") should be re-absorbed into the main port":
Forgive me, but I am having a little difficulty understanding the concern.
What is wrong with depends_lib-append port:qt5-mac-sqlite3-plugin if the plugin is necessary?
Why would "moving the applications in question to their own port" be required?

The applications I refer to are installed by the qt5 port itself, so adding a dependency on port:qt5-mac-sqlite3-plugin would mean introducing a circular dependency. The most obvious example is the Assistant, which happens to be the 1st application I always launch to see if a Qt install went OK. On a more general note, this plugin is tiny, and is required by many dependents, contrary to the other related plugins. That alone is sufficient reason for me to include it. I've never checked, but aren't they all built by default as suggested by the fact they have to disabled in the Portfile?

If few people or ports need pulseaudio support, it is probably best to create a variant.

Yes, and that could wait until there's demand for such a variant. Qt4 has been doing fine without it, apparently.

On a related note: is there reason not to provide a way to build without harfbuzz support? My own port has harfbuzz in a variant.

• If you download and install Qt, it all goes in one place, which means that it is probably better supported

That is true for Mac and MS Windows, but the intended use on those platforms is quite different than the use in MacPorts. I really regret now that I didn't keep track of the true issues I ran into with Qt4 when I installed it that way, but I seem to remember that I had issues with KDE4. KDE is built around XDG FreeDesktop assumptions on where shared things are installed, and while we currently have only KDE4 in MacPorts, a number of KDE-Mac people are preparing KF5 (and I was planning to join that effort after getting my Qt5 port accepted...) It didn't take me long to realise (working on the qt4-mac port) that I sticking as closely as possible to the current scheme would make things a lot easier and was much less likely to lead to unforeseen issues.

• Qt is a big piece of software and does not always like surprises (i.e. non-default behavior)

To counter that: Qt4 isn't significantly less complex and has proved to work fine with the layout I'm proposing. The previous qt5-mac port (5.3.2) had a layout that was close to that one, and worked well enough too (except some issues due to omissions in the Portfile).

• A simple example: without a patch, part of Qt assumes the library is ${qt_dir}/lib no matter what the configure option -libdir is (see patch-shared.diff).

Which may be one of the reasons that Michael and I populate that directory (${prefix}/libexec/qt5/lib in my case) with .dylibs that are symlinks to the actual framework binaries.

• Another simple example: we have to clear the MacPorts variables (e.g. configure.cflags) to maintain consistency throughout the build.

Do we? I've been removing part of those cleansings to get more acknowledgement of flags I pass in with configure.optflags, and I haven't noticed any adverse effects. Also, what's the relation with or relevance to the install locations?

• From my own experience: Some time ago, QtCreator was regularly crashing. When I went back to the defaults, it worked correctly. I probably should have tracked down the error, but for the most part, I just want software installation to work smoothly so I can use it.

Qt Creator works just fine with my port. One exception: I've been running into a missing symbol issue with version 3.4.1, but that has nothing to do with the install location (and probably everything with the fact that the default Qt build on 10.9 somehow uses the 10.10 SDK).

• It is much easier to maintain.

I don't agree, in part because of the things that are not found automatically as you mention below, but also because I expect things in other places, similar to where they are for Qt4 and where I find them on Linux. I can't speak for Michael of course, but I'd be surprised if he would consider making more changes than necessary to make qt4-mac co-installable...

• Making significant changes to a port requires significant testing for every major (or not so major) upgrade.

This argument would carry a lot of weight I we were discussing a layout significantly different than the one tried and tested with qt4-mac. On the other hand, moving now to a scheme where everything is in its own directory (and thus the INSTALL_PREFIX and HOST_PREFIX point there rather than to the MacPorts prefix) is likely to lead to issues down the road when KF5 makes its appearance, with its expectations of where things are to be found. We're already having significant problems with the QStandardPaths defaults on OS X (and working on a patch to make it comply with XDG/linuxy standards); having to cope with the Qt dependencies installed elsewhere is going to complicate that even further.

• This takes maintainer time (a precious commodity if ever there was one).

As mentioned before (on the ML?), I do happen to have time for such things on my hands ...

• This delays upgrades.

• The fewer changes that are made, the fewer things can go wrong.

As I argued above, the scheme I propose is well-tested, and is actually closer to what qt5-mac did not so long ago. As far as proof goes, I didn't run into any issue or delay because of this installation layout when I upgraded from 5.3.2 to 5.4.0 to 5.4.1 to 5.4.2 (and when I downgraded back from 5.4.1 to 5.3.2 for 10.6.8 systems), also when installing from scratch. Nor have I had any problem reports from the few people who installed my version and used it. Can you name 1 thing that you feel would have made upgrading easier when installing everything in a single location (and that's not the future upgrade to Qt6)?

• It makes installing multiple versions of Qt trivial (#44193).

That's the only argument that I agree with fully. I've tried to assess to what extent this is an argument that carries any weight, and I have the strong impression that MacPorts won't support use cases in which multiple Qt5 versions would be required. Qt guarantees backward compatibility within a major version, meaning that applications built against Qt 5.0.0 (and not using anything "private") will work fine running against Qt 5.4.2. Developing Qt applications using MacPorts' Qt and targeting systems that can only run older Qt5 versions would be a use case where someone might want to use that same Qt version (even if it isn't necessary), but I asked and the answer was unambiguously that that's not a supported use case . In any case there are other trivial solutions, should a supported need arise to provide multiple simultaneous versions. Just as one can install everything in libexec/qt54, libexec/qt53, libexec/qt52 etc, one can install only part in there and the shared things in share/qt54, share/qt53, share/qt52 . That doesn't really change anything.

There are, of course, a few disadvantages:

• pkgconfig files are not found automatically.
• Library and header files are not found automatically.
• qmake is not found automatically

I would actually propose to provide ${prefix}/bin/qmake-qt[45] as symlinks, and I've also submitted a port for qtchooser.

• As noted earlier, it is not the same "scheme used by Linux distros."

I cannot find that earlier note. I'm not claiming it's the exact same scheme, it's a similar scheme. Ubuntu uses /usr/share/qt[45] (so that's identical or at least equivalent):

 ll -d /usr/share/qt[45]/*
379052  13 drwxr-xr-x   2 root root      9 Apr 22 09:59 /usr/share/qt4/bin/
379053  13 drwxr-xr-x   5 root root      5 Aug  7  2014 /usr/share/qt4/doc/
379050   1 lrwxrwxrwx   1 root root     17 Mar 12  2014 /usr/share/qt4/include -> ../../include/qt4/
379054  13 drwxr-xr-x 114 root root    117 Apr 22 09:59 /usr/share/qt4/mkspecs/
364497  13 drwxr-xr-x   2 root root     15 Apr 22 09:39 /usr/share/qt4/phrasebooks/
385972   1 lrwxrwxrwx   1 root root     21 Apr 22 01:57 /usr/share/qt4/plugins -> ../../lib/qt4/plugins
395304 117 -rw-r--r--   1 root root 333221 Apr  7 11:26 /usr/share/qt4/q3porting.xml
379055  13 drwxr-xr-x   2 root root    113 Apr 22 09:59 /usr/share/qt4/translations/
497293  13 drwxr-xr-x  13 root root     47 Jul 10  2014 /usr/share/qt5/doc/
502595  13 drwxr-xr-x   2 root root     15 Jun 23  2014 /usr/share/qt5/phrasebooks/

That's Ubuntu 14.04, which has only Qt 5.2 (in my case only a partial installation at that).

Ubuntu installs the binaries in /usr/lib/qt[45] with an additional layer because it doesn't know universal binaries:

%> ll -d /usr/lib/*/qt[45]/*
1620995 13 drwxr-xr-x 12 root root 12 May 14  2014 /usr/lib/i386-linux-gnu/qt4/plugins/
 173747 13 drwxr-xr-x  2 root root 32 Apr 22 09:59 /usr/lib/x86_64-linux-gnu/qt4/bin/
 173748 13 drwxr-xr-x  6 root root  6 Apr 14 17:38 /usr/lib/x86_64-linux-gnu/qt4/imports/
 173749 13 drwxr-xr-x 19 root root 19 Oct  8  2014 /usr/lib/x86_64-linux-gnu/qt4/plugins/
 496567 13 drwxr-xr-x  2 root root 30 Jun  4 12:49 /usr/lib/x86_64-linux-gnu/qt5/bin/
 397720 13 drwxr-xr-x 26 root root 28 Jun  4 12:51 /usr/lib/x86_64-linux-gnu/qt5/examples/
 602862 13 drwxr-xr-x  2 root root  3 Jul 10  2014 /usr/lib/x86_64-linux-gnu/qt5/imports/
 499579 13 drwxr-xr-x  2 root root  4 Jun 23  2014 /usr/lib/x86_64-linux-gnu/qt5/libexec/
 496569 13 drwxr-xr-x 92 root root 96 Jun  4 12:49 /usr/lib/x86_64-linux-gnu/qt5/mkspecs/
 498759 13 drwxr-xr-x 27 root root 27 Oct 18  2014 /usr/lib/x86_64-linux-gnu/qt5/plugins/
 602894 13 drwxr-xr-x 24 root root 24 Jan 21 15:51 /usr/lib/x86_64-linux-gnu/qt5/qml/

Yes, the plugins are installed in the binary dir, but that's probably largely because of the need to install 2 trees on multiarch systems (and /usr/share/qt*/plugins exists as a symlink. Yes, qt5's mkspecs files are installed in the libdir instead of in /usr/share/qt5, but I think those are only used by qmake and possibly cmake, both of which will know how to find them.

• Other problems of which I am unaware.

I expect more unforeseen problems with a scheme that does not remain close to the one that's always been followed before...

I think the advantages outweigh the disadvantages.

Conclusion

...

I think Qt is a very useful piece of software, and it would be nice if we had a fully functioning version in MacPorts.

It certainly is, but the reason it's nice and almost unavoidable to have it in MacPorts is that it's a middleware on which so many other potential ports depend. It's the needs of those ports that need to have priority over Qt's own needs IMHO. If that means more work for the Qt5 maintainer(s), so be it... (Which is also the reason why I ended up testing a universal build [for Ryan] and going out of my way to include support for 10.6.8.)

[RJVB (René Bertin)]

I will be proposing to add a subport for KDE applications to depend on (Qt4/KDE4 as well as Qt5/KF5). Currently those subports only add a number of patches to improve user experience by addressing issues ("bugs") that won't be fixed upstream.

It would of course be conceivable to let those subports use a different install layout (even without moving it to a completely separate port), preferably by adding some logic to the PortGroup of course. But I'm not sure that'd be justified (or acceptable under some official rule I haven't yet run into).

[RJVB (René Bertin)]

On a different note: what do we know about qmake's support for out-of-tree builds?

[RJVB (René Bertin)]

One exception: I've been running into a missing symbol issue with version 3.4.1, but that has nothing to do with the install location (and probably everything with the fact that the default Qt build on 10.9 somehow uses the 10.10 SDK).

Preliminary results indicate that this is indeed the case and that at least on OS X 10.9, Qt5 should be built with -sdk macosx10.9 :

   if {${configure.sdkroot} ne ""} {
       configure.args-append \
           -sdk [string tolower [join [lrange [split [lindex [split ${configure.sdkroot} "/"] end] "."] 0 end-1] "."]]
-    }
+    } else {
+        # default: build for the current OS version, requesting the corresponding SDK explicitly
+        configure.args-append \
+            -sdk [string tolower "macosx10.[expr ${os.major} - 4]"]
+    }

unless someone can come up with a good argument why it's preferable always to build with the latest SDK available, even if it's newer than the host OS?

[mojca (Mojca Miklavec)]

Cc Me!

[RJVB (René Bertin)]

To develop this point a little bit more:

As noted earlier, it is not the same "scheme used by Linux distros."

I'm *not* saying this is how Qt installs on Linux if someone grabs the sources and follows the instructions. (Those instructions even make it clear that installation is optional, btw.) I'm referring to what's done by Linux distributions that provide and use 3rd party software like Qt in a way that's very comparable to what MacPorts aims to do. That goal is not simply to provide Qt so that one can build standalone applications that can be distributed as a self-contained ensemble (*that* is more or less Qt's principal goal). It's to provide a Qt installation that is installed such that it facilitates its shared use, in an environment based on FreeDesktop.org (and XDG) conventions. A Linux user might be running Qt-based applications while using a Gnome desktop or Gnome applications on a KDE desktop. This is not entirely relevant on OS X as no one is likely to run a "pure" KDE desktop (a Gnome desktop in addition to "Aqua" is perfectly possible, though). Still, it should be possible to share certain resources between Qt, KDE/KF5, GTk, Gnome, XFCE etc. applications if they're installed through MacPorts. I won't say that is impossible to do with Qt completely installed somewhere in its own corner (because I don't have proof for that), but I do suspect that it could be a lot harder. Nicos and/or Michael may know more about this. To look on it another way: there's nothing stopping Linux distro maintainers from sticking Qt in its own tree; for them it makes maintaining the Qt package itself a lot easier. The fact that they've gone that extra length to layout the installation the way it is done in, say, Ubuntu, cannot but mean that there are good reasons to do it in such a way, IMHO. ([EDIT for lack of morning coffee] And to give myself and Michael some credit: I think the layout I propose (derived from the one used in qt4-mac) is rather cleaner than the one in Ubuntu, shown above.)

[MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)]

r137798, r137797, r137796, and r137795 address some of the concerns expressed in this ticket.
It is not everything, but it is a start.

[RJVB (René Bertin)]

If you're going to address the issues one by one, potentially imposing frequent rebuilds on the users, would you consider moving QtWebengine to a subport as the first next step? That reduces the build overhead significantly.

[mkae (Marko Käning)]

Cc Me!

[mkae (Marko Käning)]

Replying to rjvbertin@…:

... unless someone can come up with a good argument why it's preferable always to build with the latest SDK available, even if it's newer than the host OS?

Well, I noticed already a while ago that the 10.10 SDK is usually used by default by XCode! So, perhaps this is intentional?!!

I wouldn't change this back per default so easily, if I were you.

Please lets bring that discussion to the dev-ML, to ask for other opinions.

[mkae (Marko Käning)]

René, what's the current status of this issue? In the light of Marxus' latest commits to qt5 commits...

[RJVB (René Bertin)]

Any tickets I have open about qt5-mac* are stale and can be closed (something I sadly cannot do myself).

[mkae (Marko Käning)]

Okay, closing herewith. Let me know which other ones you want to see gone.