qt5-qtbase @5.15.8: macdeployqt doesn't fix QtWebEngineProcess

[kaamui]

I'm facing an issue where my app (that contains an internal navigator) fails to run any part of the app involving the webengine on other macs than mine. It appears to be related to QtWebEngineProcess searching to resolve its dependencies, referenced with absolute path.

otool -L /opt/local/libexec/qt5/lib/QtWebEngineCore.framework/Versions/5/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess 

/opt/local/libexec/qt5/lib/QtWebEngineCore.framework/Versions/5/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess:
	/opt/local/libexec/qt5/lib/QtGui.framework/Versions/5/QtGui (compatibility version 5.15.0, current version 5.15.8)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2299.30.112)
	/System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 306.3.4)
	/opt/local/libexec/qt5/lib/QtCore.framework/Versions/5/QtCore (compatibility version 5.15.0, current version 5.15.8)
	/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/libexec/qt5/lib/QtWebEngineCore.framework/Versions/5/QtWebEngineCore (compatibility version 5.15.0, current version 5.15.12)
	/opt/local/libexec/qt5/lib/QtQuick.framework/Versions/5/QtQuick (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtQmlModels.framework/Versions/5/QtQmlModels (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtWebChannel.framework/Versions/5/QtWebChannel (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtQml.framework/Versions/5/QtQml (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtPositioning.framework/Versions/5/QtPositioning (compatibility version 5.15.0, current version 5.15.8)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

and it outputs the same result on the executable copied inside my app :

   otool -L OpenBoard.app/Contents/Frameworks/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess

/opt/local/libexec/qt5/lib/QtGui.framework/Versions/5/QtGui (compatibility version 5.15.0, current version 5.15.8)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2299.30.112)
	/System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 306.3.4)
	/opt/local/libexec/qt5/lib/QtCore.framework/Versions/5/QtCore (compatibility version 5.15.0, current version 5.15.8)
	/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/libexec/qt5/lib/QtWebEngineCore.framework/Versions/5/QtWebEngineCore (compatibility version 5.15.0, current version 5.15.12)
	/opt/local/libexec/qt5/lib/QtQuick.framework/Versions/5/QtQuick (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtQmlModels.framework/Versions/5/QtQmlModels (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtWebChannel.framework/Versions/5/QtWebChannel (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtQml.framework/Versions/5/QtQml (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.15.0, current version 5.15.8)
	/opt/local/libexec/qt5/lib/QtPositioning.framework/Versions/5/QtPositioning (compatibility version 5.15.0, current version 5.15.8)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

Here's the error message I obtain in another Mac (where MacPort and Qt are not installed)

dyld[8994]: Library not loaded: '/opt/local/libexec/qt5/lib/QtGui.framework/Versions/5/QtGui'
  Referenced from: '/Applications/OpenBoard.app/Contents/Frameworks/QtWebEngineCore.framework/Versions/5/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess'
  Reason: tried: '/opt/local/libexec/qt5/lib/QtGui.framework/Versions/5/QtGui' (no such file), '/System/Library/Frameworks/QtGui.framework/Versions/5/QtGui' (no such file)

Is it expected to be like this and I have to pass some parameter when compiling so paths are changed to @executable_path or is it an issue in MacPort's side ?

Thanks in advance for your help !

[jmroot (Joshua Root)]

Normally you would need to use a tool like dylibbundler if you want to deploy MacPorts libraries to a different location such as inside an app bundle.

[kaamui]

I'm not manually placing the QtWebEngineProcess inside my app, it is added as a dependency like any other lib (except that this one contains an executable ?)

I don't have to when using the default binaries (provided by Qt), but they aren't supporting proprietary codecs by default so I need to use the one provided by MacPorts.

Is it some kind of issue in MacPorts way to build the Qt libs ? Should these libs already reference each others with relative paths or is it intentional and wanted by you and other devs handling these Qt ports ? I'm just curious.

it looks like dylibbundler is performing what I was going to do manually : using install_name_tool everywhere it is needed. So it might indeed help, but I still wonder if it's just a workaround for a real issue that is remaining regarding dependencies.

Also, you mention that "if I want to deploy MacPorts libraries to a different location as inside an app bundle" : my understanding is that it is mandatory and already what is always done when building an app. For example :

otool -L OpenBoard.app/Contents/MacOS/OpenBoard
@executable_path/../Frameworks/libavformat.58.dylib (compatibility version 58.0.0, current version 58.76.100)
	@executable_path/../Frameworks/libavcodec.58.dylib (compatibility version 58.0.0, current version 58.134.100)
	@executable_path/../Frameworks/libswscale.5.dylib (compatibility version 5.0.0, current version 5.9.100)
	@executable_path/../Frameworks/libswresample.3.dylib (compatibility version 3.0.0, current version 3.9.100)
	@executable_path/../Frameworks/libavutil.56.dylib (compatibility version 56.0.0, current version 56.70.100)
	@executable_path/../Frameworks/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1953.255.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 169.0.0)
	/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	@executable_path/../Frameworks/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	@executable_path/../Frameworks/libquazip5.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	@executable_path/../Frameworks/libpoppler.126.dylib (compatibility version 126.0.0, current version 126.0.0)
	@executable_path/../Frameworks/QtSvg.framework/Versions/5/QtSvg (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtMultimediaWidgets.framework/Versions/5/QtMultimediaWidgets (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtWebEngineWidgets.framework/Versions/5/QtWebEngineWidgets (compatibility version 5.15.0, current version 5.15.12)
	@executable_path/../Frameworks/QtPrintSupport.framework/Versions/5/QtPrintSupport (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtWidgets.framework/Versions/5/QtWidgets (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtMultimedia.framework/Versions/5/QtMultimedia (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtWebEngineCore.framework/Versions/5/QtWebEngineCore (compatibility version 5.15.0, current version 5.15.12)
	@executable_path/../Frameworks/QtQuick.framework/Versions/5/QtQuick (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtGui.framework/Versions/5/QtGui (compatibility version 5.15.0, current version 5.15.8)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2299.30.112)
	/System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 306.3.4)
	@executable_path/../Frameworks/QtQmlModels.framework/Versions/5/QtQmlModels (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtWebChannel.framework/Versions/5/QtWebChannel (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtQml.framework/Versions/5/QtQml (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtXml.framework/Versions/5/QtXml (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtPositioning.framework/Versions/5/QtPositioning (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtConcurrent.framework/Versions/5/QtConcurrent (compatibility version 5.15.0, current version 5.15.8)
	@executable_path/../Frameworks/QtCore.framework/Versions/5/QtCore (compatibility version 5.15.0, current version 5.15.8)

As you can see here, some Qt libs are referenced, but not with absolute path.

Pleas forgive me if I say dubious things I'm not quite confortable on Mac.

[kaamui]

example with "official" Qt 5.15.2 binary :

otool -L /Applications/OpenBoard.app/Contents/Frameworks/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess

@rpath/QtGui.framework/Versions/5/QtGui (compatibility version 5.15.0, current version 5.15.2)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1671.10.106)
	/System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 1.0.0)
	@rpath/QtCore.framework/Versions/5/QtCore (compatibility version 5.15.0, current version 5.15.2)
	/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
	@rpath/QtWebEngineCore.framework/Versions/5/QtWebEngineCore (compatibility version 5.15.0, current version 5.15.2)
	@rpath/QtQuick.framework/Versions/5/QtQuick (compatibility version 5.15.0, current version 5.15.2)
	@rpath/QtQmlModels.framework/Versions/5/QtQmlModels (compatibility version 5.15.0, current version 5.15.2)
	@rpath/QtWebChannel.framework/Versions/5/QtWebChannel (compatibility version 5.15.0, current version 5.15.2)
	@rpath/QtQml.framework/Versions/5/QtQml (compatibility version 5.15.0, current version 5.15.2)
	@rpath/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.15.0, current version 5.15.2)
	@rpath/QtPositioning.framework/Versions/5/QtPositioning (compatibility version 5.15.0, current version 5.15.2)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)

[ryandesign (Ryan Carsten Schmidt)]

MacPorts does not install relocatable libraries. If you want them to be relocatable (i.e. with @executable_path), you have to use dylibbundler or install_name_tool to change it.

[kaamui]

I think I don't understand something that seems obvious to you two. From what I barely understand, you mean that without performing any install_name_tool myself, qt5-qtwebengine cannot work if I link it to my app and try my app on another Mac where MacPorts is installed ? Do you consider MacPorts should be installed on the targeted computers ? Or am I completely missing something here ?

Because I'm building an app that uses several libs that I installed with MacPorts and where I don't have any issues. I have, qt5, qt5webengine, poppler, ffmpeg, quazip5. Every one of these libs are correctly copied inside my app when I compile/macdeployqt it. And I don't have to do anything for the libs to be correctly pointed (relative to inside the app), and it works on other MacBooks. Why this difference for qt5-qtwebengine libs is mainly what I don't understand.

[kaamui]

This SO thread seems related (it actually fits pretty well with what I'm facing) => ​https://stackoverflow.com/questions/46430930/how-to-make-macdeployqt-change-the-library-names-inside-qtwebengineprocess-app-w

[ryandesign (Ryan Carsten Schmidt)]

Forgive me for not reading your messages closely enough.

MacPorts is primarily intended for use on your computer. You install MacPorts, install the software you want, it installs the dependencies you need, and the software will work on your computer. Such users are expected to leave the files (libraries/frameworks/programs) exactly where MacPorts put them. Since that's the case, the libraries/frameworks/programs are linked in the most straightforward manner: by absolute paths.

If you want to make an app that contains some libraries/frameworks that were compiled by MacPorts and that can be redistributed to users who don't have MacPorts installed, the libraries/frameworks you use have to be copied inside the app bundle and all of the linkages have to be edited to use relative paths based on @executable_path, @loader_path, or (if you only need to support Mac OS X 10.5 or later) @rpath. You can do this manually for each linkage using Apple's install_name_tool or using a third-party tool that attempts to automate the process like dylibbundler. I didn't realize until comment:7 that you were already using a different third-party tool, macdeployqt, to do this step.

After rereading your comments, I understand now that the problem here isn't that the linkages of your app and the libraries/frameworks weren't edited properly (they were); the problem is that the QtWebEngineCore framework contains a QtWebEngineProcess app, and this QtWebEngineProcess app's linkage hasn't been edited. The stackoverflow post you referred to said the same issue was seen when Qt was installed by Homebrew. This sounds like a bug in macdeployqt. Until it is fixed, you should be able to edit QtWebEngineProcess's linkage manually using install_name_tool.

I understand that the official Qt binary distribution doesn't have this issue. It is built using @rpath. Using @rpath causes some problems so in MacPorts we've decided not to ship libraries that use @rpath. The bug may be that macdeployqt expects Qt to have been built with @rpath.

Searching the ​Qt bug tracker for "macdeployqt" I found ​bug 84200 which seems to describe the problem exactly. They closed it as not their problem, referring the user back to Homebrew; I disagree with that conclusion.

I also found ​bug 41611 which appears to date from a time before Qt used @rpath and mentions a workaround that could be used at the time. It notes that the workaround "won't be necessary once we support using @rpath, which should be done in Qt 5.5.0." The operative word in that sentence to me is "support". If Qt supports using @rpath that's great but as long as they also support not using @rpath (as in MacPorts and apparently Homebrew) this is a bug in macdeployqt. If upstream will not fix that bug, maybe we will have to tackle it ourselves.

Just to close out the dylibbundler suggestion, I discovered that ​dylibbundler doesn't support frameworks so it wouldn't be suitable for use with a Qt app.

If you are still running Qt 5.15.8 as per the ticket summary, note that 5.15.10 is available in MacPorts via sudo port selfupdate and sudo port upgrade outdated.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

I also found ​bug 41611 which appears to date from a time before Qt used @rpath and mentions a workaround that could be used at the time.

And I should say: you could try that workaround and see if it still works.

[kaamui]

Sorry for the late reply. Sadly, I already had tested it before I created this ticket. It doesn't help with recent Qt 5. I also tried to manually fix absolute paths with relative one, but as explained by the user on the SO thread, it just move the issue to the libs that are no longer able to find their own dependencies.

Thanks for the good and precise explanation, it helps understanding the whole issue. I saw your comment on the closed Qt bug. I think we'll have to open a new Bug if you want to obtain Qt's attention on it.

I compiled webengine myself, both using official Qt binaries, and macports ones, and even if my tests aren't completely finished (testing with both Qt 6 and Qt 5, on intel and arm (only Qt 6 for arm, because Qt doesn't provide official Qt 5.15 packages on arm), with or without universal), I can confirm I obtained the expected result at this point : QtWebEngineProcess refer to its dependencies with relative paths when compiled with official Qt 5.15.2, and with absolute paths when using MacPorts Qt. WebEngine then only works when compiled with Official Qt.

I'll update my answer if anything new comes out from my tests