preserve_runtime_libraries convenience PortGroup

[RJVB (René Bertin)]

This is mostly for reference though I'd love to see a similar feature in MacPorts:

Many if not most Linux distributions have one way or another to keep older versions of shared libraries installed if the newer versions introduce ABI incompatibilities. Advantages are less and typically smaller upgrades for the users, less stress on the build bots and as a result, a more reliable upgrade process. That latter claim may need explanation (and may not be totally relevant for MacPorts): building binary packages takes time, and while those builds are in process users starting an upgrade may find themselves in a situation where the new packages aren't yet available. In extreme cases that may lead to uninstallation of numerous packages for which dependency criteria are no longer met. I've experienced that personally with Kubuntu during a KDE version update; I was lucky to have taken a ZFS snapshot before starting the update.

To return closer to home: I was still running a 0.4 port:webp and thought I'd update it since it's could be a dependency for QtWebEngine (if that package allowed to build against the system webp on OS X). Webp 0.5 changes the library versioning, so a number of ports broke, including a few that are quite (very) expensive to build:

--->  Found 11 broken files, matching files to ports     
--->  Found 7 broken ports:
     graphviz @2.38.0 +pangocairo+universal+x11
         /opt/local/bin/dot_builtins
         /opt/local/lib/graphviz/libgvplugin_webp.6.dylib
     ImageMagick @6.9.0-0 +x11
         /opt/local/lib/ImageMagick-6.9.0/modules-Q16/coders/webp.so
     kde4-runtime @4.16.03.80.6 +docs
         /opt/local/lib/kde4/plugins/imageformats/kimg_webp.so
     opencv @3.1.0 +contrib+eigen+python27+qt5+tbb
         /opt/local/lib/libopencv_imgcodecs.3.1.0.dylib
     webkit-gtk @2.4.7 +video
         /opt/local/lib/libwebkitgtk-1.0.0.dylib
     webkit-gtk3 @2.4.7 +video
         /opt/local/lib/libwebkitgtk-3.0.0.dylib
     qt5-kde-devel @5.6.1 +harfbuzz+langselect+qt5kde
         /opt/local/libexec/qt5/Library/Frameworks/QtWebKit.framework/Versions/5/QtWebKit
         /opt/local/share/qt5/plugins/imageformats/libqwebp.dylib

The attached PortGroup provides an easy first-approach/PoC solution to this issue: it copies the specified runtime libraries from ${prefix} to ${destroot} during the post-destroot, if they do not already exist in the destroot.

For instance, in a personal port:webp I used for testing I only needed to include the PortGroup and then add the line

preserve_libraries      [glob -nocomplain ${prefix}/lib/libwebp*.*.dylib]

This simply allows currently installed ports to remain functional using their current dependencies and alleviates the need to revbump all dependents. They'll just move to using the new dependency version when they're rebuilt for other reasons (upgrade, reinstall, ...). I'm aware that this kind of practice may lead to runtime weirdness esp. with C++ libraries if different versions cannot coexist in a single application's runtime image. Easy enough to avoid and apart from that I see only one true issue: it won't be trivial to limit the number of old versions that will be preserved via a generic mechanism.

[ryandesign (Ryan Carsten Schmidt)]

This won't work with binaries built by our buildbot system, because no ports are active when a port build is started on our buildbot system. More generally, it conflicts with our goal of ReproducibleBuilds. So I don't think we want this solution for this issue. You should bring the matter up on the macports-devel mailing list for discussion of the problem and possible solutions before implementing a particular solution.

It also seems like you're looking in part for a solution to a problem you caused for yourself. Regarding your example of webp, it was updated to version 0.5 in r144197 back in January 2016, and all the ports you show as broken were revbumped at the same time. Our build server will have built binaries of those packages at that time, if possible, such that you should now be able to upgrade those ports and receive those binaries without having to wait for compilation. kde4-runtime appears to be an exception in your case because you're running a locally-modified newer version than the one we have in MacPorts, so yes you'll have to recompile that, as well as other ports for which binaries are not available, such as ImageMagick.

[RJVB (René Bertin)]

As I said, this is mostly for reference for if anyone else has been looking at this problem.

This won't work with binaries built by our buildbot system, because no ports are active

I presume you mean that no previous version of the port being built is present?

That's OK, because this feature is handled via a variant as you must have seen. I'm not sure to what extent the reproducible build argument is a blocker in this particular case as files (themselves subjected to the principle) are added to a build that is otherwise unchanged.

There's an alternative approach in which you cache the files to be preserved, and then restore them in the post-activate. That should resolve the reproducible build thing, but leave you with a potentially growing number of files that don't belong to any port.

If useful we can continue discussing this on the ML, of course.

It also seems like you're looking in part for a solution to a problem you caused for yourself.

And for anyone running non-default variants (esp. +universal!) or a non-default prefix ... or who runs a self-upgrade before the buildbots have had a chance to build all rev-bumped ports. If I'm not mistaken you end up building those ports locally because the Portfile will have the revbump already, right?

I know I've created this for myself by having a lot of personal ports but I already stopped doing frequent self-upgrades because I have (or had) too many +universal ports and got tired of blocking my machine for long periods of time just because someone had pushed supposedly crucial update to a dependency of too many ports for which I couldn't get a prebuilt binary.

[ryandesign (Ryan Carsten Schmidt)]

We've already solved this problem: whenever a port is updated in such a way that one of its libraries changes its install_name (including changing its major version), the person who updates that port shall also revbump all ports linking with that library so that they get rebuilt. Yes, binary packages may take some time for our buildbot to compile, or may not be available in all situations so users may need to compile the upgrades on their own systems. I realize you don't like this solution, but it's the one we've chosen for now. You're welcome to discuss it on the mailing list, but your proposed portgroup in this ticket is unlikely to be an acceptable solution.

[raimue (Rainer Müller)]

Further discussions:

• ​https://lists.macosforge.org/pipermail/macports-users/2016-September/041533.html
• ​https://lists.macosforge.org/pipermail/macports-dev/2016-September/033701.html

[RJVB (René Bertin)]

In case anyone is interested: I've made this feature work a bit more cleanly but storing the legacy runtimes in a dedicated repository. That requires additional symlinks, but it does make it immediately clear which runtimes are potentially obsolete, whom they belong to - and testing whether they're still used is as easy as moving the symlink aside before running port -v rev-upgrade.