Using a newer libcurl from "base"

[RJVB (René Bertin)]

MacPorts "base" links against the system libcurl for understandable reasons but this leads to failures on older OS versions when fetching information over secure connections.

I seem to recall we already discussed this a while back, but can't seem to find traces of it.

I put a copy of the actual libcurl binary from port:curl (built +gnutls so the system libcrypto can't clash with the one from MacPorts) in $prefix/libexec/macports/lib/pextlib1.0/ and run install_name_tool (or patchelf) on the pextlib library so it will load that copy.

So far this has not led to problems for me but it's not much more than a local hack. port:curl has a few too many dependencies and it doesn't seem trivial at all to build it against static lib versions of all those dependencies, creating a standalone libcurl which could then be provided for download from macports.org .

But libcurl is only called from pextlibs curl.c as far as I've seen, and my hack has shown that a pextlib built with quite "old" curl headers from the system works fine with a newer libcurl. That should mean that pextlib ought to be able to dlopen libcurl from a number of supported locations (including for instance $prefix/libexec/curl) and load the required symbols at runtime using dlsym. The libcurl at that special location could then be provided by a properly groomed port, probably even a subport of port:curl.

Using dlopen instead of using an @rpath style dependency is more work but should provide more resilience against the special libcurl failing to load because of a missing dependency. If however dyld will in that case just load the next libcurl in the rpath then that solution would be less work, and could even be handled in a post-install script.

Thoughts? Am I overlooking something obvious?

[ryandesign (Ryan Carsten Schmidt)]

Replying to RJVB:

I seem to recall we already discussed this a while back, but can't seem to find traces of it.

#51516

[kencu (Ken)]

fyi, I’ve been using a stripped-down curl with minimal deps to support my base installs for the past few years…. much easier to bootstrap:

​https://github.com/kencu/macports-ports/blob/curl-minimal/net/minimalcurl/Portfile

[kencu (Ken)]

We had worked out in the past that you couldn’t simply swap the curl library for a newer one, as base would build differently against the different versions of curl.

So to use the dlopen method you mentioned, Clemens outlined that you would have to build base against the different curl versions and then dynamically choose the code path at runtime, using the newer path if available.

This was a bit too much for anyone to take on, so the idea dropped there.

If that is actually not necessary, and the libcurl abi has not changed in years and the calls and structures remain the exact same over all the years —- that would be…unexpected…. but would make things much simpler.

[RJVB (René Bertin)]

Replying to kencu:

If that is actually not necessary, and the libcurl abi has not changed in years and the calls and structures remain the exact same over all the years —- that would be…unexpected…. but would make things much simpler.

For OS X 10.9 my memory tells me I saw "2.7.2" in the system curl-config yesterday; I think that rather means the system has version 7.2.x . My Linux rig has system libcurl 7.35.0 . Both systems have port:curl currently at 8.7.1 . I have done the hack I described on both, making it Pextlib.so link to a copy of libcurl (+gnutls as stated) installed in the same directory. A bit more involved to achieve on Linux but we don't need to get into that here.

And as said, I have yet to notice any problems.

Which isn't in fact unexpected at all if you look at the SOVERSION, which has been 4 since those 2.7 series AFAICT. In fact, the DebUntu packaging suggests that even applications built against SOVERSION 3 can run against SOVERSION 4; the library's runtime package is called libcurl3 (or libcurl3-gnutls) and installs libcurl.so.4.3.0 which gets exposed as both libcurl.so.4 (as you'd expect) but also as libcurl.so.3 . I actually had to double-check if that wasn't a hack I made, but no, it's really packaged like that. I think we can trust the people behind Debian (and the commercial entity behind Ubuntu) to have checked the validity of this thoroughly. MacPorts probably only uses a subset of libcurl functionality, too.

As usual one should expect to be able to build against one version and then run against an older version. But that should never be the case here.

FWIW, I been using port:MacPorts-devel for years now to build+destroot+(d)pkg my MacPorts installs (sorry but not sorry to whomever it was who told me once that "MacPorts is not a build system" ;) ). That means I have a certain preference to keep the number of assumptions about local modifications/additions to system software as small as possible. (It also means I was quite annoyed I had to restore and maintain the dpkg functionality but that's yet another issue.)

Anyway, how much work and how tricky would it be to make curl.c load whatever it needs from libcurl dynamically? I'm not against digging into that but if there's already idea of what I'd be getting into I also not against knowing that beforehand :)

[RJVB (René Bertin)]

Replying to ryandesign:

#51516

Weird, I did do a search on tickets by me containing "curl" ... EDIT: and of course it didn't show up on that search, not my ticket.

Replying to kencu:

much easier to bootstrap:

My hack is actually dead simple too (on Mac), so much so that I haven't even written a script for it. You just need to remember doing it. How to bootstrap an update to Pextlib's libcurl does take some thinking out, probably, that's clear. I haven't looked at your repo, but what SSL provider does your minimal curl use? The system one uses Apple's own SSL framework, which is undoubtedly the reason why it no longer suffices. Building a newer curl version against that same framework would probably not help very much.

Thinking over the idea of a port:curl subport again I think that's not even needed. port:curl could simply have a post-activate that installs a copy of the actual libcurl binary in the designated location, doing this only for curl versions that have been verified to work against "base" built on the current host. With luck that will cover the majority of supported OS versions and on those a port -n install curl +gnutls could be the first thing that port selfupdate does. Which should take care of bootstrapping.

[kencu (Ken)]

so I guess all further discussion should join the existing ticket referenced above, where we have been talk8ng about this for years and years now.

everyone interested is cc’d there.

I’ll close this ticket here as a duplicate

[kencu (Ken)]

#51516