MacPorts should use a bundled copy of a newer libcurl and SSL library rather than the OS X version

[ryandesign (Ryan Carsten Schmidt)]

Much as we already use a bundled copy of a newer Tcl, MacPorts should use a bundled copy of a newer libcurl (and CA certificate bundle?) rather than the OS X version. On older systems, the OS X version of libcurl cannot connect to some https sites, including pypi (see #51509) and also ​https://distfiles.macports.org and ​https://packages.macports.org .

[udbraumann]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #52602.

[neverpanic (Clemens Lang)]

I don't think MacPorts should be in the business of distributing a CA bundle. We shouldn't control who our users trust.

[raimue (Rainer Müller)]

The issue is mostly not libcurl itself, but system's libcurl linked against system's OpenSSL 0.9.8, which does not support TLS >= 1.1. Therefore servers that require TLS >= 1.1 can no longer be reached (#51112, #52604). As of OS X 10.9 Mavericks, system's libcurl will use ​Secure Transport instead. Secure Transport supports TLS >= 1.1 as of OS X 10.8 Mountain Lion.

If we still wanted to support OS X <= 10.7, we would need both libcurl and a ​compatible SSL/TLS library in base.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #52604.

[ryandesign (Ryan Carsten Schmidt)]

Replying to cal@…:

I don't think MacPorts should be in the business of distributing a CA bundle. We shouldn't control who our users trust.

I don't care how we arrange it, but MacPorts should at least be able to download from the MacPorts packages and distfiles servers via SSL. If the problem isn't trusting the certificates but rather the capabilities of the SSL library, then we need to bundle a newer SSL library.

[neverpanic (Clemens Lang)]

Keep in mind that bundling an SSL library implies that we will release at least as often as this SSL library to fix any security problems with it. Personally, I'm not prepared to do that, and we should just fall back to non-SSL connections on systems that do not support modern SSL.

I doubt that we have the manpower to rescue platforms that Apple has abandoned. At some point we'll just have to admit that the system is no longer usable due to issues such as these.

[dbevans (David B. Evans)]

I agree that bundling an SSL library could lead to a lot of extra cycles to update the bundle for security issues, etc. Also probably licensing issues with a binary bundle that includes OpenSSL or LibreSSL. Not sure about other SSL libraries.

To avoid these issues, could we set it up so a source build could be built with an external library other than the system ones? An interested user could then boot strap from an outdated binary Installer like this:

• install old port binary using installer
• build the SSL library of your choice using the just installed port binary
• do a source build using the installed non-system SSL library using the appropriate configuration parameters ( say --with-openssl=${prefix}/lib )

This avoids binary licensing issues and updates for OpenSSL and the like would be handled by the normal port maintenance process.

The procedure would need to be well documented to avoid too many "I tried it but it didn't work" tickets.

Or is this too complicated for the average user to deal with? Just an idea.

[larryv (Lawrence Velázquez)]

That sounds a bit complicated.

What are the downsides of downgrading to insecure connections on old platforms? Would we have to worry about tampering? We already verify checksums.

[ryandesign (Ryan Carsten Schmidt)]

We already downgrade Leopard and earlier to http for packages.macports and distfiles.macports. r150758, r150765. Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

[neverpanic (Clemens Lang)]

The problem with linking against a copy of OpenSSL in $prefix is that it will temporarily disappear while OpenSSL itself is upgraded, but that must not interrupt the upgrade process.

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

[dbevans (David B. Evans)]

Replying to cal@…:

The problem with linking against a copy of OpenSSL in $prefix is that it will temporarily disappear while OpenSSL itself is upgraded, but that must not interrupt the upgrade process.

Good point.

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

Agreed. However, we can, perhaps, provide tools for those who are interested in using these platforms regardless of their antiquity. I'm not sure how many such people there are but they're quite enthusiastic and it would be a shame to shun them, altogether, from the MacPorts community. Best world, IMO, would be for them to form an ad hoc interest group within MacPorts so they could cooperatively work on these issues themselves.

So I'm suggesting that while we shouldn't commit (and haven't) to supporting officially anything older than 10.9, it wouldn't hurt to add a hook once in a while to allow them to do the work for themselves if they choose to do so. In this case, provide a configuration option for base to optionally build with external libraries rather than system ones to solve the SSL/TLS issues inherent in the old system libraries.

By the way, if one wanted to bundle something that is light weight with good protocol coverage and permissive licensing, wolfSSL/CyaSSL looks interesting. curl has supported it upstream as an alternative so openssl/gnutls for a while now.

[ryandesign (Ryan Carsten Schmidt)]

Replying to cal@…:

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

No matter what we do, that's a problem these platforms will always have, and will continue to do so. As I said, we cannot save abandoned platforms from dying.

I don't understand why you would say that. The problem right now is caused by older systems have an old SSL implementation. The proposed fix is to provide a newer SSL implementation in MacPorts. That fixes the problem.

I agree that if we do this, we should release a new version of MacPorts when a new version of the SSL library is available, in case the new version of the SSL library fixes security vulnerabilities. I agree that our record to date of releasing new versions of MacPorts in a timely fashion is not good, which would be a reason not to make this change right now. However, I hope that once we move to GitHub, we can improve our processes and procedures to make it much easier to release new versions of MacPorts.

[ken-cunningham-webuse]

Best world, IMO, would be for them to form an ad hoc interest group within MacPorts so they could cooperatively work on these issues themselves.

That might be a good idea. Even for example a separate trac stream for <10.9 issues would keep the main stream uncluttered, give a voice to those trying to keep these machines going, but keep the signal-to-noise ratio up for the main cadre of users. People who wanted to help out with older systems could.

Or maybe a separate stream would be the final death bell for these systems...

[larryv (Lawrence Velázquez)]

Replying to ryandesign@…:

Once we are properly mirroring distfiles before doing buildbot builds, that will probably be a sufficient fix. But it doesn't help old systems download from other sites that require https.

Other than the brief period before we mirror a new distfile, when would said systems not be able to fall back to the mirrors?

[neverpanic (Clemens Lang)]

Replying to ryandesign@…:

I don't understand why you would say that. The problem right now is caused by older systems have an old SSL implementation. The proposed fix is to provide a newer SSL implementation in MacPorts. That fixes the problem.

Yes, that fixes the problem in MacPorts, but users on these platforms will continue to experience the same problems with other services that use TLS. So we can't fix the problem completely, but would still bundle an SSL library on platforms that already ship one that's perfectly fine.

I agree with devans here, we should provide instructions, warn users about the caveats and have them compile MacPorts against MacPorts curl.

[mkae (Marko Käning)]

Cc Me!

[devernay (Frédéric Devernay)]

Cc Me!

[ken-cunningham-webuse]

I gave this a shot today. Have curl installed in /opt/local

$ port -v installed | grep curl
  curl @7.50.3_0+ssl+universal (active) platform='darwin 10' archs='i386 x86_64'
  curl-ca-bundle @7.50.3_0 (active) platform='darwin 10' archs='noarch'

configured and installed macports from source, upgrading an existing installation

./configure --with-curlprefix=/opt/local
make
sudo make install

seems to work perfectly. The recent problem downloads, like hexchat and git, download without trouble now.

That's a 4-minute fix to an irritating problem.... unless you accidentally uninstall curl, or course...

[ken-cunningham-webuse]

although this setup as described is working well, I worried it was possibly too precarious to rely on itself like that in case of any unforeseen issues that might arise in future.

I decided to install a new copy of macports into /opt/bootstrap, installed curl in that installation, and then referenced the curl in /opt/bootstrap when reinstalling macports back into /opt/local.

[RJVB (René Bertin)]

I think there might be another solution, one that doesn't require a hard dependency on curl.

If the curl function from Pextlib fails with an error that's likely to signal a TLS problem (and unlikely to have another cause), try again using the curl command (or wget). If that fails too and the command isn't found in ${prefix}/bin, instruct the user to install port:curl .

That'd be a 1-time investment to upgrade the Tcl function calling Pextlib::curl, but no cyclic or otherwise problematic dependencies will be introduced, and users don't have to follow more complex instructions than necessary.

[neverpanic (Clemens Lang)]

Please, let's not have more 'if this doesn't work let's fall back to this other lesser-tested special case'.

Have you even looked at all the options supported in pextlib1.0/curl.c? Re-creating those for a command-line client is complicated, and in some instances, impossible.

[ryandesign (Ryan Carsten Schmidt)]

The current discussion is centered around being able to reach servers running newer ssl. But the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs. That's the reason why we bundle tcl 8.5 now; we were able to remove workarounds for tcl 8.4 and start using features introduced in tcl 8.5. In the same way, the latest curl might offer some useful features not present in Tiger's curl, and there are workarounds in base for bugs in the version of curl included with Snow Leopard that we could remove if we bundled a newer version.

The problem of needing to also bundle a newer ssl library to support newer ssl sites, and to quickly release new MacPorts base versions to update the ssl library should any vulnerabilities be found, makes this idea less wonderful. On the one hand, having a recent but not completely up to date ssl library would be better for users of old OS versions that don't get security updates anymore; but on the other hand, it would be worse for users of new OS versions that do.

Maybe once we are on GitHub we will improve our testing and release process to the point where releasing a new version of MacPorts can be done very easily. If so, maybe then we can revisit this issue.

[raimue (Rainer Müller)]

In 7a428704/macports-ports:

git: add alternative download URL to master_sites

The main server at www.kernel.org has a hard requirement for TLS 1.2,
which is not available on older versions of macOS. This CDN mirror is
less restrictive and can be accessed using TLS 1.0.

Closes: #52602
Closes: #52604
Closes: #52836
See: #51516

[raimue (Rainer Müller)]

Has duplicate #46539.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs.

#51045 is an example of such a curl bug.

[mf2k (Frank Schima)]

Has duplicate #53181.

[raimue (Rainer Müller)]

Fetching Python distfiles from files.pythonhosted.org (as used by ports via master_sites pypi:) will require TLS 1.2 as of 2017-04-30. Support for TLS 1.0 and 1.1 will be disabled. ​https://pyfound.blogspot.de/2017/01/time-to-upgrade-your-python-tls-v12.html

TLS 1.2 is supported in Secure Transport as of OS X 10.8, which means fetching python ports will stop working on OS X <= 10.7 on this date.

[jduerstock (Jason Duerstock)]

Since Macports locally stores the hashes of the source files anyway, can we ignore any HTTPS failures at least with respect to files downloaded with known hashes?

[neverpanic (Clemens Lang)]

Replying to jduerstock:

Since Macports locally stores the hashes of the source files anyway, can we ignore any HTTPS failures at least with respect to files downloaded with known hashes?

Unfortunately we can't. The issue is not that the certificate presented by the servers would not verify (which we can ignore), but that all protocols implemented in older versions of libcurl (and the cryptography library it uses) shipped with older versions of macOS are no longer accepted by various servers. This means that all attempts to connect just fail. A workaround would be using an unencrypted channel to download the files, but a lot of servers nowadays redirect to SSL-only protocols.

[raimue (Rainer Müller)]

We are already downgrading connections to our own mirrors to HTTP without SSL/TLS on older systems (see ​archive_sites.tcl). This would already help with many ports on these legacy systems, but our distfiles mirroring has not been brought back yet after we left macOS Forge.

[kencu (Ken)]

Just FYI, the bootstrap method above has worked nicely on 10.4 up since I started using it last year. Haven't run into a fetch error since then on any system. I update the bootstrap install of Macports every few months, otherwise ignore it.

[kencu (Ken)]

One thing to note - when updating MacPorts versions, you will likely have to build it again from source rather than trust the sudo port selfupdate mechanism to do it properly for you. sudo port selfupdate does not presently appear to preserve configure settings like --with-curlprefix.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

But the idea to use a bundled curl is much older, and is based on the idea that we should use the latest software, because it has more features and fixes bugs.

Could we perhaps tackle this part -- bundle and link with a recent version of libcurl -- and postpone dealing with ssl issues? On older systems, we can link our bundled libcurl with the system's openssl library; on the newer systems that support it, we can use the --with-darwinssl configure flag to tell libcurl to use the macOS Secure Transport API.

[noloader (Jeffrey Walton)]

Regarding this:

Much as we already use a bundled copy of a newer Tcl, MacPorts should use a bundled copy of a newer libcurl (and CA certificate bundle?) rather than the OS X version.

I copied cURL's cacert.pem to /System/Library/OpenSSL/certs/cacert.pem, which was previously empty. It did not resolve the issue.

It looks like its the old/downlevel version of OpenSSL. OpenSSL 0.9.8 lacks a lot of goodies, like elliptic curve support and TLS 1.2 support. It is certainly not following contemporary best practices.

Thanks for Ryan's help on a related ticket.

[ryandesign (Ryan Carsten Schmidt)]

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

[neverpanic (Clemens Lang)]

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

Now that the mirroring is fixed, I'm wondering whether we should just close this ticket as wontfix. You mentioned that you want a bundled libcurl in MacPorts for other reasons, but the cosmetic issue you linked in #51045 does in my opinion not warrant the additional effort of bundling libcurl (especially since it's fixed since two OS releases and we could add a simple workaround in base for that).

Do you have any other reasons for bundling a libcurl? If not, I would close this ticket as wontfix.

[ryandesign (Ryan Carsten Schmidt)]

Replying to neverpanic:

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Now that the mirroring is fixed, I'm wondering whether we should just close this ticket as wontfix. You mentioned that you want a bundled libcurl in MacPorts for other reasons, but the cosmetic issue you linked in #51045 does in my opinion not warrant the additional effort of bundling libcurl (especially since it's fixed since two OS releases and we could add a simple workaround in base for that).

Do you have any other reasons for bundling a libcurl? If not, I would close this ticket as wontfix.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

The main difficulty or effort of bundling libcurl was originally that we didn't bundle any other software with MacPorts, so a method of doing that had to be invented. Now that we have created that method and already use it to bundle 4 libraries, it shouldn't be that difficult to bundle a few more libraries for modern ssl support or xz decompression (#52000).

[yan12125 (Chih-Hsuan Yen)]

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at ​https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

[noloader (Jeffrey Walton)]

Replying to ryandesign:

Replying to neverpanic:

Replying to ryandesign:

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: ​https://githubengineering.com/crypto-removal-notice/ .

After the GitHub change things like wget --no-check-certificate --tlsv1 and curl --insecure --tlsv1 simply will not work. It is technically impossible for some sites because of the site's configuration.

[noloader (Jeffrey Walton)]

Replying to noloader:

...

Yes, I know that merely using a bundled curl that's still linked with the system's openssl will not solve all of the problems being discussed in this ticket. But I think we should still do it because it will solve some of the problems (e.g. comment:30).

I think we all agree that we should not bundle an SSL library with MacPorts.

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: ​https://githubengineering.com/crypto-removal-notice/ .

After the GitHub change things like wget --no-check-certificate --tlsv1 and curl --insecure --tlsv1 simply will not work. It is technically impossible for some sites because of the site's configuration.

This is worth mentioning... TLS 1.3 is in Last Call (LC) status on the TLS WG mailing list. I think the future trend will be sites supporting TLS 1.2 and TLS 1.3 as sites are reconfigured for the new standard. Also see ​Last Call: <draft-ietf-tls-tls13-24.txt> (The Transport Layer Security (TLS) Protocol Version 1.3) to Proposed Standard.

Without getting too bogged down in what version of cacert.pem to use, maybe the better engineering question would be, how can MacPorts ensure TLS 1.2 and TLS 1.3 "just work" for users. If you answer that question I believe the CA list question will follow without much effort.

[kencu (Ken)]

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at ​https://github.com/yan12125/macports-base/tree/bundle-curl.

Well that looks pretty fabulous! All that is missing is the CA certificates I guess. Well done.

[ryandesign (Ryan Carsten Schmidt)]

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at ​https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

Yes thanks for this. There has been some spirited debate about the merits of openssl vs libressl, including pointing out that as of High Sierra, Apple is distributing libressl in macOS instead of openssl. I would be fine with using libressl in MacPorts instead of openssl.

I see that your branch deletes the MacPorts --with-curlprefix configure option. We probably want to keep that option for users who want to override it, but default it to using the bundled copy.

And as I mentioned above, on systems new enough to have a Secure Transport implementation that supports TLS 1.2, I'd like to use that, and only build and use the bundled libressl on older systems. /usr/bin/curl didn't start using Secure Transport until Mavericks, but it has been around a lot longer than that. I'd have to do some testing to figure out how far back it supports TLS 1.2. Ideally we would do a configure script test for Secure Transport's capabilities. The only check curl's own configure script does is for the existence of /System/Library/Frameworks/Security.framework, but that has existed since Mac OS X v10.0 so we'd need a more specific check than that.

[neverpanic (Clemens Lang)]

Replying to ryandesign:

I do not agree yet. Older OSes need a newer ssl library to fetch from some https sites. So can't we provide a newer openssl or libressl on older systems, and use Apple's Secure Transport on newer ones? Yes there might be security vulnerabilities discovered in the ssl library we bundle, but isn't it likely that there are fewer vulnerabilities in it than in the old openssl version used on those systems?

I do not want to ship a library with potentially known and dangerous security vulnerabilities to users. If we use Apple's library and it does not work, the blame is on Apple, and users on old OSes cannot expect support from Apple. I do not want to create the impression that we support this setup.

Yes, there might be fewer problems, but if there are problems they would then be *our* problems. At the moment, they are not.

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

The latter, the CA bundle would eventually be too old for new certificates to validate.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets? I'm not convinced that we should bundle every library we use.

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at ​https://github.com/yan12125/macports-base/tree/bundle-curl.

Thank you, that's helpful for users that want to run this at their own risk.

Replying to noloader:

Newer libraries with vulnerabilities seems the lesser of the two evils. Early 2000's tools with early CA lists don't help with security much. People will just disable the certificate checks, which puts them in a worse place.

Early 2000's tools are pretty much not going to give you any reasonable assumption of security at this point. I'd rather want users to realize that then to give them a false sense of security.

And newer libraries that "just work" seems like a worthy usability and security goals. People won't have to do things like wget --no-check-certificate and curl --insecure.

Also, those old libraries and programs built on them are going to cause more trouble in the future because they can't do TLS 1.2. A TLS 1.2-only configuration is just common place nowadays. For example, GitHub recently made the engineering change: ​https://githubengineering.com/crypto-removal-notice/ .

This is not the point of this ticket. Software installed by MacPorts will always use modern libraries and support modern crypto. This is about the libraries used by MacPorts itself. So either, your curl is MacPorts curl, which will work, or it is Apple's curl, which we will also not fix with this.

[ryandesign (Ryan Carsten Schmidt)]

Let me try one more time to explain my position. I still believe that we should bundle libcurl with MacPorts (either on all systems or only on older systems that don't get updates from Apple anymore), and that we should bundle a newer libressl or openssl with MacPorts (either only on older systems that don't get updates from Apple anymore or only on older systems that don't support newer TLS algorithms that many servers now require). On newer systems that still get updates from Apple we can either use our bundled curl and link it with Apple's security framework, or we can continue to use the system's libcurl as we do now.

To be concrete about it, at this time, I might propose that we use a bundled newer libcurl/libressl/openssl on OS X 10.11 El Capitan and earlier. I believe El Capitan is no longer receiving security updates as of the end of 2018. The problem I mentioned in #51045 was present in El Capitan's curl but not Sierra's, and also René recently encountered a KDE download server that could not be accessed from El Capitan's curl or earlier but worked with Sierra's. On Sierra and later, we could either use our bundled libcurl linked with the security framework or else use macOS libcurl.

Replying to neverpanic:

I do not want to ship a library with potentially known and dangerous security vulnerabilities to users.

To what library are you referring? libcurl? openssl? libressl? What are the specific known and dangerous security vulnerabilities that you're referring to?

I'm not certain what types of security vulnerabilities do or could exist in those libraries but even if there were any I'm not sure how serious that would be for us. We would only be using this libcurl for MacPorts tasks like downloading known files. We checksum our distfiles and sign our packages, so any vulnerability that caused curl to deliver data other than that which it should would be noticed by MacPorts immediately and the bad data would not be used. And we already don't consider the information we send or receive with curl to be confidential—we don't transmit private information and many of our mirrors already don't use https. As I have said before, it seems likely to me that bundling a newer libcurl/libressl/openssl will fix more problems than it causes.

Users often don't care or even know why something doesn't work, they just know it doesn't work and they blame the most obvious source. MacPorts is broken on 10.8 and earlier when downloading from many servers, and many users have reported this to us. There are some edge cases where our buildbot does not mirror distfiles immediately or at all, and sometimes it's just busy with other tasks or even offline; during those times, with no mirror to fall back on, users can encounter fetch errors. We've even had the problem on our own buildbot workers for older OS versions. We've identified the cause: outdated OS libcurl/openssl libraries and CA bundle. We have a fix: bundle newer ones. This would fix the bug, increase user happiness because things will just work for them, and free up the time MacPorts developers would otherwise spend responding to user bug reports about this problem.

Your objection in comment:3 was that we should not distribute a CA bundle. Would bundling an ssl library require us to include CA bundle? Doesn't the OS ship with one that we could use? Or is that part of the problem—the old OS's CA bundle doesn't contain the information needed to trust new sites?

The latter, the CA bundle would eventually be too old for new certificates to validate.

Ok, then we could bundle a new CA that MacPorts libcurl/libressl/openssl would use, again possibly only for older systems that won't get such an update from Apple. We could even think about a way to update it via selfupdate that would be separate from MacPorts base releases, if making a user wait for a new MacPorts base release might be undesirable. Or we could find a way to have MacPorts base optionally use the CA from the curl-ca-bundle port.

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets?

As I recall, curl on Leopard would emit a spurious error message during some downloads. As far as I remember this did not affect the ability to download the files. I'm not able to find a ticket or other reference to it as I don't remember exactly what the error message was so I can't search for it now.

Obviously this bug and the previously mentioned one are trivial but they are a reminder that there have probably been many bugs fixed in curl since the versions bundled with old macOS versions which our users would benefit from if we bundled a newer version. The benefit to us is being able to remove code from MacPorts base that works around those old bugs, thus simplifying our code and making it easier to maintain. (I currently see 3 different workarounds for bugs or missing features in various versions of curl in our curl.c.) We were able to do the same when we bundled Tcl 8.5 and could remove the Tcl 8.4 workarounds we had, as I mentioned above years ago.

I'm not convinced that we should bundle every library we use.

I'm not trying to propose that. This ticket is only about bundling curl and libressl/openssl with MacPorts. I believe this will benefit us and our users and I am not convinced that the drawbacks outweigh that.

[ryandesign (Ryan Carsten Schmidt)]

Even just bundling a newer curl on 10.8 and earlier would help. I'm discovering that the curl that comes with 10.8 and earlier is linked with openssl 0.9.8 while the curl on 10.9 and later is linked with the Security framework. openssl 0.9.8 seems to be able to do nothing newer than SSLv3. However if I compile the curl port with the +darwinssl variant to make it use the Security framework, then it is able to do TLSv1 on 10.6 and 10.7 and TLSv1.2 on 10.8. That's enough to let it get files from GitHub and SourceForge on 10.8 (and to connect to TLSv1 sites on 10.6 and 10.7 that it couldn't otherwise connect to) without putting us into the business of bundling encryption software like openssl.

[kencu (Ken)]

Edit: this post missed the point for what you were after, bundling curl without openssl. Deleted.

[fhgwright (Fred Wright)]

In the narrower case where the only issue is the certificate validation (and possibly too-old Apple root CAs), there could be a simpler fix. Currently, it's possible to get around such issues by setting fetch.ignore_sslcert. Aside from possibly using that on the command line, there are currently 11 ports and one PortGroup that set that option. It occurs to me that there could be a third value, 'auto', that maps to 'yes' or 'no' depending on circumstances.

The simplest definition of "circumstances" would be whether it's an OS version currently updated by Apple or not. It's reasonable to assume that versions still updated by Apple have sufficiently recent root CAs, while older OSes may not. If the only issue is whether the root CAs themselves have expired, then that could be checked directly, but if multiple certificates with different root CAs and different expirations are possible, then it's not so easy to check.

Doing without the certificate checking altogether isn't that awful, since distfiles need to pass checksum checks, anyway, but being a bit less heavy-handed in suppressing the check would be a plus. The 'auto' setting might even be a reasonable default.

It could also make a distinction between targeting a MacPorts distfile mirror or an upstream distfile source. In the former case, the certificate status could be known more accurately, while that isn't easily possible for arbitrary upstream sources.

[kencu (Ken)]

I suppose using a bootstrap version of MacPorts in /opt/bootstrap to provide a full current version of curl and openssl that can be used by the main installation of MacPorts in ${prefix} is felt to be just beyond the capability of average users?

Fredowski has a quite beautiful ​shell script to automate the installation of MacPorts from source. It looks like that script, with slight modification, could accomplish the bootstrap and then full installation.

FYI I am using the /opt/bootstrap system on all systems up to 10.10 now, and it appears to be 100% bulletproof after many continuous years of use. Even if MacPorts changes to some kind of darwinssl system, I'll likely revert/override that because this system is just better.

[noloader (Jeffrey Walton)]

I suppose using a bootstrap version of MacPorts in /opt/bootstrap to provide a full current version of curl and openssl that can be used by the main installation of MacPorts in ${prefix} is felt to be just beyond the capability of average users?

Yes (or is it No?). I think bootstrapping cURL is a good idea. You will need it on old PowerMacs.

In fact, for bootstrapping, I recommend you build Wget rather then cURL. Wget is easier to build because it has two dependencies - OpenSSL and Unistring. Bootstrapping Wget/OpenSSL/Unistring works back to at least OS X 10.5 on PowerMacs.

I eat my own dogfood: ​https://github.com/noloader/Build-Scripts/blob/master/bootstrap/bootstrap-wget.sh.

Wget has a minor annoyance due to dynamic linking. Sed fixes that with:

# Fix makefiles. No shared objects.
IFS="" find "$PWD" -iname 'Makefile' -print | while read -r file
do
    sed "s|-lssl|$LIBDIR/libssl.a|g" "$file" > "$file.fixed"
    mv "$file.fixed" "$file"
    sed "s|-lcrypto|$LIBDIR/libcrypto.a|g" "$file" > "$file.fixed"
    mv "$file.fixed" "$file"
    sed "s|-lunistring|$LIBDIR/libunistring.a|g" "$file" > "$file.fixed"
    mv "$file.fixed" "$file"
done

Once you bootstrap Wget you can fetch everything you need for a full-blown build of cURL: ​https://github.com/noloader/Build-Scripts/blob/master/build-curl.sh

[kencu (Ken)]

wget is not useful for this issue, though.

You build MacPorts like this:

./configure --with-curlprefix=/opt/bootstrap

and there are no more SSL issues ever.

[kencu (Ken)]

Oh, I see -- you mean use your build-curl.sh script instead of MacPorts to build curl.

Yes, perhaps. I just understand MacPorts so well, and it's so easy to use, I prefer it to a script.

[kencu (Ken)]

that is a very fancy set of scripts you've put together there... thanks for all that work. I'll take a look at them in more detail.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

#51045 was merely the first example of a curl bug that's fixed in newer versions that I found to link to this ticket. There's another cosmetic issue I found on Leopard that's been fixed. I'm sure there have been more bugs fixed in curl over the years.

Do you have specific tickets?

As I recall, curl on Leopard would emit a spurious error message during some downloads. As far as I remember this did not affect the ability to download the files. I'm not able to find a ticket or other reference to it as I don't remember exactly what the error message was so I can't search for it now.

I saw it on Tiger just now:

--->  Fetching distfiles for guile
--->  Attempting to fetch guile-2.2.7.tar.gz from https://ftp.gnu.org/gnu/guile
--->  Attempting to fetch guile-2.2.7.tar.gz from ftp://ftp.gnu.org/old-gnu/guile
WAITCONNECT => PROTOCONNECT
--->  Attempting to fetch guile-2.2.7.tar.gz from http://mirror.facebook.net/gnu/guile
--->  Verifying checksums for guile

That unnecessary WAITCONNECT => PROTOCONNECT message comes from curl and ​later versions of curl don't print it anymore.

[kencu (Ken)]

has duplicate #64081

[devernay (Frédéric Devernay)]

Any chance of getting a proper fix for that? I'm tired of manually downloading github packages (#64081), especially now that most distfiles are fetched from github rather than macports mirrors.

I would even consider it a security issue at this point, since "All older versions [of openssl] (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used." - from ​https://www.openssl.org/source/

Any software that currently ships (including MacPorts) should not use any of these versions, and thus not use the system's curl on older systems.

[danielluke (Daniel J. Luke)]

Replying to devernay:

I would even consider it a security issue at this point, since "All older versions [of openssl] (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used." - from ​https://www.openssl.org/source/

Any software that currently ships (including MacPorts) should not use any of these versions, and thus not use the system's curl on older systems.

The security issue is trying to run an old, unsupported macOS version that is no longer receiving patches from Apple. If you insist on doing that, you can work-around it. I don't think MacPorts should be in the business of providing OS patches after Apple stops, and I don't think it should be in the business of encouraging people to run obsolete OS versions.

[devernay (Frédéric Devernay)]

you can work-around it

Unfortunately no, unless by "work-around" you mean buying new hardware. In which case I don't see the point in supporting any OS version (and thus hardware) that was obsoleted by Apple. And we might as well all use homebrew, which does exactly that.

And, by the way, a newer version of OpenSSL is not an OS patch. OpenSSL is a userland library.

[kencu (Ken)]

it takes 4 minutes to rebuild and install macports from source against libcurl installed in /opt/local

it takes longer to install a macports into /opt/bootstrap and then rebuild macports in /opt/local against that, but it is more resilient.

[kencu (Ken)]

see above:

comment:19

I first did this five years ago, and have on every older system since.

works nicely

[noloader (Jeffrey Walton)]

Replying to danielluke:

Replying to devernay:

I would even consider it a security issue at this point, since "All older versions [of openssl] (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used." - from ​https://www.openssl.org/source/

Any software that currently ships (including MacPorts) should not use any of these versions, and thus not use the system's curl on older systems.

The security issue is trying to run an old, unsupported macOS version that is no longer receiving patches from Apple. If you insist on doing that, you can work-around it. I don't think MacPorts should be in the business of providing OS patches after Apple stops, and I don't think it should be in the business of encouraging people to run obsolete OS versions.

I don't think this position is correct. Macports specifically supports older versions of Apple hardware and software. Ken regularly works on support for the PowerPC machines, and I use the Macports support for the old hardware on my PowerMacs. The FAQ also includes discussions of older versions of OS X.

From the FAQ under "Why do I have to install xorg-* ports?" (wiki:FAQ#x11-libs):

This [x11 client] allows for more consistent support for older machines which are not as up to date (especially Tiger which doesn't have a pkg-config compatible X11 SDK). Most X11 packages in MacPorts are newer than those shipped by Apple and usually reflect the latest work from the ​XQuartz Project.

[udbraumann]

Replying to noloader:

I don't think this position is correct. ...

I fully agree with noloader.

[devernay (Frédéric Devernay)]

Replying to kencu:

see above:

comment:19

I first did this five years ago, and have on every older system since.

works nicely

Thanks a lot! That's a good solution.

The last phase of sudo make install (./src/images_to_archives.tcl) failed several times because some of my installed packages had wrong tarlists, and the error messages were not very informative. I had to reinstall these, and everything went fine afterwards. Maybe there's a way to skip that phase if it's not necessary?

Processing 10 of 317: lzo2-2.10_0+universal.darwin_10.i386-x86_64.tbz2
Command failed: /usr/bin/tar -cjf /opt/local-libc++/var/macports/software/lzo2/lzo2-2.10_0+universal.darwin_10.i386-x86_64.tbz2 -T /opt/local-libc++/var/macports/software/lzo2/tarlist > /opt/local-libc++/var/macports/software/lzo2/error.log 2>&1
Exit code: 1
command execution failed
    while executing
"system "$tarcmd -cjf $newlocation -T ${targetdir}/tarlist > ${targetdir}/error.log 2>&1""
    ("foreach" body line 78)
    invoked from within
"foreach installed $ilist {
    incr counter
    set iname [lindex $installed 0]
    set iversion [lindex $installed 1]
    set irevision [lindex $inst..."
    (file "./src/images_to_archives.tcl" line 36)
make: *** [install] Error 1

I wish the port command could issue a warning that it's using the system's outdated openssl/curl, and link to those special instructions to build from source with the --with-curlprefix option. Or at least make this a FAQ. Seeing the number of Cc: on this ticket, this seems to be affecting a number of users.

The real issues will come when even curl cannot be downloaded using the system's curl, but as long as pre-built packages are available (thanks again Ken) we're fine.

[noloader (Jeffrey Walton)]

Replying to devernay:

... The real issues will come when even curl cannot be downloaded using the system's curl, but as long as pre-built packages are available (thanks again Ken) we're fine.

I think this is the linchpin of the problem. A modern cURL needs to be bootstrapped.

I find cURL is not the easiest program to build because it has so many dependencies. I recommend bootstrapping a downloader by building Wget. Wget has two dependencies - libunistring and OpenSSL. Once Wget is built and installed you can download anything, including a modern cURL.

I eat my own dogfood. I bootstrap with Wget in my collection of build scripts at ​https://github.com/noloader/Build-Scripts.

[danielluke (Daniel J. Luke)]

Replying to noloader:

I don't think this position is correct. Macports specifically supports older versions of Apple hardware and software.

No, it doesn't Official support is much more limited than the number of versions we build/release for. Everything else is 'best effort' (everything more than the currently supported upstream releases is just whatever the volunteers what to support).

[evanmiller (Evan Miller)]

As noted in #64189, if an installation links against /opt/local and a dependent library is later deactivated, the port program will become unusable and the MacPorts system will have to be fixed manually or reinstalled. So I would recommend the /opt/bootstrap route until something better is implemented here.

[kencu (Ken)]

the /opt/bootstrap approach works, but for systems 10.6 to 10.8, changing macports.conf back to libstdc++ in /opt/bootstrap saves a lot of trouble, and in the case of 10.6, avoids overwriting /usr/lib/libc++.dylib by other installs of macports.

This is indeed a PITA....when I did it first 5 years ago I didn't imagine still needing it.

I support bundling some ssl as done above on older systems, and move on from this!

Throw in a dislaimer if you like.

[kencu (Ken)]

Could also perhaps give instructions on how to use Jeffery's curl-building script mechanism in some prominent place...looks more lightweight than a whole /opt/bootstrap, and much simpler perhaps tp explain (I have not actually tried it, tho).

[barracuda156]

Replying to ken-cunningham-webuse:

I gave this a shot today. Have curl installed in /opt/local

$ port -v installed | grep curl
  curl @7.50.3_0+ssl+universal (active) platform='darwin 10' archs='i386 x86_64'
  curl-ca-bundle @7.50.3_0 (active) platform='darwin 10' archs='noarch'

configured and installed macports from source, upgrading an existing installation

./configure --with-curlprefix=/opt/local
make
sudo make install

seems to work perfectly. The recent problem downloads, like hexchat and git, download without trouble now.

That's a 4-minute fix to an irritating problem.... unless you accidentally uninstall curl, or course...

Apparently failing for me.

36-9:MacPorts-2.7.1 arjuna$ port -v installed | grep curl
  curl @7.80.0_0+ssl (active) requested_variants='' platform='darwin 9' archs='ppc' date='2021-11-17T08:51:29+0800'
  curl-ca-bundle @7.80.0_0 (active) requested_variants='' platform='darwin 9' archs='noarch' date='2021-11-17T07:59:32+0800'
36-9:MacPorts-2.7.1 arjuna$  /opt/local/libexec/mpstats submit
Submitting data to https://ports.macports.org/statistics/submit/ ...
Error: SSL connect error
    while executing
"curl post "submission\[data\]=$json" $stats_url"

[kencu (Ken)]

It doesn't work to just have curl installed.

You then need to rebuild macports against that newer curl.

Then the ssl stuff will work.

But it is fragile to have macports built against it's own installed ports, so the other method (/opt/bootstrap) will likely be favoured, until a proper fix is accepted at least.

[barracuda156]

Replying to kencu:

It doesn't work to just have curl installed.

You then need to rebuild macports against that newer curl.

Then the ssl stuff will work.

But it is fragile to have macports built against it's own installed ports, so the other method (/opt/bootstrap) will likely be favoured, until a proper fix is accepted at least.

I did rebuild with:

36-9:~ arjuna$ cd /Users/arjuna/Downloads/MacPorts-2.7.1 
36-9:MacPorts-2.7.1 arjuna$ ./configure --with-curlprefix=/opt/local

and then make and sudo make install.

Will try bootstrap prefix.

[kencu (Ken)]

Oh, well -- yes that is what usually works. I'm not completely sure then why it didn't fix the ssl transports for you!

[barracuda156]

Replying to yan12125:

FWIW, I have a branch that bundles LibreSSL + libcurl with macports-base at ​https://github.com/yan12125/macports-base/tree/bundle-curl.

I choose LibreSSL just because it's based on autotools, while OpenSSL has a custom Perl-based configure system that once brought some headache to me. It should not be difficult to switch LibreSSL back to OpenSSL for macports-base.

This is outdated now? I see that repo is archived.

[jerryyhom]

I have read through most comments. I guess the issue is whether someone is willing to put in the time/effort to link macports base with a modern curl, and maybe no one is willing. I have two comments.

1. Ken (or others), you have manually compiled with a modern curl for several years. On old systems, what happens with expired certificates? Do you update them?

2. As for linking with a modern curl, I think Ryan's comment:57 makes sense, which also follows Apple's recommendation about OpenSSL[1]. I would like to nudge this issue forward, but I am not familiar with macports base. Can someone comment how much development effort is required?

[1]: <​https://developer.apple.com/library/archive/documentation/Security/Conceptual/cryptoservices/SecureNetworkCommunicationAPIs/SecureNetworkCommunicationAPIs.html#//apple_ref/doc/uid/TP40011172-CH13-SW1>

[ryandesign (Ryan Carsten Schmidt)]

Replying to jerryyhom:

I guess the issue is whether someone is willing to put in the time/effort to link macports base with a modern curl,

A good amount of that time/effort was already put in in comment:48 though I had some follow-ups about changes I would have liked to see.

1. Ken (or others), you have manually compiled with a modern curl for several years. On old systems, what happens with expired certificates? Do you update them?

Ken uses a separate MacPorts prefix to install a bootstrap copy of curl, which would use the certificates from the curl-ca-bundle port, which gets updated often. Ken would just have to selfupdate and upgrade outdated periodically in that bootstrap MacPorts prefix to receive those updates.

[jerryyhom]

More than a week has passed without objection. I quickly re-read the history of comments and wonder if anyone still has objections? There were passionate arguments 3-6 years ago, but my feeling is that the discussion veered off on a subtle tangent and missed the point. My understanding is that bundling curl will allow macports on older systems to use modern ssl/tls. If I am missing the point, kindly share your knowledge. :)

Otherwise, I think this issue is mostly beneficial for macports and a shame that it has languished for six years. Most of the work has already been prototyped. Consider the time savings by those users working around the limitations. Do we need consensus or a willing macports base developer with time? How else can this issue be nudged forward?

[tomio-arisaka (Tomio Arisaka)]

For example, Ghostscript cannot be built with MacPorts on both High-Sierra and Tiger.

On macOS High Sierra, "/usr/bin/curl" cannot download particular files due to the expired certificate.

$ /usr/bin/curl --version
curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy 
$ 
$ /usr/bin/curl -LOR https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   806    0   806    0     0   3317      0 --:--:-- --:--:-- --:--:--  3330
100   834    0   834    0     0   2071      0 --:--:-- --:--:-- --:--:--     0
100  1067    0  1067    0     0   1884      0 --:--:-- --:--:-- --:--:--  1884
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

On Mac OS X Tiger PPC, "/usr/bin/curl" cannot download any files with SSL due to the old version of curl.

$ /usr/bin/curl --version
curl 7.16.3 (powerpc-apple-darwin8.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
Protocols: tftp ftp telnet dict ldap http file https ftps 
Features: GSS-Negotiate IPv6 Largefile NTLM SSL libz 
$ 
$ /usr/bin/curl -LOR https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz

curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
$ 
$ /usr/bin/curl -v -I https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz
* About to connect() to sourceforge.net port 443 (#0)
*   Trying 172.64.153.13... connected
* Connected to sourceforge.net (172.64.153.13) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /usr/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv2, Client hello (1):
* error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
* Closing connection #0
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

My simple resolution is to add fetch phase in order to use the latest version of curl:

if {${os.major} < 18} {
    depends_build-append    port:curl

    fetch {
        if { [file isfile "${distpath}/ghostscript-fonts-other-6.0.tar.gz"] == 0 } then {
            system -W ${distpath} "curl -LOR https://sourceforge.net/projects/gs-fonts/files/gs-fonts/6.0%20%28misc%2C%20AFPL%29/ghostscript-fonts-other-6.0.tar.gz"
        }
        ...
        ...
    }
}

[kencu (Ken)]

that only is possible sometimes, depending on fetch details, and requires manual editing of each, so … ain’t likely gonna happen.

bundling curl may or may not happen eventually…influential.people don’t like macports being responsible for security updates (although we kinda already are…)

in the meantime, the /opt/bootstrap workaround above continues to work very well for those who use it, on all systems that show ssl issues, with zero changes in any portfiles or macports. So that is what I recommend.

[jerryyhom]

Thanks for the feedback. Presumably, there are over a dozen people interested in this issue, and since I am not experienced with macports base, am genuinely interested to hear objections people might have. Otherwise, I wonder why this issue is stalled when most of the development work has been done.

Ken noted the issue of macports's security. I think the issue is more fundamental. Assuming macports wants to operate on older systems, TLS has become a necessary feature. The Apple guideline I referenced above even recommends bundling your own ssl lib instead of relying upon the builtin ssl lib.

I certainly appreciate that many macports maintainers have put in significant effort to keep operating on older systems. Setting aside the issue of TLS as a necessary feature, what are the security concerns? I am genuinely asking because I think we all agree security is important, but not all of us are security experts. Again, I'm hoping to nudge this issue forward with constructive dialogue.

[barracuda156]

Replying to kencu:

in the meantime, the /opt/bootstrap workaround above continues to work very well for those who use it, on all systems that show ssl issues, with zero changes in any portfiles or macports. So that is what I recommend.

If the question is merely “If some users want this feature, there is a way to make it”, then yes, sure. But if Macports developers are interested to get more accurate statistics, then such solution is suboptimal, to say the least. Realistically speaking, pretty much no one gonna go that route, which we see from data collected. Then developers get a false impression that no one installs software, and support gets dropped. In the long run, everyone is worse off.

[kencu (Ken)]

Maybe -- but you will not get all the important people to agree to bundle curl/openssl into the MacPorts build.

So either you convince the ones that won't to agree (good luck with that) or you accept the workaround (which I know you use) or -- you do a lot of work to come up with another method.

There is an acceptable real-world workaround that has been suggested by neverpanic in the mailing lists about a year ago. It involves having the relevant part of MacPorts use the system libcurl by default, but look for an use the alternate MacPorts libcurl first if it exists. You can find the mailing list discussion in MacPorts-dev a year ago or perhaps 18 months.

You would use the relevant dyld functions to look for and open the libcurl in ${prefix} if it exists, and then use the functions needed in there (as I recall they are slightly different), or if there is no libcurl in ${prefix} fall back to the one in the system and use that.

This solution is elegant, certainly correct, robust, and does not require macports to bundle libcurl (and thereby be responsible for security breaches in so doing). It is also time consuming, requires considerable programming expertise to accomplish, and will need to be maintained based on changes in libcurl in ${prefix} as it is updated.

As the /opt/bootstrap workaround works well enough for me, I won't be spending that time to do this. I doubt, in fact, that anyone with the needed skill set will, as -- well, life is short, time is limited, and this likely won't rise to the top of anyone's list, of those who can do it.

But you never know.

[catap (Kirill A. Korinsky)]

Meanwhile we may have a much bigger issue with GitHub. On macOS 12.7 when I run s_client -showcerts -servername codeload.github.com -connect codeload.github.com:443 </dev/null I do have an output:

CONNECTED(00000005)
depth=0 C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
verify error:num=18:self-signed certificate
verify return:1
depth=0 C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
verify return:1
---
Certificate chain
 0 s:C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
   i:C = US, ST = USA, L = New York, O = Optimization, OU = Optimization, CN = tomcat
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 30 08:53:43 2019 GMT; NotAfter: Aug 25 08:53:43 2039 GMT
...

=> self-signed certificate won't pass verification. So, output is below expected but quite scary:

√ ~ % /opt/local/bin/curl -I https://codeload.github.com/   
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
? ~ % /usr/bin/curl -I https://codeload.github.com/
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
? ~ %

[catap (Kirill A. Korinsky)]

The issue is my internet connection, an ISP decided that codeload.github.com is bad web-site and redirected any DNS request to a page which says that it contains viruses and shouldn't be used.

[kencu (Ken)]

just an observation…. the current curl port has become quite onerous with recursive deps on older systems lately.

I created a “minimalcurl” port that installs libcurl with vastly fewer deps to install into /opt/bootstrap to trim this up for my own use.

Perhaps if we asked nicely Ryan might feel such a subport of the curl port might be created.

[noloader (Jeffrey Walton)]

Replying to kencu:

just an observation…. the current curl port has become quite onerous with recursive deps on older systems lately.

I created a “minimalcurl” port that installs libcurl with vastly fewer deps to install into /opt/bootstrap to trim this up for my own use.

Perhaps if we asked nicely Ryan might feel such a subport of the curl port might be created.

Wget is much easier to bootstrap than cURL. Wget has two minimal dependencies. Also see comment:62.

[kencu (Ken)]

perhaps I'll try out your build_curl.sh script instead of using macports to install curl into /opt/bootstrap. I just trust the macports installation so far, as it's so reliable.

As you know, macports can't build against wget, so you need to install curl/libcurl somehow/somewhere to get macports fully functional, then build against it.

I wouldn't want to confuse people into thinking that installing wget will make macports work, as it won't.

Add: your Build-Scripts repo is very nicely done ... great work, Jeffrey!

[kencu (Ken)]

has duplicate #70127

[RJVB (René Bertin)]

Replying to kencu:

has duplicate #70127

Two relevant comments I posted there, which may rekindle the interest in a solution loading a libcurl dynamically, from a designated private location with fallback to "just use the system one" :
comment:ticket:70127:5
comment:ticket:70127:6

[kencu (Ken)]

see previous discussion of that idea here:

​https://lists.macports.org/pipermail/macports-dev/2020-November/042650.html

and in particular Clemens' detailed response outlining an approach forward here:

​https://lists.macports.org/pipermail/macports-dev/2020-November/042653.html

In that latter message are the issues outlining the problem with just simply "switching" the libcurl versions, and indicating how base would have to be rebuilt, at least in part, against the newer libcurl and the older libcurl. I won't try to re-explain it all as it is well outlined in that message.

[RJVB (René Bertin)]

Replying to kencu:

In that latter message are the issues outlining the problem with just simply "switching" the libcurl versions, and indicating how base would have to be rebuilt, at least in part, against the newer libcurl and the older libcurl. I won't try to re-explain it all as it is well outlined in that message.

But as you indicated in the other thread, if practise shows otherwise, maybe the approach can be reconsidered and/or simply tested. And in practise it does seem to be the case that you can "just" change libcurl for a newer version, as long as it's known or proven to be backwards ABI-compatible. As stated in the other ticket, I have yet to find a version that is *not*, even w.r.t. to "ancient" 2.7.x versions.

I really don't see why Pextlib would be any different from any of the other curl dependents which AFAICT do NOT get revbumped at every curl upgrade (except p5-www-curl for a reason I cannot determine). Given Apple's track record in the matter I don't expect even its latest OS has a newer curl version installed than the one in Macports?

And just to be clear, I'm not suggesting to load ${prefix}/lib/libcurl.?. It should be loaded from a "hidden" location, installed there in (semi)-permanent fashion and be a variant that does not use OpenSSL (because Pextlib links to the system libcrypto which can clash with the one in MacPorts). It could be installed as a special port (standalone or subport of port:curl) which installs only the actual library or installed/updated from a port-activate block of one of the suitable variants of port:curl itself. A special, dedicated port would make it easier to push updates of course, if and when it's justified to do so given how "base" uses curl.

Replying to ​RJVB:

Anyway, how much work and how tricky would it be to make curl.c load whatever it needs from libcurl dynamically? I'm not against digging into that but if there's already idea of what I'd be getting into I also not against knowing that beforehand :)

TBH, I find that a quite satisfying kind of lowlevel tinkering. More so if it's not just for me of course. Wading through old mailing lists exchanges is not so much my thing...

I've done similar things before, also in such a way that the actual behaviour can be determine at build time like with Qt's use of OpenSSL (i.e. linked-in or loaded dynamically). Looking at curl.c I'd be tempted to do the loading in lazy fashion, i.e. try to load a libcurl and then load curl_global_init() from it in CurlInit(), and load each symbol when needed, in the function that needs it (and probably move them all to CurlInit() or a dedicated function in a 2nd step).

[ryandesign (Ryan Carsten Schmidt)]

Replying to RJVB:

the other curl dependents which AFAICT do NOT get revbumped at every curl upgrade (except p5-www-curl for a reason I cannot determine).

WWW::Curl is a perl interface to curl. At the time that p5-www-curl is built, it scans the curl.h header and incorporates parts of it into the perl module. When the curl port is updated to a new version, p5-www-curl is revbumped in case the new curl version's curl.h is sufficiently different from the previous one to cause the perl module to change. You can easily download for example the 4.170.0_10 and 4.170.0_11 packages from ​https://packages.macports.org/p5.34-www-curl/ and extract and compare their contents to convince yourself that they are different.

Given Apple's track record in the matter I don't expect even its latest OS has a newer curl version installed than the one in Macports?

MacPorts usually has the latest version of curl within a few days of its release. macOS usually does not.

Wading through old mailing lists exchanges is not so much my thing...

Before embarking on changes to MacPorts, it would be wise to familiarize yourself with prior discussion regarding the changes you're planning to make. This is to avoid you spending a lot of time making changes if they have already been determined to be undesirable for some reason. We don't want you to waste your time developing changes if they would not be accepted. (I'm not saying that is necessarily the case here.)

[RJVB (René Bertin)]

Come to think of it ... before my current hack I used a DYLD_LIBRARY_PATH and/or DYLD_INSERT_LIBRARIES (and their Linux equivalents) to overload the system libcurl with the one from MacPorts. That was tricky because of how MacPorts controls the environment and because of the side-effects DYLD_LIBRARY_PATH can have.

I *think* a rewrite of curl.c can be avoided if we can simply try to dlopen an appropriate (tested etc) libcurl from the port driver command before it loads the Pextlib extension. It looks like this would require either a glue library that provides at least a stub Tcl package loader function (Curl_Init(), so that Tcl's load can be used) or else adding a Tcl extension (​https://wiki.tcl-lang.org/page/dlopen+extension). A glue library seems a little more appropriate.

[RJVB (René Bertin)]

Replying to ryandesign:

Replying to RJVB: You can easily download for example the 4.170.0_10 and 4.170.0_11 packages from ​https://packages.macports.org/p5.34-www-curl/ and extract and compare their contents to convince yourself that they are different.

I simply said I couldn't determine the reason why that port is special, and of course meant from the remarks in the portfile(s). But yes, it stands to reason that curl bindings for a scripting language need to be updated to include the latest functionality (I should probably have realised that was the likely case here).

We don't want you to waste your time developing changes if they would not be accepted. (I'm not saying that is necessarily the case here.)

Truth is that it wouldn't exactly be a waste of my time to do something I don't hate doing and of which I'll reap the benefits myself. I wouldn't be jammering about this topic here if that weren't the case!

[kencu (Ken)]

your enthusiasm is commendable.

I simply wanted to point out what was raised when this was suggested previously.

At the time, I thought such a fix as Clemens suggested might be well beyond my skill level and dropped the idea. Now, I think it might be within reach, though.

[RJVB (René Bertin)]

Replying to kencu:

your enthusiasm is commendable.

If I didn't know this wasn't pure altruism I'd be blushing ;)

Since I also like *easy* lowlevel hacks I started tinkering with the glue library idea, which allows me to test an unmodified Pextlib (= still linking to the system libcurl) with the tested private copy of libcurl-gnutls I've been using, through a runtime equivalent of DYLD_INSERT_LIBRARIES. Or rather, LD_PRELOAD because I'm on my Linux rig.

This code snippet gives promising behaviour when built into a shared library: (I've called my private libcurl copy pextcurl.so.4 there)

#include <dlfcn.h>

static void* curl_handle = 0;
static void* curl_global_init = 0;

#ifndef LIBCURL
#   define LIBCURL "/opt/local/libexec/macports/lib/pextlib1.0/pextcurl.so.4"
#endif

/* A stub Tcl package init function (hence the unused ptr argument that
 * would normally point to a Tcl_Interp instance).
 * Attempts to load libcurl from a designated location and fetch the
 * curl_global_init() function address from it. Returns 0 (aka TCL_OK)
 * on success, 1 (aka TCL_ERROR) on failure.
 */
int Curlglue_Init(void *dummy)
{
#pragma unused(dummy)
    if (curl_handle || curl_global_init) {
        return 1;
    }
    int flags = RTLD_NOW | RTLD_GLOBAL;
#ifdef RTLD_DEEPBIND
    // not really certain if we need this but it seems a good idea to use
    // when available to make a dynamic load work as much as possible as a
    // DYLD_INSERT_LIBRARIES (Darwin) or LD_PRELOAD (Linux).
    flags |= RTLD_DEEPBIND;
#endif
    if (!(curl_handle = dlopen(LIBCURL, flags))) {
        return 1;
    }
    if (!(curl_global_init = dlsym(curl_handle, "curl_global_init"))) {
        return 1;
    }
    // success
    return 0;
}

For now I tried this:

> port-tclsh
% load /path/to/libcurlglue.so
% package require Pextlib 1.0
1.0
% curl
wrong # args: should be "curl option ?arg ...?"
% ^Z
[4]  +   29607 Suspended                     port-tclsh
> ll /proc/29607/map_files/ | fgrep curl
29717233 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c7a4000-7f542c7b1000 -> /opt/local/libexec/macports/lib/pextlib1.0/pextcurl.so.4*
29717234 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c7b1000-7f542c826000 -> /opt/local/libexec/macports/lib/pextlib1.0/pextcurl.so.4*
29717235 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c826000-7f542c845000 -> /opt/local/libexec/macports/lib/pextlib1.0/pextcurl.so.4*
29717236 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c845000-7f542c848000 -> /opt/local/libexec/macports/lib/pextlib1.0/pextcurl.so.4*
29717237 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c848000-7f542c84b000 -> /opt/local/libexec/macports/lib/pextlib1.0/pextcurl.so.4*
29717243 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c8a7000-7f542c8a8000 -> /path/to/libcurlglue.so*
29717244 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c8a8000-7f542c8a9000 -> /path/to/libcurlglue.so*
29717245 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c8a9000-7f542c8aa000 -> /path/to/libcurlglue.so*
29717246 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c8aa000-7f542c8ab000 -> /path/to/libcurlglue.so*
29717247 0 lr-------- 1 bertin bertin 64 Jun  2 15:25 7f542c8ab000-7f542c8ac000 -> /path/to/libcurlglue.so*

No trace of the system libcurl (that would be /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0) to be seen, so a priori Pextlib is using my dedicated copy here.

Which means that port could do something like

if {[file exists /path/to/libcurlglue.so]} {
    load /path/to/libcurlglue.so
}
package require Pextlib 1.0

and we'd be done. Or more elegantly, this could be done from pextlib1.0/pkgIndex.tcl .

[kencu (Ken)]

Alternatively, I wonder if we might create a port that uses a slightly altered version of this:

​https://github.com/macports/macports-ports/blob/master/sysutils/MacPorts/Portfile

that would build against ${prefix}/lib/libcurl, to build a completely new version of Pextlib.dylib that is linked against ${prefix}/lib/libcurl.

Then we might rename and copy that one library into the base installation, and use it if it and libcurl exist.

It would be better if the libcurl we used was an isolated, custom version rather than macports main one, to avoid breakage, but ... every extra thing adds maintenance burden.

[kencu (Ken)]

or could we statically link against /opt/local/lib/libcurl.a to keep the installation robust between updates, I wonder...

[catap (Kirill A. Korinsky)]

Replying to kencu:

or could we statically link against /opt/local/lib/libcurl.a to keep the installation robust between updates, I wonder...

that mean that in case security update of curl a brand new version of MacPorts should be released.

Last week Nginx had an update which includes fixes for 4(!) CVE in HTTP/3.

[RJVB (René Bertin)]

Replying to kencu:

that would build against ${prefix}/lib/libcurl, to build a completely new version of Pextlib.dylib that is linked against ${prefix}/lib/libcurl.

Then we might rename and copy that one library into the base installation, and use it if it and libcurl exist.

That could also be a solution, but rather than building all of MacPorts "base" it would be better in that case to add a standalone build system to the Pextlib component (and make it a git submodule of the "base" repo so it can be checked out as a standalone component too)! You don't want to build against ${prefix}/lib/libcurl though, because of the potential of openssl version clashing. Though I suppose a port:pextlib (that's what we're talking about here) could simply be built against the same openssl version that libcurl may be using too (if installed +ssl).
EDIT: for some reason even I didn't consider the possibility of a port:pextlib but it does seem to be the cleanest solution. It and the dedicated curl port could even be subports.

Replying to catap:

Last week Nginx had an update which includes fixes for 4(!) CVE in HTTP/3.

I know many people consider such things like the holy grail, but just how urgent are such hotfixes for the things that libcurl is used for in MacPorts? That is, a priori only for vetted downloads with multiple-redundant checksums. Case in point: most users do this things via a (possibly "ancient") libcurl version...

[kencu (Ken)]

Replying to catap:

Replying to kencu:

or could we statically link against /opt/local/lib/libcurl.a to keep the installation robust between updates, I wonder...

that mean that in case security update of curl a brand new version of MacPorts should be released.

just the new "pextlib" port that doesn't yet exist.

The advantage of linking against libcurl.a statically is that it would hopefully be robust to breaking changes, like a botched curl upgrade for example, or somebody uninstalling curl without knowing any better.

[kencu (Ken)]

Replying to RJVB:

That could also be a solution, but rather than building all of MacPorts "base" it would be better in that case to add a standalone build system to the Pextlib component (and make it a git submodule of the "base" repo so it can be checked out as a standalone component too)!

Being inherently lazy, I was just thinking to let all of base build itself using it's existing infrastructure, and then pick out the modified pextlib part from it.

But if you know how to make the build more elegant by building only the needed part, then by all means, that would be better.

[RJVB (René Bertin)]

Replying to RJVB:

This code snippet gives promising behaviour when built into a shared library:

Well... bummer. After adapting this to Mac and testing it via a "preload" instruction in pextlib1.0/pkgIndex.tcl it turns out that 1) my dedicated libcurl is loaded (as expected) and 2) not used by Pextlib. Sic transit etc.

After verification, OS X 10.9.5 has curl 7.30, which fails to perform port livecheck curl. My old Ubuntu rig has curl 7.35 which *is* capable of doing that livecheck, so I haven't been able to confirm nor infirm whether the overload idea will work there. Maybe there are sites with stricter requirements where curl 7.35 will no suffice either?

Just for completeness sake I'd like to know if the idea was never going to work, or if it doesn't for the same reason that DYLD_INSERT_LIBRARIES usually doesn't work on Mac (i.e. 2-level namespaces).

[RJVB (René Bertin)]

Replying to kencu:

The advantage of linking against libcurl.a statically is that it would hopefully be robust to breaking changes

That's true of course, it's equivalent to installing a shared libcurl in a dedicated location in that aspect. However, you'd still want a libcurl.a from an appropriate build variant, which is tricky to ensure and cumbersome for the enduser IMHO. Plus, you'll need to link all the indirect dependencies yourself, the ones that would normally be linked to the shared libcurl. And purely on grounds of principle I dislike installing AND using a static library, because it's formally a waste of space.

Yes, port:pextlib doesn't exist yet, but neither does the port that will provide the dedicated libcurl...

Replying to kencu:

But if you know how to make the build more elegant by building only the needed part, then by all means, that would be better.

My git-fu isn't so good that I can tell you how best to split the pextlib subdir into its own repo (maintaining the commit history and all) and then make it a submodule, but I'm sure you can google that. Once that's done (and figured out how to create release tarballs that include the corresponding pextlib sources) I'd write a simple CMake file to build Pextlib as a standalone project. Partly that's because I'm lazy too and not interested in learning how to write a configure.ac file, partly because we won't have to touch the Makefile.in and partly because that means the MacPorts "base" configure script cannot possibly get confused by the presence of an autoconf "sub buildsystem".

From there it'd be like building any other port that installs to a custom location (if the CMake file doesn't simply specify it).

For the libcurl subport I'd indeed build all of curl if there is no existing build option to generate only the shared library, and of course using a hardcoded version of the appropriate build variants from port:curl. Use a destroot that copies the generated libcurl into place, or a post-destroot that prunes everything not needed if the curl install procedure involves relinking.

I'd keep the curl version 1 step behind current, or at least avoid .0 versions.

And of course, with port:pextlib and port:pextcurl (hey, I like the name!) in a single Portfile it will be easiest to clamp the version of either one as required by the target OS or architecture, should that turn out to be required. I suppose our resident barracuda may have some experience in that domain :)

[RJVB (René Bertin)]

Replying to RJVB:

My git-fu isn't so good that I can tell you how best to split the pextlib subdir into its own repo (maintaining the commit history and all) and then make it a submodule, but I'm sure you can google that.

Of course that's not even necessary, because

I'd write a simple CMake file to build Pextlib as a standalone project.

that CMake file in src/pextlib1.0 can also just be handed to CMake if the rest of the source tree is there too.

The question that hasn't been (re)addressed is how really old and crippled system libcurl versions are going to fetch the required files to bootstrap the full-fledged Pextlib (install the 2 ports and all their dependencies, possibly from source).

[RJVB (René Bertin)]

I really should drop this for today but I realise it could be very helpful for debugging purposes if there were a curl version command in Pextlib, returning a list of relevant strings describing the libcurl currently in use.

[noloader (Jeffrey Walton)]

Replying to catap:

Replying to kencu:

or could we statically link against /opt/local/lib/libcurl.a to keep the installation robust between updates, I wonder...

that mean that in case security update of curl a brand new version of MacPorts should be released.

In the case of static linking for bootstrapping, cURL should be built with minimum components -- HTTPS and CA-certificates, and not much more.

You may need Internationalized Domain Names (IDN) and IPv6, but you don't need http/3, quic, ftp, file, ldap, ldaps, rtsp, proxy, dict, telnet, tftp, pop3, imap, smb, smtp, gopher, cookies, and other extras.

Last week Nginx had an update which includes fixes for 4(!) CVE in HTTP/3.

In this specific case (thought experiment), https/3 would not trigger a cURL rebuild since it would not be included in the bootstrap version used during static linking. The full version of cURL would need rebuild, but not programs that use the bootstrap version.

Jeff

[catap (Kirill A. Korinsky)]

Replying to noloader:

Replying to catap:

Replying to kencu:

or could we statically link against /opt/local/lib/libcurl.a to keep the installation robust between updates, I wonder...

that mean that in case security update of curl a brand new version of MacPorts should be released.

In the case of static linking for bootstrapping, cURL should be built with minimum components -- HTTPS and CA-certificates, and not much more.

Which includes modern OpenSSL.

And this is quite deep hole.

[RJVB (René Bertin)]

adds a [curl version] command to Pextlib

[RJVB (René Bertin)]

Replying to RJVB:

it could be very helpful for debugging purposes if there were a curl version command in Pextlib, returning a list of relevant strings describing the libcurl currently in use.

Dog with a bone ... couldn't resist whipping something up after all. curl version returns a minimum list of relevant strings for the current task at hand. It's not supposed to be of use beyond providing some debugging information so I haven't bothered fetching more than the information that was available from the earliest curl versions onwards.

BTW, the attentive reader will have noticed that my own "dedicated libcurl" copy is still at v8.0.1, not at 8.7.1 as I must have claimed somewhere.

On Linux:

> port-tclsh
% package require Pextlib 1.0
1.0
% curl version
3 7.35.0 x86_64-pc-linux-gnu GnuTLS/2.12.23
^D
> port-tclsh
% load libcurlglue-git/libcurlglue.so
% package require Pextlib 1.0
1.0
% curl version
3 8.0.1 x86_64-pc-linux-gnu GnuTLS/3.7.10
^D

(On Mac the output from [curl version] indeed doesn't change after loading libcurlglue.so :( )

Replying to catap:

Which includes modern OpenSSL.

And this is quite deep hole.

Yes, and as I implied, one best avoided by using a different SSL backend like GnuTLS, AT LEAST if we are NOT talking about a Pextlib built as a port, so against dependencies from $prefix.

Pextlib also provides the functions for checksum calculation, rmd160 and sha256. Those are taken from OpenSSL's libcrypto as installed in the system, which could well be from OpenSSL 1.0 (and there's nothing wrong with that for calculating just checksums, AFAICT). Curl using the OpenSSL backend will link against the libssl from that "modern OpenSSL" and libssl is linked against the matching libcrypto. I have seen port crash when those two libcrypto versions get mixed in the same binary. Took me a long time to figure out why both got loaded, too. Once I did understand the fix was easy: build a libcurl that uses the GnuTLS backend (like the system one on Ubuntu, btw).

[kencu (Ken)]

the “minimalcurl” port I use to build base against, with as few deps as I could manage:

​https://github.com/kencu/macports-ports/blob/curl-minimal/net/minimalcurl/Portfile

[RJVB (René Bertin)]

seeing the size of that Portfile I am beginning to think that the most efficient implementation would be as a subport of port:curl, even if that means keeping a few patches for (say) an LTS version in a subdirectory. The curl port maintainer may have a different opinion of course.

• why curl 8.2.0 and not 8.7.1 for instance?
• I'm confused about what exactly is meant with "for bootstrapping" here. Is this port meant to be temporary, replaced by a less-minimal (lib)curl? Apparently so, because I don't see it being configured for installing in a dedicated location.
• That in turn means that Pextlib shouldn't be linking to it directly - are we back to considering an approach using dlopen/dlsym?
• My memo about avoiding the SSL backend has not made a lot of impact it seems. Again, that argument is moot if we can build a Pextlib copy against the same OpenSSL version as the one used by libcurl, or let Pextlib use an alternative to OpenSSL's libcrypto. If not, and if the bootstrap curl won't be built against the same major OpenSSL version as the system provides (seems counterproductive to me) then that bootstrap curl should use a different backend.

Is there any reason NOT to use the GnuTLS backend for the purpose at hand? It seems to be good enough to be used as the default in Ubuntu...

If SSL really is to be preferred, what other options are there to avoid different versions clashing, that don't involve a complete overhaul of the Pextlib parts that depend on libcrypto? What's the status of LibreSSL for instance? Now that there is infrastructure for using different, co-installed OpenSSL versions, could LibreSSL be somehow included or at least get a similar treatment (installation under $prefix/libexec) and would that be of any help here? IOW, will curl's SSL backend work with LibreSSL and can LibreSSL and OpenSSL be used in a single running application without biting each other?

[noloader (Jeffrey Walton)]

Replying to catap:

Replying to noloader:

Replying to catap:

Replying to kencu:

or could we statically link against /opt/local/lib/libcurl.a to keep the installation robust between updates, I wonder...

that mean that in case security update of curl a brand new version of MacPorts should be released.

In the case of static linking for bootstrapping, cURL should be built with minimum components -- HTTPS and CA-certificates, and not much more.

Which includes modern OpenSSL.

OpenSSL is easy to build because it has almost no dependencies.

Nowadays, for OpenSSL 3.x, the pain point is Perl. Perl is used to configure OpenSSL. Earlier versions of OpenSSL did not use Perl or require a modern version of Perl. OS X 10.4 and 10.5 does not have a new enough Perl, but that may not matter (if you are cross-compiling on a modern host).

And this is quite deep hole.

Here is my experience with this problem...

I have a set of scripts I use to build modern tools on old OSes, like Ubuntu 4, Fedora 1 and OS X 10.5. The scripts use a bootstrapped Wget instead of a bootstrapped cURL. (You only need one of them. Once you have a bootstrapped downloader, you can build everything else).

The bootstrap version of cURL or Wget has just enough features to download other packages. About all you need is a modern TLS library so you can do HTTPS. The requirements for TLS is TLS v1.0 or above, and a modern set of CA-Certificates.

The bootstrapped version of cURL or Wget uses static linking. You don't have to worry about API and ABI compatibility because nothing gets loaded at runtime. In fact, you can move the bootstrapped cURL or Wget to anywhere on the filesystem and it just works because the programs use static linking. There's no need to solve runtime linking problems because of static linking.

A bootstrapped version of cURL is what I am proposing for MacPorts. Once MacPorts has a downloader, it can download and build everything else. "Everything else" includes a fully featured version of OpenSSL and cURL.

[RJVB (René Bertin)]

Replying to noloader:

OpenSSL is easy to build because it has almost no dependencies.

True, contrary to GnuTLS...

The bootstrapped version of cURL or Wget uses static linking. You don't have to worry about API and ABI compatibility because nothing gets loaded at runtime.

This would be true if "base" invoked the actual external curl command. It doesn't; instead Pextlib provides a Tcl command curl (see my attached patch). I still have a patch for portfetch.tcl which adds a fallback to doing that if the built-in command fails (without crashing). That worked for fetching distfiles but not for other things like port livecheck. Hence my post-hoc relinking of Pextlib, and hence the rabbit hole of OpenSSL ABI clashing that I already went down once when port started crashing on me.

Re: dependencies: libressl has *no* dependencies at all. It also builds a lot faster than OpenSSL and is almost half the size, I just noticed. Plus, curl has detection of LibreSSL vs. OpenSSL in place when configuring the SSL backend.

I just added this to porrt:libressl:

subport ${name}-co {
    description-prepend Co-installable
    long_description-append \nThis port is co-installable with port:openssl

    # should libressl and libressl-co conflict with each other? I don't see
    # a good reason given that the purpose of libressl-co is to allow other
    # ports to use LibreSSL regardless of what SSL variant other ports use
    # by default.
    conflicts-replace \
                    openssl \
                    libressl
    set thePrefix   ${prefix}/libexec/libressl
    configure.pre_args \
                    --prefix=${thePrefix}
    # avoid having a conflicts_build with port:openssl:
    configure.ldflags-delete \
                    -L${prefix}/lib
    configure.ldflags-append \
                    -Wl,-rpath,${thePrefix}/lib
    if {${os.platform} eq "darwin"} {
        set shext "dylib"
    } else {
        set shext "so"
    }
    post-destroot {
        # also make certain no one can re-arrange the order of the linker command line
        # and -L and -l arguments cannot get separated causing the wrong libraries to be linked.
        foreach lib {crypto ssl tls} {
            reinplace -q "s|Libs: |Libs: -Wl,-rpath,\$\{libdir\} |g" \
                ${destroot}${thePrefix}/lib/pkgconfig/lib${lib}.pc
        }
    }
}

and this variant to my port:curl(@8.7.1):

    variant libre requires ssl description {Use the LibreSSL instead of OpenSSL} {
        PortGroup               conflicts_build 1.0
        depends_lib-replace     path:lib/libssl.dylib:openssl \
                                port:libressl-co
        configure.pkg_config_path-prepend \
                                ${prefix}/libexec/libressl/lib/pkgconfig
        depends_build-delete    port:pkgconfig
        depends_build-append    port:pkgconfig
        conflicts_build         openssl-dev
    }

Next step is to test if this gives a fully functional Pextlib without libcrypto ABI clashes.

EDIT: I've been able to fetch+checksum a distfile, so a priori LibreSSL is a suitable SSL provider for the ditto curl background. I've not been able to prevent a build conflict with port:openssl due to the curl build system but that won't be an issue for the majority of users who pull binary packages.

> port-tclsh
% package require Pextlib 1.0
1.0
% curl version
3 7.35.0 8.7.1 x86_64-pc-linux-gnu LibreSSL/3.9.1

[RJVB (René Bertin)]

Something I noticed in kencu's minimal curl port too: is there a use-case for building a universal MacPorts base installation?

I'm attaching an example minimal subport for port:curl that installs just $prefix/libexec/macports/lib/pextlib1.0/libcurl.dylib . Next step: figuring out a build system for a standalone Pextlib build that links against that library.

[RJVB (René Bertin)]

adds a MacPorts-libcurl subport to port:curl

[RJVB (René Bertin)]

I've tweaked that MacPorts-libcurl port a bit, dropping some dependencies but adding port:gss so as to provide about the same features as my system libcurl on 10.9.5 also provides. (Why does port:curl *not* use the gss & kerberos5 ports but rather the libraries from the system, btw? I haven't followed that here.

My port:MacPorts-libcurl (still at 8.7.1) : ​https://github.com/RJVB/macstrop/tree/master/net/curl

For completeness, my port:libressl-co: ​https://github.com/RJVB/macstrop/tree/master/security/libressl

First quick checks suggest that Pextlib just works when relinked against that new libcurl. I'll use the system as is for a while before I start looking at bundling Pextlib in a port.

[kencu (Ken)]

well the focus at the moment is not really who can come up with the leanest minimalcurl port, or even which ssl backend has the fewest deps.

it really is : can the idea work?

Can we build and insert a new Pextlib into base (this part should be easy enough) ?

Can we rewrite the TCL code in base to use the new library, if it exists, in a way that works reliably and Josh can accept (this is the million dollar question)?

[RJVB (René Bertin)]

Replying to kencu:

well the focus at the moment is not really who can come up with the leanest minimalcurl port, or even which ssl backend has the fewest deps.

True. I just jumped on it because I like the idea of using a backend that uses a library that's OpenSSL-compatible but otherwise independent of OpenSSL. The only nuisance is the fact that currently the entire port:openssl has to be deactivated to build curl against a co-installable LibreSSL. That's curl's fault, but given what port:openssl is nowadays it should be trivial to split it up into runtime and a builders'-only subports.

it really is : can the idea work?

I'm currently getting a longer-term answer if it can work with my usual hack of a relinked Pextlib.so but I already have sufficient practical proof that that approach worked with a previous version of libcurl (using GnuTLS).

Can we build and insert a new Pextlib into base (this part should be easy enough) ?

Yes (to the easy enough bit). Building the library could probably even be handled with a hand-written Makefile.something if a dependency on another build system is not desirable

Can we rewrite the TCL code in base to use the new library, if it exists, in a way that works reliably and Josh can accept (this is the million dollar question)?

From what I've seen this could be as simple as rewriting pkgIndex.tcl from an appropriate phase of the hypothetical port:MacPorts-pextlib. That's the actual loader for the package require Pextlib 1.0 syntax from what I understand. It is (presumably) generated when MacPorts is being built so rewriting the installed version should be safe enough. I've already done this to preload my curl glue library, and that worked so this should too:

# Tcl package index file, version 1.1
# This file is generated by the "pkg_mkIndex" command
# and sourced either when an application starts up or
# by a "package unknown" script.  It invokes the
# "package ifneeded" command to set up package-related
# information so that packages will be loaded automatically
# in response to "package require" commands.  When this
# script is sourced, the variable $dir must contain the
# full path name of this file's directory.

if {![file exists ${dir}/Pextlib.dylib]} {
      set PEXTLIB ${dir}/Pextlib.dylib.orig
} else {
      set PEXTLIB ${dir}/Pextlib.dylib
}
package ifneeded Pextlib 1.0 [list load [file join $dir ${PEXTLIB}]]

In case it isn't clear ;), this presumes that port:MacPorts-pextlib installs ${prefix}/libexec/macports/pextlib1.0/Pextlib.${os.shext}, renaming the original library first. The loader script then falls back to loading the renamed library if the new one isn't there, allowing to deactivate port:MacPorts-pextlib in case there's a reason to trouble-shoot it, or to maintain "base" functionality if ever an update glitches for some reason.

port:MacPorts-pextlib would have a pre-activate :

proc rewrite_pkgIndex {} {
# add a fallback to pkgIndex.tcl
}

pre-activate {
    # move the original library out of the way if not already done
    if {![file exists ${prefix}/libexec/macports/pextlib1.0/Pextlib.${os.shext}.orig]} {
        file rename ${prefix}/libexec/macports/pextlib1.0/Pextlib.${os.shext} \
            ${prefix}/libexec/macports/pextlib1.0/Pextlib.${os.shext}.orig
        rewrite_pkgIndex 
    } else {
        # just in case!
        file delete -force ${prefix}/libexec/macports/pextlib1.0/Pextlib.${os.shext}
    }
}

I use a pre-activate to avoid an activation conflict, and the presence or not of the backed-up library as a condition for rewriting the pkgIndex loader. This will break after updating "base"; it would be better to use the content of the loader script to determine whether a Pextlib.dylib.orig from a previous MacPorts version needs to be replaced with the current Pextlib.dylib and pkgIndex.tcl rewritten again. That was just a bit too cumbersome to type into this editor :)

(BTW: it would be really handy if an os.shext or similar variable were defined by "base"!)

[RJVB (René Bertin)]

In the the above we shouldn't have to worry about possible race conditions inside the running port instance because the old/orig Pextlib will already be in memory (and we're not on MSWin where the renaming might fail because of that).

[kencu (Ken)]

We won't be able to use a relinked Pextlib, so even if it somehow works today, it is unreliable and will never be accepted by Josh and Ryan long-term.

We might as well build the infrastructure to use the newer Pextlilb if it exists, and fall back on the default one built against the system curl if it doesn't.

That is the only thing that will be accepted into base/Macports as a long term solution.

[kencu (Ken)]

Your code to use the newer Pextlib looks very promising!

[RJVB (René Bertin)]

Replying to kencu:

We won't be able to use a relinked Pextlib, so even if it somehow works today, it is unreliable and will never be accepted by Josh and Ryan long-term.

Just see it as a cheap way of testing ideas, equivalent to but much simpler than rewriting curl.c to load Pextlib dynamically. I'll be throwing together a port:MacPorts-pexlib one of these days, but that also takes a bit more time than I currently want to commit (these things always take more time than planned).

And not just accessible to me, either - all of us here know how to "relink" a shared library with install_name_tool. Heck, even the pkgIndex rewrite can be tested with what we have right now! (Rename the original Pextlib, make a copy of it under the old name, relink it to e.g. my port:MacPorts-libcurl payload, and voilà...)

[RJVB (René Bertin)]

Replying to kencu:

Your code to use the newer Pextlib looks very promising!

I'd *almost* put my hand in the fire that it'll work ;)

Actually, I think we should even be able to rewrite the pkgIndex loader in such a way that it'll load the original Pextlib as a fallback not just for a missing new Pextlib but also if that new Pextlib fails to load. Say because of a missing or botched dependency.

And that should make it possible at least technically speaking to simply use the installed libcurl as a dependency of the new Pextlib. And port:MacPorts-pextlib would then get a revbump just like every other dependent of port:curl.

I don't know if I find that a reassuring idea, but it'll make things a lot easier if the fallback idea works. Only 1 new port to maintain, for instance, and building that new Pextlib will also be a little easier.

A hybrid scheme would also be possible, where port:MacPorts-pextlib builds against the installed libcurl but then relinks to a dedicated libcurl copy of that library. A MacPorts-libcurl subport which follows port:curl at a short distance would then do just that copying. That would give a little testing time after updating port:curl.

I kind of like that idea though not everyone might like the fact that it means the user gets direct control over what SSL backend is used by Pextlib.

[RJVB (René Bertin)]

Replying to RJVB:

I kind of like that idea though not everyone might like the fact that it means the user gets direct control over what SSL backend is used by Pextlib.

Plus copying a file from another port would violate the reproducible build principle. So this approach would only be acceptable at that level if it is possible to create and host distfiles that hold an appropriate libcurl for every possible OS version/architecture.

[RJVB (René Bertin)]

Replying to RJVB:

Replying to kencu:

Your code to use the newer Pextlib looks very promising!

I'd *almost* put my hand in the fire that it'll work ;)

No need. Tried a version of the pkgIndex mod yesterday (on Mac) and it works as expected.

What I didn't expect is that a Pextlib.dylib built from the current HEAD of the master branch (so against Tcl 8.6 headers) works with a "base" install that's maybe 2 years old now.

This does beg the question to the powers that be: to what extent would a hypothetical port:MacPorts-pextlib have to check for/protect against installing a new pextlib into an older "base" install?

I started writing a CMake file for building just Pextlib using -I$prefix/libexec/macports/include but it quickly became clear that I'd have to write an as-yet unknown number of macro definitions in a dedicated config.h file (or as -D arguments).

If this pextlib port has to be built against the installed Tcl and other MacPorts headers I will need to continue working on a dedicated buildsystem. If not, the easiest approach will be to run configure in the toplevel directory of MacPorts source tree, and invoke make (and make install) in the src/pextlib1.0 source directory. That should work.

Either way it would be helpful if someone who knows the pextlib sources really well could dress a list of the various HAVE_??? defines determined by the configure script and required by pextlib.

[kencu (Ken)]

I would humbly suggest we just forget about the fancy new build system, and just build all of base using it's existing build system (and using the base source files downloaded in every macports install, why not?), against a newer libcurl (statically linking if that is possible).

Then we just copy out Pextlib.dylib, rename it to something else, install it into the existing base install, and see what breaks.

This seems 100x easier, and has zero maintenance burden, at the cost of a trivial amount of building time.

[kencu (Ken)]

with two level namespaces, can we just rename Pextlib.dylib to Pextlib-updated-curl.dylib and expect it to work like that...?

Is there anything that links against Pextlib.dylib?

[RJVB (René Bertin)]

Replying to kencu:

with two level namespaces, can we just rename Pextlib.dylib to Pextlib-updated-curl.dylib and expect it to work like that...?

I'd say yes, given that loading Pextlib.dylib.orig worked. Semantically speaking I prefer to rename the old library and install the new one under the expected name. It's only a fallback after all.

Is there anything that links against Pextlib.dylib?

Not as far as I can tell. And if the trace functionality is standalone enough that it could be moved into the registry library Pextlib also wouldn't have to link against libcregistry. (Standalone as in not requiring other functions that should remain in Pextlib.)

Replying to kencu:

I would humbly suggest we just forget about the fancy new build system, and just build all of base using it's existing build system (and using the base source files downloaded in every macports install, why not?)[...]

This seems 100x easier, and has zero maintenance burden, at the cost of a trivial amount of building time.

As you may have guessed I'm also not looking for more work than required. If it's OK to build against (header)files from the source bundle rather than against the already install files than that makes things a lot easier, at the risk of making the Portfile a little more complex.

It's probably possible too to add a configure argument to make the build use headers from $prefix/libexec/macports/include instead of from vendor/vendor-destroot/$prefix/libexec/macports/include.

It won't be necessary to build all of the base install, I think, but I'll find out. We definitely shouldn't have to destroot the entire base install.

We will need to look into that libcregistry dependency though... can we build against the one in the source tree and then use the original one at runtime or will we actually need to install newer versions of both. The test I ran yesterday suggests we don't but I am not familiar with that part of Pextlib so don't really know how to test it.

(My experience with trace mode from long ago is that it always found something it wasn't happy with.)

[RJVB (René Bertin)]

I suppose the underlying issue we're dealing with here is also the reason why many native apps on legacy OS versions will no longer connect to websites and services?

There's also a way to fix that with a proxy server: ​https://jonathanalland.com/downloads/legacy-mac-proxy.dmg (from ​https://jonathanalland.com/old-osx-projects.html). That particular solution works at the system-wide level and would interfere with the functioning of MacPorts applications using up-to-date secure-connection libraries but it might be possible to use a dedicated proxy used only by MacPorts itself? Mostly thinking aloud: this might be usable as a bootstrapping solution?

And out of curiosity, I think we cannot build a sufficiently feature-full version of Apple's SecureTransport fw ourselves, but could we redistribute a sufficiently current version, (to be) relinked with libMacportsLegacySupport.dylib?

[RJVB (René Bertin)]

Some more thinking aloud, sorry.

If we want port:MacPorts-libcurl to follow the latest curl release at some distance or with some delay, we will in fact not be able to do without having both a runtime version (installing $prefix/libexec/macports/lib/pextlib1.0/libcurl-mp.dylib) and a "-dev" port that installs all the rest in an appropriately hidden location (also under libexec/macports?) but with its libcurl-mp.4.8.dylib or whatever turned into a symlink into the pextlib directory.

Rationale: people need to be to install the dependent port:MacPorts-pextlib from source even if port:curl is at a newer version, and for that we'll need to be able to declare a build dependency on a developers' install of port:MacPorts-libcurl.

This overhead can be prevented if we synchronise port:MacPorts-libcurl with port:curl (port:MacPorts-pextlib would depends_build port:curl and depends_lib port:MacPorts-libcurl) but that will mean testing if port:MacPorts-pextlib needs a revbump every time a curl update is pushed.

[neverpanic (Clemens Lang)]

This all sounds like great progress.

Note that I don't think this port should modify any of the files installed by base (things might end up breaking when base is updated, or the port is updated). Instead, just modify the pkgIndex.tcl shipped by base to opportunistically load the library from $prefix if it is installed.

That would also give us the option to not blow up things should Pextlib.dylib ever change ABI, because it could then introduce a version number (e.g., Pextlib.2.dylib), and installing that base update would automatically attempt to load Pextlib.2.dylib from $prefix, instead of loading the old one and potentially crashing.

As for whether the port-installed Pextlib.dylib needs to support old base versions, our general stance has always been that we expect users to update their base two weeks after a new release, as that's the cutoff date where we will start using new Portfile features not understood by older releases. As a consequence, I don't think you need any kind of compatibility for older base releases.

[RJVB (René Bertin)]

Replying to neverpanic:

Note that I don't think this port should modify any of the files installed by base (things might end up breaking when base is updated, or the port is updated). Instead, just modify the pkgIndex.tcl shipped by base to opportunistically load the library from $prefix if it is installed.

We could call the new Pextlib Pextlib.dylib.new or some variation on that, rather than renaming the old version but I don't think there's any difference from the point of view of pkgIndex.tcl. Just the content of a variable that changes; the old version will always be loaded as a fallback if the new version fails to load, in the scheme I propose.

The only difference where installing the new library under a new name might make a practical difference is that there would not be a dangling, old Pextlib.dylib.orig . That seems cleaner, but I am not yet convinced that it is the better option since I was planning to use its presence and timestamp as a state variable to determine whether pkgIndex.tcl needed to be rewritten. Maybe we can do without that, I'll see.

Either way, reinstalling "base" will also reinstall pkgIndex.tcl and thus undo the changes made by port:MacPorts-pextlib. I would expect that in case of an "upgrade reinstall" there would also be an accompanying upgrade to port:MacPorts-pextlib to restore use of port:MacPorts-libcurl. The user reinstalling "base" for some other reason is another can of worms; no clever ideas how to re-activate port:MacPorts-pextlib reliably and without modifying "base" come to mind. (We could install a copy of the modified pkgIndex.tcl and check for differences between the 2 loaders).

I do think it's better if the new version of the library is installed in the designated directory (libexec/macports/lib/pextlib1.0) rather than in a more common location.

That would also give us the option to not blow up things should Pextlib.dylib ever change ABI, because it could then introduce a version number (e.g., Pextlib.2.dylib), and installing that base update would automatically attempt to load Pextlib.2.dylib from $prefix, instead of loading the old one and potentially crashing.

I was already proposing to rewrite pkgIndex.tcl to load the new Pextlib and fall back to loading the original version in case of failure, so I think I already had this covered. Wouldn't a v2.0 ABI Pextlib install to lib/pextlib2.0/Pextlib.dylib, so that the Tcl syntax could also be changed to package require Pextlib 2.0? But ... changing the ABI of a plugin, isn't that a little bit farfetched or if not, shouldn't it already have changed a few releases back when vercmp was added? ;)

As a consequence, I don't think you need any kind of compatibility for older base releases.

Good, that makes things easier. Not that it seems to be a big problem anyway; the Pextlib from MacPorts 2.9.3 seems to work just fine in a 2.7.1.x install (but I haven't yet done any port install or de/activation).

I still think it could be wise to drop the dependency on the registry dylib by moving the function(s) that need it into that library itself. Would that be an option?

(PS: Trac is getting on my nerve when writing longer replies like this because it shifts the edit box out of view when refreshing the preview. Frequently but not systematically and I haven't yet figured out when it does and when it doesn't. Is there something I can do about that from my end?)

[RJVB (René Bertin)]

And here we are:

​https://github.com/RJVB/macstrop/tree/master/sysutils/MacPorts-pextlib

For now this has only an implementation for the MacPorts-pextlib-devel subport which follows the MacPorts master branch and has my personal patches for Linux and features - in addition to (and distinguishable from) a number of patchfiles that are required to get things to build against installed ports. I've also figured out how to build only in the few directories that are of direct interest without having to build the entire project. This was as simple as using port:sqlite3 as a build dependency rather than building the copy hidden in the vendored Tcl source tree. There are no hidden costs to it either; port:sqlite3 doesn't add new, indirect dependencies.

I tested (on Mac and Linux):

• port fetch
• port checksum
• a rebuild and reinstall (upgrade --force) of MacPorts-pextlib-devel itself

One of my feature patches reintroduces the Tcl binding for mkstemp that was dropped, so I could test with an older "base" install. Being able to do this is not a goal as discussed above but I think that the fact that everything I tested works normally indicates that the approach used here is reliable.

[RJVB (René Bertin)]

Replying to RJVB:

I tested (on Mac):

• upgrading port:libressl-co to 3.9.2
• upgrading port:curl (and thus port:MacPorts-libcurl which uses port:libressl-co) to 8.8.0

Both upgrades were applied without glitches using the installed libcurl-mp.dylib and Pextlib-mp.dylib.

As a reminder: port:MacPorts-libcurl is a subport of ​https://github.com/RJVB/macstrop/tree/master/net/curl .

NB @ryandesign : port:curl-ca-bundle has an upstream update according to liveck (2024-06-08).

[RJVB (René Bertin)]

Same tests performed on Linux too now, IMHO confirming the validity of the underlying ideas I used.

I could start working on filtering out my own tweaks and a non-devel port:MacPorts-pextlib to put in a PR (preferably a single one, possibly with several commits) but I'd like to have some feedback first. Esp. about the idea of using a new, co-installable libressl subport (IMHO not a bad option to have anyway) and port:MacPorts-libcurl.

Reading ​https://www.libressl.org/goals.html and comparing the port:openssl3 and port:libressl Portfiles only strengthens my idea that LibreSSL is a good choice for MacPorts's own SSL backend (one originating on another BSD derivative, even) but I have no hard arguments against OpenSSL if the new Pextlib will use the corresponding libcrypto anyway.

Regarding curl: I mentioned the possibility to keep port:MacPorts-libcurl on the previous rather than the current release. I didn't yet implement that to keep the subport as simple as possible so I'll have to make the necessary modifications if it is deemed wise to impose a time buffer for testing a new curl release "in the wild" and beyond a simple port -v test MacPorts-libcurl.

[RJVB (René Bertin)]

​https://github.com/RJVB/macstrop/tree/master/sysutils/MacPorts-pextlib

Main port implemented and a more robust pkgIndex wrapper implemented.

I'm not even sure why the previous version worked as apparently the script passed to package ifneeded cannot see variables defined in the same file.

But the main problem was that it was too easy to cripple an installation by deactivating port:MacPorts-libcurl or port:MacPorts-pextlib itself. Kind of dumb that I didn't realise that it wouldn't be possible to port activate MacPorts-libcurl (or pextlib) after doing that!

So the new pkgIndex wrapper installs an "ifneeded script" that catches any error in load Pextlib-mp.dylib and then falls back to load Pextlib.dylib. For some reason the $dir variable is not accessible/defined in this new implementation so I had to use the hardcoded install path but that should be a problem AFAICT.

[RJVB (René Bertin)]

As an upbeat to a PR for the new ports (and their proposed LibreSSL dependency):

​https://github.com/macports/macports-base/pull/339

[RJVB (René Bertin)]

I've updated (and somewhat improved) my port:MacPorts-libcurl and port:MacPorts-pexlib :

​https://github.com/RJVB/macstrop/tree/master/net/curl ​https://github.com/RJVB/macstrop/tree/master/sysutils/MacPorts-pextlib