openssh @9.3p1_0+kerberos5+xauth not compatible with openssl @3_10--segmentation fault

[EJFielding (Eric Fielding)]

I recently installed a port that apparently triggered an update to the openssl port so I have openssl @3_10 installed and active.

I tried to use the ssh program from the openssh port (version @9.2p1_0+kerberos5+xauth) and I got an error about a version mismatch:

OpenSSL version mismatch. Built against 30000080, you have 30100000

I then ran port selfupdate and port upgrade outdated and it installed a new version of the openssh @9.3p1_0+kerberos5+xauth.

The new version 9.3 now simply fails with a Segmentation fault.

I checked what ssh is linked to with:

otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libgssapi_krb5.2.2.dylib (compatibility version 2.0.0, current version 2.2.0)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

I am on macOS 13.3.1(a) and this is an Intel Mac.

[artkiver (グレェ)]

Replying to EJFielding:

I recently installed a port that apparently triggered an update to the openssl port so I have openssl @3_10 installed and active.

I tried to use the ssh program from the openssh port (version @9.2p1_0+kerberos5+xauth) and I got an error about a version mismatch:

OpenSSL version mismatch. Built against 30000080, you have 30100000

I then ran port selfupdate and port upgrade outdated and it installed a new version of the openssh @9.3p1_0+kerberos5+xauth.

The new version 9.3 now simply fails with a Segmentation fault.

I checked what ssh is linked to with:

otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libgssapi_krb5.2.2.dylib (compatibility version 2.0.0, current version 2.2.0)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

I am on macOS 13.3.1(a) and this is an Intel Mac.

Thank you for that.

I was previously aware that there were already compatibility issues with LibreSSL and Kerberos5 (also see: ​https://github.com/macports/macports-ports/pull/16927 which was an attempt to fix that) but as I do not currently have a kerberized environment with which to test that variant, I don't tend to do much aside from see if it builds.

Moreover, OpenBSD (the parent project for OpenSSH) defaults to LibreSSL, as does macOS, so compatibility with OpenSSL, while certainly not without merits, I don't think is as much of a priority for the upstream project.

Additionally, in the upstream project Kerberos is only in the -portable branch (albeit that is what we use in MacPorts, but my guess is it has less scrutiny by many of the OpenSSH developers as a result). If you scan ​https://www.openssh.com/releasenotes.html you'll note that the most recent time it is even mentioned was in the 7.4 portable release, and prior to that was 6.1 with the following:

"ssh(1): Don't link in the Kerberos libraries. They aren't necessary
   on the client, just on sshd(8). bz#2072"

I can look into this some, but I am afraid my preliminary gut feeling is I won't be able to do much.

Presumably it functions OK with removing the -kerberos5 variant? Though doubtlessly you want that for other reasons related to your environment.

[EJFielding (Eric Fielding)]

Thanks for checking on my report. I had not heard about LibreSSL before. I re-installed the port openssh @9.3p1_0+xauth without the Kerberos variant, and it still gives a segmentation fault on my Intel Mac. Curiously, the ssh program from the same openssh version with the +kerberos5 variant works fine on my new M1 Mac, also on macOS 13.3.1 (a).

Can I just install LibreSSL or do I have to uninstall openssl first? I had not heard of LibreSSL before.

[EJFielding (Eric Fielding)]

This is the library linkage on my Intel Mac:

otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

It is no longer linked to the kerberos library but still segFaults.

[artkiver (グレェ)]

Thanks for the update!

In looking at your earlier output a little more carefully, I also noticed this:

/opt/local/lib/libgssapi_krb5.2.2.dylib (compatibility version 2.0.0, current version 2.2.0)

Which, isn't reflected in your most recent otool output.

The GSSAPI/gsskex variant was deprecated a while ago (9.0p1 e.g. ​https://trac.macports.org/ticket/64982 ).

I also can't claim to have thoroughly tested the xauth variant, so while I might suggest removing that with -xauth, your mention that it is functioning OK on M1 but not on an Intel Mac has me scratching my head a bit. I do still have some Intel Macs, but in storage and running rather old macOS variants (I don't think they're even capable of running Ventura without using 3rd party tools such as ​https://dortania.github.io/OpenCore-Legacy-Patcher/).

LibreSSL has been used as the default macOS TLS library since at least High Sierra macOS 10.13 released circa 2017 if not earlier. However, I am reticent to recommend simply removing OpenSSL and installing LibreSSL for a MacPorts user as a fix all, because some ports have not been updated to accomodate the change gracefully. It has been a long and involved process, not just for MacPorts developers (who are, near as I can discern, myself included, all volunteers) but for other projects. FreeBSD has relatively simple toggles to rebuild /usr/src with LibreSSL, OpenBSD (obviously) defaulted to LibreSSL before any other OS, but some projects such as HardenBSD which had migrated to LibreSSL, eventually undid that due to challenges (though, AFAIK HardenBSD only has one "full time" developer and he isn't particularly well funded so that may not be a great example).

e.g. ​https://trac.macports.org/ticket/66740 is one ticket highlighting some similar challenges, though at least personally on some of my systems I have found that LibreSSL and OpenSSL can peacefully coexist, that is going to be pretty contingent upon what other dependencies and their TLS libraries may be.

Some things (rpki-client for example) have a bit more robust handling for either TLS library beyond the dylib and will attempt to determine which library is already installed and work with that and otherwise default to (at least in that instance, installing libressl as needed).

It is probably a bit of my bias showing, but since Apple already started shipping LibreSSL years ago and increasingly more projects have adapted compatibility with it where it wasn't as easy of a "drop in replacement" as maybe intended, but my hope is to continue to adapt software within MacPorts to be happy with LibreSSL.

But, I am a volunteer and have a LOT of other things on my plate.

Tomorrow it's possible I can pick up one of my older Intel based Macs from my storage unit and attempt to do some more thorough testing to see if I can replicate the segfault you have reported. However, I am already booked to work Saturday and Sunday, have at least one other obligation on Monday and am doing my damndest to try to fit in some time to apply to a job requisition which is hopefully still open as I am a far cry from working a normal "40 hour work week" am thousands of dollars in debt, and sleeping in a car, which is a hardship to understate it.

So, when I state that "I won't be able to do much" that is unpacking that a little bit, particularly as, and I am doing my best to phrase this as delicately as possible, but if you have never heard of LibreSSL until my reply, you're fundamentally (2023-2014[when LibreSSL was originally released] = 9 years behind OpenBSD's complete open source release announcement of the LibreSSL project and there's only so much advocacy I can do for libre/free open source software projects when I myself am a volunteer).

Rest assured, I have dragged older code kicking and screaming into the future than this, but even in those instances it was a laborious and time consuming process and more often than not done for free, or under the auspices of registered 501c3 non-profit organizations who, when they would invoice attorneys for my time would charge say, Fortune 25 (e.g. Alphabet Inc./Google) $300/hour for my time, but I was *lucky* if they paid me $15-$30/hour for my troubles out of that. Meanwhile, MacPorts, as far as I know, has even less funding than that and I have seen precisely nothing other than mutual good will from fellow MacPorts contributors, so it's been an endeavor in which I am happy to continue to co-create with other seemingly good natured folks.

Having written as much, one of the Trac tickets I recently closed was 9 years old too. So, if things progress slightly slower than desireable, there's only so much we can do collectively; particularly as at the moment: we are still shipping newer SSH and TLS libraries than Apple is with what was it, trillions of dollars in market valuation? I think we (humble brag) do a fair bit better than Homebrew with supporting MUCH older versions of OS X, to say nothing of Fink and other "also ran" port managers for macOS/OS X. All of that, and I do not even have commit access to the project! Yet, it has been my experience that those who do, have been great about providing worthwhile perspective and modifications and merging PRs quickly as merited.

My apologies for the TL;DR! I most certainly do want to help you, and others who may be experiencing similar issues. At best, I may be able to take a more in depth look at it tomorrow. Hopefully that is acceptable?

I am also more than willing to review patches if you have any to submit! Moreover, I only added myself as "maintainer" (in conjunction with "openmaintainer") because it was my experience that I had submitted several PRs updating OpenSSH and it had been a "nomaintainer" MacPort and it seemed that since I was presumably going to keep putting my time and attention torwards at least doing my best to keep it as up to date with the stock "vanilla" OpenSSH -portable branch, that I may as well make my efforts slightly known. By no means do I claim to be as well versed as djm, Niels Provos, Theo de Raadt or other key OpenSSH developers. I don't think I have, to date, even submitted a patch to upstream that has been merged, but maybe one will be necessary in this instance? Hard to say! I did help get a modification merged into the aforemention rpki-client somewhat recently, as well as some changes to the -portable branch of Got (GameofTrees) both of which are also OpenBSD affiliated projects.

Hopefully that sheds a little more light into my immediate takeaway (which I realize isn't too helpful with good advice) and my own work process and constraints?

In summation:

If you *want* to try to uninstall OpenSSL and install LibreSSL *maybe* that will function more harmoniously? But I would caution against that as a first approach, mostly because you may have other dependencies, and you also mentioned specific strife on an Intel Mac, but not an M1, so maybe it is more archictecture dependent?

Similarly removing the -xauth variant might simplify things (I am currently second guessing things at the moment, because I swear in the past I saw that installed by default even though I never explicitly specified it, and maybe I need to comb over the Portfile a bit more to have a cleaner default)?

Alternatively, MacPorts actually has multiple OpenSSL ports, the dylib attempts to be mindful of this too, but OpenSSL3 (​https://ports.macports.org/port/openssl3/) would probably be recommended as contrasted with openssl (which is more of a "catch all") or older "openssl10" (​https://ports.macports.org/port/openssl10/) or "openssl11" (​https://ports.macports.org/port/openssl11/) which are presumably there more for compatibility with even older more brittle MacPorts? That is just a guess though!

Oh, one more thing, you can run:

ssh -V

And it should display the OpenSSH version as well as what TLS library it is linked against.

e.g. locally:

% ssh -V
OpenSSH_9.3p1, LibreSSL 3.7.2

However, if SSH is continuing to segfault, the otool methodology you are using may be necessary.

In my experience with building OpenSSH against different TLS libraries, typically I will get more verbose error messages that display information related specifically to TLS libraries than segfaults, so your problem has me scratching my head in some other ways, but I have already probably written WAY TOO MUCH here and I don't want to cause reader fatigue, so much as be thorough in my thought process and documenting some ideas to try until I am in front of hardware that I can do more with hands on (lamentably, due to my life circumstances and employer dynamics, I do not currently have servers racked in datacenters or even VMs running vastly variant OS variants to test with at the moment, and my relationship with GitHub is about as far of a cry as "harmonious" as can be imagined and I am vehemently opposed to using them, not just "oh codespaces could solve this" ignorant, using anything tied to GitHub/M$/AWS and many other vendors, to me personally, is antipode to the reality I am striving to co-create and actually actively working against my ideals when I taint myself with it, but that's scraping the surface of ethics and morality and I am already way too deep in this reply).

Replying to EJFielding:

This is the library linkage on my Intel Mac:

otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

It is no longer linked to the kerberos library but still segFaults.

[bK4gYuRo]

I see the binary package for Ventura is built with openssl3:

mbp141:~ pike$ ssh -V
OpenSSH_9.3p1, OpenSSL 3.1.0 14 Mar 2023
mbp141:~ pike$ otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

and it works.

On the other hand, if I build openssh against LibreSSL, it segfaults, even in "ssh -V" invocation:

mbp141:~ pike$ otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libcrypto.50.dylib (compatibility version 51.0.0, current version 51.2.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
mbp141:~ pike$ ssh -V
Segmentation fault: 11

mbp141:~ pike$ port provides /opt/local/lib/libcrypto.50.dylib
/opt/local/lib/libcrypto.50.dylib is provided by: libressl

[EJFielding (Eric Fielding)]

Thanks @artkiver for the explanation. I can use the /usr/bin/ssh or a "conda" installed ssh for now. I see the Apple version in macOS 13.3.1 (a) is linked to LibreSSL, as you mentioned OpenSSH_9.0p1, LibreSSL 3.3.6 but is a slightly older version of OpenSSH.

[EJFielding (Eric Fielding)]

In case this is helpful, here is the information on the openssh that is installed on my M1 Mac, which works.

 % ssh -V
OpenSSH_9.3p1, OpenSSL 3.1.0 14 Mar 2023
 % otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libgssapi_krb5.2.2.dylib (compatibility version 2.0.0, current version 2.2.0)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

[artkiver (グレェ)]

Replying to EJFielding:

Thank you for the updates and your patience. I was able to retrieve an older laptop from storage last week but have only been able to sit down somewhere with decent WiFi today to take a deeper look at this.

That is also great that it seems as if you were at least able to get OpenSSH 9.3p1 with OpenSSL3 as a workaround.

As previously alluded to, I was able to retrieve one of my Intel based MacBook Pros from storage and have been seeing if I can replicate what you have encountered.

So far, OpenSSH seems to be functioning OK on my older Intel based MacBook with LibreSSL 3.7.2 though, so: I am presently unable to replicate your issue.

Some details below:

ssh -V
OpenSSH_9.3p1, LibreSSL 3.7.2

otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libcrypto.50.dylib (compatibility version 51.0.0, current version 51.2.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

uname -a
Darwin redacted.local 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64

port installed openssh
The following ports are currently installed:
  openssh @9.3p1_0 (active)

(You'll note: I am not using any variants such as xauth or kerberos5 though)

sh -c 'echo "macOS $(sw_vers -productVersion) $(sw_vers -buildVersion) $(uname -m)"; xcode=$(xcodebuild -version 2>/dev/null); if [ $? == 0 ]; then echo "$(echo "$xcode" | awk '\''NR==1{x=$0}END{print x" "$NF}'\'')"; else echo "Command Line Tools $(pkgutil --pkg-info=com.apple.pkg.CLTools_Executables | awk '\''/version:/ {print $2}'\'')"; fi' | tee /dev/tty | pbcopy
macOS 10.11.6 15G31 x86_64
Command Line Tools 8.2.0.0.1.1480973914

(The above is essentially what we use when submitting a PR to MacPorts on GitHub.)

From "About This Mac"

"OS X El Capitan Version 10.11.6 Processor 2.9GHz Intel Core i7"

While I was in the middle of running port upgrade outdated, I did, temporarily observe the following error:

ssh -V
dyld: Library not loaded: /opt/local/lib/libcrypto.49.dylib
  Referenced from: /opt/local/bin/ssh
  Reason: image not found
Trace/BPT trap: 5

However, once the "sudo port upgrade outdated"/"sudo port rev-upgrade" processes completed successfully even the above error was no longer able to be produced.

When this system started, it had the following OpenSSH and LibreSSL versions:

ssh -V
OpenSSH_9.0p1, LibreSSL 3.5.3

uninstalling/deactivating libressl and activating openssl3 and rebuilding as necessary, still seems to produce a functioning OpenSSH. Albeit, this is perhaps closer to what you are currently running?

ssh -V
OpenSSH_9.3p1, OpenSSL 3.1.0 14 Mar 2023

So, as it currently stands, I am wondering if the segfaults you have encountered have less to do with the CPU being Apple Silicon or Intel, and maybe more to do with the kerberos5 and xauth variants?

[artkiver (グレェ)]

I've done some additional testing as follows:

ssh -V
OpenSSH_9.3p1, OpenSSL 3.1.0 14 Mar 2023

port installed openssh
The following ports are currently installed:
  openssh @9.3p1_0+kerberos5 (active)

So, no segfault there either. As suspected, kerberos5 seems OK as long as OpenSSL(3) is the active TLS library.

Similarly, reinstalling OpenSSH with +xauth also does not segfault for me:

ssh -V
OpenSSH_9.3p1, OpenSSL 3.1.0 14 Mar 2023

port installed openssh
The following ports are currently installed:
  openssh @9.3p1_0+kerberos5+xauth (active)

My thought is that perhaps until ​https://github.com/macports/macports-ports/pull/16927 is closed/fixed that I should add a "conflicts libressl" parameter to the kerberos5 variant as that will, for the time being, not function unless using OpenSSL(3).

I'll prep a diff for that, but I'll wait to submit the PR until you have a chance to respond to see if that may be an acceptable alternative for the time being; if you can test on your Intel Mac and report back that would be ideal since I do not think this is related to CPU architecture so much as an issue between the active TLS library and kerberos5, which is already a known defect though the present patch/PR to restore kerberos5 compatibility with LibreSSL has not been functional in my testing, and presumably others' which is why it remains unmerged.

Thank you again for your patience! I hope this provides some additional insights from another vantage?

[artkiver (グレェ)]

I'm think that while it may probably still be wise to add a "conflicts libressl" stanza to the Kerberos5 variant (until such a time as the aforementioned PR has a resolution that is satisfactory) that this may actually be the same as what is being reported in this Trac issue: ​https://trac.macports.org/ticket/67539#comment:35

Subsequently, it would be great to have some additional insights into which version of clang/llvm/Xcode was being run on the systems where openssh was segfaulting.

Thanks!

[artkiver (グレェ)]

I've added a Portfile to this Trac issue with a potential fix: ​https://trac.macports.org/ticket/67539#comment:44

If anyone impacted by this issue could test that Portfile and let me know if it rectifies the issue for you, please let me know!

Thank you!

[artkiver (グレェ)]

I haven't seen a reply from anyone who originally reported this, so I am kind of going out on a limb, but I think perhaps this PR hopefully fixes this issue as well?

​https://github.com/macports/macports-ports/pull/19061

If not, please let me know and feel free to reopen this ticket.

[artkiver (グレェ)]

In a120a75a2a08eb553dfc6606771d3803c6c3bebd/macports-ports (master):

openssh: update compiler.blacklist-append to avoid segfault. (​https://github.com/macports/macports-ports/pull/19061)

• closes #67539
  • closes #67442

[janv37]

I'm running Ventura with macports and can confirm this fixes the segfault of ssh client linked against latest openssl 3.1.1. JV