transmission @3.00: segfault involving libcrypto.3.dylib that is not present in prior builds or official builds

[kakuhen]

When using Transmission 3.00 compiled with the latest dependents from MacPorts, the program now segfaults on startup, producing stack traces such as the following.

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [2546]

...

Thread 2 Crashed:
0   libcrypto.3.dylib             	0x000000010ceeaa6c EVP_CIPHER_CTX_set_key_length + 36
1   org.m0k.transmission          	0x000000010cb80f6a 0x10cae2000 + 651114
2   org.m0k.transmission          	0x000000010cb6c4f8 0x10cae2000 + 566520
3   org.m0k.transmission          	0x000000010cb6c58a 0x10cae2000 + 566666
4   org.m0k.transmission          	0x000000010cb6d4f0 0x10cae2000 + 570608

The exact version of Transmission on MacPorts is "Transmission/3.00 (bb6b5a0)". The release on GitHub reports "Transmission/3.00 (f4489c9)".

I suspect the cause to be some API or ABI breakage, but no linker errors are reported post-destroot. Here is the output of otool(1) on the MacPorts build.

% otool -L /Applications/MacPorts/Transmission.app/Contents/MacOS/Transmission 
/Applications/MacPorts/Transmission.app/Contents/MacOS/Transmission:
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	@rpath/Sparkle.framework/Versions/A/Sparkle (compatibility version 1.6.0, current version 1.23.0)
	/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/opt/local/libexec/openssl3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libcurl.4.dylib (compatibility version 13.0.0, current version 13.0.0)
	/opt/local/libexec/openssl3/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59306.140.5)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 162.0.0)
	/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1677.104.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1894.60.100)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1677.104.0)
	/System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage (compatibility version 1.0.1, current version 5.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1069.24.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)

Then for the official GitHub release, which does not have this problem.

% otool -L /Applications/Transmission.app/Contents/MacOS/Transmission
/Applications/Transmission.app/Contents/MacOS/Transmission:
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	@rpath/Sparkle.framework/Versions/A/Sparkle (compatibility version 1.6.0, current version 1.23.0)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
	/usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59306.101.1)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 162.0.0)
	/System/Library/Frameworks/Quartz.framework/Versions/A/Quartz (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1675.129.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1894.40.150)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1675.129.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1069.22.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)

Additional Information

macOS 10.15.7 19H2026 x86_64

Xcode 12.4 12D4e

[Gandoon (Erik Hedlund)]

That explains a lot. When it kept failing for me, despite it being the most recent available on MacPorts, I pulled the GitHub origin. When I built and tested that I had no problems. But of course, since I built it with Xcode so I could properly sign it, I may have used some "Fruit company" provided dependencies, rather than MacPorts ones, and that is why it worked smoothly. I maybe should check that out of interest one of these days.

In the end I actually also got a slightly more modern version. The Transmission version numbers are up in the 4.n series now.

[ryandesign (Ryan Carsten Schmidt)]

See #67150 for another openssl3-related crash just reported in another port. Maybe the recent openssl 3.1.0 update or the even more recent security update are not as backward-compatible as we thought.

[neverpanic (Clemens Lang)]

Does this problem still occur if Transmission is recompiled against OpenSSL 3.1?

[ryandesign (Ryan Carsten Schmidt)]

Replying to neverpanic:

Does this problem still occur if Transmission is recompiled against OpenSSL 3.1?

In other words, try:

sudo port -ns upgrade --force transmission

[ryandesign (Ryan Carsten Schmidt)]

In #67150 the fix turned out to be:

sudo port upgrade --enforce-variants openssl3 +legacy

So you could try that too. Let us know if that fixes it.

[MichaelJFischer]

The suggestion from comment:7 fixes a problem I have been having with subversion (svn) using sasl2 authentication. It seems that OpenSSL 3.1 is not downward compatible with OpenSSL 2, and it breaks sasl authentication in subversion. The +legacy variant fixes the problem.

[neverpanic (Clemens Lang)]

Replying to MichaelJFischer:

The suggestion from comment:7 fixes a problem I have been having with subversion (svn) using sasl2 authentication. It seems that OpenSSL 3.1 is not downward compatible with OpenSSL 2, and it breaks sasl authentication in subversion. The +legacy variant fixes the problem.

That very likely happens because your SVN server uses RC4 in SASL authentication, which OpenSSL 3 only contains in the legacy provider: ​https://www.openssl.org/docs/man3.0/man7/EVP_CIPHER-RC4.html

RC4 is insecure and broken. You should ask your server administrator to switch to a newer algorithm.

[neverpanic (Clemens Lang)]

I compiled Transmission 4.0.3 (6b0e49bbb2), which is the current version in MacPorts, and cannot reproduce this. I'm assuming this is now fixed, whatever the issue was. Please re-open if you can reproduce.

[neverpanic (Clemens Lang)]

See also ​https://www.rfc-editor.org/rfc/rfc6331.html, which marks the Digest-MD5 SASL authentication mechanism that uses RC4 as historic due to its security problems.