openssl crashes svn on ppc due to asm code in OPENSSL_crypto207_probe ()

[kencu (Ken)]

Stumbled across this chasing down a crashing bug in svn on Tiger (same thing happens on Leopard PPC).

Host Name:      tigerg5
Date/Time:      2017-03-29 12:06:17.692 -0700
OS Version:     10.4.11 (Build 8S165)
Report Version: 4

Command: svn
Path:    /opt/local/bin/svn
Parent:  bash [2829]

Version: ??? (???)

PID:    2831
Thread: 0

Exception:  EXC_BAD_INSTRUCTION (0x0002)
Code[0]:    0x00000002
Code[1]:    0x01185920


Thread 0 Crashed:
0   libcrypto.1.0.0.dylib   	0x01185920 OPENSSL_crypto207_probe + 0
1   libcrypto.1.0.0.dylib   	0x01185cec OPENSSL_cpuid_setup + 360
2   libcrypto.1.0.0.dylib   	0x0122e608 OPENSSL_add_all_algorithms_noconf + 20
3   libsrp.plugin           	0x01646fa4 srp_client_plug_init + 88
4   libsasl2.dylib          	0x004b5384 sasl_client_add_plugin + 84
5   libsasl2.dylib          	0x004c1324 _sasl_load_plugins + 1296
6   libsasl2.dylib          	0x004b5810 sasl_client_init + 288
7   libsvn_ra_svn-1.0.dylib 	0x0049c2dc sasl_init_cb + 60
8   libsvn_subr-1.0.dylib   	0x0052a0b0 svn_atomic__init_once + 80
9   libsvn_ra-1.0.dylib     	0x000ce290 svn_ra_print_modules + 140
10  svn                     	0x00012a98 svn_cl__help + 448
11  svn                     	0x0001eca8 main + 544
12  svn                     	0x0000b09c _start + 760
13  svn                     	0x0000ada0 start + 48

Thread 0 crashed with PPC Thread State 64:
  srr0: 0x0000000001185920 srr1: 0x100000000208f030                        vrsave: 0x0000000000000000
    cr: 0x24008402          xer: 0x0000000000000000   lr: 0x0000000001185cec  ctr: 0x00000000900019c0
    r0: 0x0000000000000000   r1: 0x00000000bfffd620   r2: 0x0000000000000003   r3: 0x0000000000000000
    r4: 0x0000000000000000   r5: 0x0000000001185ce0   r6: 0x00000000012fac18   r7: 0x00000000000000ff
    r8: 0x00000000012fac04   r9: 0x00000000012f5b8c  r10: 0x000000000000000e  r11: 0x00000000012e744c
   r12: 0x00000000900019c0  r13: 0x00000000bfffe007  r14: 0x00000000bfffdc04  r15: 0x00000000bfffe001
   r16: 0x00000000bfffe004  r17: 0x00000000004c4d08  r18: 0x00000000bfffd7f8  r19: 0x00000000bfffe806
   r20: 0x00000000bfffd804  r21: 0x00000000004c533c  r22: 0x0000000000000015  r23: 0x0000000000000015
   r24: 0x00000000bfffdc04  r25: 0x00000000004c51d4  r26: 0x00000000bfffd778  r27: 0x00000000bfffd77c
   r28: 0x00000000bfffd780  r29: 0x00000000004c51d4  r30: 0x00000000fffff927  r31: 0x0000000001185b8c

Binary Images Description:
    0x1000 -    0x39fff svn 	/opt/local/bin/svn
   0x47000 -    0x4bfff libsvn_fs_util-1.0.dylib 	/opt/local/lib/libsvn_fs_util-1.0.dylib
   0x4e000 -    0x50fff libanonymous.plugin 	/opt/local/lib/sasl2/libanonymous.plugin
   0x61000 -    0xb3fff libsvn_client-1.0.dylib 	/opt/local/lib/libsvn_client-1.0.dylib
   0xc2000 -    0xd1fff libsvn_ra-1.0.dylib 	/opt/local/lib/libsvn_ra-1.0.dylib
   0xd6000 -    0xe8fff libsvn_diff-1.0.dylib 	/opt/local/lib/libsvn_diff-1.0.dylib
   0xee000 -    0xf8fff libsvn_ra_local-1.0.dylib 	/opt/local/lib/libsvn_ra_local-1.0.dylib
  0x205000 -   0x288fff libsvn_wc-1.0.dylib 	/opt/local/lib/libsvn_wc-1.0.dylib
  0x29a000 -   0x2c5fff libsvn_repos-1.0.dylib 	/opt/local/lib/libsvn_repos-1.0.dylib
  0x2cf000 -   0x2dafff libsvn_fs-1.0.dylib 	/opt/local/lib/libsvn_fs-1.0.dylib
  0x2df000 -   0x31bfff libsvn_fs_fs-1.0.dylib 	/opt/local/lib/libsvn_fs_fs-1.0.dylib
  0x328000 -   0x366fff libsvn_fs_x-1.0.dylib 	/opt/local/lib/libsvn_fs_x-1.0.dylib
  0x373000 -   0x396fff libsvn_fs_base-1.0.dylib 	/opt/local/lib/libsvn_fs_base-1.0.dylib
  0x39f000 -   0x47cfff libdb-4.6.dylib 	/opt/local/lib/db46/libdb-4.6.dylib
  0x490000 -   0x4a8fff libsvn_ra_svn-1.0.dylib 	/opt/local/lib/libsvn_ra_svn-1.0.dylib
  0x4b0000 -   0x4c4fff libsasl2.dylib 	/opt/local/lib/libsasl2.dylib
  0x4ca000 -   0x4eafff libsvn_ra_serf-1.0.dylib 	/opt/local/lib/libsvn_ra_serf-1.0.dylib
  0x4f7000 -   0x509fff libserf-1.dylib 	/opt/local/lib/libserf-1.dylib
  0x511000 -   0x520fff libsvn_delta-1.0.dylib 	/opt/local/lib/libsvn_delta-1.0.dylib
  0x526000 -   0x5affff libsvn_subr-1.0.dylib 	/opt/local/lib/libsvn_subr-1.0.dylib
  0x610000 -   0x634fff libexpat.1.dylib 	/opt/local/lib/libexpat.1.dylib
  0x63b000 -   0x64cfff libz.1.dylib 	/opt/local/lib/libz.1.dylib
  0x650000 -   0x725fff libsqlite3.0.dylib 	/opt/local/lib/libsqlite3.0.dylib
  0x73d000 -   0x75afff libmagic.1.dylib 	/opt/local/lib/libmagic.1.dylib
  0x760000 -   0x779fff libaprutil-1.0.dylib 	/opt/local/lib/libaprutil-1.0.dylib
  0x783000 -   0x7a1fff libapr-1.0.dylib 	/opt/local/lib/libapr-1.0.dylib
  0x7ac000 -   0x7b5fff libintl.8.dylib 	/opt/local/lib/libintl.8.dylib
  0x7b9000 -   0x7c3fff libgcc_s.1.dylib 	/opt/local/lib/libgcc/libgcc_s.1.dylib
  0x7ef000 -   0x7f1fff libcrammd5.plugin 	/opt/local/lib/sasl2/libcrammd5.plugin
  0x7f4000 -   0x7f9fff libgs2.plugin 	/opt/local/lib/sasl2/libgs2.plugin
 0x1008000 -  0x1100fff libiconv.2.dylib 	/opt/local/lib/libiconv.2.dylib
 0x110c000 -  0x115afff libssl.1.0.0.dylib 	/opt/local/lib/libssl.1.0.0.dylib
 0x1180000 -  0x12e6fff libcrypto.1.0.0.dylib 	/opt/local/lib/libcrypto.1.0.0.dylib
 0x13d5000 -  0x13dffff libdigestmd5.plugin 	/opt/local/lib/sasl2/libdigestmd5.plugin
 0x13e2000 -  0x13f3fff libk5crypto.3.1.dylib 	/opt/local/lib/libk5crypto.3.1.dylib
 0x13fa000 -  0x13fbfff libcom_err.1.1.dylib 	/opt/local/lib/libcom_err.1.1.dylib
 0x1505000 -  0x1535fff libgssapi_krb5.2.2.dylib 	/opt/local/lib/libgssapi_krb5.2.2.dylib
 0x1542000 -  0x15c7fff libkrb5.3.3.dylib 	/opt/local/lib/libkrb5.3.3.dylib
 0x15f9000 -  0x15fffff libkrb5support.1.1.dylib 	/opt/local/lib/libkrb5support.1.1.dylib
 0x1603000 -  0x1608fff libgssapiv2.plugin 	/opt/local/lib/sasl2/libgssapiv2.plugin
 0x160b000 -  0x160dfff liblogin.plugin 	/opt/local/lib/sasl2/liblogin.plugin
 0x1610000 -  0x1615fff libntlm.plugin 	/opt/local/lib/sasl2/libntlm.plugin
 0x1618000 -  0x1620fff libotp.plugin 	/opt/local/lib/sasl2/libotp.plugin
 0x1629000 -  0x162bfff libplain.plugin 	/opt/local/lib/sasl2/libplain.plugin
 0x162e000 -  0x1639fff libsasldb.plugin 	/opt/local/lib/sasl2/libsasldb.plugin
 0x163d000 -  0x1643fff libscram.plugin 	/opt/local/lib/sasl2/libscram.plugin
 0x1646000 -  0x164efff libsrp.plugin 	/opt/local/lib/sasl2/libsrp.plugin
0x8fe00000 - 0x8fe52fff dyld 46.16	/usr/lib/dyld
0x90000000 - 0x901bcfff libSystem.B.dylib 	/usr/lib/libSystem.B.dylib
0x90214000 - 0x90219fff libmathCommon.A.dylib 	/usr/lib/system/libmathCommon.A.dylib
0x907bb000 - 0x90895fff com.apple.CoreFoundation 6.4.11 (368.35)	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x908de000 - 0x908defff com.apple.CoreServices 10.4 (???)	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
0x908e0000 - 0x909e2fff libicucore.A.dylib 	/usr/lib/libicucore.A.dylib
0x90a3c000 - 0x90ac0fff libobjc.A.dylib 	/usr/lib/libobjc.A.dylib
0x90aea000 - 0x90b5cfff com.apple.framework.IOKit 1.4 (???)	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x90b72000 - 0x90b84fff libauto.dylib 	/usr/lib/libauto.dylib
0x90b8b000 - 0x90e62fff com.apple.CoreServices.CarbonCore 681.19 (681.21)	/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0x90ec8000 - 0x90f48fff com.apple.CoreServices.OSServices 4.1	/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
0x90f92000 - 0x90fd4fff com.apple.CFNetwork 4.0 (129.24)	/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
0x90fe9000 - 0x91001fff com.apple.WebServices 1.1.2 (1.1.0)	/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/WebServicesCore.framework/Versions/A/WebServicesCore
0x91011000 - 0x91092fff com.apple.SearchKit 1.0.8	/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
0x910d8000 - 0x91101fff com.apple.Metadata 10.4.4 (121.36)	/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
0x91112000 - 0x91120fff libz.1.dylib 	/usr/lib/libz.1.dylib
0x91123000 - 0x912defff com.apple.security 4.6 (29770)	/System/Library/Frameworks/Security.framework/Versions/A/Security
0x913dd000 - 0x913e6fff com.apple.DiskArbitration 2.1.2	/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
0x913ed000 - 0x913f5fff libbsm.dylib 	/usr/lib/libbsm.dylib
0x913f9000 - 0x91421fff com.apple.SystemConfiguration 1.8.3	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
0x91434000 - 0x9143ffff libgcc_s.1.dylib 	/usr/lib/libgcc_s.1.dylib
0x9506a000 - 0x95087fff libresolv.9.dylib 	/usr/lib/libresolv.9.dylib
0x969a2000 - 0x969b5fff libpam.1.dylib 	/usr/lib/libpam.1.dylib

The assembly code for this call is in openssl/ppccpuil.pl. It looks to me like it might be ppc64 code.

Disabling assembly code by adding this in the os.major <= 9 block:

append merger_configure_args(ppc) { no-asm}

fixes the svn crash.

I'm not certain this needs to be fixed generally. I haven't seen this crash with anything else, so perhaps leaving the asm code in there is the better option generally for performance for web servers, etc.

[kencu (Ken)]

sorry, that asm in in openssl/crypto/ppccpuid.pl.

<​https://github.com/openssl/openssl/blob/master/crypto/ppccpuid.pl>

[kencu (Ken)]

I wonder if this asm code is Power8-specific: <​https://www.ibm.com/developerworks/library/se-power8-in-core-cryptography/index.html>

[neverpanic (Clemens Lang)]

Can you provide a patch that makes the proposed change, ideally in a pull request against ​https://github.com/macports/macports-ports?

I don't object to the change, but cannot test on old PPC platforms and would rather not spend too much time with them.

[kencu (Ken)]

I'm just going to ask <​https://github.com/openssl/openssl/issues/3096> if there is a way to include the "good" PPC asm and leave out the "bad" Power8 specific parts.

[kencu (Ken)]

There appear to be differences between MacOS versions - 10.5PPC runs the crypto207 probe code silently, and the bad instruction error is only noted when in the debugger, which is standard expected behaviour when probing for the asm instruction.

on 10.4PPC, it works differently. the crypto207 probe code pops up a dialog box indicating an execution error, and writes a backtrace and debugging info into CrashReporter for a 'bad instruction' error every single time. I suspect this behaviour is not correct based on this discussion, and it was "fixed" in OS 10.5.

So it looks like this "error" can either just be ignored (and I can wipe out all the erroneous error logs on 10.4 from time to time) or I could patch the probe out on 10.4 if I really wanted to. 10.4 is ancient, and not many of us left even try to run it, so not worth more time from you I would say. Thanks for helping me understand what's going on.

I think this ticket can be closed, and the ppc asm code just left in place for performance reasons.

[neverpanic (Clemens Lang)]

OK, closing as requested.

[kencu (Ken)]

Ultimately I think this behaviour on Tiger is due to some inconsistency in the signal handling -- Leopard correctly handles SIGILL during the probes in openssl to not cause a crash. On Tiger however it goes to the default behavour and writes a full crashlog every time the probe fails on the illegal instruction, which is every time any program uses openssl. sigaction does not appear to be picking up the requested change on Tiger.

I thought it might just be a POSIX guard thing in signal.h (these errors are common on Tiger), but the first attempt at that didn't fix it -- so I just #ifdef'd the crashing call out, and now it works correctly.

[kencu (Ken)]

small patch for this error for anyone on TIger who is interested

[kencu (Ken)]

Oh, hey look -- the openssl people fixed it even better than I did, in trunk. What nice folks!

<​https://github.com/openssl/openssl/commit/0bd93bbe4ae60e5f318b298bfe617e468a7b71d0>