openssl-1.0.1e broken with key_from_blob error messages

[davidfavor (David Favor)]

openssl-1.0.1e failing.

scp -l 5000 -P 8933 -i /Users/david/.ssh/dfavor.dsa xhtml11.conf.patch root@68.233.248.187:.
buffer_get_bignum2_ret: BN_bin2bn failed
key_from_blob: can't read ecdsa key point
key_read: key_from_blob AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKjE4pdShkPwQMxc83R4rcIlwC6c66gcurdiyZtWiTAKZFhy45qKmTa/OEWMotNz/S6Fw7ktQHCa7rQNYwSx7Hs=
 failed
Connection closed by 68.233.248.187
lost connection

Workaround is to rollback to openssl-1.0.1c as openssl-1.0.1d fails in other ways.

[davidfavor (David Favor)]

Odd... now this is showing up in openssl-1.0.1c too. Trying to get a version of openssl installed that works.

Will update this ticket if I make progress.

[davidfavor (David Favor)]

Other error reports suggest this can be fixed by switching from RSA keys to DSA.

Tried this and same error.

Also, ssh works with 1.0.1c + 1.0.1d + 1.0.1e and scp seems to fail with them all.

Server end is running this version of openssl... OpenSSL 1.0.1c 10 May 2012

Still trying to find a solution.

[davidfavor (David Favor)]

Debugging the session shows...

Client end...

 scp -l 5000 -v -P 9999 -i /Users/david/.ssh/dfavor.dsa xhtml11.conf.patch root@68.233.248.187:.
Executing: program /opt/local/bin/ssh host 68.233.248.187, user root, command scp -v -t .
OpenSSH_6.1p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /opt/local/etc/ssh/ssh_config
debug1: Connecting to 68.233.248.187 [68.233.248.187] port 9999.
debug1: Connection established.
debug1: identity file /Users/david/.ssh/dfavor.dsa type 2
debug1: identity file /Users/david/.ssh/dfavor.dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-3ubuntu1
debug1: match: OpenSSH_6.0p1 Debian-3ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 68.233.248.187
lost connection

Server end...

 /usr/sbin/sshd -p 9999 -d
debug1: sshd version OpenSSH_6.0p1 Debian-3ubuntu1
debug1: read PEM private key done: type RSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: private host key: #1 type 2 DSA
debug1: read PEM private key done: type ECDSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.ECDSA-256
debug1: Checking blacklist file /etc/ssh/blacklist.ECDSA-256
debug1: private host key: #2 type 3 ECDSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='9999'
debug1: rexec_argv[3]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 9999 on 0.0.0.0.
Server listening on 0.0.0.0 port 9999.
debug1: Bind to port 9999 on ::.
Server listening on :: port 9999.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 173.174.85.112 port 48690
debug1: Client protocol version 2.0; client software version OpenSSH_6.1
debug1: match: OpenSSH_6.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-3ubuntu1
debug1: permanently_set_uid: 105/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr hmac-md5 none [preauth]
debug1: kex: server->client aes128-ctr hmac-md5 none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
buffer_get_bignum2_ret: BN_bin2bn failed [preauth]
buffer_get_ecpoint: buffer error [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 29221

[davidfavor (David Favor)]

Work around for this problem is to use the native /usr/bin/ssh + /usr/bin/scp programs, rather than Macports versions.

The native /usr/bin/ssh + /usr/bin/scp work. Macports /opt/local/bin/ssh + /opt/local/bin/scp fail.

Macports fails with both 1.0.1c and 1.0.1e versions of openssl.

Here's the version info.

David-Favor-iMac> port -v installed openssl
The following ports are currently installed:
  openssl @1.0.1c_0+rfc3779 (active) platform='darwin 12' archs='x86_64'
  openssl @1.0.1e_0+rfc3779 platform='darwin 12' archs='x86_64'
David-Favor-iMac> /usr/bin/ssh -V
OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
David-Favor-iMac> /opt/local/bin/ssh -V
OpenSSH_6.1p1, OpenSSL 1.0.1c 10 May 2012

The debug conversation is very different for /usr/bin/ssh + /opt/local/bin/ssh.

Here's the debug conversation from /usr/bin/ssh...

/usr/bin/ssh -v -p 8933 -i /Users/david/.ssh/dfavor.dsa root@net1.bizcooker.com
OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 50: Applying options for *
debug1: Connecting to net1.bizcooker.com [68.233.248.187] port 8933.
debug1: Connection established.
debug1: identity file /Users/david/.ssh/dfavor.dsa type 2
debug1: identity file /Users/david/.ssh/dfavor.dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-3ubuntu1
debug1: match: OpenSSH_6.0p1 Debian-3ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA c3:05:17:fa:53:5a:31:88:9a:f3:ff:e9:55:9d:81:87
debug1: Host '[net1.bizcooker.com]:8933' is known and matches the RSA host key.
debug1: Found key in /Users/david/.ssh/known_hosts:11
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /Users/david/.ssh/dfavor.dsa
debug1: Server accepts key: pkalg ssh-dss blen 817
debug1: Authentication succeeded (publickey).
Authenticated to net1.bizcooker.com ([68.233.248.187]:8933).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Requesting authentication agent forwarding.
Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-23-generic x86_64)

Here's the /opt/local/bin/ssh debug conversation...

/opt/local/bin/ssh -v -p 8933 -i /Users/david/.ssh/dfavor.dsa root@net1.bizcooker.com
OpenSSH_6.1p1, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /opt/local/etc/ssh/ssh_config
debug1: Connecting to net1.bizcooker.com [68.233.248.187] port 8933.
debug1: Connection established.
debug1: identity file /Users/david/.ssh/dfavor.dsa type 2
debug1: identity file /Users/david/.ssh/dfavor.dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-3ubuntu1
debug1: match: OpenSSH_6.0p1 Debian-3ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.1
buffer_get_bignum2_ret: BN_bin2bn failed
key_from_blob: can't read ecdsa key point
key_read: key_from_blob AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKjE4pdShkPwQMxc83R4rcIlwC6c66gcurdiyZtWiTAKZFhy45qKmTa/OEWMotNz/S6Fw7ktQHCa7rQNYwSx7Hs=
 failed
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 68.233.248.187

Unsure what to do next.

Suggestions for getting Macports versions of ssh + scp to have similar conversation style, so they work?

[jmroot (Joshua Root)]

So given that you're seeing the problem with every version of openssl you tried, couldn't the problem just as easily be in the openssh port?

[aaron@…]

See also #38015 and #38017. This is fairly severe and renders openssh and openssl entirely unusable with RSA keys. A concise resolution would be appreciated.

[davidfavor (David Favor)]

Replying to jmr@…:

So given that you're seeing the problem with every version of openssl you tried, couldn't the problem just as easily be in the openssh port?

Unfortunately this is correct. I should have stated this above.

The only way I can fix this right now is to use the Apple shipped ssh + scp, as the Macports versions fail.

I also tried building an openssh variant (+ldns) which forces a true configure + make, rather than downloading a binary.

This failed too.

It's a bit odd no one has reported this because looking at a random openssh mirror (​http://mirror.esc7.net/pub/OpenBSD/OpenSSH/portable/) shows that OpenSSH-6.1p1 released on 29-Aug-2012 so there should have been bug reports generated against Macports ssh + scp long before now.

Only thing I can determine is something has changed which is escaping me.

I'm running openssl-1.0.1c which still gives the same problem, so neither the 1.0.1d or 1.0.1e openssl releases appear to be the culprit.

Now that I think about it, maybe it's some other openssh library...

David-Favor-iMac> /opt/local/bin/ssh -V
OpenSSH_6.1p1, OpenSSL 1.0.1c 10 May 2012
David-Favor-iMac> otool -L /opt/local/bin/ssh
/opt/local/bin/ssh:
	/opt/local/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.7)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/local/lib/libgssapi_krb5.2.2.dylib (compatibility version 2.0.0, current version 2.2.0)
	/opt/local/lib/libkrb5.3.3.dylib (compatibility version 3.0.0, current version 3.3.0)
	/opt/local/lib/libk5crypto.3.1.dylib (compatibility version 3.0.0, current version 3.1.0)
	/opt/local/lib/libcom_err.1.1.dylib (compatibility version 1.0.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Maybe one of the other libraries change underneath openssh.

I'll hack on /opt/local/etc/ssh/ssh_config and see if I can come up with some config that fixes the problem.

No great hope for this though, as this code is completely new to me.

[davidfavor (David Favor)]

Replying to aaron@…:

See also #38015 and #38017. This is fairly severe and renders openssh and openssl entirely unusable with RSA keys. A concise resolution would be appreciated.

Per ticket #38015 I ran port test openssl and got a failure.

Failure log posted in the #38015 ticket.

[davidfavor (David Favor)]

Per ticket #38015, rebuilding openssl-1.0.1e with no-asm creates a working Macports ssh + scp.

This ticket can be closed.

[ruben+macports@…]

Cc Me!