Git clone fails - SSL certificate problem: Couldn't understand the server certificate format

[breun (Nils Breunese)]

I'm running OS X 10.11.3 and have both Apple's /usr/bin/git (version 2.5.4) and MacPort's /opt/local/bin/git (version 2.7.0) installed.

I can use both git clients to clone the ansible git repository:

$ /usr/bin/git clone https://github.com/ansible/ansible.git
Cloning into 'ansible'...
remote: Counting objects: 107584, done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 107584 (delta 35), reused 0 (delta 0), pack-reused 107506
Receiving objects: 100% (107584/107584), 37.61 MiB | 935.00 KiB/s, done.
Resolving deltas: 100% (64374/64374), done.
Checking connectivity... done.

$ /opt/local/bin/git clone https://github.com/ansible/ansible.git
Cloning into 'ansible'...
remote: Counting objects: 107584, done.
remote: Compressing objects: 100% (70/70), done.
remote: Total 107584 (delta 35), reused 0 (delta 0), pack-reused 107506
Receiving objects: 100% (107584/107584), 37.61 MiB | 591.00 KiB/s, done.
Resolving deltas: 100% (64374/64374), done.
Checking connectivity... done.

However, when I try to install the ansible port I get the following error:

:info:fetch fatal: unable to access 'https://github.com/ansible/ansible.git/': SSL certificate problem: Couldn't understand the server certificate format
:info:fetch Command failed: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 2>&1
:info:fetch Exit code: 128
:error:fetch org.macports.fetch for port ansible returned: Git clone failed
:debug:fetch Error code: NONE
:debug:fetch Backtrace: Git clone failed
    while executing
"gitfetch"
    (procedure "portfetch::fetch_main" line 14)
    invoked from within
"portfetch::fetch_main org.macports.fetch"
    ("eval" body line 1)
    invoked from within
"eval $procedure $targetname"

I have attached the full log. How can I proceed to solve this issue?

[breun (Nils Breunese)]

(Initially created as #50467 as an issue for the ansible port, but the maintainers think the issue is not directly related to the port.)

[danielluke (Daniel J. Luke)]

just as a data point, sudo port -dv fetch ansible works for me on 10.11.3:

DEBUG: Executing org.macports.fetch (ansible)
DEBUG: Executing: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_Volumes_Extra_dluke_Projects_macports_dports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 2>&1
DEBUG: Executing GIT_DIR=/opt/local/var/macports/build/_Volumes_Extra_dluke_Projects_macports_dports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1/.git GIT_WORK_TREE=/opt/local/var/macports/build/_Volumes_Extra_dluke_Projects_macports_dports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 /usr/bin/git checkout -q v2.0.0.2-1 2>&1
DEBUG: Executing proc-post-org.macports.fetch-fetch-0
Submodule 'lib/ansible/modules/core' (https://github.com/ansible/ansible-modules-core) registered for path 'lib/ansible/modules/core'
Submodule 'lib/ansible/modules/extras' (https://github.com/ansible/ansible-modules-extras) registered for path 'lib/ansible/modules/extras'
Cloning into 'lib/ansible/modules/core'...
Submodule path 'lib/ansible/modules/core': checked out 'ce6619bf5db87f94001625c991d02960109dee2d'
Cloning into 'lib/ansible/modules/extras'...
Submodule path 'lib/ansible/modules/extras': checked out '29af26884ea11639f38c145b348afccdb6923285'

[danielluke (Daniel J. Luke)]

It's maybe also worth noting that github has been having issues: ​http://www.theregister.co.uk/2016/01/28/github_outage/

[breun (Nils Breunese)]

Doesn't seem to be related to GitHub's outage. Right now git clone https://github.com/ansible/ansible.git still works fine, but sudo port -dv fetch ansible still fails.

[danielluke (Daniel J. Luke)]

Replying to nils@…:

Doesn't seem to be related to GitHub's outage. Right now git clone https://github.com/ansible/ansible.git still works fine, but sudo port -dv fetch ansible still fails.

... but works for me.

Which suggests that either we're connecting to different github servers, or there is something different between our setups that is causing the issue.

[breun (Nils Breunese)]

This suggests port changes something from running git clone directly, at least on my install. I have no problems running git clone against repositories on GitHub. Sadly I have no idea how to find out what is different when port tries to clone this repository for me. Does port maybe use a different CA bundle from straight up git?

[danielluke (Daniel J. Luke)]

The most common difference is that port sanitized the environment before it runs (see src/macports1.0/macports.tcl ~ line 1038)

[ryandesign (Ryan Carsten Schmidt)]

Replying to nils@…:

This suggests port changes something from running git clone directly, at least on my install. I have no problems running git clone against repositories on GitHub. Sadly I have no idea how to find out what is different when port tries to clone this repository for me. Does port maybe use a different CA bundle from straight up git?

If you run with debug mode (sudo port -d fetch) you can see the exact git command MacPorts runs and the environment variables it sets when doing so.

[breun (Nils Breunese)]

The information about the git command being run is also in the issue and and the log I attached to this issue. I don't see anything about environment variables though:

$ sudo port clean ansible
--->  Cleaning ansible
$ sudo port -d fetch ansible
DEBUG: Copying /Users/breun/Library/Preferences/com.apple.dt.Xcode.plist to /opt/local/var/macports/home/Library/Preferences
DEBUG: Changing to port directory: /opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/ansible
DEBUG: OS darwin/15.3.0 (Mac OS X 10.11) arch i386
DEBUG: Re-registering default for build.cmd
DEBUG: Re-registering default for destroot.cmd
DEBUG: Re-registering default for destroot.destdir
DEBUG: Sourcing PortGroup python 1.0 from /opt/local/var/macports/sources/rsync.macports.org/release/ports/_resources/port1.0/group/python-1.0.tcl
DEBUG: Sourcing PortGroup github 1.0 from /opt/local/var/macports/sources/rsync.macports.org/release/ports/_resources/port1.0/group/github-1.0.tcl
DEBUG: only one arch supported, so not adding the default universal variant
DEBUG: Running callback portconfigure::add_automatic_compiler_dependencies
DEBUG: Finished running callback portconfigure::add_automatic_compiler_dependencies
DEBUG: Running callback portbuild::add_automatic_buildsystem_dependencies
DEBUG: Finished running callback portbuild::add_automatic_buildsystem_dependencies
DEBUG: Attempting ln -sf /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work /opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/ansible/work
DEBUG: Starting logging for ansible
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
--->  Computing dependencies for ansible
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
DEBUG: Searching for dependency: git
DEBUG: Didn't find receipt, going to depspec regex for: git
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
DEBUG: Executing org.macports.main (ansible)
DEBUG: fetch phase started at Sat Jan 30 15:13:22 CET 2016
--->  Fetching distfiles for ansible
DEBUG: Executing org.macports.fetch (ansible)
DEBUG: Executing: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 2>&1
fatal: unable to access 'https://github.com/ansible/ansible.git/': SSL certificate problem: Couldn't understand the server certificate format
Command failed: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 2>&1
Exit code: 128
Error: org.macports.fetch for port ansible returned: Git clone failed
DEBUG: Error code: NONE
DEBUG: Backtrace: Git clone failed
    while executing
"gitfetch"
    (procedure "portfetch::fetch_main" line 14)
    invoked from within
"portfetch::fetch_main org.macports.fetch"
    ("eval" body line 1)
    invoked from within
"eval $procedure $targetname"
Warning: targets not executed for ansible: org.macports.fetch
Please see the log file for port ansible for details:
    /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/main.log
To report a bug, follow the instructions in the guide:
    http://guide.macports.org/#project.tickets
Error: Processing of port ansible failed

When I run the command myself that MacPorts says it ran (using sudo, of course), then there is no problem and the git clone works just fine:

$ sudo /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 2>&1
$ echo $?
0
$ ls /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1
CHANGELOG.md             MANIFEST.in              ansible-core-sitemap.xml examples                 setup.py
CODING_GUIDELINES.md     Makefile                 bin                      hacking                  test
CONTRIBUTING.md          README.md                contrib                  lib                      test-requirements.txt
COPYING                  RELEASES.txt             docs                     packaging                ticket_stubs
ISSUE_TEMPLATE.md        VERSION                  docsite                  samples                  tox.ini

Apparently there is something different when this command is executed by MacPorts on my computer. I also tried this on my work computer and there it works fine. Any idea what I could check for next?

[breun (Nils Breunese)]

It seems the SSL certificate problem: Couldn't understand the server certificate format error I'm getting comes from curl as I found it in the curl source code, but using curl directly for requests to github.com also seems to work fine:

$ curl -vi https://github.com/ansible/ansible.git
*   Trying 192.30.252.129...
* Connected to github.com (192.30.252.129) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: github.com
* Server certificate: DigiCert SHA2 Extended Validation Server CA
* Server certificate: DigiCert High Assurance EV Root CA
> GET /ansible/ansible.git HTTP/1.1
> Host: github.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: GitHub.com
Server: GitHub.com
< Date: Sun, 31 Jan 2016 17:25:40 GMT
Date: Sun, 31 Jan 2016 17:25:40 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 178
Content-Length: 178
< Location: https://github.com/ansible/ansible
Location: https://github.com/ansible/ansible
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Served-By: 362482c1f05726391203e2d2c32818a4
X-Served-By: 362482c1f05726391203e2d2c32818a4
< X-GitHub-Request-Id: 4DAC817E:1607:656AA4A:56AE4394
X-GitHub-Request-Id: 4DAC817E:1607:656AA4A:56AE4394
< X-Frame-Options: DENY
X-Frame-Options: DENY

< 
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host github.com left intact

[breun (Nils Breunese)]

Apparently Git support environment variables like GIT_CURL_VERBOSE and GIT_SSL_NO_VERIFY: ​https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables#Networking

Is there any way I could pass these to the git command run by port?

[neverpanic (Clemens Lang)]

Add the variable name to extra_env in $prefix/etc/macports/macports.conf and set them in the environment where you run port(1). Note that if you use sudo, you'll usually have to set the environment variables after the privilege elevation using sudo env GIT_CURL_VERBOSE=1 port ....

[breun (Nils Breunese)]

Ok, so I edited macports.conf:

$ grep ^extra_env /opt/local/etc/macports/macports.conf
extra_env		GIT_CURL_VERBOSE

And I tried the install again with the environment variable and debug output:

$ sudo env GIT_CURL_VERBOSE=1 port -d install ansible

But I don't find any new verbose output on either the command line output or in the main.log. I also don't see any mentions of environments variables in either of those places. Should I?

[neverpanic (Clemens Lang)]

Replying to nils@…:

But I don't find any new verbose output on either the command line output or in the main.log. I also don't see any mentions of environments variables in either of those places. Should I?

You should see the verbose output, not necessarily the environment variables. For you use case of the ansible fetch, it might however be easier to run port edit ansible and add fetch.env-append GIT_CURL_VERBOSE=1, then re-try.

[breun (Nils Breunese)]

I get invalid command name "fetch.env-append" when I try that. Looking at ​https://guide.macports.org/chunked/reference.phases.html it seems MacPorts only supports configure.env-append, build.env-append and test.env-append. Is there another way to add the environment variable to the git fetch phase?

[ryandesign (Ryan Carsten Schmidt)]

fetch is not what MacPorts internally calls a "command", so there isn't any corresponding fetch.env. There should be git.env, but there isn't, because the git and hg fetch types were not implemented as commands either (though the cvs, svn and bzr fetch types were). This seems like a bug; git and hg fetch types should have been commands too.

You could modify the portfetch::gitfetch procedure in /opt/local/libexec/macports/lib/port1.0/portfetch.tcl. You'll see where a local env variable is set.

[breun (Nils Breunese)]

It seems the git clone operation doesn't use an env variable at all, the one on line 443 is only used for git checkout. I just added GIT_CURL_VERBOSE=1 to the command on line 436 and that got me this output:

$ sudo port -d fetch ansible
(...)
--->  Fetching distfiles for ansible
DEBUG: Executing org.macports.fetch (ansible)
DEBUG: Executing: GIT_CURL_VERBOSE=1 /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 2>&1
* Couldn't find host github.com in the .netrc file; using defaults
*   Trying 192.30.252.128...
* Connected to github.com (192.30.252.128) port 443 (#0)
* SSL certificate problem: Couldn't understand the server certificate format
* Closing connection 0
fatal: unable to access 'https://github.com/ansible/ansible.git/': SSL certificate problem: Couldn't understand the server certificate format
Command failed: GIT_CURL_VERBOSE=1 /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.0.0.2-1 2>&1
Exit code: 128
Error: org.macports.fetch for port ansible returned: Git clone failed
DEBUG: Error code: NONE
DEBUG: Backtrace: Git clone failed
    while executing
"gitfetch"
    (procedure "portfetch::fetch_main" line 14)
    invoked from within
"portfetch::fetch_main org.macports.fetch"
    ("eval" body line 1)
    invoked from within
"eval $procedure $targetname"
Warning: targets not executed for ansible: org.macports.fetch
Please see the log file for port ansible for details:
    /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/main.log
To report a bug, follow the instructions in the guide:
    http://guide.macports.org/#project.tickets
Error: Processing of port ansible failed

The lines starting with the asterisks seem to be the result of setting GIT_CURL_VERBOSE=1. Sadly this output doesn't really tell me more than I already knew.

For laughs I also tried adding GIT_SSL_NO_VERIFY=1 and tried the fetch again, but I got the same error message. The problem really seems to be the certificate format and not the certificate verification failing. Still, when I execute /usr/bin/git clone https://github.com/ansible/ansible.git directly on the command line - with or without sudo - the operation succeeds without any error messages, so somehow the certificate format is only an issue when the git clone operation is running via port. I have no idea how this is possible. It doesn't sound like a problem with the github.com certificate itself.

[breun (Nils Breunese)]

I've been reading about curl-ca-bundle and certsync and thought installing either one of those might help, but that doesn't seem to be the case. That also makes sense, since /usr/bin/git is used, which probably doesn't know about those ports.

[breun (Nils Breunese)]

I've also encountered a problem with npm install (npm installed via MacPorts) failing with an error that seems to refer to TLS: ​https://github.com/hoodiehq/hoodie-server-account/issues/85#issuecomment-186884993 The logging is this case also doesn't quite tell me what's wrong exactly.

I'm guessing there is some sort of SSL/TLS certificate issue with my MacPorts installation, although there is of course chance this is a separate issue.

[breun (Nils Breunese)]

Upgraded to OS X 10.11.4, but I still have the same issue and I still can't install ansible via MacPorts.

[nhojpatrick (John Patrick)]

Any update as I would like to install ansible using macports.

[nhojpatrick (John Patrick)]

I've got 2 installations of git, I assume the native mac git from apple, plus also the one via mac ports.

I've tried as my regular user and as root.

All 4 combinations work, and successfully clone the repo which port "install" reports as failing.

So I assume this is an issue with port specific code and how it's setup's the environment/shell prior to it's clone.

john$ which git
/opt/local/bin/git
john$ git clone https://github.com/ansible/ansible.git/ ansible-0
Cloning into 'ansible-0'...
remote: Counting objects: 119091, done.
remote: Compressing objects: 100% (75/75), done.
remote: Total 119091 (delta 35), reused 1 (delta 1), pack-reused 119004
Receiving objects: 100% (119091/119091), 36.06 MiB | 630.00 KiB/s, done.
Resolving deltas: 100% (73428/73428), done.
Checking connectivity... done.
john$ /usr/bin/git clone https://github.com/ansible/ansible.git/ ansible-1
Cloning into 'ansible-1'...
remote: Counting objects: 119091, done.
remote: Compressing objects: 100% (75/75), done.
remote: Total 119091 (delta 35), reused 1 (delta 1), pack-reused 119004
Receiving objects: 100% (119091/119091), 36.06 MiB | 303.00 KiB/s, done.
Resolving deltas: 100% (73428/73428), done.
Checking connectivity... done.
john$ sudo su -
Password:
root# which git
/usr/bin/git
root# git clone https://github.com/ansible/ansible.git/ ansible-2
Cloning into 'ansible-2'...
remote: Counting objects: 119091, done.
remote: Compressing objects: 100% (75/75), done.
remote: Total 119091 (delta 35), reused 1 (delta 1), pack-reused 119004
Receiving objects: 100% (119091/119091), 36.06 MiB | 698.00 KiB/s, done.
Resolving deltas: 100% (73428/73428), done.
Checking connectivity... done.
root# /opt/local/bin/git clone https://github.com/ansible/ansible.git/ ansible-3
Cloning into 'ansible-3'...
remote: Counting objects: 119091, done.
remote: Compressing objects: 100% (75/75), done.
remote: Total 119091 (delta 35), reused 1 (delta 1), pack-reused 119004
Receiving objects: 100% (119091/119091), 36.06 MiB | 357.00 KiB/s, done.
Resolving deltas: 100% (73428/73428), done.
Checking connectivity... done.
root#

[g5pw (Aljaž Srebrnič)]

To add more statistics, ansible and git-flow are fetching perfectly fine on my machine. OS X 10.11.4, I run macports built from source though.

[breun (Nils Breunese)]

Interesting to hear that someone else has the exact same problem. The port fetch is still failing for me as well. I've tried the Git debug environment variables by modifying line 436 of /opt/local/libexec/macports/lib/port1.0/portfetch.tcl, but even GIT_SSL_NO_VERIFY=1 didn't make a difference.

[macports@…]

I am running into the same problem: "manual" git clone works while port refuses to perform the same (OSX 11.5)!

My work around (as root): Make port use it's own git distribution (of course, you have to "port install" it first)

cd /usr/bin
mv git git-orig-osx
ln -s /opt/local/bin/git .

Then ansible install works fine!

[ryandesign (Ryan Carsten Schmidt)]

You should not make changes to /usr/bin, and OS X's ​SIP feature should have prevented you from doing so.

[macports@…]

• I am administering UNIXes for ~30 Years now. Believe me, I know, what I am doing!
• You should write software which allows to replace the git binary, at least for power users ;-)

Nevertheless: Thanks for providing Mac Ports, it's a great tool set which makes my life much easier!

[breun (Nils Breunese)]

Does anyone have any idea why OS X's /usr/bin/git seems to work for some people, but not for others? Or know a way to find out?

Is there a way to make the ansible port/Git fetch use an alternative git binary without have to change /usr/bin/git?

[breun (Nils Breunese)]

I updated to macOS Sierra 10.12.1 with Apple's Git 2.9.3 and MacPorts 2.3.5, but I still get the 'SSL certificate problem: Couldn't understand the server certificate format' error:

~ $ which git
/usr/bin/git
~ $ git --version
git version 2.9.3 (Apple Git-75)
~ $ sudo port -dv fetch ansible
DEBUG: Copying /Users/breun/Library/Preferences/com.apple.dt.Xcode.plist to /opt/local/var/macports/home/Library/Preferences
DEBUG: Changing to port directory: /Users/breun/Projects/macports-ports/sysutils/ansible
DEBUG: OS darwin/16.1.0 (Mac OS X 10.12) arch i386
DEBUG: Re-registering default for build.cmd
DEBUG: Re-registering default for destroot.cmd
DEBUG: Re-registering default for destroot.destdir
DEBUG: Sourcing PortGroup python 1.0 from /Users/breun/Projects/macports-ports/_resources/port1.0/group/python-1.0.tcl
DEBUG: Sourcing PortGroup github 1.0 from /Users/breun/Projects/macports-ports/_resources/port1.0/group/github-1.0.tcl
DEBUG: only one arch supported, so not adding the default universal variant
DEBUG: Running callback portconfigure::add_automatic_compiler_dependencies
DEBUG: Preferred compilers: clang macports-clang-3.8 macports-clang-3.7
DEBUG: Finished running callback portconfigure::add_automatic_compiler_dependencies
DEBUG: Running callback portbuild::add_automatic_buildsystem_dependencies
DEBUG: Finished running callback portbuild::add_automatic_buildsystem_dependencies
DEBUG: Starting logging for ansible
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
--->  Computing dependencies for ansible
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
DEBUG: Searching for dependency: git
DEBUG: Didn't find receipt, going to depspec regex for: git
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
DEBUG: Executing org.macports.main (ansible)
DEBUG: fetch phase started at Sat Nov 19 12:57:03 CET 2016
--->  Fetching distfiles for ansible
DEBUG: Executing org.macports.fetch (ansible)
DEBUG: Executing: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_Users_breun_Projects_macports-ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1 2>&1
fatal: unable to access 'https://github.com/ansible/ansible.git/': SSL certificate problem: Couldn't understand the server certificate format
Command failed: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_Users_breun_Projects_macports-ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1 2>&1
Exit code: 128
Error: org.macports.fetch for port ansible returned: Git clone failed
DEBUG: Error code: NONE
DEBUG: Backtrace: Git clone failed
    while executing
"gitfetch"
    (procedure "portfetch::fetch_main" line 14)
    invoked from within
"portfetch::fetch_main org.macports.fetch"
    ("eval" body line 1)
    invoked from within
"eval $procedure $targetname"
Warning: targets not executed for ansible: org.macports.fetch
Please see the log file for port ansible for details:
    /opt/local/var/macports/logs/_Users_breun_Projects_macports-ports_sysutils_ansible/ansible/main.log
To report a bug, follow the instructions in the guide:
    http://guide.macports.org/#project.tickets
Error: Processing of port ansible failed

[ryandesign (Ryan Carsten Schmidt)]

All I can suggest is that you contact GitHub for support.

[breun (Nils Breunese)]

Running git clone directly works fine, so I doubt that GitHub is causing the issue. Other GitHub-based ports also work fine, so it seems something special is going on with this andible port. I have no idea where to look further though.

[ken-cunningham-webuse]

so I just tried this on 10.6.8, using an installed version of git at /usr/bin/git that is older now (2011)

$ /usr/bin/git --version
git version 1.7.5.4

and it works fine

DEBUG: Executing org.macports.fetch (ansible)
DEBUG: Executing: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1 2>&1
DEBUG: Executing GIT_DIR=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1/.git GIT_WORK_TREE=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1 /usr/bin/git checkout -q v2.1.2.0-1 2>&1
DEBUG: Executing proc-post-org.macports.fetch-fetch-0
Submodule 'lib/ansible/modules/core' (https://github.com/ansible/ansible-modules-core) registered for path 'lib/ansible/modules/core'
Submodule 'lib/ansible/modules/extras' (https://github.com/ansible/ansible-modules-extras) registered for path 'lib/ansible/modules/extras'
Cloning into '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1/lib/ansible/modules/core'...
Cloning into '/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1/lib/ansible/modules/extras'...
Submodule path 'lib/ansible/modules/core': checked out '17ee1cfaf960d99bd3fee7631f86b1013957ec70'
Submodule path 'lib/ansible/modules/extras': checked out 'd312f34d9bbb722bf13314102d7373f1bd82e433'

I notice you have macports installed in your home directory, so my guess is it is a permissions problem (macports would be running git as the macports user most likely).

So I would try either working out the chain of permissions (ugh), or installing macports in /opt/something (like /opt/local) or in /Users/Shared or somewhere where the permissions issue wouldn't be a problem. -- hope this is helpful - K

[breun (Nils Breunese)]

MacPorts is actually not installed in my home directory, but in /opt/local. Since I maintain a few ports I have a got clone of the ports tree in my home directory for which an entry exists in my sources.conf.

[ken-cunningham-webuse]

looks like it's accessing your home directory

Changing to port directory: /Users/breun/Projects/macports-ports/sysutils/ansible

I'd disable that in your sources.conf and see what happens then, but hey, up to you.

[breun (Nils Breunese)]

MacPorts is accessing my home directory, because of the local git clone of the macports-ports repository I added in /opt/local/etc/macports/sources.conf, but the MacPorts installation itself is in the default /opt/local directory.

When I comment the local entry (file:///Users/breun/Projects/macports-ports) from sources.conf and leave just the default rsync entry the fetch fails in the same way:

$ sudo port -dv fetch ansible
DEBUG: Copying /Users/breun/Library/Preferences/com.apple.dt.Xcode.plist to /opt/local/var/macports/home/Library/Preferences
DEBUG: Changing to port directory: /opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/ansible
DEBUG: OS darwin/16.1.0 (Mac OS X 10.12) arch i386
DEBUG: Re-registering default for build.cmd
DEBUG: Re-registering default for destroot.cmd
DEBUG: Re-registering default for destroot.destdir
DEBUG: Sourcing PortGroup python 1.0 from /opt/local/var/macports/sources/rsync.macports.org/release/ports/_resources/port1.0/group/python-1.0.tcl
DEBUG: Sourcing PortGroup github 1.0 from /opt/local/var/macports/sources/rsync.macports.org/release/ports/_resources/port1.0/group/github-1.0.tcl
DEBUG: only one arch supported, so not adding the default universal variant
DEBUG: Running callback portconfigure::add_automatic_compiler_dependencies
DEBUG: Preferred compilers: clang macports-clang-3.8 macports-clang-3.7
DEBUG: Finished running callback portconfigure::add_automatic_compiler_dependencies
DEBUG: Running callback portbuild::add_automatic_buildsystem_dependencies
DEBUG: Finished running callback portbuild::add_automatic_buildsystem_dependencies
DEBUG: Attempting ln -sf /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work /opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/ansible/work
DEBUG: Checksum recorded in statefile '0eb70249efbb7b3bc062270873345cfcbbcbac32a917dac44ce8aa500d1eb526' differs from Portfile checksum '9caae729c44a6276b200899983edf2b6352ea1f37a3a305fbecebc2c1d781d2e'
Portfile changed since last build; discarding previous state.
DEBUG: Starting logging for ansible
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
--->  Computing dependencies for ansible
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
DEBUG: Searching for dependency: git
DEBUG: Didn't find receipt, going to depspec regex for: git
DEBUG: Found Dependency: path: /usr/bin filename: git regex: ^git$
DEBUG: Executing org.macports.main (ansible)
DEBUG: fetch phase started at Sun Nov 20 12:40:42 CET 2016
--->  Fetching distfiles for ansible
DEBUG: Executing org.macports.fetch (ansible)
DEBUG: Executing: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1 2>&1
fatal: unable to access 'https://github.com/ansible/ansible.git/': SSL certificate problem: Couldn't understand the server certificate format
Command failed: /usr/bin/git clone -q https://github.com/ansible/ansible.git /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/work/ansible-2.1.2.0-1 2>&1
Exit code: 128
Error: org.macports.fetch for port ansible returned: Git clone failed
DEBUG: Error code: NONE
DEBUG: Backtrace: Git clone failed
    while executing
"gitfetch"
    (procedure "portfetch::fetch_main" line 14)
    invoked from within
"portfetch::fetch_main org.macports.fetch"
    ("eval" body line 1)
    invoked from within
"eval $procedure $targetname"
Warning: targets not executed for ansible: org.macports.fetch
Please see the log file for port ansible for details:
    /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_ports_sysutils_ansible/ansible/main.log
To report a bug, follow the instructions in the guide:
    http://guide.macports.org/#project.tickets
Error: Processing of port ansible failed

[ken-cunningham-webuse]

Thanks for trying. I just fetched it on my Sierra system as well, and of course it worked fine.

That was all I could think of -- sorry I'm out of other ideas. Good luck. At least it works manually for you.

[breun (Nils Breunese)]

Well, I still can't install the ansible port, because when the git clone command is run by MacPorts it still fails. Running the exact same git clone command directly works, but that doesn't help me getting the port installed.

[larryv (Lawrence Velázquez)]

Are you running /usr/bin/git for your manual testing?

[breun (Nils Breunese)]

Yes, that's currently the only git I have installed. I also use this git installation for my own development projects on GitHub, without any issues. I have had git installed from MacPorts as well before, but that didn't help either. I think the log even showed that installation wasn't used. The port fetch seems to prefer /usr/bin/git.

[ken-cunningham-webuse]

If those manual installation ideas I emailed you didn't fix it for you, and if you still wanted to pursue this further, I think the next step would be to take 5 minutes and install a separate installation of macports from source in /opt/something, configure your path to that, don't add any extra repos, and see if you still get the same problem on your machine with a stock fresh install.

If it works, then diff your installations and see what happened to the old install.

If it still doesn't work -- well at least you'd know more. You don't have anything lying around in /usr/local do you?

[raimue (Rainer Müller)]

I cannot reproduce this problem on macOS 10.12 Sierra, using either base master or 2.3.5.

This error message originates in ​libcurl and is a reaction to the error code returned from SecureTransport.

As this only happens with /usr/bin/git running under MacPorts, could you please check Console.app for any messages that could be related to this (Keychain, Sandbox, etc.)?

[breun (Nils Breunese)]

I have just done a fresh installation of MacPorts 2.3.5 in /opt/local-alt. I temporarily renamed /Applications/MacPorts because the python27 port complained about files being already there but not belonging to any port (yeah, that's from the python27 installation from the MacPorts installation in /opt/local), but after that I managed to install the ansible port!

$ which port
/opt/local-alt/bin/port
$ sudo port install ansible
(...)
$ which ansible
/opt/local-alt/bin/ansible
$ ansible --version
ansible 2.1.2.0
  config file = 
  configured module search path = Default w/o overrides

One difference I found is that /opt/local/etc/macports/macports.conf contains macportsuser root. I guess I'll just clean up everything and do a fresh installation in /opt/local.

[raimue (Rainer Müller)]

To avoid the issue with /Applications/MacPorts, you could have used ./configure ... --with-applications-dir=..., as described at ​https://guide.macports.org/#installing.macports.source.multiple

Setting macportsuser root is not a good idea, as it effectively disables privilege dropping, but I could reproduce this problem with this. According to the messages in Console.app, it is indeed a sandbox violation:

SandboxViolation: git-remote-http(16876) deny file-write-data /private/var/db/mds/system/mds.lock

[breun (Nils Breunese)]

I don't recall setting macportsuser root myself, but the MacPorts installation I had was a few years old, so I'm not sure. Maybe this used to be a default? Anyway, I started fresh, reinstalled all my ports and all seems fine now.

I don't know if the fact that you get an Git SSL certificate format error when using root as the macportsuser is a bug. It's certainly kind of unexpected and misleading.

[ken-cunningham-webuse]

But -- we have SUCCESS at tracking down a year old headache! That is progress!

[breun (Nils Breunese)]

Yeah, thanks for the suggestion of trying a fresh MacPorts install.

[ryandesign (Ryan Carsten Schmidt)]

Replying to breun:

SSL certificate problem: Couldn't understand the server certificate format

The Homebrew folks ran into and analyzed this problem ​here. They say the combination of using the root user, plus the use of sandbox-exec, leads to this problem, because:

Aug 22 14:53:23 kernel[0]: Sandbox: curl(43548) deny(1) file-write-data /private/var/db/mds/system/mds.lock

MacPorts does use sandbox-exec when using system, and does use system when fetching from git. Can we add that location to the list of locations allowed by the sandbox? From the end of portsandbox.tcl it looks like we already allow everything in /private/var/tmp and /private/var/folders so adding /private/var/db might be reasonable.

[neverpanic (Clemens Lang)]

I'm not quite sure why curl (actually probably CommonCrypto) uses Spotlight (which is what these mds files are, if I'm not mistaken) to find its certificates, but I guess we can add a sandbox exception.

I'd favor not making this file-write-data /private/var/db, though, because /private/var/db contains a number of important system configuration files, such as the DirectoryService database, which contains all your users, in /private/var/db/dslocal, the launchd overrides, the dyld cache and a file suspiciously named auth.db. Let's add /private/var/db/mds as an exception.

[ryandesign (Ryan Carsten Schmidt)]

Replying to neverpanic:

Let's add /private/var/db/mds as an exception.

Ok, can you do that?

[neverpanic (Clemens Lang)]

In 1f76522228abba0d8383071b059daff726a8c9f6/macports-base:

sandbox: Allow file-write-data for /var/db/mds

It seems CommonCrypto sometimes uses mds, which can occasionally make
'git clone' fail if write access to /var/db/mds/system/mds.lock is
denied. Fix this by adding an exception.

Closes: #50469

[neverpanic (Clemens Lang)]

In 401aa37bdad31a8094cd1a08853e1a6ca2747711/macports-base:

sandbox: Allow file-write-data for /var/db/mds

It seems CommonCrypto sometimes uses mds, which can occasionally make
'git clone' fail if write access to /var/db/mds/system/mds.lock is
denied. Fix this by adding an exception.

Closes: #50469
(cherry picked from commit 1f76522228abba0d8383071b059daff726a8c9f6)