RFE: Have curl-ca-bundle install individual .pem files in ${prefix}/etc/openssl/certs

[landonf (Landon Fuller)]

curl-ca-bundle currently installs a single cacerts.pem file. This makes it difficult to add new CA certificates to OpenSSL's default set, as cacerts.pem takes precedence over the ${prefix}/etc/openssl/certs directory.

Rather than install a single file, curl-ca-bundle could instead install individual certificates in ${prefix}/etc/openssl/certs. This would make it easy for users to add new certificates without modifying a port-installed cacerts file.

Debian/Ubuntu implement this mechanism in their ca-certificates package, along with some helpful scripts (update-ca-certificates) to generate the certs/ directory and keep it up-to-date: ​http://archive.ubuntu.com/ubuntu/pool/main/c/ca-certificates/ca-certificates_20111211.tar.gz

Note that Debian/Ubuntu use Mozilla's trust store instead of curl-ca-bundle; I'm not sure which is really better.

[nonstop.server@…]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

I haven't looked into this, but if you'd like to make this change, go ahead.

[landonf (Landon Fuller)]

Just a note, that we probably ought to also solve this problem for java, eg: ​http://archive.ubuntu.com/ubuntu/pool/main/c/ca-certificates-java/ca-certificates-java_20120721.tar.gz

A proper Mac OS X solution would actually be to synchronize the OpenSSL and Java keystore(s) from the system keychain (which would require writing some original software, and possibly slightly outside the purview of MacPorts). That said, this is somewhat necessary to make the keychain/openssl/java work nicely together, and would make sense for some level of integration into MacPorts.

[larryv (Lawrence Velázquez)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Cc Me!

[cooljeanius (Eric Gallager)]

Cc Me!

[landonf (Landon Fuller)]

I finally had time to look at this; it turned out to be pretty easy (well, the non-Java part, anyway). See the attached certsync.m; it exports anchor certificates from the system keychain. It should run on Mac OS X all the way back to 10.6 (and possibly earlier), but I only tested on 10.8.

To compile:

clang -mmacosx-version-min=10.6 certsync.m -o certsync -framework Foundation -framework Security -fobjc-arc

Example usage:

./certsync -o /opt/local/etc/openssl/cert.pem (this overwrites the curl-ca-bundle symlink)

This will write out all OS-included CA certs, as well as any CA certs that have been installed system-wide by the user -- this ought to make life a *lot* easier for anyone that works for a company that manages internal private CAs.

I'm open to suggestions on how best to integrate this into MacPorts; ideally it would be run automatically at activate-time, along with a user-executable script to update the certificates (or via cron, or via monitoring the keychain, or ...).

Since curl is configured with a non-standard certificate path, the above certsync command won't affect it. You can test with curl by overwriting the actual path curl is configured with:

./certsync -o /opt/local/etc/openssl/cert.pem (this overwrites the curl-ca-bundle symlink)

Short of patching curl to use SSL_CTX_set_default_verify_paths(), we should probably change the curl port to use:

--with-ca-bundle=${prefix}/etc/openssl/cert.pem

[landonf (Landon Fuller)]

I just added an experimental certsync port; it should be a drop-in replacement for curl-ca-bundle:

sudo port -f deactivate curl-ca-bundle
sudo port install certsync

Bumping over to Ryan to get his input.

[landonf (Landon Fuller)]

The port now also automatically updates the cert.pem when any of the keychain CAs are updated/changed/modified -- if you run port load certsync. As a side note, this is a rare case where I think a startup item should auto-enable.

[ryandesign (Ryan Carsten Schmidt)]

Replying to landonf@…:

Bumping over to Ryan to get his input.

I don't have a particular opinion on this. I don't use custom certificate authorities so I haven't run into the problems this is designed to solve. If the new certsync port meets all the needs that curl-ca-bundle meets, and then some, I have no objection to retiring curl-ca-bundle. curl-ca-bundle is currently used by glib-networking, mercurial, osc and vantages in addition to curl, so we would want to make sure it works with all of those before retiring it.

[landonf (Landon Fuller)]

The only thing that concerns me regarding throwing the switch is that the auto-updating only kicks in if the user runs 'port load'. I'm not sure how much of a problem that is?

[landonf (Landon Fuller)]

Attached a patch for review that updates all the ports in question.

[ryandesign (Ryan Carsten Schmidt)]

The portion of the patch that applies to certsync itself needn't increase the revision, since it does not affect any files the port installs; also, the deactivate hack has a copypasto; it still says kerberos5 on one line where it needs to say curl-ca-bundle.

In the name of a better user experience at the expense of some magic, we could have the certsync port automatically load itself in post-activate and unload in pre-deactivate. We could create the launchd plists manually, instead of using the startupitem commands, so that the user is not presented with any irrelevant instructions.

[neverpanic (Clemens Lang)]

Replying to ryandesign@…:

In the name of a better user experience at the expense of some magic, we could have the certsync port automatically load itself in post-activate and unload in pre-deactivate.

I agree.

We could create the launchd plists manually, instead of using the startupitem commands, so that the user is not presented with any irrelevant instructions.

We could also extend the startupitem mechanism to support this out of the box.

[ryandesign (Ryan Carsten Schmidt)]

Replying to cal@…:

We could also extend the startupitem mechanism to support this out of the box.

Yes; I was just trying to suggest a solution that would work immediately, without having to wait for a new release of MacPorts base.

[neverpanic (Clemens Lang)]

Replying to ryandesign@…:

Yes; I was just trying to suggest a solution that would work immediately, without having to wait for a new release of MacPorts base.

We're in control of new base releases and we're not in a hurry with this change. I think we should rather do this right than quick.

[landonf (Landon Fuller)]

Replying to cal@…:

We're in control of new base releases and we're not in a hurry with this change. I think we should rather do this right than quick.

Agreed on the lack of rush. I wouldn't mind seeing more testing, too.

How about 'startupitem.enable' as an option that defaults to no?

[neverpanic (Clemens Lang)]

startupitem.autostart does this and has landed in r106810. Do we want to ship this in 2.2?

[landonf (Landon Fuller)]

Replying to cal@…:

startupitem.autostart does this and has landed in r106810. Do we want to ship this in 2.2?

Awesome! An (obvious) +1 vote from me.

[neverpanic (Clemens Lang)]

Merged into release_2_2 in r109245.

[neverpanic (Clemens Lang)]

Enabled startupitem.autostart, if available, for certsync in r109247.

[neverpanic (Clemens Lang)]

Since the startupitem.autostart change shipped in 2.2.1, we could remove the forward compatibility hack committed in r109247. I think we might also consider this bug fixed. Opinions?

[ryandesign (Ryan Carsten Schmidt)]

We would still want to replace curl-ca-bundle with certsync, but I'd like it to build on Tiger first (#41716).

[neverpanic (Clemens Lang)]

Of course, but that doesn't affect this ticket, does it?

[dClauzel (Damien Clauzel)]

Replying to landonf@…:

I just added an experimental certsync port; it should be a drop-in replacement for curl-ca-bundle:

sudo port -f deactivate curl-ca-bundle
sudo port install certsync

Bumping over to Ryan to get his input.

Tested with curl and wget. Works fine; I don't notice any bad side effects.