openssl claims to use /private/etc/ssl, but appears not to, What The Heck?

Question

In the course of trying to help a friend with a problem with pip and ssl sites (GitHub issue here), I've become confused about how the High Sierra /usr/bin/openssl finds its certificates. My "keg-only" openssl does not have any trouble with the site.

Here's the test case that I've been playing with:

(alice)[14:22:06]~>>/usr/bin/openssl s_client -connect files.pythonhosted.org:443 | head 2>&1
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign CloudSSL CA - SHA256 - G3
verify error:num=20:unable to get local issuer certificate
verify return:0
CONNECTED(00000005)
---
Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=Fastly, Inc/CN=r.ssl.fastly.net
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign CloudSSL CA - SHA256 - G3
 1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign CloudSSL CA - SHA256 - G3
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
^C

I've been scratching my head because an appropriate key resides in the Keychain utility (Determined by downloading the Mozilla cert bundle from the Curl site, finding the one cert that rescues the test case when provided via -CAfile, and comparing its fingerprint to certs in the Keychain app. See the pip issue for gory details).

The value of OPENSSLDIR in the openssl version -a output suggests that /usr/bin/openssl should be using /private/etc/ssl:

(alice)[14:05:27]~>>/usr/bin/openssl version -a
LibreSSL 2.2.7
built on: date not available
platform: information not available
options:  bn(64,64) rc4(ptr,int) des(idx,cisc,16,int) blowfish(idx)
compiler: information not available
OPENSSLDIR: "/private/etc/ssl"

And, in fact, pointing at that directory with the -CApath command line option rescues the test case:

(alice)[14:26:32]~>>/usr/bin/openssl s_client -connect files.pythonhosted.org:443 -CApath /private/etc/ssl | head 2>&1 < /dev/null
depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign CloudSSL CA - SHA256 - G3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Fastly, Inc", CN = r.ssl.fastly.net
verify return:1
^C
(alice)[15:21:22]~>>

What's going on? Do the CApath/CAfile commands enable behavior that doesn't otherwise occur?

I'd love to understand what's going on.

Answer 1

This issue is the result of a known bug in OpenSSL that was reported in February 2013 and tracked here (log in using username guest and password guest). At the time, the latest public release of OpenSSL was OpenSSL 1.0.2. The issue was marked resolved in October 2016, and the bug no longer exists in releases after this date, the earliest of which is OpenSSL 1.1.1. Some users have reported that the bug was also not present in "version 1.1", to which they may be referring to OpenSSL 1.1.0 or forks of it — I have not confirmed whether the bug exists in OpenSSL 1.1.0.

The openssl binary included with macOS is LibreSSL, which is a fork of OpenSSL that originated from OpenSSL 1.0.1g. Since the bug existed in that version of OpenSSL, it persisted in LibreSSL until it was noticed and fixed by their developers, which was somewhere between the version you're using (LibreSSL 2.2.7) and the version I'm using (LibreSSL 2.8.3, which is packaged with macOS Catalina).