Catalina and pyenv install python versions fine, but anything needing ssl fails

Question

I seek to use pyenv instead of manually installing an official python release. My goal is to run the included script titled “Install Certificates.command”.

My assessment of the situation is pyenv is supposed to handle the same tasks that script performs during the automated install of a python version. Every python install via pyenv allows setting my global or local version just fine. after it’s installed without error. I assume pyenv could not miss reporting install errors for any of the 6-8 official versions of python install, uninstall, and reinstall several times now.

To narrow things, the following 3 versions of python matter:

3.10.0
3.9.8
3.8.12
system

I seek to migrate a few packages to 3.10.0 and updating some of the ones I'm personally working on to make sure my required site-packages are all there for these versions.

• I've had no trouble accomplishing this task under Ubuntu 20.04, but under MacOS Catalina 10.15.7 I am unable to make this happen with working results.
• I used brew to install both pyenv and pyenv virtualenv. Both install with zero errors.
• I have followed the most current MacOS specific installation instructions on the respective github repository page for each item (pyenv and pyenv-virtualenv).
• I have read through the Troubleshooting/FAQ and README's for each project.
• I've also reviewed the fairly conclusive install guide posted here: https://wilsonmar.github.io/pyenv/

Finally, I have removed all versions of python I successfully installed with pyenv (pyenv uninstall) and then uninstalled pyenv/pyenv-virtual with brew. Reinstalled both [email protected] and openssl@3 (The only two required casks for pyenv and pyenv-virtual, depending on the python version, as of this writing according to brew)

Even after reinstalling everything and looking over my .bashrc/.zshrc/.bash_profile, no matter what I do, I am unable to do even the most basic things that utilize ssl/certs.

If I use the system version of python (2.7.16), the same things that fail on my pyenv versions work fine.

Here is an example of the failure I get on any of the installed python versions I have except for system

catalina10_15_7:~ user$ pyenv --version
pyenv 2.2.0
catalina10_15_7:~ user$ pyenv virtualenv --version
pyenv-virtualenv 1.1.5 (virtualenv unknown)
catalina10_15_7:~ user$ pyenv versions
* system (set by /Users/user/.pyenv/version)
catalina10_15_7:~ user$ pyenv install 3.10.0
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.0.tar.xz...
-> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz
Installing Python-3.10.0...
patching file aclocal.m4
patching file configure
Hunk #5 succeeded at 10537 (offset -15 lines).
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.0 to /Users/user/.pyenv/versions/3.10.0

catalina10_15_7:~ user$ pyenv rehash
catalina10_15_7:~ user$ pyenv versions
* system (set by /Users/user/.pyenv/version)
  3.10.0
catalina10_15_7:~ user$ pyenv global 3.10.0
catalina10_15_7:~ user$ python --version
Python 3.10.0
catalina10_15_7:~ user$ cd dev/somegit
catalina10_15_7:somegit user$ ./download.py
Traceback (most recent call last):
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/http/client.py", line 1276, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/http/client.py", line 1322, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/http/client.py", line 1271, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/http/client.py", line 1031, in _send_output
    self.send(msg)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/http/client.py", line 969, in send
    self.connect()
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/http/client.py", line 1448, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/dev/somegit/./download.py", line 56, in <module>
    r = urlopen(manifest)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
catalina10_15_7:somegit user$

catalina10_15_7:somegit user$

catalina10_15_7:somegit user$
catalina10_15_7:somegit user$ cat ~/.bashrc
# .bashrc

# PATH
export PATH="$(brew --prefix sqlite)/bin:\
$(brew --prefix [email protected])/bin:\
$(brew --prefix openssl@3)/bin:\
$(brew --prefix bison)/bin:\
$HOME/Library/Android/sdk/platform-tools:\
$HOME/.pyenv/bin:\
$PATH"

# Exports
export PATH="$HOME/Library/Android/sdk/platform-tools:$PATH"
export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
export CPPFLAGS="-I/usr/local/opt/[email protected]/include"
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"
export LDFLAGS="-L/usr/local/opt/openssl@3/lib"
export CPPFLAGS="-I/usr/local/opt/openssl@3/include"
export LDFLAGS="-L/usr/local/opt/readline/lib"
export CPPFLAGS="-I/usr/local/opt/readline/include"
export LDFLAGS="-L/usr/local/opt/sqlite/lib"
export CPPFLAGS="-I/usr/local/opt/sqlite/include"
export LDFLAGS="-L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"
export LDFLAGS="-L/usr/local/opt/bison/lib"

# pyenv
eval "$(pyenv virtualenv-init -)"
export PYENV_ROOT="$HOME/.pyenv"

# gpip
gpip() {
    PIP_REQUIRE_VIRTUALENV=false pip "$@"
}

# Aliases
alias ls='ls -G'
alias ll='ls -laG'

catalina10_15_7:somegit user$
catalina10_15_7:somegit user$ brew list
==> Formulae
autoconf        dmg2img         libidn2         m4          pcre2           readline        xz
autossh         gettext         libunistring        ntfs-3g-mac     pkg-config      sqlite          zlib
bash            git         libusb          openjdk         pv          squashfs        zstd
bison           gzip            lz4         [email protected]     pyenv           tree
ca-certificates     javacc          lzo         openssl@3       pyenv-virtualenv    wget

==> Casks
font-hack-nerd-font
catalina10_15_7:somegit user$

Might there be a guide on resolving OpenSSL errors in this case?

Answer 1

For anyone else who finds this, had the same errors/issues. You can verify that your pyenv-installed Python isn't finding the SSL certificates by:

>>> import ssl
>>> ssl.create_default_context().cert_store_stats()
{'x509': 0, 'crl': 0, 'x509_ca': 0}

Decided to let pyenv (re)build the previously brew-installed openssl@3, ca-certificates packages, so brew uninstalled them and then re-installed the pyenv-installed pythons and let the pyenv installation install the brew dependencies.

Here's the step-by-step process that fixed it. YMMV, etc. First:

$ brew uninstall --ignore-dependencies ca-certificates

and brew will tell you Error: Could not remove ... and instruct you to remove it manually and supply you with the appropriate sudo rm -rf ... command.

Next:

$ brew uninstall --ignore-dependencies openssl@3

and again, brew will say it needs to be done manually and supply a sudo rm -rf ... command.

Next, use pyenv to uninstall any pythons, so pyenv uninstall <version> for each pyenv-installed python, then re-install with pyenv install <version> and the ca-certificates and openssl@3 will get re-installed as dependencies of the pyenv install.

To verify the fix, select a newly reinstalled pyenv python and repeat the

>>> import ssl
>>> ssl.create_default_context().cert_store_stats()

test from above. This time you should get something like

{'x509': 140, 'crl': 0, 'x509_ca': 140}

Hope this helps!