Can't Receive Peer Certificate In Python Client Using OpenSSL's Ssl.SSLContext()

I am a Windows user. I use Python 3.6.5 and I import this version of OpenSSL OpenSSL 1.0.2k. I need to write a script for a python TLS client that I can customize in terms of the s

Solution 1:

After a number of attempts, some failed, some partial successful, I found a way that should work (didn't test it with self signed certificates, though). Also, I wiped out everything from the previous attempts.

There are 2 necessary steps:

1. Get the server certificate using [Python 3.Docs]: (ssl.get_server_certificate(addr, ssl_version=PROTOCOL_TLS, ca_certs=None), which returns it as a PEM encoded string (e.g.: ours - pretty printed):

   '-----BEGIN CERTIFICATE-----'
   'MIIIPjCCByagAwIBAgIICG/ofYt2G48wDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE`
   'BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl'
   
   ...
   
   'L2KuOvWZ40sTVCJdWPUMtT9VP7VHfLNTFft/IhR+bUPkr33xjOa0Idq6cL89oufn'
   '-----END CERTIFICATE-----'
   

2. Decode the certificate using (!!!undocumented!!!) ssl._ssl._test_decode_cert (present in Python 3 / Python 2)

3. Due to the fact that ssl._ssl._test_decode_cert can only read the certificate from a file, 2 additional steps are needed:
   • Save the certificate from #1. in a temporary file (before #2., obviously)
   • Delete that file when done with it

I would like to emphasize [Python 3.Docs]: SSLSocket.getpeercert(binary_form=False), which contains lots of info (that I missed the last time(s)).
Also, I found out about ssl._ssl._test_decode_cert, by looking at SSLSocket.getpeercert implementation ("${PYTHON_SRC_DIR}/Modules/_ssl.c").

code00.py:

#!/usr/bin/env python3

import sys
import os
import socket
import ssl
import itertools


def _get_tmp_cert_file_name(host, port):
    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "_".join(("cert", host, str(port), str(os.getpid()), ".crt")))


def _decode_cert(cert_pem, tmp_cert_file_name):
    #print(tmp_cert_file_name)
    with open(tmp_cert_file_name, "w") as fout:
        fout.write(cert_pem)
    try:
        return ssl._ssl._test_decode_cert(tmp_cert_file_name)
    except Exception as e:
        print("Error decoding certificate:", e)
        return dict()
    finally:
        os.unlink(tmp_cert_file_name)


def get_srv_cert_0(host, port=443):
    try:
        cert_pem = ssl.get_server_certificate((host, port))
    except Exception as e:
        print("Error getting certificate:", e)
        return dict()
    tmp_cert_file_name = _get_tmp_cert_file_name(host, port)
    return _decode_cert(cert_pem, tmp_cert_file_name)


def get_srv_cert_1(host, port=443):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    context = ssl.SSLContext()
    ssl_sock = context.wrap_socket(sock, server_hostname=host)
    try:
        ssl_sock.connect((host, port))
    except Exception as e:
        print("Error connecting:\n", e)
        return dict()
    try:
        cert_der = ssl_sock.getpeercert(True)  # NOTE THE ARGUMENT!!!
    except Exception as e:
        print("Error getting cert:\n", e)
        return dict()
    tmp_cert_file_name = _get_tmp_cert_file_name(host, port)
    return _decode_cert(ssl.DER_cert_to_PEM_cert(cert_der), tmp_cert_file_name)


def main(argv):
    domain = "google.com"
    if argv:
        print("Using custom method")
        get_srv_cert_func = get_srv_cert_1
    else:
        print("Using regular method")
        get_srv_cert_func = get_srv_cert_0

    cert = get_srv_cert_func(domain)
    print("====== peer's certificate ======")
    try:
        print("Issued To:", dict(itertools.chain(*cert["subject"]))["commonName"])
        print("Issued By:", dict(itertools.chain(*cert["issuer"]))["commonName"])
        print("Valid From:", cert["notBefore"])
        print("Valid To:", cert["notAfter"])
        if (cert == None):
            print("no certificate")
    except Exception as e:
        print("Error getting certificate:", e)


if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main(sys.argv[1:])

Notes:

• _get_tmp_cert_file_name: generates the temporary file name (located in the same dir as the script) that will store the certificate
• _decode_cert: saves the certificate in the file, then decodes the file and returns the resulting dict
• get_srv_cert_0: gets the certificate form server, then decodes it
• get_srv_cert_1: same thing that get_srv_cert_0 does, but "manually"
  • Its advantage is controlling the SSL context creation / manipulation (which I think was the main point of the question)
• main:
  • Gets the server certificate using one of the 2 methods above (based on an argument being / not being passed to the script)
  • Prints certificate data (your code with some small corrections)

Output:

(py35x64_test) e:\Work\Dev\StackOverflow\q050055935>"e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" code00.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

Using regular method
====== peer's certificate ======
Issued To: *.google.com
Issued By: Google Internet Authority G2
Valid From: Apr 10 18:58:05 2018 GMT
Valid To: Jul  3 18:33:00 2018 GMT

(py35x64_test) e:\Work\Dev\StackOverflow\q050055935>"e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" code00.py 1
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

Using custom method
====== peer's certificate ======
Issued To: *.google.com
Issued By: Google Internet Authority G2
Valid From: Apr 10 18:55:13 2018 GMT
Valid To: Jul  3 18:33:00 2018 GMT

Check [SO]: How can I decode a SSL certificate using python? (@CristiFati's answer) for the decoding part only.

Solution 2:

Here is a 9 line snippet that grabs cert data from any url.

import ssl
import socket

def getcertmeta(url="",port=443):
    hostname = socket.gethostbyaddr(url)[0]
    context = ssl.create_default_context()
    with socket.create_connection((hostname, port)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            return ssock.context.get_ca_certs()

Tin Foil Note: Something fishy is going with this cert stuff. They are the source of all kinds of problems but tech companies are obsessed with them. They provide no security benefit, every single cert authority has been hacked(except LetsEncrypt iirc) this is irrefutable and can be looked up and found online in idk 5mins, there's something else going on but they are so damn complicated and needlessly complex that its very difficult to figure out what. Imo they are being used along with intel ME to backdoor server data to central locations(like the hack revealed by wileaks v7), the certs on my phone included like 3-4 governments, just straight up says Japanese Government or Chinese Government and it has a bunch of intel agency companies like 'starlight technologies' its all very shady imo.

EDIT2: The above snippet is wrong, it only gets the certificate authority certs not the actual url, here is a 13 line snippet to get the cert for the actual url.

import socket,ssl
from contextlib import contextmanager
@contextmanager
def fetch_certificate(url="", port=443, timeout=0.5):
    cxt = ssl.create_default_context()
    sslctxsock = cxt.wrap_socket(socket.socket(), server_hostname=hostname)
    sslctxsock.settimeout(timeout)
    sslctxsock.connect((url,port))
    cert = sslctxsock.getpeercert()
    try:
        yield cert
    finally:
        sslctxsock.close()

Use it like this:

with fetch_certificate(url=url) as cert:
   print(cert)

Solution 3:

It looks like Python doesn't parse/store the client's cert when ssl.CERT_NONE is set which is the default for SSLContext

import socket
import ssl

dest = ("www.google.com", 443)

sock = socket.socket(socket.AF_INET)
ctx = ssl.SSLContext()

# You will either need to add the self-signed certs to the OS cert store (varies by OS)
# See https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations for loading non-defaults
ctx.set_default_verify_paths()

ctx.verify_mode = ssl.CERT_REQUIRED
ssock = ctx.wrap_socket(sock, server_hostname=dest[0])
result = ssock.connect(dest)
print(ssock.getpeercert())

See https://github.com/python/cpython/blob/main/Modules/_ssl.c#L1823 It's doing bitwise & so when verify mode is CERT_NONE (0) then the code block is skipped. CERT_OPTIONAL=1 or CERT_REQUIRED=2 is needed for the cert parsing/populating to work (otherwise it just sets an empty dict)

For your self-signed certs, you should either import them into the OS certificate authority store or use the method SSLContext.load_verify_locations to load the self-signed cert (since it's self-signed, you can load the server/leaf cert as the CA cert)

If you go the OS-store path, you can probably just use the default context. I think on Windows there's a per-user CA store (not positive, though). I don't think such a thing exists on Linux so you'd need to add it system-wide. Not sure about macOS or other operating systems