Skip to contentSkip to conversation

Add CRL extractor script for --crl-verify dir mode

PATCH In review Vladislav Grishenko 2 participants 3 messages Oct 2, 2020 – May 7, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add CRL extractor script for --crl-verify dir mode

When --crl-verify is enabled, specified CRL file gets reloaded on every client connection. With huge CRL files it may take a significant amount of time - seconds and tens of seconds, during which OpenVPN is blocked and can't serve existing and/or incoming connections due its singlethread nature.
In alternative mode --crl-verify option takes directory containing files named as decimal serial numbers of the revoked certificates and 'dir' flag, revoked certificate check is being done by checking the presence of client's certificate number in that directory.

This script allow to perform incremental extraction of revoked serial numbers from CRL by adding absent ones and removing excess ones.

Usage example:
extractcrl.py -f pem /path/to/crl.pem /path/to/outdir extractcrl.py -f der /path/to/crl.crl /path/to/outdir cat /path/to/crl.pem | extractcrl.py -f pem - /path/to/outdir cat /path/to/crl.crl | extractcrl.py -f der - /path/to/outdir

Output example:

    Loaded:  309797 revoked certs in 4.136s
    Scanned: 312006 files in 0.61s
    Created: 475 files in 0.05s
    Removed: 2684 files in 0.116s
---
 contrib/extract-crl/extractcrl.py | 138 ++++++++++++++++++++++++++++++
 1 file changed, 138 insertions(+)
 create mode 100755 contrib/extract-crl/extractcrl.py
contrib/extract-crl/extractcrl.py new file +138 -0
new file mode 100755
index 00000000..441464e9
@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+'''
+Helper script for CRL (certificate revocation list) file extraction
+to a directory containing files named as decimal serial numbers of
+the revoked certificates, to be used with OpenVPN CRL directory
+verify mode. To enable this mode, directory and 'dir' flag needs to
+be specified as parameters of '--crl-verify' option.
+For more information refer OpenVPN tls-options.rst.
+
+Usage example:
+    extractcrl.py -f pem /path/to/crl.pem /path/to/outdir
+    extractcrl.py -f der /path/to/crl.crl /path/to/outdir
+    cat /path/to/crl.pem | extractcrl.py -f pem - /path/to/outdir
+    cat /path/to/crl.crl | extractcrl.py -f der - /path/to/outdir
+
+Output example:
+    Loaded:  309797 revoked certs in 4.136s
+    Scanned: 312006 files in 0.61s
+    Created: 475 files in 0.05s
+    Removed: 2684 files in 0.116s
+'''
+
+import argparse
+import os
+import sys
+import time
+from subprocess import check_output
+
+FILETYPE_PEM = 'PEM'
+FILETYPE_DER = 'DER'
+
+def measure_time(method):
+    def elapsed(*args, **kwargs):
+        start = time.time()
+        result = method(*args, **kwargs)
+        return result, round(time.time() - start, 3)
+    return elapsed
+
+@measure_time
+def load_crl(filename, format):
+
+    def try_openssl_module(filename, format):
+        from OpenSSL import crypto
+        types = {
+            FILETYPE_PEM: crypto.FILETYPE_PEM,
+            FILETYPE_DER: crypto.FILETYPE_ASN1
+        }
+        if filename == '-':
+            crl = crypto.load_crl(types[format], sys.stdin.buffer.read())
+        else:
+            with open(filename, 'rb') as f:
+                crl = crypto.load_crl(types[format], f.read())
+        return set(int(r.get_serial(), 16) for r in crl.get_revoked())
+
+    def try_openssl_exec(filename, format):
+        args = ['openssl', 'crl', '-inform', format, '-text']
+        if filename != '-':
+            args += ['-in', filename]
+        serials = set()
+        for line in check_output(args, universal_newlines=True).splitlines():
+            _, _, serial = line.partition('Serial Number:')
+            if serial:
+                serials.add(int(serial.strip(), 16))
+        return serials
+
+    try:
+        return try_openssl_module(filename, format)
+    except ImportError:
+        return try_openssl_exec(filename, format)
+
+@measure_time
+def scan_dir(dirname):
+    _, _, files = next(os.walk(dirname))
+    return set(int(f) for f in files if f.isdigit())
+
+@measure_time
+def create_new_files(dirname, newset, oldset):
+    addset = newset.difference(oldset)
+    for serial in addset:
+        try:
+            with open(os.path.join(dirname, str(serial)), 'xb'): pass
+        except FileExistsError:
+            pass
+    return addset
+
+@measure_time
+def remove_old_files(dirname, newset, oldset):
+    delset = oldset.difference(newset)
+    for serial in delset:
+        try:
+            os.remove(os.path.join(dirname, str(serial)))
+        except FileNotFoundError:
+            pass
+    return delset
+
+def check_crlfile(arg):
+    if arg == '-' or os.path.isfile(arg):
+        return arg
+    raise argparse.ArgumentTypeError('No such file "{}"'.format(arg))
+
+def check_outdir(arg):
+    if os.path.isdir(arg):
+        return arg
+    raise argparse.ArgumentTypeError('No such directory: "{}"'.format(arg))
+
+def main():
+    parser = argparse.ArgumentParser(description='OpenVPN CRL extractor')
+    parser.add_argument('-f', '--format',
+        type=str.upper,
+        default=FILETYPE_PEM, choices=[FILETYPE_PEM, FILETYPE_DER],
+        help='input CRL format - default {}'.format(FILETYPE_PEM)
+    )
+    parser.add_argument('crlfile', metavar='CRLFILE|-',
+        type=lambda x: check_crlfile(x),
+        help='input CRL file or "-" for stdin'
+    )
+    parser.add_argument('outdir', metavar='OUTDIR',
+        type=lambda x: check_outdir(x),
+        help='output directory for serials numbers'
+    )
+    args = parser.parse_args()
+
+    certs, t = load_crl(args.crlfile, args.format)
+    print('Loaded:  {} revoked certs in {}s'.format(len(certs), t))
+
+    files, t = scan_dir(args.outdir)
+    print('Scanned: {} files in {}s'.format(len(files), t))
+
+    created, t = create_new_files(args.outdir, certs, files)
+    print('Created: {} files in {}s'.format(len(created), t))
+
+    removed, t = remove_old_files(args.outdir, certs, files)
+    print('Removed: {} files in {}s'.format(len(removed), t))
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.17.1
Acked-by on v1 1/1
#1 lore raw

We discussed this in the community meeting today, and came to the conclusion that this is a nice and helpful addition. It could be argued that python is somewhat heavy to "run an openssl binary and do something with the result", but it's neither the first nor last python script in our repo, and "have a script" is better than "I could write one in shell" :-)

I have not actually tested the script. David has taken a look and said it looks reasonable. Rules for contrib/ are not as strict as for "openvpn main source", so that's good enough.

Thanks.

Your patch has been applied to the master branch.

commit 4c2549ba5d8b1b449acc62a46692345710965647

Author: Vladislav Grishenko
Date:   Sat Oct 3 02:51:46 2020 +0500
     Add CRL extractor script for --crl-verify dir mode

     Signed-off-by: Vladislav Grishenko <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg21154.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#2 lore raw

Thanks!
Need to say, implemented "run an openssl binary" internal method is a bit faster than python-native crl parsing, according our tests and usage experience.

--
Best Regards, Vladislav Grishenko

Quoted 6 lines

-----Original Message-----
From: Gert Doering <[email protected]...>
Sent: Thursday, May 6, 2021 1:12 AM
To: Vladislav Grishenko <[email protected]...>
Cc: [email protected]
Subject: [PATCH applied] Re: Add CRL extractor script for --crl-verify dir

mode

Quoted 4 lines

We discussed this in the community meeting today, and came to the

conclusion

Quoted 1 line

that this is a nice and helpful addition. It could be argued that python

is

Quoted 1 line

somewhat heavy to "run an openssl binary and do something with the

result",

Quoted 1 line

but it's neither the first nor last python script in our repo, and "have a

script" is

Quoted 3 lines

better than "I could write one in shell" :-)

I have not actually tested the script. David has taken a look and said it

looks

Quoted 1 line

reasonable. Rules for contrib/ are not as strict as for "openvpn main

source", so

Quoted 24 lines

that's good enough.

Thanks.

Your patch has been applied to the master branch.

commit 4c2549ba5d8b1b449acc62a46692345710965647

Author: Vladislav Grishenko
Date:   Sat Oct 3 02:51:46 2020 +0500
     Add CRL extractor script for --crl-verify dir mode

     Signed-off-by: Vladislav Grishenko <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/openvpn-
[email protected]/msg21154.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering