Skip to contentSkip to conversation

add engine keys

PATCH Superseded James Bottomley 3 participants 8 messages May 24, 2020 – May 25, 2020

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v5 cover add engine keys

This update tries to cope with the fact that the test engine dynamic extension is different on macos (.dylib) and linux (.so) by dynamically building the openssl.cnf file with the correct one

Note: I don't have any MacOS machines to test this on, so I only know it works on Linux if someone with a Mac could check, I'd be grateful.

---

Engine keys are an openssl concept for a key file which can only be understood by an engine (usually because it's been wrapped by the engine itself). We use this for TPM engine keys, so you can either generate them within your TPM or wrap them from existing private keys.
Once wrapped, the keys will only function in the TPM that generated
them, so it means the VPN keys are tied to the physical platform, which is very useful. Engine keys have to be loaded via a specific callback, so use this as a fallback in openvpn if an engine is specified and if the PEM read of the private key fails.

Adding a unit test for this type of key proved particularly
problematic: there's apparently no simple engine you can use to check the functionality, so after a bit of googling, I just wrote one as part of the test. You can see that the unit test converts an existing key to engine format (which is simply changing the PEM guards), tries to start openvpn with the key and verifies that the engine methods are called and the password correctly retrieved. To make the test simple, it relies on openssl detecting a mismatch between the certificate and the key after the key has been loaded rather than going on to bring up an openvpn loop, but I think that's sufficient to test out the engine patch fully.

---

James Bottomley (2):
openssl: add engine method for loading the key
Add unit tests for engine keys

 configure.ac                                  |   5 +
 src/openvpn/crypto_openssl.c                  |  57 ++++++++++
 src/openvpn/crypto_openssl.h                  |  12 +++
 src/openvpn/ssl_openssl.c                     |   5 +
 tests/unit_tests/Makefile.am                  |   3 +
 tests/unit_tests/engine-key/Makefile.am       |  24 +++++
 .../engine-key/check_engine_keys.sh           |  30 ++++++
 tests/unit_tests/engine-key/libtestengine.c   | 101 ++++++++++++++++++
 tests/unit_tests/engine-key/openssl.cnf.in    |  12 +++
 9 files changed, 249 insertions(+)
 create mode 100644 tests/unit_tests/engine-key/Makefile.am
 create mode 100755 tests/unit_tests/engine-key/check_engine_keys.sh
 create mode 100644 tests/unit_tests/engine-key/libtestengine.c
 create mode 100644 tests/unit_tests/engine-key/openssl.cnf.in
Signature · 1 line
2.26.2
#1 lore raw

PATCH v5 1/2 openssl: add engine method for loading the key

As well as doing crypto acceleration, engines can also be used to load key files. If the engine is set, and the private key loading fails for bio methods, this patch makes openvpn try to get the engine to load the key. If that succeeds, we end up using an engine based key. This can be used with the openssl tpm engines to make openvpn use a TPM wrapped key file.

---

v2: add better configuration guarding

v4: - use crypto_msg() instead of raw openssl prints
- remove ENGINE_init/finish(). Openvpn already initializes the engine so doing a second initialization is wrong.
- don't clear the openssl errors from the BIO_read failure just in case they might be useful.
- ad ui.h include for openssl 1.1 build failure

v5: add engine init

---
 src/openvpn/crypto_openssl.c | 56 ++++++++++++++++++++++++++++++++++++
 src/openvpn/crypto_openssl.h | 12 ++++++++
 src/openvpn/ssl_openssl.c    |  5 ++++
 3 files changed, 73 insertions(+)
src/openvpn/crypto_openssl.c +56 -0
index a5b2c45a..a7569623 100644
@@ -63,6 +63,7 @@
 #endif
 
 #if HAVE_OPENSSL_ENGINE
+#include <openssl/ui.h>
 #include <openssl/engine.h>
 
 static bool engine_initialized = false; /* GLOBAL */
@@ -1070,4 +1071,59 @@ memcmp_constant_time(const void *a, const void *b, size_t size)
 {
     return CRYPTO_memcmp(a, b, size);
 }
+
+#if HAVE_OPENSSL_ENGINE
+static int
+ui_reader(UI *ui, UI_STRING *uis)
+{
+    SSL_CTX *ctx = UI_get0_user_data(ui);
+
+    if (UI_get_string_type(uis) == UIT_PROMPT) {
+        pem_password_cb *cb = SSL_CTX_get_default_passwd_cb(ctx);
+        void *d = SSL_CTX_get_default_passwd_cb_userdata(ctx);
+        char password[64];
+
+        cb(password, sizeof(password), 0, d);
+        UI_set_result(ui, uis, password);
+
+        return 1;
+    }
+    return 0;
+}
+#endif
+
+EVP_PKEY *
+engine_load_key(const char *file, SSL_CTX *ctx)
+{
+#if HAVE_OPENSSL_ENGINE
+    UI_METHOD *ui;
+    EVP_PKEY *pkey;
+
+    if (!engine_persist)
+        return NULL;
+
+    /* this will print out the error from BIO_read */
+    crypto_msg(M_INFO, "PEM_read_bio failed, now trying engine method to load private key");
+
+    ui = UI_create_method("openvpn");
+    if (!ui) {
+	crypto_msg(M_FATAL, "Engine UI creation failed");
+        return NULL;
+    }
+
+    UI_method_set_reader(ui, ui_reader);
+
+    ENGINE_init(engine_persist);
+    pkey = ENGINE_load_private_key(engine_persist, file, ui, ctx);
+    ENGINE_finish(engine_persist);
+    if (!pkey)
+	crypto_msg(M_FATAL, "Engine could not load key file");
+ out:
+    UI_destroy_method(ui);
+    return pkey;
+#else
+    return NULL;
+#endif
+}
+
 #endif /* ENABLE_CRYPTO_OPENSSL */
src/openvpn/crypto_openssl.h +12 -0
index 64754480..7449fbd3 100644
@@ -107,4 +107,16 @@ cipher_kt_var_key_size(const cipher_kt_t *cipher)
     return EVP_CIPHER_flags(cipher) & EVP_CIPH_VARIABLE_LENGTH;
 }
 
+/**
+ * Load a key file from an engine
+ *
+ * @param file	The engine file to load
+ * @param ui	The UI method for the password prompt
+ * @param data	The data to pass to the UI method
+ *
+ * @return	The private key if successful or NULL if not
+ */
+EVP_PKEY *
+engine_load_key(const char *file, SSL_CTX *ctx);
+
 #endif /* CRYPTO_OPENSSL_H_ */
src/openvpn/ssl_openssl.c +5 -0
index 06c836da..a489053b 100644
@@ -1020,6 +1020,11 @@ tls_ctx_load_priv_file(struct tls_root_ctx *ctx, const char *priv_key_file,
     pkey = PEM_read_bio_PrivateKey(in, NULL,
                                    SSL_CTX_get_default_passwd_cb(ctx->ctx),
                                    SSL_CTX_get_default_passwd_cb_userdata(ctx->ctx));
+    if (!pkey)
+    {
+        pkey = engine_load_key(priv_key_file, ctx->ctx);
+    }
+
     if (!pkey || !SSL_CTX_use_PrivateKey(ssl_ctx, pkey))
     {
 #ifdef ENABLE_MANAGEMENT
Signature · 1 line
2.26.2
#2 lore raw

PATCH v5 2/2 Add unit tests for engine keys

In reply to James Bottomley

Testing engines is problematic, so one of the prerequisites built for the tests is a simple openssl engine that reads a non-standard PEM guarded key. The test is simply can we run a client/server configuration with the usual sample key replaced by an engine key. The trivial engine prints out some operations and we check for these in the log to make sure the engine was used to load the key and that it correctly got the password.

---
v5: do not hard code dynamic library extension into openssl.cnf (MacOS)
v4: add OPENSSL_config(NULL) so debian checks will work
v3: added this patch

---
 configure.ac                                  |   5 +
 src/openvpn/crypto_openssl.c                  |   1 +
 tests/unit_tests/Makefile.am                  |   3 +
 tests/unit_tests/engine-key/Makefile.am       |  24 +++++
 .../engine-key/check_engine_keys.sh           |  30 ++++++
 tests/unit_tests/engine-key/libtestengine.c   | 101 ++++++++++++++++++
 tests/unit_tests/engine-key/openssl.cnf.in    |  12 +++
 7 files changed, 176 insertions(+)
 create mode 100644 tests/unit_tests/engine-key/Makefile.am
 create mode 100755 tests/unit_tests/engine-key/check_engine_keys.sh
 create mode 100644 tests/unit_tests/engine-key/libtestengine.c
 create mode 100644 tests/unit_tests/engine-key/openssl.cnf.in
configure.ac +5 -0
index 273a8d1b..92d4eeba 100644
@@ -1387,6 +1387,10 @@ AM_CONDITIONAL([GIT_CHECKOUT], [test "${GIT_CHECKOUT}" = "yes"])
 AM_CONDITIONAL([ENABLE_PLUGIN_AUTH_PAM], [test "${enable_plugin_auth_pam}" = "yes"])
 AM_CONDITIONAL([ENABLE_PLUGIN_DOWN_ROOT], [test "${enable_plugin_down_root}" = "yes"])
 AM_CONDITIONAL([HAVE_LD_WRAP_SUPPORT], [test "${have_ld_wrap_support}" = "yes"])
+AM_CONDITIONAL([OPENSSL_ENGINE], [test "${have_openssl_engine}" = "yes"])
+
+shrext=$shrext_cmds
+AC_SUBST([shrext])
 
 sampledir="\$(docdir)/sample"
 AC_SUBST([plugindir])
@@ -1448,6 +1452,7 @@ AC_CONFIG_FILES([
         tests/unit_tests/openvpn/Makefile
         tests/unit_tests/plugins/Makefile
         tests/unit_tests/plugins/auth-pam/Makefile
+	tests/unit_tests/engine-key/Makefile
 	sample/Makefile
 ])
 AC_CONFIG_FILES([tests/t_client.sh], [chmod +x tests/t_client.sh])
src/openvpn/crypto_openssl.c +1 -0
index a7569623..34637ebf 100644
@@ -92,6 +92,7 @@ setup_engine(const char *engine)
 {
     ENGINE *e = NULL;
 
+    OPENSSL_config(NULL);
     ENGINE_load_builtin_engines();
 
     if (engine)
tests/unit_tests/Makefile.am +3 -0
index 33fefaac..f27cd90f 100644
@@ -2,4 +2,7 @@ AUTOMAKE_OPTIONS = foreign
 
 if ENABLE_UNITTESTS
 SUBDIRS = example_test openvpn plugins
+if OPENSSL_ENGINE
+SUBDIRS += engine-key
+endif
 endif
tests/unit_tests/engine-key/Makefile.am new file +24 -0
new file mode 100644
index 00000000..05f56bfd
@@ -0,0 +1,24 @@
+AUTOMAKE_OPTIONS = foreign
+
+check_LTLIBRARIES = libtestengine.la
+conffiles = openssl.cnf
+
+TESTS_ENVIRONMENT = srcdir="$(abs_srcdir)"; \
+	builddir="$(abs_builddir)"; \
+	top_builddir="$(top_builddir)"; \
+	top_srcdir="$(top_srcdir)"; \
+	export srcdir builddir top_builddir top_srcdir;
+
+TESTS = check_engine_keys.sh
+check_engine_keys.sh: $(conffiles)
+
+clean-local:
+	rm -f $(conffiles)
+
+$(builddir)/%.cnf: $(srcdir)/%.cnf.in
+	sed 's/SHREXT/@shrext@/' < $< > $@
+
+libtestengine_la_SOURCES = libtestengine.c
+libtestengine_la_LDFLAGS = @TEST_LDFLAGS@ -rpath /lib -avoid-version -module -shared -export-dynamic
+libtestengine_la_CFLAGS = @TEST_CFLAGS@ -I$(openvpn_srcdir) -I$(compat_srcdir)
+
tests/unit_tests/engine-key/check_engine_keys.sh new file +30 -0
new file mode 100755
index 00000000..e0c9d7b0
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+OPENSSL_CONF="${builddir}/openssl.cnf"
+export OPENSSL_CONF
+
+password='AT3S4PASSWD'
+
+key="${builddir}/client.key"
+pwdfile="${builddir}/passwd"
+
+# create an engine key for us
+sed 's/PRIVATE KEY/TEST ENGINE KEY/' < ${top_srcdir}/sample/sample-keys/client.key > ${key}
+echo "$password" > $pwdfile
+
+# note here we've induced a mismatch in the client key and the server
+# cert which openvpn should report and die.  Check that it does.  Note
+# also that this mismatch depends on openssl not openvpn, so it is
+# somewhat fragile
+${top_builddir}/src/openvpn/openvpn --cd ${top_srcdir}/sample --config sample-config-files/loopback-server --engine testengine --key ${key} --askpass $pwdfile > log.txt 2>&1
+
+# first off check we died because of a key mismatch.  If this doesn't
+# pass, suspect openssl of returning different messages and update the
+# test accordingly
+grep -q 'X509_check_private_key:key values mismatch' log.txt || { echo "Key mismatch not detected"; exit 1; }
+
+# now look for the engine prints (these are under our control)
+grep -q 'ENGINE: engine_init called' log.txt || { echo "Engine initialization not detected"; exit 1; }
+grep -q 'ENGINE: engine_load_key called' log.txt || { echo "Key was not loaded from engine"; exit 1; }
+grep -q "ENGINE: engine_load_key got password ${password}" log.txt || { echo "Key password was not retrieved by the engine"; exit 1; }
+exit 0
tests/unit_tests/engine-key/libtestengine.c new file +101 -0

101 lines not on this page · raw message

tests/unit_tests/engine-key/openssl.cnf.in new file +12 -0
new file mode 100644
index 00000000..93edd483
@@ -0,0 +1,12 @@
+HOME		= .
+openssl_conf	= openssl_init
+
+[req]
+[openssl_init]
+engines		= engines_section
+
+[engines_section]
+testengine	= testengine_section
+
+[testengine_section]
+dynamic_path	= $ENV::srcdir/.libs/libtestengineSHREXT
Signature · 1 line
2.26.2
#3 lore raw

Re: [Openvpn-devel] [PATCH v5 2/2] Add unit tests for engine keys

Hi,

I see the granularity of your patch set as "not right":

Quoted 7 lines · James Bottomley

Testing engines is problematic, so one of the prerequisites built for the tests is a simple openssl engine that reads a non-standard PEM guarded key. The test is simply can we run a client/server configuration with the usual sample key replaced by an engine key. The trivial engine prints out some operations and we check for these in the log to make sure the engine was used to load the key and that it correctly got the password.

This patch says "add unit tests", but it contains changes to configure.ac and OpenVPN code

Quoted 2 lines
 configure.ac                                  |   5 +
 src/openvpn/crypto_openssl.c                  |   1 +

These two hunks should go to the first patch.

Every patch should be fully testable on its own - so if I apply only the first hunk, I should be able to use and test engine keys, without having to apply the "add a unit test" patch set.

src/openvpn/crypto_openssl.c setup_engine(const char *engine) · line 98

Quoted 12 lines
src/openvpn/crypto_openssl.c +1 -0
index a7569623..34637ebf 100644
@@ -92,6 +92,7 @@ setup_engine(const char *engine)
 {
     ENGINE *e = NULL;
 
+    OPENSSL_config(NULL);
     ENGINE_load_builtin_engines();
 
     if (engine)

For that change, I wonder what side effects it might have on existing setups. Arne, can you help? Is this "safe"?

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...
#4 lore raw

Re: [Openvpn-devel] [PATCH v5 2/2] Add unit tests for engine keys

Am 25.05.20 um 08:04 schrieb Gert Doering:

Quoted 33 lines

Hi,

I see the granularity of your patch set as "not right":

Quoted 7 lines · James Bottomley

Testing engines is problematic, so one of the prerequisites built for the tests is a simple openssl engine that reads a non-standard PEM guarded key. The test is simply can we run a client/server configuration with the usual sample key replaced by an engine key. The trivial engine prints out some operations and we check for these in the log to make sure the engine was used to load the key and that it correctly got the password.

This patch says "add unit tests", but it contains changes to configure.ac and OpenVPN code

Quoted 2 lines
 configure.ac                                  |   5 +
 src/openvpn/crypto_openssl.c                  |   1 +

These two hunks should go to the first patch.

Every patch should be fully testable on its own - so if I apply only the first hunk, I should be able to use and test engine keys, without having to apply the "add a unit test" patch set.

Quoted 8 lines
src/openvpn/crypto_openssl.c +0 -0
index a7569623..34637ebf 100644
@@ -92,6 +92,7 @@ setup_engine(const char *engine)
 {
     ENGINE *e = NULL;
 
Quoted 8 lines

src/openvpn/crypto_openssl.c setup_engine(const char *engine) · line 98

Quoted 4 lines
src/openvpn/crypto_openssl.c +1 -0
+    OPENSSL_config(NULL);
     ENGINE_load_builtin_engines();
 
     if (engine)

For that change, I wonder what side effects it might have on existing setups. Arne, can you help? Is this "safe"?

For OpenSSL 1.1 this is actually a NACK since it is depreacted API and we want to wrap those into ifdefs to be able to remove them when we drop support for older version.

For OpenSSL 1.1 this is a deprecated function to contrary to other depreacted function like OpenSSL_init() not a noop.

But this command even though it does not have a return value or reports erro, can still put errors on the error stack.

Reading the general openssl.cnf at engine_setup also feels a bit like the wrong place but since this code is only executed when --engine is present in the config and we will remove that OPENSSL_config in the future anyway (with the ifdef) it a low risk to put it here instead of a more proper place, so I am fine to do here.

As a side note, engines are deprecated in OpenSSL 3.0.0 and are replaced by providers. But that will probably take a long time until that is really depreacted and removed, so adding/fixing this for now is okay.

Arne

#5 lore raw

Re: [Openvpn-devel] [PATCH v5 2/2] Add unit tests for engine keys

Quoted 45 lines · Arne Schwabe

Am 25.05.20 um 08:04 schrieb Gert Doering:

Quoted 34 lines

Hi,

I see the granularity of your patch set as "not right":

Quoted 7 lines · James Bottomley

Testing engines is problematic, so one of the prerequisites built for the tests is a simple openssl engine that reads a non- standard PEM guarded key. The test is simply can we run a client/server configuration with the usual sample key replaced by an engine key. The trivial engine prints out some operations and we check for these in the log to make sure the engine was used to load the key and that it correctly got the password.

This patch says "add unit tests", but it contains changes to configure.ac and OpenVPN code

Quoted 2 lines
 configure.ac                                  |   5 +
 src/openvpn/crypto_openssl.c                  |   1 +

These two hunks should go to the first patch.

Every patch should be fully testable on its own - so if I apply only the first hunk, I should be able to use and test engine keys, without having to apply the "add a unit test" patch set.

Quoted 9 lines
src/openvpn/crypto_openssl.c +0 -0
index a7569623..34637ebf 100644
@@ -92,6 +92,7 @@ setup_engine(const char *engine)
 {
     ENGINE *e = NULL;
 
Quoted 9 lines

src/openvpn/crypto_openssl.c setup_engine(const char *engine) · line 98

Quoted 4 lines
src/openvpn/crypto_openssl.c +1 -0
+    OPENSSL_config(NULL);
     ENGINE_load_builtin_engines();
 
     if (engine)

For that change, I wonder what side effects it might have on existing
setups. Arne, can you help? Is this "safe"?

Quoted 3 lines

For OpenSSL 1.1 this is actually a NACK since it is depreacted API and we want to wrap those into ifdefs to be able to remove them when we drop support for older version.

OK ... it's replacement is a simple

OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CONFIG, NULL);

which is easy to #ifdef around.

Quoted 11 lines

For OpenSSL 1.1 this is a deprecated function to contrary to other depreacted function like OpenSSL_init() not a noop.

But this command even though it does not have a return value or reports erro, can still put errors on the error stack.

Reading the general openssl.cnf at engine_setup also feels a bit like the wrong place but since this code is only executed when --engine is present in the config and we will remove that OPENSSL_config in the future anyway (with the ifdef) it a low risk to put it here instead of a more proper place, so I am fine to do here.

So if the desire is to make openvpn fully consistent with regard to local configuration files, the statement should be in crypto_lib_init() rather than conditioning the behaviour on setting the engines parameter. I can certainly do that, as a separate patch, but it would have a slightly wider blast radius.

Quoted 4 lines

As a side note, engines are deprecated in OpenSSL 3.0.0 and are replaced by providers. But that will probably take a long time until that is really depreacted and removed, so adding/fixing this for now is okay.

Heh, yes, we're all waiting to see how that works out.

James

#6 lore raw

Re: [Openvpn-devel] [PATCH v5 2/2] Add unit tests for engine keys

In reply to Gert Doering
Quoted 16 lines · Gert Doering

Hi,

I see the granularity of your patch set as "not right":

Quoted 10 lines · James Bottomley

Testing engines is problematic, so one of the prerequisites built for
the tests is a simple openssl engine that reads a non-standard PEM guarded key. The test is simply can we run a client/server configuration with the usual sample key replaced by an engine key. The trivial engine prints out some operations and we check for these
in the log to make sure the engine was used to load the key and that
it correctly got the password.

Quoted 8 lines

This patch says "add unit tests", but it contains changes to configure.ac
and OpenVPN code

Quoted 2 lines
 configure.ac                                  |   5 +
 src/openvpn/crypto_openssl.c                  |   1 +

These two hunks should go to the first patch.

The configure.ac one adds the test Makefile ... it can't move. The other pieces figure out what the dynamic extension is, which is only used by the test, so I think everything in the configure.ac change is in the right place.

The second is a bit more problematic: Right at the moment openvpn is blind to local openssl configuration files. To make the test work, it needs to be able to add an engine, which requires a local file, so it's only function is to enable the test, but it does add to openvpn capabilities, I suppose.

James

Quoted 22 lines

Every patch should be fully testable on its own - so if I apply only the first hunk, I should be able to use and test engine keys, without having to apply the "add a unit test" patch set.

src/openvpn/crypto_openssl.c setup_engine(const char *engine) · line 98

Quoted 13 lines
src/openvpn/crypto_openssl.c +1 -0
index a7569623..34637ebf 100644
@@ -92,6 +92,7 @@ setup_engine(const char *engine)
 {
     ENGINE *e = NULL;
 
+    OPENSSL_config(NULL);
     ENGINE_load_builtin_engines();
 
     if (engine)

For that change, I wonder what side effects it might have on existing setups. Arne, can you help? Is this "safe"?

gert

#7 lore raw

Re: [Openvpn-devel] [PATCH v5 2/2] Add unit tests for engine keys

Hi,

Quoted 9 lines · James Bottomley
Quoted 4 lines
Quoted 2 lines
 configure.ac                                  |   5 +
 src/openvpn/crypto_openssl.c                  |   1 +

These two hunks should go to the first patch.

The configure.ac one adds the test Makefile ... it can't move. The other pieces figure out what the dynamic extension is, which is only used by the test, so I think everything in the configure.ac change is in the right place.

Yeah, I misread the part about

+AM_CONDITIONAL([OPENSSL_ENGINE], [test "${have_openssl_engine}" = "yes"])

and assumed it's needed for the #ifdef in the openvpn code.

Quoted 5 lines

The second is a bit more problematic: Right at the moment openvpn is blind to local openssl configuration files. To make the test work, it needs to be able to add an engine, which requires a local file, so it's only function is to enable the test, but it does add to openvpn capabilities, I suppose.

I think it should go to the first patch - so "all changes to openvpn behaviour" are caused by this patch, and the test patch will only add testing, not change behaviour.

On the actual patch itself, I do not understand enough of the engine stuff to fully ACK it. Arne or Steffan understand OpenSSL better.

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...