Skip to contentSkip to conversation

nvme: In-kernel TLS support for TCP

PATCH Superseded Hannes Reinecke 4 participants 69 messages Mar 29, 2023 – Apr 17, 2023

Subsystems (8)

v2 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v2 cover nvme: In-kernel TLS support for TCP

Hi all,

finally I've managed to put all things together and enable in-kernel TLS support for NVMe-over-TCP.

The patchset is based on the TLS upcall mechanism from Chuck Lever (cf '[PATCH v7 0/2] Another crack at a handshake upcall mechanism' posted to the linux netdev list), and requires the 'tlshd' userspace daemon (https://github.com/oracle/ktls-utils) for the actual TLS handshake. Changes for nvme-cli are already included in the upstream repository.

Theory of operation:
A dedicated '.nvme' keyring is created to hold the pre-shared keys (PSKs) for the TLS handshake. Keys will have to be provisioned before TLS handshake is attempted; that can be done with the 'nvme gen-tls-key' command for nvme-cli (patches are already merged upstream).
After connection to the remote TCP port the client side will use the 'best' PSK (as inferred from the NVMe TCP spec) or the PSK specified by the '--tls_key' option to nvme-cli and call the TLS userspace daemon to initiate a TLS handshake.
The server side will then invoke the TLS userspace daemon to run the TLS handshake.
If the TLS handshake succeeds the userspace daemon will be activating kTLS on the socket, and control is passed back to the kernel.

To make this work I had to implement the 'read_sock()' functionality for TLS; it seems to be holding up well enough (for me), but it really could do with reviews from persons with more network stack knowledge.

As usual, comments and reviews are welcome.

Changes to the original RFC:
- Add a CONFIG_NVME_TLS config option
- Use a single PSK for the TLS handshake
- Make TLS connections mandatory
- Do not peek messages for the server
- Simplify data_ready callback
- Implement read_sock() for TLS

Hannes Reinecke (18):
nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
nvme-keyring: define a 'psk' keytype
nvme: add TCP TSAS definitions
nvme-tcp: add definitions for TLS cipher suites net/tls: implement ->read_sock()
nvme/tcp: allocate socket file
nvme-keyring: implement nvme_tls_psk_default() security/keys: export key_lookup()
nvme-tcp: enable TLS handshake upcall
nvme-tcp: fixup send workflow for kTLS
nvme-tcp: control message handling for recvmsg()
nvme-fabrics: parse options 'keyring' and 'tls_key'
nvmet: make TCP sectype settable via configfs
nvmet-tcp: allocate socket file
nvmet-tcp: enable TLS handshake upcall
nvmet-tcp: rework sendpage for kTLS
nvmet-tcp: control messages for recvmsg()
nvmet-tcp: add configfs attribute 'param_keyring'

 drivers/nvme/common/Kconfig    |   9 ++
 drivers/nvme/common/Makefile   |   1 +
 drivers/nvme/common/keyring.c  | 181 ++++++++++++++++++++++
 drivers/nvme/host/core.c       |  44 +++++-
 drivers/nvme/host/fabrics.c    |  90 ++++++++++-
 drivers/nvme/host/fabrics.h    |   9 ++
 drivers/nvme/host/nvme.h       |   4 +-
 drivers/nvme/host/tcp.c        | 213 +++++++++++++++++++++++---
 drivers/nvme/target/configfs.c | 165 ++++++++++++++++++++-
 drivers/nvme/target/nvmet.h    |   1 +
 drivers/nvme/target/tcp.c      | 264 ++++++++++++++++++++++++++++++---
 include/linux/nvme-keyring.h   |  20 +++
 include/linux/nvme-tcp.h       |   6 +
 include/linux/nvme.h           |  10 ++
 net/tls/tls.h                  |   2 +
 net/tls/tls_main.c             |   2 +
 net/tls/tls_sw.c               |  71 +++++++++
 security/keys/key.c            |   1 +
 18 files changed, 1050 insertions(+), 43 deletions(-)
 create mode 100644 drivers/nvme/common/keyring.c
 create mode 100644 include/linux/nvme-keyring.h
Signature · 1 line
2.35.3
#1 lore raw

PATCH v2 1/18 nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS

Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option NVME_TLS to enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.

---
 drivers/nvme/common/Kconfig   |  9 +++++++++
 drivers/nvme/common/Makefile  |  1 +
 drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c      | 19 +++++++++++++++---
 include/linux/nvme-keyring.h  | 12 ++++++++++++
 5 files changed, 74 insertions(+), 3 deletions(-)
 create mode 100644 drivers/nvme/common/keyring.c
 create mode 100644 include/linux/nvme-keyring.h
drivers/nvme/common/Kconfig +9 -0
index 4514f44362dd..b6fff16da1fb 100644
@@ -2,3 +2,12 @@
 
 config NVME_COMMON
        tristate
+
+config NVME_TLS
+	bool "NVMe/TCP TLS encryption support"
+	depends on NVME_COMMON
+	select KEYS
+	help
+	  Enables TLS encryption for NVMe/TCP using the netlink handshake API.
+
+	  If unsure, say N.
drivers/nvme/common/Makefile +1 -0
index 720c625b8a52..4bf8c08293f3 100644
@@ -5,3 +5,4 @@ ccflags-y			+= -I$(src)
 obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
 
 nvme-common-y			+= auth.o
+nvme-common-$(CONFIG_NVME_TLS)	+= keyring.o
drivers/nvme/common/keyring.c new file +36 -0
new file mode 100644
index 000000000000..3a6e8a0b38e2
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2020 Hannes Reinecke, SUSE Linux
+ */
+
+#include <linux/module.h>
+#include <linux/nvme.h>
+#include <linux/seq_file.h>
+#include <linux/key-type.h>
+#include <keys/user-type.h>
+
+static struct key *nvme_keyring;
+
+int nvme_keyring_init(void)
+{
+	int err;
+
+	nvme_keyring = keyring_alloc(".nvme",
+				     GLOBAL_ROOT_UID, GLOBAL_ROOT_GID,
+				     current_cred(),
+				     (KEY_POS_ALL & ~KEY_POS_SETATTR) |
+				     (KEY_USR_ALL & ~KEY_USR_SETATTR),
+				     KEY_ALLOC_NOT_IN_QUOTA, NULL, NULL);
+	if (IS_ERR(nvme_keyring))
+		return PTR_ERR(nvme_keyring);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_keyring_init);
+
+void nvme_keyring_exit(void)
+{
+	key_revoke(nvme_keyring);
+	key_put(nvme_keyring);
+}
+EXPORT_SYMBOL_GPL(nvme_keyring_exit);
drivers/nvme/host/core.c +16 -3
index d4be525f8100..416d0a898f56 100644
@@ -25,6 +25,9 @@
 #include "nvme.h"
 #include "fabrics.h"
 #include <linux/nvme-auth.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#endif
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
 		return 0;
 #endif
-
 	return a->mode;
 }
 
@@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void)
 		result = PTR_ERR(nvme_ns_chr_class);
 		goto unregister_generic_ns;
 	}
-
-	result = nvme_init_auth();
+#ifdef CONFIG_NVME_TLS
+	result = nvme_keyring_init();
 	if (result)
 		goto destroy_ns_chr;
+#endif
+	result = nvme_init_auth();
+	if (result)
+		goto keyring_exit;
 	return 0;
 
+keyring_exit:
+#ifdef CONFIG_NVME_TLS
+	nvme_keyring_exit();
 destroy_ns_chr:
+#endif
 	class_destroy(nvme_ns_chr_class);
 unregister_generic_ns:
 	unregister_chrdev_region(nvme_ns_chr_devt, NVME_MINORS);
@@ -5443,6 +5453,9 @@ static int __init nvme_core_init(void)
 static void __exit nvme_core_exit(void)
 {
 	nvme_exit_auth();
+#ifdef CONFIG_NVME_TLS
+	nvme_keyring_exit();
+#endif
 	class_destroy(nvme_ns_chr_class);
 	class_destroy(nvme_subsys_class);
 	class_destroy(nvme_class);
include/linux/nvme-keyring.h new file +12 -0
new file mode 100644
index 000000000000..a875c06cc922
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_KEYRING_H
+#define _NVME_KEYRING_H
+
+int nvme_keyring_init(void);
+void nvme_keyring_exit(void);
+
+#endif
Signature · 1 line
2.35.3
maintainer
#2 lore raw

Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS

Quoted 23 lines

Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option NVME_TLS to enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.

---
  drivers/nvme/common/Kconfig   |  9 +++++++++
  drivers/nvme/common/Makefile  |  1 +
  drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
  drivers/nvme/host/core.c      | 19 +++++++++++++++---
  include/linux/nvme-keyring.h  | 12 ++++++++++++
  5 files changed, 74 insertions(+), 3 deletions(-)

create mode 100644 drivers/nvme/common/keyring.c create mode 100644 include/linux/nvme-keyring.h

drivers/nvme/common/Kconfig +0 -0
index 4514f44362dd..b6fff16da1fb 100644

drivers/nvme/common/Kconfig line 8

Quoted 7 lines
drivers/nvme/common/Kconfig +4 -0
@@ -2,3 +2,12 @@
 
  config NVME_COMMON
         tristate
+
+config NVME_TLS
+	bool "NVMe/TCP TLS encryption support"
+	depends on NVME_COMMON

depends on TLS as well? Or maybe select would be more appropriate?

drivers/nvme/common/Kconfig line 13

Quoted 5 lines
drivers/nvme/common/Kconfig +5 -0
+	select KEYS
+	help
+	  Enables TLS encryption for NVMe/TCP using the netlink handshake API.
+
+	  If unsure, say N.

Would it not make sense to default it to y?

#3 lore raw

Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS

Quoted 18 lines · Sagi Grimberg
Quoted 18 lines

Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option NVME_TLS to enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.

---
  drivers/nvme/common/Kconfig   |  9 +++++++++ drivers/nvme/common/Makefile  |  1 + drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/core.c      | 19 +++++++++++++++--- include/linux/nvme-keyring.h  | 12 ++++++++++++

  5 files changed, 74 insertions(+), 3 deletions(-)

  create mode 100644 drivers/nvme/common/keyring.c create mode 100644 include/linux/nvme-keyring.h

Quoted 14 lines

drivers/nvme/common/Kconfig line 8

Quoted 11 lines
drivers/nvme/common/Kconfig +0 -0
index 4514f44362dd..b6fff16da1fb 100644
@@ -2,3 +2,12 @@

  config NVME_COMMON
         tristate

drivers/nvme/common/Kconfig +4 -0
+
+config NVME_TLS
+    bool "NVMe/TCP TLS encryption support"
+    depends on NVME_COMMON

depends on TLS as well? Or maybe select would be more appropriate?

I would argue 'depends', as we cannot use the standard session/process/thread keyrings, as they'll vanish after the initial 'connect', and then there's no keyring to lookup keys for subsection reconnects.

Quoted 8 lines
Quoted 6 lines

+    select KEYS
+    help
+      Enables TLS encryption for NVMe/TCP using the netlink handshake API.
+
+      If unsure, say N.

Would it not make sense to default it to y?

Yeah.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#4 lore raw

Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS

In reply to Hannes Reinecke
Quoted 23 lines

Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option NVME_TLS to enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.

---
  drivers/nvme/common/Kconfig   |  9 +++++++++
  drivers/nvme/common/Makefile  |  1 +
  drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++
  drivers/nvme/host/core.c      | 19 +++++++++++++++---
  include/linux/nvme-keyring.h  | 12 ++++++++++++
  5 files changed, 74 insertions(+), 3 deletions(-)

create mode 100644 drivers/nvme/common/keyring.c create mode 100644 include/linux/nvme-keyring.h

drivers/nvme/common/Kconfig +0 -0
index 4514f44362dd..b6fff16da1fb 100644

drivers/nvme/common/Kconfig line 6

Quoted 5 lines
drivers/nvme/common/Kconfig +2 -0
@@ -2,3 +2,12 @@
 
  config NVME_COMMON
         tristate
+
+config NVME_TLS

Better to call it NVME_TCP_TLS... although not a biggie, should be well understood this is tcp

#5 lore raw

Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS

Quoted 18 lines · Sagi Grimberg
Quoted 18 lines

Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option NVME_TLS to enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.

---
  drivers/nvme/common/Kconfig   |  9 +++++++++ drivers/nvme/common/Makefile  |  1 + drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/core.c      | 19 +++++++++++++++--- include/linux/nvme-keyring.h  | 12 ++++++++++++

  5 files changed, 74 insertions(+), 3 deletions(-)

  create mode 100644 drivers/nvme/common/keyring.c create mode 100644 include/linux/nvme-keyring.h

Quoted 13 lines

drivers/nvme/common/Kconfig line 6

Quoted 9 lines
drivers/nvme/common/Kconfig +0 -0
index 4514f44362dd..b6fff16da1fb 100644
@@ -2,3 +2,12 @@

  config NVME_COMMON
         tristate

drivers/nvme/common/Kconfig +2 -0
+
+config NVME_TLS

Better to call it NVME_TCP_TLS... although not a biggie, should be well understood this is tcp

Sure.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
#6 lore raw

Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS

In reply to Hannes Reinecke
Quoted 20 lines · Hannes Reinecke

+++ b/drivers/nvme/host/core.c

drivers/nvme/host/core.c +3 -1
@@ -25,6 +25,9 @@
 #include "nvme.h"
 #include "fabrics.h"
 #include <linux/nvme-auth.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#endif
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
 		return 0;
 #endif
-
 	return a->mode;
 }
 

drivers/nvme/host/core.c static int __init nvme_core_init(void) · line 5423

Quoted 10 lines
drivers/nvme/host/core.c +3 -2
@@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void)
 		result = PTR_ERR(nvme_ns_chr_class);
 		goto unregister_generic_ns;
 	}
-
-	result = nvme_init_auth();
+#ifdef CONFIG_NVME_TLS
+	result = nvme_keyring_init();
 	if (result)
 		goto destroy_ns_chr;
+#endif

Would it make sense to introduce empty nvme_keyring_init() function for !CONFIG_NVME_TLS and avoid this ifdefery here?

maintainer
#7 lore raw

Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS

Quoted 20 lines · Daniel Wagner
Quoted 19 lines · Hannes Reinecke

+++ b/drivers/nvme/host/core.c

drivers/nvme/host/core.c +3 -1
@@ -25,6 +25,9 @@
  #include "nvme.h"
  #include "fabrics.h"
  #include <linux/nvme-auth.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#endif
 
  #define CREATE_TRACE_POINTS
  #include "trace.h"
@@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
  	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
  		return 0;
  #endif
-
  	return a->mode;
  }
 
Quoted 14 lines

drivers/nvme/host/core.c static int __init nvme_core_init(void) · line 5423

Quoted 11 lines
drivers/nvme/host/core.c +3 -2
@@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void)
  		result = PTR_ERR(nvme_ns_chr_class);
  		goto unregister_generic_ns;
  	}
-
-	result = nvme_init_auth();
+#ifdef CONFIG_NVME_TLS
+	result = nvme_keyring_init();
  	if (result)
  		goto destroy_ns_chr;
+#endif

Would it make sense to introduce empty nvme_keyring_init() function for !CONFIG_NVME_TLS and avoid this ifdefery here?

Yes.

#8 lore raw

PATCH v2 2/18 nvme-keyring: define a 'psk' keytype

In reply to Hannes Reinecke

Define a 'psk' keytype to hold the NVMe TLS PSKs.

---
 drivers/nvme/common/keyring.c | 98 +++++++++++++++++++++++++++++++++++
 include/linux/nvme-keyring.h  |  6 +++
 2 files changed, 104 insertions(+)
drivers/nvme/common/keyring.c +98 -0
index 3a6e8a0b38e2..4ac33538f839 100644
@@ -5,12 +5,104 @@
 
 #include <linux/module.h>
 #include <linux/nvme.h>
+#include <linux/nvme-tcp.h>
 #include <linux/seq_file.h>
 #include <linux/key-type.h>
 #include <keys/user-type.h>
 
 static struct key *nvme_keyring;
 
+key_serial_t nvme_keyring_id(void)
+{
+	return nvme_keyring->serial;
+}
+EXPORT_SYMBOL_GPL(nvme_keyring_id);
+
+static void nvme_tls_psk_describe(const struct key *key, struct seq_file *m)
+{
+	seq_puts(m, key->description);
+	seq_printf(m, ": %u", key->datalen);
+}
+
+static bool nvme_tls_psk_match(const struct key *key,
+			       const struct key_match_data *match_data)
+{
+	const char *match_id;
+	size_t match_len;
+
+	if (!key->description) {
+		pr_debug("%s: no key description\n", __func__);
+		return false;
+	}
+	match_len = strlen(key->description);
+	pr_debug("%s: id %s len %zd\n", __func__, key->description, match_len);
+
+	if (!match_data->raw_data) {
+		pr_debug("%s: no match data\n", __func__);
+		return false;
+	}
+	match_id = match_data->raw_data;
+	pr_debug("%s: match '%s' '%s' len %zd\n",
+		 __func__, match_id, key->description, match_len);
+	return !memcmp(key->description, match_id, match_len);
+}
+
+static int nvme_tls_psk_match_preparse(struct key_match_data *match_data)
+{
+	match_data->lookup_type = KEYRING_SEARCH_LOOKUP_ITERATE;
+	match_data->cmp = nvme_tls_psk_match;
+	return 0;
+}
+
+static struct key_type nvme_tls_psk_key_type = {
+	.name           = "psk",
+	.flags          = KEY_TYPE_NET_DOMAIN,
+	.preparse       = user_preparse,
+	.free_preparse  = user_free_preparse,
+	.match_preparse = nvme_tls_psk_match_preparse,
+	.instantiate    = generic_key_instantiate,
+	.revoke         = user_revoke,
+	.destroy        = user_destroy,
+	.describe       = nvme_tls_psk_describe,
+	.read           = user_read,
+};
+
+struct key *nvme_tls_psk_lookup(struct key *keyring,
+		const char *hostnqn, const char *subnqn,
+		int hmac, bool generated)
+{
+	char *identity;
+	size_t identity_len = (NVMF_NQN_SIZE) * 2 + 11;
+	key_ref_t keyref;
+	key_serial_t keyring_id;
+
+	identity = kzalloc(identity_len, GFP_KERNEL);
+	if (!identity)
+		return ERR_PTR(-ENOMEM);
+
+	snprintf(identity, identity_len, "NVMe0%c%02d %s %s",
+		 generated ? 'G' : 'R', hmac, hostnqn, subnqn);
+
+	if (!keyring)
+		keyring = nvme_keyring;
+	keyring_id = key_serial(keyring);
+	pr_debug("keyring %x lookup tls psk '%s'\n",
+		 keyring_id, identity);
+	keyref = keyring_search(make_key_ref(keyring, true),
+				&nvme_tls_psk_key_type,
+				identity, false);
+	if (IS_ERR(keyref)) {
+		pr_debug("lookup tls psk '%s' failed, error %ld\n",
+			 identity, PTR_ERR(keyref));
+		kfree(identity);
+		return ERR_PTR(-ENOKEY);
+	}
+	kfree(identity);
+
+	return key_ref_to_ptr(keyref);
+}
+EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
+
 int nvme_keyring_init(void)
 {
 	int err;
@@ -24,12 +116,18 @@ int nvme_keyring_init(void)
 	if (IS_ERR(nvme_keyring))
 		return PTR_ERR(nvme_keyring);
 
+	err = register_key_type(&nvme_tls_psk_key_type);
+	if (err) {
+		key_put(nvme_keyring);
+		return err;
+	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(nvme_keyring_init);
 
 void nvme_keyring_exit(void)
 {
+	unregister_key_type(&nvme_tls_psk_key_type);
 	key_revoke(nvme_keyring);
 	key_put(nvme_keyring);
 }
include/linux/nvme-keyring.h +6 -0
index a875c06cc922..5be60485ddfa 100644
@@ -6,6 +6,12 @@
 #ifndef _NVME_KEYRING_H
 #define _NVME_KEYRING_H
 
+struct key *nvme_tls_psk_lookup(struct key *keyring,
+				const char *hostnqn, const char *subnqn,
+				int hmac, bool generated);
+
+key_serial_t nvme_keyring_id(void);
+
 int nvme_keyring_init(void);
 void nvme_keyring_exit(void);
 
Signature · 1 line
2.35.3
#9 lore raw

PATCH v2 3/18 nvme: add TCP TSAS definitions

In reply to Hannes Reinecke
---
 include/linux/nvme.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/linux/nvme.h +10 -0
index 779507ac750b..ea961ca2022d 100644
@@ -108,6 +108,13 @@ enum {
 	NVMF_RDMA_CMS_RDMA_CM	= 1, /* Sockets based endpoint addressing */
 };
 
+/* TSAS SECTYPE for TCP transport */
+enum {
+	NVMF_TCP_SECTYPE_NONE = 0, /* No Security */
+	NVMF_TCP_SECTYPE_TLS12 = 1, /* TLSv1.2, NVMe-oF 1.1 and NVMe-TCP 3.6.1.1 */
+	NVMF_TCP_SECTYPE_TLS13 = 2, /* TLSv1.3, NVMe-oF 1.1 and NVMe-TCP 3.6.1.1 */
+};
+
 #define NVME_AQ_DEPTH		32
 #define NVME_NR_AEN_COMMANDS	1
 #define NVME_AQ_BLK_MQ_DEPTH	(NVME_AQ_DEPTH - NVME_NR_AEN_COMMANDS)
@@ -1458,6 +1465,9 @@ struct nvmf_disc_rsp_page_entry {
 			__u16	pkey;
 			__u8	resv10[246];
 		} rdma;
+		struct tcp {
+			__u8	sectype;
+		} tcp;
 	} tsas;
 };
 
Signature · 1 line
2.35.3
#10 lore raw

PATCH v2 4/18 nvme-tcp: add definitions for TLS cipher suites

In reply to Hannes Reinecke
---
 include/linux/nvme-tcp.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/linux/nvme-tcp.h +6 -0
index 75470159a194..19ca863813f1 100644
@@ -18,6 +18,12 @@ enum nvme_tcp_pfv {
 	NVME_TCP_PFV_1_0 = 0x0,
 };
 
+enum nvme_tcp_tls_cipher {
+	NVME_TCP_TLS_CIPHER_INVALID     = 0,
+	NVME_TCP_TLS_CIPHER_SHA256      = 1,
+	NVME_TCP_TLS_CIPHER_SHA384      = 2,
+};
+
 enum nvme_tcp_fatal_error_status {
 	NVME_TCP_FES_INVALID_PDU_HDR		= 0x01,
 	NVME_TCP_FES_PDU_SEQ_ERR		= 0x02,
Signature · 1 line
2.35.3
#11 lore raw

PATCH v2 5/18 net/tls: implement ->read_sock()

In reply to Hannes Reinecke

Implement ->read_sock() function for use with nvme-tcp.

---
 net/tls/tls.h      |  2 ++
 net/tls/tls_main.c |  2 ++
 net/tls/tls_sw.c   | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+)
net/tls/tls.h +2 -0
index 804c3880d028..a5bf3a9ce142 100644
@@ -113,6 +113,8 @@ bool tls_sw_sock_is_readable(struct sock *sk);
 ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,
 			   struct pipe_inode_info *pipe,
 			   size_t len, unsigned int flags);
+int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
+		     sk_read_actor_t read_actor);
 
 int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
 int tls_device_sendpage(struct sock *sk, struct page *page,
net/tls/tls_main.c +2 -0
index 3735cb00905d..9b097f2902f6 100644
@@ -934,9 +934,11 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG]
 
 	ops[TLS_BASE][TLS_SW  ] = ops[TLS_BASE][TLS_BASE];
 	ops[TLS_BASE][TLS_SW  ].splice_read	= tls_sw_splice_read;
+	ops[TLS_BASE][TLS_SW  ].read_sock	= tls_sw_read_sock;
 
 	ops[TLS_SW  ][TLS_SW  ] = ops[TLS_SW  ][TLS_BASE];
 	ops[TLS_SW  ][TLS_SW  ].splice_read	= tls_sw_splice_read;
+	ops[TLS_SW  ][TLS_SW  ].read_sock	= tls_sw_read_sock;
 
 #ifdef CONFIG_TLS_DEVICE
 	ops[TLS_HW  ][TLS_BASE] = ops[TLS_BASE][TLS_BASE];
net/tls/tls_sw.c +71 -0

77 lines not on this page · raw message

Signature · 1 line
2.35.3
#13 lore raw

Re: [PATCH 05/18] net/tls: implement ->read_sock()

Quoted 9 lines · Sagi Grimberg
Quoted 2 lines

Implement ->read_sock() function for use with nvme-tcp.

Nice.

Quoted 2 lines

Would be great to see a tls_device implementation as well.

Sheesh. But first we would need to update the tls_device implementation to TLS 1.3:

int tls_set_device_offload(struct sock *sk, struct tls_context *ctx) [ .. ]

        crypto_info = &ctx->crypto_send.info;
        if (crypto_info->version != TLS_1_2_VERSION) {
                rc = -EOPNOTSUPP;
                goto release_netdev;
        }

which I'd rather defer to Boris or whoever maintains the hardware.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#14 lore raw

Re: [PATCH 05/18] net/tls: implement ->read_sock()

Quoted 11 lines
Quoted 9 lines
Quoted 2 lines

Implement ->read_sock() function for use with nvme-tcp.

Nice.

Quoted 2 lines

Would be great to see a tls_device implementation as well.

Sheesh. But first we would need to update the tls_device implementation to TLS 1.3:

;)

The comment was for the nvidia folks..

Quoted 9 lines

int tls_set_device_offload(struct sock *sk, struct tls_context *ctx) [ .. ]
        crypto_info = &ctx->crypto_send.info;
        if (crypto_info->version != TLS_1_2_VERSION) { rc = -EOPNOTSUPP;
                goto release_netdev;
        }

which I'd rather defer to Boris or whoever maintains the hardware

maintainer
#15 lore raw

Re: [PATCH 05/18] net/tls: implement ->read_sock()

In reply to Hannes Reinecke
Quoted 78 lines

Implement ->read_sock() function for use with nvme-tcp.

---
  net/tls/tls.h      |  2 ++
  net/tls/tls_main.c |  2 ++
  net/tls/tls_sw.c   | 71 ++++++++++++++++++++++++++++++++++++++++++++++
  3 files changed, 75 insertions(+)
net/tls/tls.h +2 -0
index 804c3880d028..a5bf3a9ce142 100644
@@ -113,6 +113,8 @@ bool tls_sw_sock_is_readable(struct sock *sk);
  ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,
  			   struct pipe_inode_info *pipe,
  			   size_t len, unsigned int flags);
+int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
+		     sk_read_actor_t read_actor);
 
  int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
  int tls_device_sendpage(struct sock *sk, struct page *page,
net/tls/tls_main.c +2 -0
index 3735cb00905d..9b097f2902f6 100644
@@ -934,9 +934,11 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG]
 
  	ops[TLS_BASE][TLS_SW  ] = ops[TLS_BASE][TLS_BASE];
  	ops[TLS_BASE][TLS_SW  ].splice_read	= tls_sw_splice_read;
+	ops[TLS_BASE][TLS_SW  ].read_sock	= tls_sw_read_sock;
 
  	ops[TLS_SW  ][TLS_SW  ] = ops[TLS_SW  ][TLS_BASE];
  	ops[TLS_SW  ][TLS_SW  ].splice_read	= tls_sw_splice_read;
+	ops[TLS_SW  ][TLS_SW  ].read_sock	= tls_sw_read_sock;
 
  #ifdef CONFIG_TLS_DEVICE
  	ops[TLS_HW  ][TLS_BASE] = ops[TLS_BASE][TLS_BASE];
net/tls/tls_sw.c +30 -0
index 782d3701b86f..77d57066daf4 100644
@@ -2210,6 +2210,77 @@ ssize_t tls_sw_splice_read(struct socket *sock,  loff_t *ppos,
  	goto splice_read_end;
  }
 
+int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,
+		     sk_read_actor_t read_actor)
+{
+	struct tls_context *tls_ctx = tls_get_ctx(sk);
+	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
+	struct strp_msg *rxm = NULL;
+	struct tls_msg *tlm;
+	struct sk_buff *skb;
+	ssize_t copied = 0;
+	int err, used;
+
+	if (!skb_queue_empty(&ctx->rx_list)) {
+		skb = __skb_dequeue(&ctx->rx_list);
+	} else {
+		struct tls_decrypt_arg darg;
+
+		err = tls_rx_rec_wait(sk, NULL, true, true);
+		if (err <= 0)
+			return err;
+
+		memset(&darg.inargs, 0, sizeof(darg.inargs));
+
+		err = tls_rx_one_record(sk, NULL, &darg);
+		if (err < 0) {
+			tls_err_abort(sk, -EBADMSG);
+			return err;
+		}
+
+		tls_rx_rec_done(ctx);
+		skb = darg.skb;

net/tls/tls_sw.c ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos, · line 2252

Quoted 10 lines
net/tls/tls_sw.c +10 -0
+	}
+
+	do {
+		rxm = strp_msg(skb);
+		tlm = tls_msg(skb);
+
+		/* read_sock does not support reading control messages */
+		if (tlm->control != TLS_RECORD_TYPE_DATA) {
+			err = -EINVAL;
+			goto read_sock_requeue;

In the general case, this could just pass in the skb to recv_actor and have it access the tlm and decide what to do with it... (will require moving tls_msg() to include/net/strparser.h)

I agree that for nvme-tcp this would not have anything useful other than breaking the connection and re-establish it later...

#16 lore raw

PATCH v2 6/18 nvme/tcp: allocate socket file

In reply to Hannes Reinecke

When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.

---
 drivers/nvme/host/tcp.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/nvme/host/tcp.c +15 -2
index 42c0598c31f2..fddf785aba74 100644
@@ -115,6 +115,7 @@ enum nvme_tcp_recv_state {
 struct nvme_tcp_ctrl;
 struct nvme_tcp_queue {
 	struct socket		*sock;
+	struct file		*sock_file;
 	struct work_struct	io_work;
 	int			io_cpu;
 
@@ -1330,7 +1331,10 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	}
 
 	noreclaim_flag = memalloc_noreclaim_save();
-	sock_release(queue->sock);
+	/* ->sock will be released by fput() */
+	fput(queue->sock_file);
+	queue->sock_file = NULL;
+	queue->sock = NULL;
 	memalloc_noreclaim_restore(noreclaim_flag);
 
 	kfree(queue->pdu);
@@ -1526,6 +1530,13 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
 		goto err_destroy_mutex;
 	}
 
+	queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL);
+	if (IS_ERR(queue->sock_file)) {
+		sock_release(queue->sock);
+		ret = PTR_ERR(queue->sock_file);
+		queue->sock_file = NULL;
+		goto err_sock;
+	}
 	nvme_tcp_reclassify_socket(queue->sock);
 
 	/* Single syn retry */
@@ -1647,7 +1658,9 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
 	if (queue->hdr_digest || queue->data_digest)
 		nvme_tcp_free_crypto(queue);
 err_sock:
-	sock_release(queue->sock);
+	/* ->sock will be released by fput() */
+	fput(queue->sock_file);
+	queue->sock_file = NULL;
 	queue->sock = NULL;
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
Signature · 1 line
2.35.3
maintainer
#17 lore raw

Re: [PATCH 06/18] nvme/tcp: allocate socket file

Quoted 13 lines

When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.

---
  drivers/nvme/host/tcp.c | 17 +++++++++++++++--
  1 file changed, 15 insertions(+), 2 deletions(-)
drivers/nvme/host/tcp.c +0 -0
index 42c0598c31f2..fddf785aba74 100644

drivers/nvme/host/tcp.c enum nvme_tcp_recv_state { · line 118

Quoted 4 lines
drivers/nvme/host/tcp.c +1 -0
@@ -115,6 +115,7 @@ enum nvme_tcp_recv_state {
  struct nvme_tcp_ctrl;
  struct nvme_tcp_queue {
  	struct socket		*sock;
+	struct file		*sock_file;

I think that sock->file is storing the file, no need to store it as well.

drivers/nvme/host/tcp.c static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) · line 1337

Quoted 12 lines
drivers/nvme/host/tcp.c +4 -1
  	struct work_struct	io_work;
  	int			io_cpu;
 
@@ -1330,7 +1331,10 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
  	}
 
  	noreclaim_flag = memalloc_noreclaim_save();
-	sock_release(queue->sock);
+	/* ->sock will be released by fput() */
+	fput(queue->sock_file);
+	queue->sock_file = NULL;
+	queue->sock = NULL;

Is the queue->sock = NULL here required because we allocate a sock_file?

#18 lore raw

PATCH v2 7/18 nvme-keyring: implement nvme_tls_psk_default()

In reply to Hannes Reinecke

Implement a function to select the 'best' PSK for TLS.

---
 drivers/nvme/common/keyring.c | 47 +++++++++++++++++++++++++++++++++++
 include/linux/nvme-keyring.h  |  2 ++
 2 files changed, 49 insertions(+)
drivers/nvme/common/keyring.c +47 -0
index 4ac33538f839..ca36a061bd48 100644
@@ -103,6 +103,53 @@ struct key *nvme_tls_psk_lookup(struct key *keyring,
 }
 EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
 
+/*
+ * NVMe PSK priority list
+ *
+ * 'Retained' PSKs (ie 'generated == false')
+ * should be preferred to 'generated' PSKs,
+ * and SHA-384 should be preferred to SHA-256.
+ */
+struct nvme_psk_priority_list {
+	bool generated;
+	enum nvme_tcp_tls_cipher cipher;
+} nvme_psk_prio[] = {
+	{ .generated = false,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
+	{ .generated = false,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
+	{ .generated = true,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
+	{ .generated = true,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
+};
+
+/*
+ * nvme_tls_psk_default - Return 'best' PSK to use for TLS ClientHello
+ */
+key_serial_t nvme_tls_psk_default(struct key *keyring,
+		      const char *hostnqn, const char *subnqn)
+{
+	struct key *tls_key;
+	key_serial_t tls_key_id;
+	int prio;
+
+	for (prio = 0; prio < ARRAY_SIZE(nvme_psk_prio); prio++) {
+		bool generated = nvme_psk_prio[prio].generated;
+		enum nvme_tcp_tls_cipher cipher = nvme_psk_prio[prio].cipher;
+
+		tls_key = nvme_tls_psk_lookup(keyring, hostnqn, subnqn,
+					      cipher, generated);
+		if (!IS_ERR(tls_key)) {
+			tls_key_id = tls_key->serial;
+			key_put(tls_key);
+			return tls_key_id;
+		}
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_tls_psk_default);
+
 int nvme_keyring_init(void)
 {
 	int err;
include/linux/nvme-keyring.h +2 -0
index 5be60485ddfa..5293e0a90167 100644
@@ -9,6 +9,8 @@
 struct key *nvme_tls_psk_lookup(struct key *keyring,
 				const char *hostnqn, const char *subnqn,
 				int hmac, bool generated);
+key_serial_t nvme_tls_psk_default(struct key *keyring,
+				  const char *hostnqn, const char *subnqn);
 
 key_serial_t nvme_keyring_id(void);
 
Signature · 1 line
2.35.3
maintainer
#19 lore raw

Re: [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default()

Quoted 14 lines · Hannes Reinecke

Implement a function to select the 'best' PSK for TLS.

---
  drivers/nvme/common/keyring.c | 47 +++++++++++++++++++++++++++++++++++
  include/linux/nvme-keyring.h  |  2 ++
  2 files changed, 49 insertions(+)
drivers/nvme/common/keyring.c +0 -0
index 4ac33538f839..ca36a061bd48 100644
@@ -103,6 +103,53 @@ struct key *nvme_tls_psk_lookup(struct key *keyring,
  }

drivers/nvme/common/keyring.c struct key *nvme_tls_psk_lookup(struct key *keyring, · line 113

Quoted 10 lines
drivers/nvme/common/keyring.c +8 -0
  EXPORT_SYMBOL_GPL(nvme_tls_psk_lookup);
 
+/*
+ * NVMe PSK priority list
+ *
+ * 'Retained' PSKs (ie 'generated == false')
+ * should be preferred to 'generated' PSKs,
+ * and SHA-384 should be preferred to SHA-256.
+ */
+struct nvme_psk_priority_list {

nvme_tls_psk_priority_list

drivers/nvme/common/keyring.c struct key *nvme_tls_psk_lookup(struct key *keyring, · line 116

Quoted 3 lines
drivers/nvme/common/keyring.c +3 -0
+	bool generated;
+	enum nvme_tcp_tls_cipher cipher;
+} nvme_psk_prio[] = {

nvme_tls_psk_prio

drivers/nvme/common/keyring.c struct key *nvme_tls_psk_lookup(struct key *keyring, · line 128

Quoted 12 lines
drivers/nvme/common/keyring.c +12 -0
+	{ .generated = false,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
+	{ .generated = false,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
+	{ .generated = true,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA384, },
+	{ .generated = true,
+	  .cipher = NVME_TCP_TLS_CIPHER_SHA256, },
+};
+
+/*
+ * nvme_tls_psk_default - Return 'best' PSK to use for TLS ClientHello

s/'best'/the preferred/...

#20 lore raw

PATCH v2 8/18 security/keys: export key_lookup()

In reply to Hannes Reinecke

For in-kernel consumers one cannot readily assign a user (eg when running from a workqueue), so the normal key search permissions cannot be applied.
This patch exports the 'key_lookup()' function for a simple lookup of keys without checking for permissions.

---
 security/keys/key.c | 1 +
 1 file changed, 1 insertion(+)
security/keys/key.c +1 -0
index 5c0c7df833f8..bd1b7d45df90 100644
@@ -693,6 +693,7 @@ struct key *key_lookup(key_serial_t id)
 	spin_unlock(&key_serial_lock);
 	return key;
 }
+EXPORT_SYMBOL_GPL(key_lookup);
 
 /*
  * Find and lock the specified key type against removal.
Signature · 1 line
2.35.3
#21 lore raw

PATCH v2 9/18 nvme-tcp: enable TLS handshake upcall

In reply to Hannes Reinecke

Add a fabrics option 'tls' and start the TLS handshake upcall with the default PSK. When TLS is started the PSK key serial number is displayed in the sysfs attribute 'tls_key'

---
 drivers/nvme/host/core.c    |  25 ++++++-
 drivers/nvme/host/fabrics.c |  11 +++
 drivers/nvme/host/fabrics.h |   3 +
 drivers/nvme/host/nvme.h    |   4 +-
 drivers/nvme/host/tcp.c     | 141 ++++++++++++++++++++++++++++++++++--
 5 files changed, 174 insertions(+), 10 deletions(-)
drivers/nvme/host/core.c +24 -1
index 416d0a898f56..869396058e2e 100644
@@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
 	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
 #endif
 
+#ifdef CONFIG_NVME_TLS
+static ssize_t tls_key_show(struct device *dev,
+			    struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	if (!ctrl->tls_key)
+		return 0;
+	return sysfs_emit(buf, "%08x", key_serial(ctrl->tls_key));
+}
+static DEVICE_ATTR_RO(tls_key);
+#endif
+
 static struct attribute *nvme_dev_attrs[] = {
 	&dev_attr_reset_controller.attr,
 	&dev_attr_rescan_controller.attr,
@@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {
 #ifdef CONFIG_NVME_AUTH
 	&dev_attr_dhchap_secret.attr,
 	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
+#ifdef CONFIG_NVME_TLS
+	&dev_attr_tls_key.attr,
 #endif
 	NULL
 };
@@ -3956,6 +3972,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
 		return 0;
 	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
 		return 0;
+#endif
+#ifdef CONFIG_NVME_TLS
+	if (a == &dev_attr_tls_key.attr && !ctrl->opts)
+		return 0;
 #endif
 	return a->mode;
 }
@@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)
 
 	if (!subsys || ctrl->instance != subsys->instance)
 		ida_free(&nvme_instance_ida, ctrl->instance);
-
+#ifdef CONFIG_NVME_TLS
+	if (ctrl->tls_key)
+		key_put(ctrl->tls_key);
+#endif
 	nvme_free_cels(ctrl);
 	nvme_mpath_uninit(ctrl);
 	nvme_auth_stop(ctrl);
drivers/nvme/host/fabrics.c +11 -0
index bbaa04a0c502..3e4f0e45b58f 100644
@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
 	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
 	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
+	{ NVMF_OPT_TLS,			"tls"			},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -632,6 +633,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	opts->hdr_digest = false;
 	opts->data_digest = false;
 	opts->tos = -1; /* < 0 == use transport default */
+	opts->tls = false;
 
 	options = o = kstrdup(buf, GFP_KERNEL);
 	if (!options)
@@ -918,6 +920,15 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			kfree(opts->dhchap_ctrl_secret);
 			opts->dhchap_ctrl_secret = p;
 			break;
+		case NVMF_OPT_TLS:
+#ifdef CONFIG_NVME_TLS
+			opts->tls = true;
+			break;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
drivers/nvme/host/fabrics.h +3 -0
index dcac3df8a5f7..5db36e250e7a 100644
@@ -70,6 +70,7 @@ enum {
 	NVMF_OPT_DISCOVERY	= 1 << 22,
 	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
 	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
+	NVMF_OPT_TLS		= 1 << 25,
 };
 
 /**
@@ -102,6 +103,7 @@ enum {
  * @dhchap_secret: DH-HMAC-CHAP secret
  * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
  *              authentication
+ * @tls:        Start TLS encrypted connections (TCP)
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)
@@ -128,6 +130,7 @@ struct nvmf_ctrl_options {
 	int			max_reconnects;
 	char			*dhchap_secret;
 	char			*dhchap_ctrl_secret;
+	bool			tls;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
drivers/nvme/host/nvme.h +3 -1
index bf46f122e9e1..421d01f849d1 100644
@@ -347,7 +347,9 @@ struct nvme_ctrl {
 	struct nvme_dhchap_key *ctrl_key;
 	u16 transaction;
 #endif
-
+#ifdef CONFIG_NVME_TLS
+	struct key *tls_key;
+#endif
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;
drivers/nvme/host/tcp.c +133 -8

194 lines not on this page · raw message

Signature · 1 line
2.35.3
#22 lore raw

Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall

On Wed, Mar 29, 2023 at 03:59:29PM +0200, Hannes Reinecke wrote:
--- a/drivers/nvme/host/fabrics.c

drivers/nvme/host/fabrics.c static const match_table_t opt_tokens = { · line 614

Quoted 8 lines

+++ b/drivers/nvme/host/fabrics.c

drivers/nvme/host/fabrics.c +1 -0
@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
 	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
 	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
+	{ NVMF_OPT_TLS,			"tls"			},
 	{ NVMF_OPT_ERR,			NULL			}
 };

This should be ifdef out for !CONFIG_NVME_TLS

#23 lore raw

Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall

Quoted 12 lines · Daniel Wagner

On Wed, Mar 29, 2023 at 03:59:29PM +0200, Hannes Reinecke wrote:
--- a/drivers/nvme/host/fabrics.c

drivers/nvme/host/fabrics.c static const match_table_t opt_tokens = { · line 614

Quoted 8 lines

+++ b/drivers/nvme/host/fabrics.c

drivers/nvme/host/fabrics.c +1 -0
@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
  	{ NVMF_OPT_DISCOVERY,		"discovery"		},
  	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
  	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
+	{ NVMF_OPT_TLS,			"tls"			},
  	{ NVMF_OPT_ERR,			NULL			}
  };

This should be ifdef out for !CONFIG_NVME_TLS

No, I did that on purpose.
Error handling in the fabrics option parsing is notoriously bad, and if we disable that nvme-cli will fail with annoying interfaces. As it stands, the 'tls' option will be accepted, but you'll get a kernel message telling you it's not compiled in, and the function returns -EINVAL.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)
maintainer
#24 lore raw

Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall

In reply to Hannes Reinecke
Quoted 93 lines · Hannes Reinecke

Add a fabrics option 'tls' and start the TLS handshake upcall with the default PSK. When TLS is started the PSK key serial number is displayed in the sysfs attribute 'tls_key'

---
  drivers/nvme/host/core.c    |  25 ++++++-
  drivers/nvme/host/fabrics.c |  11 +++
  drivers/nvme/host/fabrics.h |   3 +
  drivers/nvme/host/nvme.h    |   4 +-
  drivers/nvme/host/tcp.c     | 141 ++++++++++++++++++++++++++++++++++--
  5 files changed, 174 insertions(+), 10 deletions(-)
drivers/nvme/host/core.c +24 -1
index 416d0a898f56..869396058e2e 100644
@@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR,
  	nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
  #endif
 
+#ifdef CONFIG_NVME_TLS
+static ssize_t tls_key_show(struct device *dev,
+			    struct device_attribute *attr, char *buf)
+{
+	struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+	if (!ctrl->tls_key)
+		return 0;
+	return sysfs_emit(buf, "%08x", key_serial(ctrl->tls_key));
+}
+static DEVICE_ATTR_RO(tls_key);
+#endif
+
  static struct attribute *nvme_dev_attrs[] = {
  	&dev_attr_reset_controller.attr,
  	&dev_attr_rescan_controller.attr,
@@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {
  #ifdef CONFIG_NVME_AUTH
  	&dev_attr_dhchap_secret.attr,
  	&dev_attr_dhchap_ctrl_secret.attr,
+#endif
+#ifdef CONFIG_NVME_TLS
+	&dev_attr_tls_key.attr,
  #endif
  	NULL
  };
@@ -3956,6 +3972,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,
  		return 0;
  	if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
  		return 0;
+#endif
+#ifdef CONFIG_NVME_TLS
+	if (a == &dev_attr_tls_key.attr && !ctrl->opts)
+		return 0;
  #endif
  	return a->mode;
  }
@@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)
 
  	if (!subsys || ctrl->instance != subsys->instance)
  		ida_free(&nvme_instance_ida, ctrl->instance);
-
+#ifdef CONFIG_NVME_TLS
+	if (ctrl->tls_key)
+		key_put(ctrl->tls_key);
+#endif
  	nvme_free_cels(ctrl);
  	nvme_mpath_uninit(ctrl);
  	nvme_auth_stop(ctrl);
drivers/nvme/host/fabrics.c +2 -0
index bbaa04a0c502..3e4f0e45b58f 100644
@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
  	{ NVMF_OPT_DISCOVERY,		"discovery"		},
  	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
  	{ NVMF_OPT_DHCHAP_CTRL_SECRET,	"dhchap_ctrl_secret=%s"	},
+	{ NVMF_OPT_TLS,			"tls"			},
  	{ NVMF_OPT_ERR,			NULL			}
  };
 
@@ -632,6 +633,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
  	opts->hdr_digest = false;
  	opts->data_digest = false;
  	opts->tos = -1; /* < 0 == use transport default */
+	opts->tls = false;
 
  	options = o = kstrdup(buf, GFP_KERNEL);
  	if (!options)
@@ -918,6 +920,15 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
  			kfree(opts->dhchap_ctrl_secret);
  			opts->dhchap_ctrl_secret = p;

drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct nvmf_ctrl_options *opts, · line 931

Quoted 10 lines
drivers/nvme/host/fabrics.c +9 -0
  			break;
+		case NVMF_OPT_TLS:
+#ifdef CONFIG_NVME_TLS
+			opts->tls = true;
+			break;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif

This can become:

			if (!IS_ENABLED(CONFIG_NVME_TLS)) {
				pr_err("TLS is not supported\n");
				ret = -EINVAL;
				goto out;
			}
			opts->tls = true;
			break;
Quoted 36 lines
drivers/nvme/host/fabrics.c +0 -0
  		default:
  			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
  				p);
drivers/nvme/host/fabrics.h +3 -0
index dcac3df8a5f7..5db36e250e7a 100644
@@ -70,6 +70,7 @@ enum {
  	NVMF_OPT_DISCOVERY	= 1 << 22,
  	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
  	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
+	NVMF_OPT_TLS		= 1 << 25,
  };
 
  /**
@@ -102,6 +103,7 @@ enum {
   * @dhchap_secret: DH-HMAC-CHAP secret
   * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
   *              authentication
+ * @tls:        Start TLS encrypted connections (TCP)
   * @disable_sqflow: disable controller sq flow control
   * @hdr_digest: generate/verify header digest (TCP)
   * @data_digest: generate/verify data digest (TCP)
@@ -128,6 +130,7 @@ struct nvmf_ctrl_options {
  	int			max_reconnects;
  	char			*dhchap_secret;
  	char			*dhchap_ctrl_secret;
+	bool			tls;
  	bool			disable_sqflow;
  	bool			hdr_digest;
  	bool			data_digest;
drivers/nvme/host/nvme.h +0 -0
index bf46f122e9e1..421d01f849d1 100644

drivers/nvme/host/nvme.h struct nvme_ctrl { · line 352

Quoted 7 lines
drivers/nvme/host/nvme.h +3 -1
@@ -347,7 +347,9 @@ struct nvme_ctrl {
  	struct nvme_dhchap_key *ctrl_key;
  	u16 transaction;
  #endif
-
+#ifdef CONFIG_NVME_TLS
+	struct key *tls_key;
+#endif

I'd add a comment that this is nvme_tcp specific.

Quoted 41 lines
drivers/nvme/host/nvme.h +0 -0
  	/* Power saving configuration */
  	u64 ps_max_latency_us;
  	bool apst_enabled;
drivers/nvme/host/tcp.c +16 -0
index fddf785aba74..fdb564d0b9f4 100644
@@ -8,9 +8,15 @@
  #include <linux/init.h>
  #include <linux/slab.h>
  #include <linux/err.h>
+#include <linux/key.h>
  #include <linux/nvme-tcp.h>
  #include <net/sock.h>
  #include <net/tcp.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#include <net/tls.h>
+#include <net/handshake.h>
+#endif
  #include <linux/blk-mq.h>
  #include <crypto/hash.h>
  #include <net/busy_poll.h>
@@ -31,6 +37,16 @@ static int so_priority;
  module_param(so_priority, int, 0644);
  MODULE_PARM_DESC(so_priority, "nvme tcp socket optimize priority");
 
+#ifdef CONFIG_NVME_TLS
+/*
+ * TLS handshake timeout
+ */
+static int tls_handshake_timeout = 10;
+module_param(tls_handshake_timeout, int, 0644);
+MODULE_PARM_DESC(tls_handshake_timeout,
+		 "nvme TLS handshake timeout in seconds (default 10)");
+#endif
+
  #ifdef CONFIG_DEBUG_LOCK_ALLOC
  /* lockdep can detect a circular dependency of the form
   *   sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock

drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 167

Quoted 6 lines
drivers/nvme/host/tcp.c +2 -1
@@ -147,7 +163,10 @@ struct nvme_tcp_queue {
  	struct ahash_request	*snd_hash;
  	__le32			exp_ddgst;
  	__le32			recv_ddgst;
-
+#ifdef CONFIG_NVME_TLS
+	struct completion       *tls_complete;

Why is this a pointer?

Quoted 9 lines
drivers/nvme/host/tcp.c +2 -0
+	int                     tls_err;
+#endif
  	struct page_frag_cache	pf_cache;
 
  	void (*state_change)(struct sock *);
@@ -1503,7 +1522,80 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue)
  	queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false);
  }
 

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1533

Quoted 10 lines
drivers/nvme/host/tcp.c +9 -1
-static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
+#ifdef CONFIG_NVME_TLS
+static void nvme_tcp_tls_done(void *data, int status, key_serial_t peerid)
+{
+	struct nvme_tcp_queue *queue = data;
+	struct nvme_tcp_ctrl *ctrl = queue->ctrl;
+	int qid = nvme_tcp_queue_id(queue);
+
+	dev_dbg(ctrl->ctrl.device, "queue %d: TLS handshake done, key %x, status %d\n",
+		qid, peerid, status);

Can we avoid naming the key peerid? pskid perhaps.

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1536

Quoted 3 lines
drivers/nvme/host/tcp.c +3 -0
+
+	if (status)
+		queue->tls_err = -status;

Maybe just return or goto err, instead of the else case?

Quoted 15 lines
drivers/nvme/host/tcp.c +15 -0
+	else {
+		struct key *tls_key = key_lookup(peerid);
+		if (IS_ERR(tls_key)) {
+			dev_warn(ctrl->ctrl.device, "queue %d: Invalid key %x\n",
+				 qid, peerid);
+			queue->tls_err = -ENOKEY;
+		} else {
+			ctrl->ctrl.tls_key = tls_key;
+			queue->tls_err = 0;
+		}
+	}
+
+	if (queue->tls_complete)
+		complete(queue->tls_complete);
+}

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1561

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+
+static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
+			      struct nvme_tcp_queue *queue,
+			      key_serial_t peerid)
+{
+	int qid = nvme_tcp_queue_id(queue);
+	int ret;
+	struct tls_handshake_args args;
+	unsigned long tmo = tls_handshake_timeout * HZ;
+	DECLARE_COMPLETION_ONSTACK(tls_complete);

I think we can have this completion a queue member.

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1572

Quoted 11 lines
drivers/nvme/host/tcp.c +11 -0
+	key_serial_t keyring = nvme_keyring_id();
+
+	dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n",
+		qid, peerid);
+	args.ta_sock = queue->sock;
+	args.ta_done = nvme_tcp_tls_done;
+	args.ta_data = queue;
+	args.ta_my_peerids[0] = peerid;
+	args.ta_num_peerids = 1;
+	args.ta_keyring = keyring;
+	args.ta_timeout_ms = tls_handshake_timeout * 2 * 1000;

remove the 2x.

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1578

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+	queue->tls_err = -EOPNOTSUPP;
+	queue->tls_complete = &tls_complete;
+	ret = tls_client_hello_psk(&args, GFP_KERNEL);
+	if (ret) {
+		dev_dbg(nctrl->device, "queue %d: failed to start TLS: %d\n",
+			qid, ret);

dev_err

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1583

Quoted 5 lines
drivers/nvme/host/tcp.c +5 -0
+		return ret;
+	}
+	if (wait_for_completion_timeout(queue->tls_complete, tmo) == 0) {
+		dev_dbg(nctrl->device,
+			"queue %d: TLS handshake timeout\n", qid);

dev_err

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1592

Quoted 9 lines
drivers/nvme/host/tcp.c +9 -0
+		queue->tls_complete = NULL;
+		ret = -ETIMEDOUT;
+	} else {
+		dev_dbg(nctrl->device,
+			"queue %d: TLS handshake complete, error %d\n",
+			qid, queue->tls_err);
+		ret = queue->tls_err;
+	}
+	queue->tls_complete = NULL;

as mentioned, lets make the completion a queue member and not a pointer that we need to pay attention to.

Quoted 34 lines
drivers/nvme/host/tcp.c +18 -2
+	return ret;
+}
+#endif
+
+static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
+				key_serial_t peerid)
  {
  	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
  	struct nvme_tcp_queue *queue = &ctrl->queues[qid];
@@ -1627,6 +1719,14 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
  		goto err_rcv_pdu;
  	}
 
+#ifdef CONFIG_NVME_TLS
+	/* If PSKs are configured try to start TLS */
+	if (peerid) {
+		ret = nvme_tcp_start_tls(nctrl, queue, peerid);
+		if (ret)
+			goto err_init_connect;
+	}
+#endif
  	ret = nvme_tcp_init_connection(queue);
  	if (ret)
  		goto err_init_connect;
@@ -1771,10 +1871,23 @@ static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl,
  static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
  {
  	int ret;
-
-	ret = nvme_tcp_alloc_queue(ctrl, 0);
+	key_serial_t psk_id = 0;
+
+#ifdef CONFIG_NVME_TLS
+	if (ctrl->opts->tls) {

drivers/nvme/host/tcp.c static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl, · line 1887

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+		psk_id = nvme_tls_psk_default(NULL,
+					      ctrl->opts->host->nqn,
+					      ctrl->opts->subsysnqn);
+		if (!psk_id) {
+			dev_err(ctrl->device, "no valid PSK found\n");
+			ret = -ENOKEY;
+			goto out_free_queue;
+		}
+	}
+#endif

ctrl->opts->tls == true guarantes that CONFIG_NVME_TLS is enabled. No need for it, as long as you give a nvme_tls_psk_default stub when it is not.

Quoted 15 lines
drivers/nvme/host/tcp.c +5 -2
+	ret = nvme_tcp_alloc_queue(ctrl, 0, psk_id);
  	if (ret)
-		return ret;
+		goto out_free_queue;
 
  	ret = nvme_tcp_alloc_async_req(to_tcp_ctrl(ctrl));
  	if (ret)
@@ -1790,9 +1903,21 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
  static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
  {
  	int i, ret;
-
+	key_serial_t psk_id = 0;
+
+#ifdef CONFIG_NVME_TLS

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) · line 1918

Quoted 10 lines
drivers/nvme/host/tcp.c +10 -0
+	if (ctrl->opts->tls) {
+		psk_id = nvme_tls_psk_default(NULL,
+					      ctrl->opts->host->nqn,
+					      ctrl->opts->subsysnqn);
+		if (!psk_id) {
+			dev_err(ctrl->device, "no valid PSK found\n");
+			return -ENOKEY;
+		}
+	}
+#endif

Do we need this for io queues? why not just store it on nvme_tcp_ctrl and just use it?

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -1
  	for (i = 1; i < ctrl->queue_count; i++) {
-		ret = nvme_tcp_alloc_queue(ctrl, i);
+		ret = nvme_tcp_alloc_queue(ctrl, i, psk_id);
  		if (ret)
  			goto out_free_queues;
  	}

drivers/nvme/host/tcp.c static struct nvmf_transport_ops nvme_tcp_transport = { · line 2832

Quoted 8 lines
drivers/nvme/host/tcp.c +1 -1
@@ -2701,7 +2826,7 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
  			  NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO |
  			  NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |
  			  NVMF_OPT_NR_WRITE_QUEUES | NVMF_OPT_NR_POLL_QUEUES |
-			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+			  NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE | NVMF_OPT_TLS,
  	.create_ctrl	= nvme_tcp_create_ctrl,
  };
 

Overall this looks much better Hannes!

#25 lore raw

Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall

Quoted 107 lines · Sagi Grimberg
Quoted 93 lines · Hannes Reinecke

Add a fabrics option 'tls' and start the TLS handshake upcall with the default PSK. When TLS is started the PSK key serial number is displayed in the sysfs attribute 'tls_key'

---
  drivers/nvme/host/core.c    |  25 ++++++- drivers/nvme/host/fabrics.c |  11 +++ drivers/nvme/host/fabrics.h |   3 + drivers/nvme/host/nvme.h    |   4 +- drivers/nvme/host/tcp.c     | 141 ++++++++++++++++++++++++++++++++++--

  5 files changed, 174 insertions(+), 10 deletions(-)
drivers/nvme/host/core.c +16 -0
index 416d0a898f56..869396058e2e 100644
@@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO 
 | S_IWUSR,
       nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);
   #endif
+#ifdef CONFIG_NVME_TLS
+static ssize_t tls_key_show(struct device *dev,
+                struct device_attribute *attr, char *buf)
+{
+    struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
+
+    if (!ctrl->tls_key)
+        return 0;
+    return sysfs_emit(buf, "%08x", key_serial(ctrl->tls_key));
+}
+static DEVICE_ATTR_RO(tls_key);
+#endif
+
   static struct attribute *nvme_dev_attrs[] = {
       &dev_attr_reset_controller.attr,
       &dev_attr_rescan_controller.attr,
@@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {
   #ifdef CONFIG_NVME_AUTH
       &dev_attr_dhchap_secret.attr,
       &dev_attr_dhchap_ctrl_secret.attr,
+#endif
+#ifdef CONFIG_NVME_TLS
+    &dev_attr_tls_key.attr,
   #endif

      NULL
  };

drivers/nvme/host/core.c +4 -0
@@ -3956,6 +3972,10 @@ static umode_t 
 nvme_dev_attrs_are_visible(struct kobject *kobj,
           return 0;
       if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)
           return 0;
+#endif
+#ifdef CONFIG_NVME_TLS
+    if (a == &dev_attr_tls_key.attr && !ctrl->opts)
+        return 0;
   #endif
       return a->mode;

  }

drivers/nvme/host/core.c +4 -1
@@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)
       if (!subsys || ctrl->instance != subsys->instance)
           ida_free(&nvme_instance_ida, ctrl->instance);
-
+#ifdef CONFIG_NVME_TLS
+    if (ctrl->tls_key)
+        key_put(ctrl->tls_key);
+#endif
       nvme_free_cels(ctrl);
       nvme_mpath_uninit(ctrl);
       nvme_auth_stop(ctrl);
drivers/nvme/host/fabrics.c +1 -0
index bbaa04a0c502..3e4f0e45b58f 100644
@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = {
       { NVMF_OPT_DISCOVERY,        "discovery"        },
       { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
       { NVMF_OPT_DHCHAP_CTRL_SECRET,    "dhchap_ctrl_secret=%s"    },
+    { NVMF_OPT_TLS,            "tls"            },
       { NVMF_OPT_ERR,            NULL            }
   };
@@ -632,6 +633,7 @@ static int nvmf_parse_options(struct 
 nvmf_ctrl_options *opts,
       opts->hdr_digest = false;
       opts->data_digest = false;

      opts->tos = -1; /* < 0 == use transport default */ +    opts->tls = false;
      options = o = kstrdup(buf, GFP_KERNEL); if (!options)

drivers/nvme/host/fabrics.c +0 -0
@@ -918,6 +920,15 @@ static int nvmf_parse_options(struct 
 nvmf_ctrl_options *opts,
               kfree(opts->dhchap_ctrl_secret);
               opts->dhchap_ctrl_secret = p;

drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct · line 932

Quoted 10 lines
drivers/nvme/host/fabrics.c +9 -0
               break;
+        case NVMF_OPT_TLS:
+#ifdef CONFIG_NVME_TLS
+            opts->tls = true;
+            break;
+#else
+            pr_err("TLS is not supported\n");
+            ret = -EINVAL;
+            goto out;
+#endif
Quoted 9 lines

This can become:
            if (!IS_ENABLED(CONFIG_NVME_TLS)) { pr_err("TLS is not supported\n"); ret = -EINVAL;
                goto out;
            }
            opts->tls = true;
            break;

Yes.

Quoted 36 lines
Quoted 36 lines

          default:
              pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n", p);

drivers/nvme/host/fabrics.h +3 -0
index dcac3df8a5f7..5db36e250e7a 100644
@@ -70,6 +70,7 @@ enum {
       NVMF_OPT_DISCOVERY    = 1 << 22,
       NVMF_OPT_DHCHAP_SECRET    = 1 << 23,
       NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
+    NVMF_OPT_TLS        = 1 << 25,
   };
   /**
@@ -102,6 +103,7 @@ enum {
    * @dhchap_secret: DH-HMAC-CHAP secret
    * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
    *              authentication
+ * @tls:        Start TLS encrypted connections (TCP)
    * @disable_sqflow: disable controller sq flow control
    * @hdr_digest: generate/verify header digest (TCP)
    * @data_digest: generate/verify data digest (TCP)
@@ -128,6 +130,7 @@ struct nvmf_ctrl_options {
       int            max_reconnects;
       char            *dhchap_secret;
       char            *dhchap_ctrl_secret;
+    bool            tls;
       bool            disable_sqflow;
       bool            hdr_digest;
       bool            data_digest;
drivers/nvme/host/nvme.h +0 -0
index bf46f122e9e1..421d01f849d1 100644
@@ -347,7 +347,9 @@ struct nvme_ctrl {
       struct nvme_dhchap_key *ctrl_key;
Quoted 9 lines

drivers/nvme/host/nvme.h struct nvme_ctrl { · line 352

Quoted 6 lines
drivers/nvme/host/nvme.h +3 -1
       u16 transaction;
   #endif
-
+#ifdef CONFIG_NVME_TLS
+    struct key *tls_key;
+#endif

I'd add a comment that this is nvme_tcp specific.

Okay.

Quoted 40 lines
Quoted 40 lines
drivers/nvme/host/nvme.h +0 -0
       /* Power saving configuration */
       u64 ps_max_latency_us;
       bool apst_enabled;
drivers/nvme/host/tcp.c +16 -0
index fddf785aba74..fdb564d0b9f4 100644
@@ -8,9 +8,15 @@
   #include <linux/init.h>
   #include <linux/slab.h>
   #include <linux/err.h>
+#include <linux/key.h>
   #include <linux/nvme-tcp.h>
   #include <net/sock.h>
   #include <net/tcp.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#include <net/tls.h>
+#include <net/handshake.h>
+#endif
   #include <linux/blk-mq.h>
   #include <crypto/hash.h>
   #include <net/busy_poll.h>
@@ -31,6 +37,16 @@ static int so_priority;
   module_param(so_priority, int, 0644);
   MODULE_PARM_DESC(so_priority, "nvme tcp socket optimize priority");
+#ifdef CONFIG_NVME_TLS
+/*
+ * TLS handshake timeout
+ */
+static int tls_handshake_timeout = 10;
+module_param(tls_handshake_timeout, int, 0644);
+MODULE_PARM_DESC(tls_handshake_timeout,
+         "nvme TLS handshake timeout in seconds (default 10)");
+#endif
+
   #ifdef CONFIG_DEBUG_LOCK_ALLOC
   /* lockdep can detect a circular dependency of the form
    *   sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
@@ -147,7 +163,10 @@ struct nvme_tcp_queue {
Quoted 9 lines

drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 167

Quoted 6 lines
drivers/nvme/host/tcp.c +2 -1
       struct ahash_request    *snd_hash;
       __le32            exp_ddgst;
       __le32            recv_ddgst;
-
+#ifdef CONFIG_NVME_TLS
+    struct completion       *tls_complete;

Why is this a pointer?

Hmm. I'll change it.

Quoted 14 lines
Quoted 14 lines
drivers/nvme/host/tcp.c +6 -1
+    int                     tls_err;
+#endif
       struct page_frag_cache    pf_cache;
       void (*state_change)(struct sock *);
@@ -1503,7 +1522,80 @@ static void nvme_tcp_set_queue_io_cpu(struct 
 nvme_tcp_queue *queue)
       queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false);
   }
-static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
+#ifdef CONFIG_NVME_TLS
+static void nvme_tcp_tls_done(void *data, int status, key_serial_t 
 peerid)
+{
+    struct nvme_tcp_queue *queue = data;
Quoted 9 lines

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1533

Quoted 6 lines
drivers/nvme/host/tcp.c +5 -0
+    struct nvme_tcp_ctrl *ctrl = queue->ctrl;
+    int qid = nvme_tcp_queue_id(queue);
+
+    dev_dbg(ctrl->ctrl.device, "queue %d: TLS handshake done, key %x, 
 status %d\n",
+        qid, peerid, status);

Can we avoid naming the key peerid? pskid perhaps.

Okay.

Quoted 6 lines

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1536

Quoted 3 lines
drivers/nvme/host/tcp.c +3 -0
+
+    if (status)
+        queue->tls_err = -status;

Maybe just return or goto err, instead of the else case?

Okay.

Quoted 19 lines
Quoted 19 lines
drivers/nvme/host/tcp.c +19 -0
+    else {
+        struct key *tls_key = key_lookup(peerid);
+        if (IS_ERR(tls_key)) {
+            dev_warn(ctrl->ctrl.device, "queue %d: Invalid key %x\n",
+                 qid, peerid);
+            queue->tls_err = -ENOKEY;
+        } else {
+            ctrl->ctrl.tls_key = tls_key;
+            queue->tls_err = 0;
+        }
+    }
+
+    if (queue->tls_complete)
+        complete(queue->tls_complete);
+}
+
+static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
+                  struct nvme_tcp_queue *queue,
+                  key_serial_t peerid)
Quoted 9 lines

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1561

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+{
+    int qid = nvme_tcp_queue_id(queue);
+    int ret;
+    struct tls_handshake_args args;
+    unsigned long tmo = tls_handshake_timeout * HZ;
+    DECLARE_COMPLETION_ONSTACK(tls_complete);

I think we can have this completion a queue member.

Let's see.

Quoted 12 lines

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1572

Quoted 11 lines
drivers/nvme/host/tcp.c +11 -0
+    key_serial_t keyring = nvme_keyring_id();
+
+    dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n",
+        qid, peerid);
+    args.ta_sock = queue->sock;
+    args.ta_done = nvme_tcp_tls_done;
+    args.ta_data = queue;
+    args.ta_my_peerids[0] = peerid;
+    args.ta_num_peerids = 1;
+    args.ta_keyring = keyring;
+    args.ta_timeout_ms = tls_handshake_timeout * 2 * 1000;
Quoted 2 lines

remove the 2x.

Leftover, indeed.

Quoted 27 lines

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1578

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+    queue->tls_err = -EOPNOTSUPP;
+    queue->tls_complete = &tls_complete;
+    ret = tls_client_hello_psk(&args, GFP_KERNEL);
+    if (ret) {
+        dev_dbg(nctrl->device, "queue %d: failed to start TLS: %d\n",
+            qid, ret);

dev_err

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1583

Quoted 5 lines
drivers/nvme/host/tcp.c +5 -0
+        return ret;
+    }
+    if (wait_for_completion_timeout(queue->tls_complete, tmo) == 0) {
+        dev_dbg(nctrl->device,
+            "queue %d: TLS handshake timeout\n", qid);

dev_err

drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1592

Quoted 9 lines
drivers/nvme/host/tcp.c +9 -0
+        queue->tls_complete = NULL;
+        ret = -ETIMEDOUT;
+    } else {
+        dev_dbg(nctrl->device,
+            "queue %d: TLS handshake complete, error %d\n",
+            qid, queue->tls_err);
+        ret = queue->tls_err;
+    }
+    queue->tls_complete = NULL;
Quoted 3 lines

as mentioned, lets make the completion a queue member and not a pointer that we need to pay attention to.

Okay.

Quoted 41 lines
Quoted 41 lines
drivers/nvme/host/tcp.c +6 -0
+    return ret;
+}
+#endif
+
+static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
+                key_serial_t peerid)
   {

      struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid];

drivers/nvme/host/tcp.c +18 -2
@@ -1627,6 +1719,14 @@ static int nvme_tcp_alloc_queue(struct 
 nvme_ctrl *nctrl, int qid)
           goto err_rcv_pdu;
       }
+#ifdef CONFIG_NVME_TLS
+    /* If PSKs are configured try to start TLS */
+    if (peerid) {
+        ret = nvme_tcp_start_tls(nctrl, queue, peerid);
+        if (ret)
+            goto err_init_connect;
+    }
+#endif
       ret = nvme_tcp_init_connection(queue);
       if (ret)
           goto err_init_connect;
@@ -1771,10 +1871,23 @@ static int nvme_tcp_start_io_queues(struct 
 nvme_ctrl *ctrl,
   static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
   {
       int ret;
-
-    ret = nvme_tcp_alloc_queue(ctrl, 0);
+    key_serial_t psk_id = 0;
+
+#ifdef CONFIG_NVME_TLS
+    if (ctrl->opts->tls) {
+        psk_id = nvme_tls_psk_default(NULL,
+                          ctrl->opts->host->nqn,
+                          ctrl->opts->subsysnqn);
+        if (!psk_id) {
+            dev_err(ctrl->device, "no valid PSK found\n");
+            ret = -ENOKEY;
Quoted 9 lines

drivers/nvme/host/tcp.c static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl, · line 1887

Quoted 4 lines
drivers/nvme/host/tcp.c +4 -0
+            goto out_free_queue;
+        }
+    }
+#endif

ctrl->opts->tls == true guarantes that CONFIG_NVME_TLS is enabled. No need for it, as long as you give a nvme_tls_psk_default stub when it is not.

Hmm. I deliberately had the 'tls' option always enabled to avoid parsing errors. But sure, if we make that a compile condition this can be simplified.

Quoted 20 lines
Quoted 20 lines
drivers/nvme/host/tcp.c +10 -2
+    ret = nvme_tcp_alloc_queue(ctrl, 0, psk_id);
       if (ret)
-        return ret;
+        goto out_free_queue;
       ret = nvme_tcp_alloc_async_req(to_tcp_ctrl(ctrl));
       if (ret)
@@ -1790,9 +1903,21 @@ static int nvme_tcp_alloc_admin_queue(struct 
 nvme_ctrl *ctrl)
   static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
   {
       int i, ret;
-
+    key_serial_t psk_id = 0;
+
+#ifdef CONFIG_NVME_TLS
+    if (ctrl->opts->tls) {
+        psk_id = nvme_tls_psk_default(NULL,
+                          ctrl->opts->host->nqn,
+                          ctrl->opts->subsysnqn);
+        if (!psk_id) {
Quoted 9 lines

drivers/nvme/host/tcp.c static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) · line 1918

Quoted 5 lines
drivers/nvme/host/tcp.c +5 -0
+            dev_err(ctrl->device, "no valid PSK found\n");
+            return -ENOKEY;
+        }
+    }
+#endif

Do we need this for io queues? why not just store it on nvme_tcp_ctrl and just use it?

Guess we can; current debate at fmds is heading to having just one PSK for both admin and I/O queues anyway.

Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +1 -1
       for (i = 1; i < ctrl->queue_count; i++) {
-        ret = nvme_tcp_alloc_queue(ctrl, i);
+        ret = nvme_tcp_alloc_queue(ctrl, i, psk_id);
           if (ret)
               goto out_free_queues;

      }

drivers/nvme/host/tcp.c +0 -0
@@ -2701,7 +2826,7 @@ static struct nvmf_transport_ops 
 nvme_tcp_transport = {

                NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO | NVMF_OPT_HDR_DIGEST | NVMF_OPT_DATA_DIGEST |

Quoted 8 lines
Quoted 5 lines

                NVMF_OPT_NR_WRITE_QUEUES | NVMF_OPT_NR_POLL_QUEUES | -              NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE,
+              NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE | NVMF_OPT_TLS, .create_ctrl    = nvme_tcp_create_ctrl,
  };

Overall this looks much better Hannes!

Thanks.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
#26 lore raw

PATCH v2 10/18 nvme-tcp: fixup send workflow for kTLS

In reply to Hannes Reinecke

kTLS does not support MSG_EOR flag for sendmsg(), and in general is really picky about invalid MSG_XXX flags.
So ensure that the MSG_EOR flags is blanked out for TLS, and that the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

---
 drivers/nvme/host/tcp.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +20 -8
index fdb564d0b9f4..54d27873227a 100644
@@ -218,6 +218,15 @@ static inline struct blk_mq_tags *nvme_tcp_tagset(struct nvme_tcp_queue *queue)
 	return queue->ctrl->tag_set.tags[queue_idx - 1];
 }
 
+static inline bool nvme_tcp_tls_enabled(struct nvme_tcp_queue *queue)
+{
+#ifdef CONFIG_NVME_TLS
+	return (queue->ctrl->ctrl.tls_key != NULL);
+#else
+	return false;
+#endif
+}
+
 static inline u8 nvme_tcp_hdgst_len(struct nvme_tcp_queue *queue)
 {
 	return queue->hdr_digest ? NVME_TCP_DIGEST_LENGTH : 0;
@@ -1021,12 +1030,14 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 		int req_data_sent = req->data_sent;
 		int ret, flags = MSG_DONTWAIT;
 
-		if (last && !queue->data_digest && !nvme_tcp_queue_more(queue))
+		if (!last || queue->data_digest || nvme_tcp_queue_more(queue))
+			flags |= MSG_MORE;
+		else if (!nvme_tcp_tls_enabled(queue))
 			flags |= MSG_EOR;
-		else
-			flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
 
 		if (sendpage_ok(page)) {
+			if (flags & MSG_MORE)
+				flags |= MSG_SENDPAGE_NOTLAST;
 			ret = kernel_sendpage(queue->sock, page, offset, len,
 					flags);
 		} else {
@@ -1077,9 +1088,11 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	int flags = MSG_DONTWAIT;
 	int ret;
 
-	if (inline_data || nvme_tcp_queue_more(queue))
-		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	if (inline_data || nvme_tcp_queue_more(queue)) {
+		flags |= MSG_MORE;
+		if (!nvme_tcp_tls_enabled(queue))
+			flags |= MSG_SENDPAGE_NOTLAST;
+	} else if (!nvme_tcp_tls_enabled(queue))
 		flags |= MSG_EOR;
 
 	if (queue->hdr_digest && !req->offset)
@@ -1154,9 +1167,8 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)
 
 	if (nvme_tcp_queue_more(queue))
 		msg.msg_flags |= MSG_MORE;
-	else
+	else if (!nvme_tcp_tls_enabled(queue))
 		msg.msg_flags |= MSG_EOR;
-
 	ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);
 	if (unlikely(ret <= 0))
 		return ret;
Signature · 1 line
2.35.3
maintainer
#27 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Quoted 3 lines

kTLS does not support MSG_EOR flag for sendmsg(), and in general is really picky about invalid MSG_XXX flags.

CC'ing TLS folks.

Can't tls simply ignore MSG_EOR instead of consumers having to be careful over it?

Quoted 2 lines

So ensure that the MSG_EOR flags is blanked out for TLS, and that the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

You mean MSG_SENDPAGE_NOTLAST.

It is also a bit annoying that a tls socket dictates different behavior than a normal socket.

The current logic is rather simple:
if more data comming:
flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
else:
flags = MSG_EOR

Would like to keep it that way for tls as well. Can someone explain why this is a problem with tls?

#28 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Quoted 16 lines · Sagi Grimberg
Quoted 3 lines

kTLS does not support MSG_EOR flag for sendmsg(), and in general is really picky about invalid MSG_XXX flags.

CC'ing TLS folks.

Can't tls simply ignore MSG_EOR instead of consumers having to be careful over it?

Quoted 2 lines

So ensure that the MSG_EOR flags is blanked out for TLS, and that the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

You mean MSG_SENDPAGE_NOTLAST.

It is also a bit annoying that a tls socket dictates different behavior than a normal socket.

Quoted 8 lines

The current logic is rather simple:
if more data comming:
    flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
else:
    flags = MSG_EOR

Would like to keep it that way for tls as well. Can someone explain why this is a problem with tls?

TLS is using MSG_EOR internally (to control the flow of the underlying tcp stream), so it's meaningless for the ULP. I've no idea about SENDPAGE_NOTLAST; that one is particularly annoying
as we have to check whether we do sendpage() or sendmsg().

But TLS really should blank out invalid flags, as it has different rules for them than anyone else. And the caller really shouldn't be burdened with checking whether TLS is enabled or not.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#29 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

In reply to Sagi Grimberg
Quoted 7 lines · On Thu, 30 Mar 2023 18:24:04 +0300 Sagi Grimberg wrote:
Quoted 2 lines

kTLS does not support MSG_EOR flag for sendmsg(), and in general is really picky about invalid MSG_XXX flags.

CC'ing TLS folks.

Can't tls simply ignore MSG_EOR instead of consumers having to be careful over it?

I think we can support EOR, I don't see any fundamental problem there.

Quoted 8 lines
Quoted 2 lines

So ensure that the MSG_EOR flags is blanked out for TLS, and that the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

You mean MSG_SENDPAGE_NOTLAST.

It is also a bit annoying that a tls socket dictates different behavior than a normal socket.

Quoted 8 lines

The current logic is rather simple:
if more data comming:
flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
else:
flags = MSG_EOR

Would like to keep it that way for tls as well. Can someone explain why this is a problem with tls?

Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(

#30 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Quoted 25 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 30 Mar 2023 18:24:04 +0300 Sagi Grimberg wrote:
Quoted 2 lines

kTLS does not support MSG_EOR flag for sendmsg(), and in general is really picky about invalid MSG_XXX flags.

CC'ing TLS folks.

Can't tls simply ignore MSG_EOR instead of consumers having to be careful over it?

I think we can support EOR, I don't see any fundamental problem there.

Quoted 14 lines
Quoted 2 lines

So ensure that the MSG_EOR flags is blanked out for TLS, and that the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

You mean MSG_SENDPAGE_NOTLAST.

It is also a bit annoying that a tls socket dictates different behavior than a normal socket.

The current logic is rather simple:
if more data comming:
flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
else:
flags = MSG_EOR

Quoted 7 lines
Quoted 2 lines

Would like to keep it that way for tls as well. Can someone explain why this is a problem with tls?

Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(

A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#31 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Hey Jakub, Hannes.

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

kTLS does not support MSG_EOR flag for sendmsg(), and in general is really picky about invalid MSG_XXX flags.

CC'ing TLS folks.

Can't tls simply ignore MSG_EOR instead of consumers having to be careful over it?

I think we can support EOR, I don't see any fundamental problem there.

It would help at least one consumer (nvme-tcp) to not change its behavior between tls and non-tls. At the very minimum don't fail the send operation (just do the same as if it wasn't passed).

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 2 lines

So ensure that the MSG_EOR flags is blanked out for TLS, and that the MSG_SENDPAGE_LAST is only set if we actually do sendpage().

You mean MSG_SENDPAGE_NOTLAST.

It is also a bit annoying that a tls socket dictates different behavior than a normal socket.

The current logic is rather simple:
if more data comming:
    flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
else:
    flags = MSG_EOR

Would like to keep it that way for tls as well. Can someone explain why this is a problem with tls?

Quoted 8 lines
Quoted 4 lines

Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(

A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?

MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was essentially similar semantics to MSG_MORE but for sendpage. It'd be great if this can be allowed in tls (again, at the very least don't fail but continue as if it wasn't passed).

If this turns out to be a big project, I would prefer to change nvme-tcp for now in order not to block nvme tls support (although it is a hidden capability interface, which is always bad).

maintainer
#32 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Quoted 5 lines · On Mon, 3 Apr 2023 15:20:13 +0300 Sagi Grimberg wrote:
Quoted 5 lines
Quoted 4 lines

Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(

Quoted 8 lines
Quoted 3 lines

A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?

MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was essentially similar semantics to MSG_MORE but for sendpage. It'd be great if this can be allowed in tls (again, at the very least don't fail but continue as if it wasn't passed).

.. but.. MSG_SENDPAGE_NOTLAST is supported in TLS, isn't it? Why are we talking about it?

maintainer
#33 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(

A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?

Quoted 7 lines
Quoted 4 lines

MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was essentially similar semantics to MSG_MORE but for sendpage. It'd be great if this can be allowed in tls (again, at the very least don't fail but continue as if it wasn't passed).

.. but.. MSG_SENDPAGE_NOTLAST is supported in TLS, isn't it? Why are we talking about it?

Ah, right.

What I'm assuming that Hannes is tripping on is that tls does not accept when this flag is sent to sock_no_sendpage, which is simply calling sendmsg. TLS will not accept this flag when passed to sendmsg IIUC.

Today the rough logic in nvme send path is:

	if (more_coming(queue)) {
		flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
	} else {
		flags = MSG_EOR;
	}

	if (!sendpage_ok(page)) {
		kernel_sendpage();
	} else {
		sock_no_sendpage();
	}

This pattern (note that sock_no_sednpage was added later following bug reports where nvme attempted to sendpage a slab allocated page), is perfectly acceptable with normal sockets, but not with TLS.

So there are two options:
1. have tls accept MSG_SENDPAGE_NOTLAST in sendmsg (called from sock_no_sendpage)
2. Make nvme set MSG_SENDPAGE_NOTLAST only when calling kernel_sendpage and clear it when calling sock_no_sendpage

If you say that MSG_SENDPAGE_NOTLAST must be cleared when calling sock_no_sendpage and it is a bug that it isn't enforced for normal tcp sockets, then we need to change nvme, but I did not find any documentation that indicates it, and right now, normal sockets behave differently than tls sockets (wrt this flag in particular).

Hope this clarifies.

maintainer
#34 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Quoted 29 lines · On Mon, 3 Apr 2023 18:51:09 +0300 Sagi Grimberg wrote:

What I'm assuming that Hannes is tripping on is that tls does not accept when this flag is sent to sock_no_sendpage, which is simply calling sendmsg. TLS will not accept this flag when passed to sendmsg IIUC.

Today the rough logic in nvme send path is:

	if (more_coming(queue)) {
		flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
	} else {
		flags = MSG_EOR;
	}

	if (!sendpage_ok(page)) {
		kernel_sendpage();
	} else {
		sock_no_sendpage();
	}

This pattern (note that sock_no_sednpage was added later following bug reports where nvme attempted to sendpage a slab allocated page), is perfectly acceptable with normal sockets, but not with TLS.

So there are two options:
1. have tls accept MSG_SENDPAGE_NOTLAST in sendmsg (called from sock_no_sendpage)
2. Make nvme set MSG_SENDPAGE_NOTLAST only when calling kernel_sendpage and clear it when calling sock_no_sendpage

Quoted 7 lines

If you say that MSG_SENDPAGE_NOTLAST must be cleared when calling sock_no_sendpage and it is a bug that it isn't enforced for normal tcp sockets, then we need to change nvme, but I did not find any documentation that indicates it, and right now, normal sockets behave differently than tls sockets (wrt this flag in particular).

Hope this clarifies.

Oh right, it does, the context evaporated from my head over the weekend.

IMHO it's best if the caller passes the right flags. The semantics of MSG_MORE vs NOTLAST are quite murky and had already caused bugs in the past :(

See commit d452d48b9f8b ("tls: prevent oversized sendfile() hangs by ignoring MSG_MORE")

Alternatively we could have sock_no_sendpage drop NOTLAST to help all protos. But if we consider sendfile behavior as the standard simply clearing it isn't right, it should be a:

	more = (flags & (MORE | NOTLAST)) == MORE | NOTLAST
	flags &= ~(MORE | NOTLAST)
	if (more)
		flags |= MORE
maintainer
#35 lore raw

Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS

Quoted 38 lines · Jakub Kicinski
Quoted 29 lines · On Mon, 3 Apr 2023 18:51:09 +0300 Sagi Grimberg wrote:

What I'm assuming that Hannes is tripping on is that tls does not accept when this flag is sent to sock_no_sendpage, which is simply calling sendmsg. TLS will not accept this flag when passed to sendmsg IIUC.

Today the rough logic in nvme send path is:

	if (more_coming(queue)) {
		flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
	} else {
		flags = MSG_EOR;
	}

	if (!sendpage_ok(page)) {
		kernel_sendpage();
	} else {
		sock_no_sendpage();
	}

This pattern (note that sock_no_sednpage was added later following bug reports where nvme attempted to sendpage a slab allocated page), is perfectly acceptable with normal sockets, but not with TLS.

So there are two options:
1. have tls accept MSG_SENDPAGE_NOTLAST in sendmsg (called from sock_no_sendpage)
2. Make nvme set MSG_SENDPAGE_NOTLAST only when calling kernel_sendpage and clear it when calling sock_no_sendpage

Quoted 7 lines

If you say that MSG_SENDPAGE_NOTLAST must be cleared when calling sock_no_sendpage and it is a bug that it isn't enforced for normal tcp sockets, then we need to change nvme, but I did not find any documentation that indicates it, and right now, normal sockets behave differently than tls sockets (wrt this flag in particular).

Hope this clarifies.

Quoted 8 lines

Oh right, it does, the context evaporated from my head over the weekend.

IMHO it's best if the caller passes the right flags. The semantics of MSG_MORE vs NOTLAST are quite murky and had already caused bugs in the past :(

See commit d452d48b9f8b ("tls: prevent oversized sendfile() hangs by ignoring MSG_MORE")

Well, that is fine with me. This may change anyways with the new MSG_SPLICE_PAGES from David.

Quoted 8 lines

Alternatively we could have sock_no_sendpage drop NOTLAST to help all protos. But if we consider sendfile behavior as the standard simply clearing it isn't right, it should be a:

	more = (flags & (MORE | NOTLAST)) == MORE | NOTLAST
	flags &= ~(MORE | NOTLAST)
	if (more)
		flags |= MORE

I don't think this would be the best option. Requiring callers to clear NOTLAST if not calling sendpages is reasonable, but we need to have this consistent. And also fix this pattern for the rest of the kernel socket consumers that use this flag.

#36 lore raw

PATCH v2 11/18 nvme-tcp: control message handling for recvmsg()

In reply to Hannes Reinecke

kTLS is sending TLS ALERT messages as control messages for recvmsg(). As we can't do anything sensible with it just abort the connection and let the userspace agent to a re-negotiation.

---
 drivers/nvme/host/tcp.c | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +22 -1
index 54d27873227a..2e15fa83b725 100644
@@ -1377,6 +1377,11 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
 {
 	struct nvme_tcp_icreq_pdu *icreq;
 	struct nvme_tcp_icresp_pdu *icresp;
+#ifdef CONFIG_NVME_TLS
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	struct cmsghdr *cmsg;
+	unsigned char ctype;
+#endif
 	struct msghdr msg = {};
 	struct kvec iov;
 	bool ctrl_hdgst, ctrl_ddgst;
@@ -1414,11 +1419,27 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
 	memset(&msg, 0, sizeof(msg));
 	iov.iov_base = icresp;
 	iov.iov_len = sizeof(*icresp);
+#ifdef CONFIG_NVME_TLS
+	msg.msg_control = cbuf;
+	msg.msg_controllen = sizeof(cbuf);
+#endif
 	ret = kernel_recvmsg(queue->sock, &msg, &iov, 1,
 			iov.iov_len, msg.msg_flags);
 	if (ret < 0)
 		goto free_icresp;
-
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d: unhandled TLS record %d\n",
+			       nvme_tcp_queue_id(queue), ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif
 	ret = -EINVAL;
 	if (icresp->hdr.type != nvme_tcp_icresp) {
 		pr_err("queue %d: bad type returned %d\n",
Signature · 1 line
2.35.3
#38 lore raw

PATCH v2 12/18 nvme-fabrics: parse options 'keyring' and 'tls_key'

In reply to Hannes Reinecke

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

---
 drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/fabrics.h |  6 +++
 drivers/nvme/host/tcp.c     | 20 +++++++---
 3 files changed, 98 insertions(+), 7 deletions(-)
drivers/nvme/host/fabrics.c +78 -1
index 3e4f0e45b58f..5f5e487d498c 100644
@@ -605,6 +605,8 @@ static const match_table_t opt_tokens = {
 	{ NVMF_OPT_NR_WRITE_QUEUES,	"nr_write_queues=%d"	},
 	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
 	{ NVMF_OPT_TOS,			"tos=%d"		},
+	{ NVMF_OPT_KEYRING,		"keyring=%d"		},
+	{ NVMF_OPT_TLS_KEY,		"tls_key=%d"		},
 	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
 	{ NVMF_OPT_DISCOVERY,		"discovery"		},
 	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
@@ -620,8 +622,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	char *options, *o, *p;
 	int token, ret = 0;
 	size_t nqnlen  = 0;
-	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
+	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO, key_id;
 	uuid_t hostid;
+	struct key *key = NULL;
 
 	/* Set defaults */
 	opts->queue_size = NVMF_DEF_QUEUE_SIZE;
@@ -889,6 +892,74 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			}
 			opts->tos = token;
 			break;
+		case NVMF_OPT_KEYRING:
+#ifdef CONFIG_NVME_TLS
+			if (match_int(args, &key_id)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (key_id < 0) {
+				pr_err("Invalid keyring id %d\n", key_id);
+				ret = -EINVAL;
+				goto out;
+			}
+			if (!key_id) {
+				pr_debug("Using default keyring\n");
+				if (opts->keyring) {
+					key_put(opts->keyring);
+					opts->keyring = NULL;
+				}
+				break;
+			}
+			key = key_lookup(key_id);
+			if (!key) {
+				pr_err("Keyring id %08x not found\n", key_id);
+				ret = -ENOKEY;
+				goto out;
+			}
+			if (opts->keyring)
+				key_put(opts->keyring);
+			opts->keyring = key;
+			break;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
+		case NVMF_OPT_TLS_KEY:
+#ifdef CONFIG_NVME_TLS
+			if (match_int(args, &key_id)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (key_id < 0) {
+				pr_err("Invalid key id %d\n", key_id);
+				ret = -EINVAL;
+				goto out;
+			}
+			if (!key_id) {
+				pr_debug("Using 'best' PSK\n");
+				if (opts->tls_key) {
+					key_put(opts->tls_key);
+					opts->tls_key = NULL;
+				}
+				break;
+			}
+			key = key_lookup(key_id);
+			if (!key) {
+				pr_err("Key id %08x not found\n", key_id);
+				ret = -ENOKEY;
+				goto out;
+			}
+			if (opts->tls_key)
+				key_put(opts->tls_key);
+			opts->tls_key = key;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
+			break;
 		case NVMF_OPT_DISCOVERY:
 			opts->discovery_nqn = true;
 			break;
@@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts,
 void nvmf_free_options(struct nvmf_ctrl_options *opts)
 {
 	nvmf_host_put(opts->host);
+#ifdef CONFIG_NVME_TLS
+	if (opts->keyring)
+		key_put(opts->keyring);
+	if (opts->tls_key)
+		key_put(opts->tls_key);
+#endif
 	kfree(opts->transport);
 	kfree(opts->traddr);
 	kfree(opts->trsvcid);
drivers/nvme/host/fabrics.h +6 -0
index 5db36e250e7a..2ff7b7168a40 100644
@@ -71,6 +71,8 @@ enum {
 	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
 	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 	NVMF_OPT_TLS		= 1 << 25,
+	NVMF_OPT_KEYRING	= 1 << 26,
+	NVMF_OPT_TLS_KEY	= 1 << 27,
 };
 
 /**
@@ -103,6 +105,8 @@ enum {
  * @dhchap_secret: DH-HMAC-CHAP secret
  * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
  *              authentication
+ * @keyring:    Keyring to use for key lookups
+ * @tls_key:    TLS key for encrypted connections (TCP)
  * @tls:        Start TLS encrypted connections (TCP)
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
@@ -130,6 +134,8 @@ struct nvmf_ctrl_options {
 	int			max_reconnects;
 	char			*dhchap_secret;
 	char			*dhchap_ctrl_secret;
+	struct key		*keyring;
+	struct key		*tls_key;
 	bool			tls;
 	bool			disable_sqflow;
 	bool			hdr_digest;
drivers/nvme/host/tcp.c +14 -6
index 2e15fa83b725..00a3f18a69af 100644
@@ -1596,6 +1596,8 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
 
 	dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n",
 		qid, peerid);
+	if (nctrl->opts->keyring)
+		keyring = key_serial(nctrl->opts->keyring);
 	args.ta_sock = queue->sock;
 	args.ta_done = nvme_tcp_tls_done;
 	args.ta_data = queue;
@@ -1908,9 +1910,12 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)
 
 #ifdef CONFIG_NVME_TLS
 	if (ctrl->opts->tls) {
-		psk_id = nvme_tls_psk_default(NULL,
-					      ctrl->opts->host->nqn,
-					      ctrl->opts->subsysnqn);
+		if (ctrl->opts->tls_key)
+			psk_id = key_serial(ctrl->opts->tls_key);
+		else
+			psk_id = nvme_tls_psk_default(ctrl->opts->keyring,
+						      ctrl->opts->host->nqn,
+						      ctrl->opts->subsysnqn);
 		if (!psk_id) {
 			dev_err(ctrl->device, "no valid PSK found\n");
 			ret = -ENOKEY;
@@ -1940,9 +1945,12 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
 
 #ifdef CONFIG_NVME_TLS
 	if (ctrl->opts->tls) {
-		psk_id = nvme_tls_psk_default(NULL,
-					      ctrl->opts->host->nqn,
-					      ctrl->opts->subsysnqn);
+		if (ctrl->opts->tls_key)
+			psk_id = key_serial(ctrl->opts->tls_key);
+		else
+			psk_id = nvme_tls_psk_default(ctrl->opts->keyring,
+						      ctrl->opts->host->nqn,
+						      ctrl->opts->subsysnqn);
 		if (!psk_id) {
 			dev_err(ctrl->device, "no valid PSK found\n");
 			return -ENOKEY;
Signature · 1 line
2.35.3
maintainer
#39 lore raw

Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'

Quoted 2 lines

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?

Quoted 33 lines
---
  drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++-
  drivers/nvme/host/fabrics.h |  6 +++
  drivers/nvme/host/tcp.c     | 20 +++++++---
  3 files changed, 98 insertions(+), 7 deletions(-)
drivers/nvme/host/fabrics.c +4 -1
index 3e4f0e45b58f..5f5e487d498c 100644
@@ -605,6 +605,8 @@ static const match_table_t opt_tokens = {
  	{ NVMF_OPT_NR_WRITE_QUEUES,	"nr_write_queues=%d"	},
  	{ NVMF_OPT_NR_POLL_QUEUES,	"nr_poll_queues=%d"	},
  	{ NVMF_OPT_TOS,			"tos=%d"		},
+	{ NVMF_OPT_KEYRING,		"keyring=%d"		},
+	{ NVMF_OPT_TLS_KEY,		"tls_key=%d"		},
  	{ NVMF_OPT_FAIL_FAST_TMO,	"fast_io_fail_tmo=%d"	},
  	{ NVMF_OPT_DISCOVERY,		"discovery"		},
  	{ NVMF_OPT_DHCHAP_SECRET,	"dhchap_secret=%s"	},
@@ -620,8 +622,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
  	char *options, *o, *p;
  	int token, ret = 0;
  	size_t nqnlen  = 0;
-	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
+	int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO, key_id;
  	uuid_t hostid;
+	struct key *key = NULL;
 
  	/* Set defaults */
  	opts->queue_size = NVMF_DEF_QUEUE_SIZE;

drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct nvmf_ctrl_options *opts, · line 896

Quoted 5 lines
drivers/nvme/host/fabrics.c +2 -0
@@ -889,6 +892,74 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
  			}
  			opts->tos = token;
  			break;
+		case NVMF_OPT_KEYRING:
+#ifdef CONFIG_NVME_TLS

Same comment as before:

			if (!IS_ENABLED(CONFIG_NVME_TLS)) {
				pr_err("TLS is not supported\n");
				ret = -EINVAL;
				goto out;
			}
Quoted 70 lines
drivers/nvme/host/fabrics.c +66 -0
+			if (match_int(args, &key_id)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (key_id < 0) {
+				pr_err("Invalid keyring id %d\n", key_id);
+				ret = -EINVAL;
+				goto out;
+			}
+			if (!key_id) {
+				pr_debug("Using default keyring\n");
+				if (opts->keyring) {
+					key_put(opts->keyring);
+					opts->keyring = NULL;
+				}
+				break;
+			}
+			key = key_lookup(key_id);
+			if (!key) {
+				pr_err("Keyring id %08x not found\n", key_id);
+				ret = -ENOKEY;
+				goto out;
+			}
+			if (opts->keyring)
+				key_put(opts->keyring);
+			opts->keyring = key;
+			break;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
+		case NVMF_OPT_TLS_KEY:
+#ifdef CONFIG_NVME_TLS
+			if (match_int(args, &key_id)) {
+				ret = -EINVAL;
+				goto out;
+			}
+			if (key_id < 0) {
+				pr_err("Invalid key id %d\n", key_id);
+				ret = -EINVAL;
+				goto out;
+			}
+			if (!key_id) {
+				pr_debug("Using 'best' PSK\n");
+				if (opts->tls_key) {
+					key_put(opts->tls_key);
+					opts->tls_key = NULL;
+				}
+				break;
+			}
+			key = key_lookup(key_id);
+			if (!key) {
+				pr_err("Key id %08x not found\n", key_id);
+				ret = -ENOKEY;
+				goto out;
+			}
+			if (opts->tls_key)
+				key_put(opts->tls_key);
+			opts->tls_key = key;
+#else
+			pr_err("TLS is not supported\n");
+			ret = -EINVAL;
+			goto out;
+#endif
+			break;
  		case NVMF_OPT_DISCOVERY:
  			opts->discovery_nqn = true;
  			break;

drivers/nvme/host/fabrics.c static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts, · line 1132

Quoted 8 lines
drivers/nvme/host/fabrics.c +5 -0
@@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts,
  void nvmf_free_options(struct nvmf_ctrl_options *opts)
  {
  	nvmf_host_put(opts->host);
+#ifdef CONFIG_NVME_TLS
+	if (opts->keyring)
+		key_put(opts->keyring);
+	if (opts->tls_key)
+		key_put(opts->tls_key);

I think key_put is null safe.

#40 lore raw

Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'

Quoted 6 lines · Sagi Grimberg
Quoted 2 lines

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?

Choice.
With a single keyring we can only have a single identity.
But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).
With this option we can prepare a new keyring, and use that instead of the old one.

(And it really doesn't add much complexity...)

Quoted 38 lines
Quoted 38 lines

---
  drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++- drivers/nvme/host/fabrics.h |  6 +++ drivers/nvme/host/tcp.c     | 20 +++++++---

  3 files changed, 98 insertions(+), 7 deletions(-)
drivers/nvme/host/fabrics.c +5 -1
index 3e4f0e45b58f..5f5e487d498c 100644
@@ -605,6 +605,8 @@ static const match_table_t opt_tokens = {
       { NVMF_OPT_NR_WRITE_QUEUES,    "nr_write_queues=%d"    },
       { NVMF_OPT_NR_POLL_QUEUES,    "nr_poll_queues=%d"    },
       { NVMF_OPT_TOS,            "tos=%d"        },
+    { NVMF_OPT_KEYRING,        "keyring=%d"        },
+    { NVMF_OPT_TLS_KEY,        "tls_key=%d"        },
       { NVMF_OPT_FAIL_FAST_TMO,    "fast_io_fail_tmo=%d"    },
       { NVMF_OPT_DISCOVERY,        "discovery"        },
       { NVMF_OPT_DHCHAP_SECRET,    "dhchap_secret=%s"    },
@@ -620,8 +622,9 @@ static int nvmf_parse_options(struct 
 nvmf_ctrl_options *opts,
       char *options, *o, *p;
       int token, ret = 0;
       size_t nqnlen  = 0;
-    int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
+    int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO, key_id;
       uuid_t hostid;
+    struct key *key = NULL;
       /* Set defaults */
       opts->queue_size = NVMF_DEF_QUEUE_SIZE;
@@ -889,6 +892,74 @@ static int nvmf_parse_options(struct 
 nvmf_ctrl_options *opts,
               }
               opts->tos = token;
               break;
+        case NVMF_OPT_KEYRING:
Quoted 9 lines

drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct · line 897

Quoted 1 line
drivers/nvme/host/fabrics.c +1 -0
+#ifdef CONFIG_NVME_TLS

Same comment as before:
            if (!IS_ENABLED(CONFIG_NVME_TLS)) { pr_err("TLS is not supported\n"); ret = -EINVAL;
                goto out;
            }

OK.

Quoted 73 lines
Quoted 73 lines
drivers/nvme/host/fabrics.c +66 -0
+            if (match_int(args, &key_id)) {
+                ret = -EINVAL;
+                goto out;
+            }
+            if (key_id < 0) {
+                pr_err("Invalid keyring id %d\n", key_id);
+                ret = -EINVAL;
+                goto out;
+            }
+            if (!key_id) {
+                pr_debug("Using default keyring\n");
+                if (opts->keyring) {
+                    key_put(opts->keyring);
+                    opts->keyring = NULL;
+                }
+                break;
+            }
+            key = key_lookup(key_id);
+            if (!key) {
+                pr_err("Keyring id %08x not found\n", key_id);
+                ret = -ENOKEY;
+                goto out;
+            }
+            if (opts->keyring)
+                key_put(opts->keyring);
+            opts->keyring = key;
+            break;
+#else
+            pr_err("TLS is not supported\n");
+            ret = -EINVAL;
+            goto out;
+#endif
+        case NVMF_OPT_TLS_KEY:
+#ifdef CONFIG_NVME_TLS
+            if (match_int(args, &key_id)) {
+                ret = -EINVAL;
+                goto out;
+            }
+            if (key_id < 0) {
+                pr_err("Invalid key id %d\n", key_id);
+                ret = -EINVAL;
+                goto out;
+            }
+            if (!key_id) {
+                pr_debug("Using 'best' PSK\n");
+                if (opts->tls_key) {
+                    key_put(opts->tls_key);
+                    opts->tls_key = NULL;
+                }
+                break;
+            }
+            key = key_lookup(key_id);
+            if (!key) {
+                pr_err("Key id %08x not found\n", key_id);
+                ret = -ENOKEY;
+                goto out;
+            }
+            if (opts->tls_key)
+                key_put(opts->tls_key);
+            opts->tls_key = key;
+#else
+            pr_err("TLS is not supported\n");
+            ret = -EINVAL;
+            goto out;
+#endif
+            break;

          case NVMF_OPT_DISCOVERY:
              opts->discovery_nqn = true;
              break;

drivers/nvme/host/fabrics.c +0 -0
@@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct 
 nvmf_ctrl_options *opts,
   void nvmf_free_options(struct nvmf_ctrl_options *opts)
   {
Quoted 8 lines

drivers/nvme/host/fabrics.c static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts, · line 1132

Quoted 6 lines
drivers/nvme/host/fabrics.c +5 -0
       nvmf_host_put(opts->host);
+#ifdef CONFIG_NVME_TLS
+    if (opts->keyring)
+        key_put(opts->keyring);
+    if (opts->tls_key)
+        key_put(opts->tls_key);

I think key_put is null safe.

Possibly. I'll check.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#41 lore raw

Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?

Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).

How do you expect that rotation would work with this?

How does nvmet handle a non-nvme keyring?

Quoted 1 line

With this option we can prepare a new keyring, and use that instead of the old one.

On an existing controller?

Quoted 1 line

(And it really doesn't add much complexity...)

I know, it just adds one more argument, and I want to understand if it is really needed.

#42 lore raw

Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'

Quoted 6 lines · Sagi Grimberg
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?

Quoted 7 lines
Quoted 4 lines

Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).

How do you expect that rotation would work with this?

The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyring

Quoted 2 lines

How does nvmet handle a non-nvme keyring?

There is a configfs attribute 'param_keyring', allowing you to use a different keyring (the nvme one is just the default).

Quoted 4 lines
Quoted 1 line

With this option we can prepare a new keyring, and use that instead of the old one.

On an existing controller?

Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.

Quoted 4 lines
Quoted 1 line

(And it really doesn't add much complexity...)

I know, it just adds one more argument, and I want to understand if it is really needed.

I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.

So I'd prefer to keep it.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)
maintainer
#43 lore raw

Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?

Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).

Quoted 7 lines
Quoted 2 lines

How do you expect that rotation would work with this?

The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyring

And the user cannot update the nvme keyring?

Quoted 3 lines
Quoted 2 lines

How does nvmet handle a non-nvme keyring?

There is a configfs attribute 'param_keyring', allowing you to use a different keyring (the nvme one is just the default).

OK.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

With this option we can prepare a new keyring, and use that instead of the old one.

On an existing controller?

Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(And it really doesn't add much complexity...)

I know, it just adds one more argument, and I want to understand if it is really needed.

I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.

I'm not sure I understand why authentication move to keyrings requires that the user is able to bring its own keyring.

Quoted 1 line

So I'd prefer to keep it.

I'm not against it, but I failing to understand in what situation a user _must_ send a different keyring to the driver (and not use the nvme keyring).

#44 lore raw

Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'

Quoted 14 lines · Sagi Grimberg
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?

Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).

How do you expect that rotation would work with this?

Quoted 8 lines
Quoted 5 lines

The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyring

And the user cannot update the nvme keyring?

Sure, he can. All I'm saying is that he might choose to create a different one (Different tenants, setups, what do I know).

Quoted 20 lines
Quoted 3 lines
Quoted 2 lines

How does nvmet handle a non-nvme keyring?

There is a configfs attribute 'param_keyring', allowing you to use a different keyring (the nvme one is just the default).

OK.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

With this option we can prepare a new keyring, and use that instead of the old one.

On an existing controller?

Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(And it really doesn't add much complexity...)

I know, it just adds one more argument, and I want to understand if it is really needed.

I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.

Quoted 8 lines

I'm not sure I understand why authentication move to keyrings requires that the user is able to bring its own keyring.

Quoted 1 line

So I'd prefer to keep it.

I'm not against it, but I failing to understand in what situation a user _must_ send a different keyring to the driver (and not use the nvme keyring).

Oh, there is no 'must'. He _can_. And under certain circumstances
(in a multi-tenant environment, say) there are reasons to keep the keys separate into several keyrings.
Bit like the '--key' parameter; the code will try to select the 'default' key, but if the user wants to use a different one by all means, do. Same for the keyring; we do have the default keyring, but if the user for whatever reason doesn't want to use it we're giving him the possibility for that.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)
maintainer
#45 lore raw

Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'

Quoted 50 lines · Hannes Reinecke
Quoted 14 lines · Sagi Grimberg
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

Parse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.

Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?

Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).

How do you expect that rotation would work with this?

Quoted 8 lines
Quoted 5 lines

The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyring

And the user cannot update the nvme keyring?

Sure, he can. All I'm saying is that he might choose to create a different one (Different tenants, setups, what do I know).

Quoted 25 lines
Quoted 3 lines
Quoted 2 lines

How does nvmet handle a non-nvme keyring?

There is a configfs attribute 'param_keyring', allowing you to use a different keyring (the nvme one is just the default).

OK.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

With this option we can prepare a new keyring, and use that instead of the old one.

On an existing controller?

Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

(And it really doesn't add much complexity...)

I know, it just adds one more argument, and I want to understand if it is really needed.

I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.

I'm not sure I understand why authentication move to keyrings requires that the user is able to bring its own keyring.

Quoted 1 line

So I'd prefer to keep it.

Quoted 8 lines
Quoted 3 lines

I'm not against it, but I failing to understand in what situation a user _must_ send a different keyring to the driver (and not use the nvme keyring).

Oh, there is no 'must'. He _can_. And under certain circumstances
(in a multi-tenant environment, say) there are reasons to keep the keys separate into several keyrings.
Bit like the '--key' parameter; the code will try to select the 'default' key, but if the user wants to use a different one by all means, do. Same for the keyring; we do have the default keyring, but if the user for whatever reason doesn't want to use it we're giving him the possibility for that.

I'm not overly keen to overload the argument space without proper real-life justification. But maybe it does make sense to allow users to store their secrets in separate keyrings...

So I'm not really opposed to this one.

#46 lore raw

PATCH v2 13/18 nvmet: make TCP sectype settable via configfs

In reply to Hannes Reinecke

Add a new configfs attribute 'addr_tsas' to make the TCP sectype settable via configfs.

---
 drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
drivers/nvme/target/configfs.c +67 -0
index 907143870da5..ca66ee6dc153 100644
@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port)
 	port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
 }
 
+static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int sectype)
+{
+	port->disc_addr.tsas.tcp.sectype = sectype;
+}
+
 static ssize_t nvmet_addr_trtype_store(struct config_item *item,
 		const char *page, size_t count)
 {
@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct config_item *item,
 	port->disc_addr.trtype = nvmet_transport[i].type;
 	if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
 		nvmet_port_init_tsas_rdma(port);
+	else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
+		nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
 	return count;
 }
 
 CONFIGFS_ATTR(nvmet_, addr_trtype);
 
+static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
+	{ NVMF_TCP_SECTYPE_NONE,	"none" },
+	{ NVMF_TCP_SECTYPE_TLS13,	"tls1.3" },
+};
+
+static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
+	{ NVMF_RDMA_QPTYPE_CONNECTED,	"connected" },
+	{ NVMF_RDMA_QPTYPE_DATAGRAM,	"datagram"  },
+};
+
+static ssize_t nvmet_addr_tsas_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	int i;
+
+	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+			if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type)
+				return sprintf(page, "%s\n", nvmet_addr_tsas_tcp[i].name);
+		}
+		return sprintf(page, "reserved\n");
+	} else if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) {
+		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_rdma); i++) {
+			if (port->disc_addr.tsas.rdma.qptype == nvmet_addr_tsas_rdma[i].type)
+				return sprintf(page, "%s\n", nvmet_addr_tsas_rdma[i].name);
+		}
+		return sprintf(page, "reserved\n");
+	}
+	return sprintf(page, "not required\n");
+}
+
+static ssize_t nvmet_addr_tsas_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	int i;
+
+	if (nvmet_is_port_enabled(port, __func__))
+		return -EACCES;
+
+	if (port->disc_addr.trtype != NVMF_TRTYPE_TCP)
+		return -EINVAL;
+
+	for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+		if (sysfs_streq(page, nvmet_addr_tsas_tcp[i].name))
+			goto found;
+	}
+
+	pr_err("Invalid value '%s' for tsas\n", page);
+	return -EINVAL;
+
+found:
+	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_, addr_tsas);
+
 /*
  * Namespace structures & file operation functions below
  */
@@ -1741,6 +1807,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = {
 	&nvmet_attr_addr_traddr,
 	&nvmet_attr_addr_trsvcid,
 	&nvmet_attr_addr_trtype,
+	&nvmet_attr_addr_tsas,
 	&nvmet_attr_param_inline_data_size,
 #ifdef CONFIG_BLK_DEV_INTEGRITY
 	&nvmet_attr_param_pi_enable,
Signature · 1 line
2.35.3
maintainer
#47 lore raw

Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs

Quoted 48 lines

Add a new configfs attribute 'addr_tsas' to make the TCP sectype settable via configfs.

---
  drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++
  1 file changed, 67 insertions(+)
drivers/nvme/target/configfs.c +19 -0
index 907143870da5..ca66ee6dc153 100644
@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port)
  	port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
  }
 
+static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int sectype)
+{
+	port->disc_addr.tsas.tcp.sectype = sectype;
+}
+
  static ssize_t nvmet_addr_trtype_store(struct config_item *item,
  		const char *page, size_t count)
  {
@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct config_item *item,
  	port->disc_addr.trtype = nvmet_transport[i].type;
  	if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
  		nvmet_port_init_tsas_rdma(port);
+	else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
+		nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
  	return count;
  }
 
  CONFIGFS_ATTR(nvmet_, addr_trtype);
 
+static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
+	{ NVMF_TCP_SECTYPE_NONE,	"none" },
+	{ NVMF_TCP_SECTYPE_TLS13,	"tls1.3" },
+};
+
+static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
+	{ NVMF_RDMA_QPTYPE_CONNECTED,	"connected" },
+	{ NVMF_RDMA_QPTYPE_DATAGRAM,	"datagram"  },
+};
+
+static ssize_t nvmet_addr_tsas_show(struct config_item *item,
+		char *page)

drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct config_item *item, · line 361

Quoted 10 lines
drivers/nvme/target/configfs.c +10 -0
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	int i;
+
+	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+			if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type)
+				return sprintf(page, "%s\n", nvmet_addr_tsas_tcp[i].name);
+		}
+		return sprintf(page, "reserved\n");

What is reserved?

Quoted 48 lines
drivers/nvme/target/configfs.c +38 -0
+	} else if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) {
+		for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_rdma); i++) {
+			if (port->disc_addr.tsas.rdma.qptype == nvmet_addr_tsas_rdma[i].type)
+				return sprintf(page, "%s\n", nvmet_addr_tsas_rdma[i].name);
+		}
+		return sprintf(page, "reserved\n");
+	}
+	return sprintf(page, "not required\n");
+}
+
+static ssize_t nvmet_addr_tsas_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	int i;
+
+	if (nvmet_is_port_enabled(port, __func__))
+		return -EACCES;
+
+	if (port->disc_addr.trtype != NVMF_TRTYPE_TCP)
+		return -EINVAL;
+
+	for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+		if (sysfs_streq(page, nvmet_addr_tsas_tcp[i].name))
+			goto found;
+	}
+
+	pr_err("Invalid value '%s' for tsas\n", page);
+	return -EINVAL;
+
+found:
+	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_, addr_tsas);
+
  /*
   * Namespace structures & file operation functions below
   */
@@ -1741,6 +1807,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = {
  	&nvmet_attr_addr_traddr,
  	&nvmet_attr_addr_trsvcid,
  	&nvmet_attr_addr_trtype,
+	&nvmet_attr_addr_tsas,
  	&nvmet_attr_param_inline_data_size,
  #ifdef CONFIG_BLK_DEV_INTEGRITY
  	&nvmet_attr_param_pi_enable,
#48 lore raw

Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs

Quoted 54 lines · Sagi Grimberg
Quoted 54 lines

Add a new configfs attribute 'addr_tsas' to make the TCP sectype settable via configfs.

---
  drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++

  1 file changed, 67 insertions(+)
drivers/nvme/target/configfs.c +5 -0
index 907143870da5..ca66ee6dc153 100644
@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct 
 nvmet_port *port)
       port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
   }
+static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int 
 sectype)
+{
+    port->disc_addr.tsas.tcp.sectype = sectype;
+}
+
   static ssize_t nvmet_addr_trtype_store(struct config_item *item,
           const char *page, size_t count)

  {

drivers/nvme/target/configfs.c +20 -0
@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct 
 config_item *item,
       port->disc_addr.trtype = nvmet_transport[i].type;
       if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
           nvmet_port_init_tsas_rdma(port);
+    else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
+        nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
       return count;
   }
   CONFIGFS_ATTR(nvmet_, addr_trtype);
+static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
+    { NVMF_TCP_SECTYPE_NONE,    "none" },
+    { NVMF_TCP_SECTYPE_TLS13,    "tls1.3" },
+};
+
+static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
+    { NVMF_RDMA_QPTYPE_CONNECTED,    "connected" },
+    { NVMF_RDMA_QPTYPE_DATAGRAM,    "datagram"  },
+};
+
+static ssize_t nvmet_addr_tsas_show(struct config_item *item,
+        char *page)
+{
+    struct nvmet_port *port = to_nvmet_port(item);
+    int i;
+
+    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
Quoted 9 lines

drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct · line 362

Quoted 6 lines
drivers/nvme/target/configfs.c +4 -0
+            if (port->disc_addr.tsas.tcp.sectype == 
 nvmet_addr_tsas_tcp[i].type)
+                return sprintf(page, "%s\n", 
 nvmet_addr_tsas_tcp[i].name);
+        }
+        return sprintf(page, "reserved\n");

What is reserved?

All other values?

Quoted 43 lines
Quoted 43 lines
drivers/nvme/target/configfs.c +37 -0
+    } else if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) {
+        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_rdma); i++) {
+            if (port->disc_addr.tsas.rdma.qptype == 
 nvmet_addr_tsas_rdma[i].type)
+                return sprintf(page, "%s\n", 
 nvmet_addr_tsas_rdma[i].name);
+        }
+        return sprintf(page, "reserved\n");
+    }
+    return sprintf(page, "not required\n");
+}
+
+static ssize_t nvmet_addr_tsas_store(struct config_item *item,
+        const char *page, size_t count)
+{
+    struct nvmet_port *port = to_nvmet_port(item);
+    int i;
+
+    if (nvmet_is_port_enabled(port, __func__))
+        return -EACCES;
+
+    if (port->disc_addr.trtype != NVMF_TRTYPE_TCP)
+        return -EINVAL;
+
+    for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+        if (sysfs_streq(page, nvmet_addr_tsas_tcp[i].name))
+            goto found;
+    }
+
+    pr_err("Invalid value '%s' for tsas\n", page);
+    return -EINVAL;
+
+found:
+    nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
+    return count;
+}
+
+CONFIGFS_ATTR(nvmet_, addr_tsas);
+

  /*
   * Namespace structures & file operation functions below */

drivers/nvme/target/configfs.c +0 -0
@@ -1741,6 +1807,7 @@ static struct configfs_attribute 
Quoted 8 lines
Quoted 8 lines
drivers/nvme/target/configfs.c +1 -0
 *nvmet_port_attrs[] = {
       &nvmet_attr_addr_traddr,
       &nvmet_attr_addr_trsvcid,
       &nvmet_attr_addr_trtype,
+    &nvmet_attr_addr_tsas,
       &nvmet_attr_param_inline_data_size,
   #ifdef CONFIG_BLK_DEV_INTEGRITY

      &nvmet_attr_param_pi_enable,

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
maintainer
#49 lore raw

Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs

Quoted 63 lines
Quoted 54 lines
Quoted 54 lines

Add a new configfs attribute 'addr_tsas' to make the TCP sectype settable via configfs.

---
  drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++

  1 file changed, 67 insertions(+)
drivers/nvme/target/configfs.c +5 -0
index 907143870da5..ca66ee6dc153 100644
@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct 
 nvmet_port *port)
       port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
   }
+static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int 
 sectype)
+{
+    port->disc_addr.tsas.tcp.sectype = sectype;
+}
+
   static ssize_t nvmet_addr_trtype_store(struct config_item *item,
           const char *page, size_t count)

  {

drivers/nvme/target/configfs.c +20 -0
@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct 
 config_item *item,
       port->disc_addr.trtype = nvmet_transport[i].type;
       if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
           nvmet_port_init_tsas_rdma(port);
+    else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
+        nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
       return count;
   }
   CONFIGFS_ATTR(nvmet_, addr_trtype);
+static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
+    { NVMF_TCP_SECTYPE_NONE,    "none" },
+    { NVMF_TCP_SECTYPE_TLS13,    "tls1.3" },
+};
+
+static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
+    { NVMF_RDMA_QPTYPE_CONNECTED,    "connected" },
+    { NVMF_RDMA_QPTYPE_DATAGRAM,    "datagram"  },
+};
+
+static ssize_t nvmet_addr_tsas_show(struct config_item *item,
+        char *page)
+{
+    struct nvmet_port *port = to_nvmet_port(item);
+    int i;
+
+    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
Quoted 9 lines

drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct · line 362

Quoted 6 lines
drivers/nvme/target/configfs.c +4 -0
+            if (port->disc_addr.tsas.tcp.sectype == 
 nvmet_addr_tsas_tcp[i].type)
+                return sprintf(page, "%s\n", 
 nvmet_addr_tsas_tcp[i].name);
+        }
+        return sprintf(page, "reserved\n");

What is reserved?

Quoted 1 line

All other values?

Yes, you should just consolidate it outside the if-else instead of "not required" (which is a treq value iirc).

#50 lore raw

Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs

Quoted 64 lines · Sagi Grimberg
Quoted 64 lines
Quoted 64 lines
Quoted 51 lines

Add a new configfs attribute 'addr_tsas' to make the TCP sectype settable via configfs.

---
  drivers/nvme/target/configfs.c | 67
++++++++++++++++++++++++++++++++++

  1 file changed, 67 insertions(+)
drivers/nvme/target/configfs.c +5 -0
index 907143870da5..ca66ee6dc153 100644
@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct 
 nvmet_port *port)
       port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM;
   }
+static void nvmet_port_init_tsas_tcp(struct nvmet_port *port, int 
 sectype)
+{
+    port->disc_addr.tsas.tcp.sectype = sectype;
+}
+
   static ssize_t nvmet_addr_trtype_store(struct config_item *item,
           const char *page, size_t count)

  {

drivers/nvme/target/configfs.c +16 -0
@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct 
 config_item *item,
       port->disc_addr.trtype = nvmet_transport[i].type;
       if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA)
           nvmet_port_init_tsas_rdma(port);
+    else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP)
+        nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE);
       return count;
   }
   CONFIGFS_ATTR(nvmet_, addr_trtype);
+static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = {
+    { NVMF_TCP_SECTYPE_NONE,    "none" },
+    { NVMF_TCP_SECTYPE_TLS13,    "tls1.3" },
+};
+
+static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = {
+    { NVMF_RDMA_QPTYPE_CONNECTED,    "connected" },
+    { NVMF_RDMA_QPTYPE_DATAGRAM,    "datagram"  },
+};
+
+static ssize_t nvmet_addr_tsas_show(struct config_item *item,
+        char *page)
+{
+    struct nvmet_port *port = to_nvmet_port(item);

drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct · line 362

Quoted 10 lines
drivers/nvme/target/configfs.c +8 -0
+    int i;
+
+    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+        for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
+            if (port->disc_addr.tsas.tcp.sectype == 
 nvmet_addr_tsas_tcp[i].type)
+                return sprintf(page, "%s\n", 
 nvmet_addr_tsas_tcp[i].name);
+        }
+        return sprintf(page, "reserved\n");

What is reserved?

Quoted 5 lines
Quoted 1 line

All other values?

Yes, you should just consolidate it outside the if-else instead of "not required" (which is a treq value iirc).

Okay, will do.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)
#51 lore raw

PATCH v2 14/18 nvmet-tcp: allocate socket file

In reply to Hannes Reinecke

When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.

---
 drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++----------
 1 file changed, 38 insertions(+), 13 deletions(-)
drivers/nvme/target/tcp.c +38 -13
index 66e8f9fd0ca7..5931971d715f 100644
@@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {
 
 enum nvmet_tcp_queue_state {
 	NVMET_TCP_Q_CONNECTING,
+	NVMET_TCP_Q_TLS_HANDSHAKE,
 	NVMET_TCP_Q_LIVE,
 	NVMET_TCP_Q_DISCONNECTING,
 };
 
 struct nvmet_tcp_queue {
 	struct socket		*sock;
+	struct file		*sock_file;
 	struct nvmet_tcp_port	*port;
 	struct work_struct	io_work;
 	struct nvmet_cq		nvme_cq;
@@ -1406,6 +1408,19 @@ static void nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)
 	write_unlock_bh(&sock->sk->sk_callback_lock);
 }
 
+static void nvmet_tcp_close_sock(struct nvmet_tcp_queue *queue)
+{
+	if (queue->sock_file) {
+		fput(queue->sock_file);
+		queue->sock_file = NULL;
+		queue->sock = NULL;
+	} else {
+		WARN_ON(!queue->sock->ops);
+		sock_release(queue->sock);
+		queue->sock = NULL;
+	}
+}
+
 static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue)
 {
 	struct nvmet_tcp_cmd *cmd = queue->cmds;
@@ -1455,12 +1470,11 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
 	nvmet_sq_destroy(&queue->nvme_sq);
 	cancel_work_sync(&queue->io_work);
 	nvmet_tcp_free_cmd_data_in_buffers(queue);
-	sock_release(queue->sock);
+	nvmet_tcp_close_sock(queue);
 	nvmet_tcp_free_cmds(queue);
 	if (queue->hdr_digest || queue->data_digest)
 		nvmet_tcp_free_crypto(queue);
 	ida_free(&nvmet_tcp_queue_ida, queue->idx);
-
 	page = virt_to_head_page(queue->pf_cache.va);
 	__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
 	kfree(queue);
@@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)
 	return ret;
 }
 
-static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
+static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 		struct socket *newsock)
 {
 	struct nvmet_tcp_queue *queue;
@@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 
 	queue = kzalloc(sizeof(*queue), GFP_KERNEL);
 	if (!queue)
-		return -ENOMEM;
+		return;
 
 	INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work);
 	INIT_WORK(&queue->io_work, nvmet_tcp_io_work);
@@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	queue->port = port;
 	queue->nr_cmds = 0;
 	spin_lock_init(&queue->state_lock);
-	queue->state = NVMET_TCP_Q_CONNECTING;
+	if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
+	    NVMF_TCP_SECTYPE_TLS13)
+		queue->state = NVMET_TCP_Q_TLS_HANDSHAKE;
+	else
+		queue->state = NVMET_TCP_Q_CONNECTING;
 	INIT_LIST_HEAD(&queue->free_list);
 	init_llist_head(&queue->resp_list);
 	INIT_LIST_HEAD(&queue->resp_send_list);
 
+	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
+		queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL);
+		if (IS_ERR(queue->sock_file)) {
+			ret = PTR_ERR(queue->sock_file);
+			queue->sock_file = NULL;
+			goto out_free_queue;
+		}
+	}
+
 	queue->idx = ida_alloc(&nvmet_tcp_queue_ida, GFP_KERNEL);
 	if (queue->idx < 0) {
 		ret = queue->idx;
-		goto out_free_queue;
+		goto out_sock;
 	}
 
 	ret = nvmet_tcp_alloc_cmd(queue, &queue->connect);
@@ -1628,7 +1655,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	if (ret)
 		goto out_destroy_sq;
 
-	return 0;
+	return;
 out_destroy_sq:
 	mutex_lock(&nvmet_tcp_queue_mutex);
 	list_del_init(&queue->queue_list);
@@ -1638,9 +1665,11 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	nvmet_tcp_free_cmd(&queue->connect);
 out_ida_remove:
 	ida_free(&nvmet_tcp_queue_ida, queue->idx);
+out_sock:
+	nvmet_tcp_close_sock(queue);
 out_free_queue:
 	kfree(queue);
-	return ret;
+	pr_err("failed to allocate queue");
 }
 
 static void nvmet_tcp_accept_work(struct work_struct *w)
@@ -1657,11 +1686,7 @@ static void nvmet_tcp_accept_work(struct work_struct *w)
 				pr_warn("failed to accept err=%d\n", ret);
 			return;
 		}
-		ret = nvmet_tcp_alloc_queue(port, newsock);
-		if (ret) {
-			pr_err("failed to allocate queue\n");
-			sock_release(newsock);
-		}
+		nvmet_tcp_alloc_queue(port, newsock);
 	}
 }
 
Signature · 1 line
2.35.3
maintainer
#52 lore raw

Re: [PATCH 14/18] nvmet-tcp: allocate socket file

Quoted 91 lines · Hannes Reinecke

When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.

---
  drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++----------
  1 file changed, 38 insertions(+), 13 deletions(-)
drivers/nvme/target/tcp.c +23 -5
index 66e8f9fd0ca7..5931971d715f 100644
@@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {
 
  enum nvmet_tcp_queue_state {
  	NVMET_TCP_Q_CONNECTING,
+	NVMET_TCP_Q_TLS_HANDSHAKE,
  	NVMET_TCP_Q_LIVE,
  	NVMET_TCP_Q_DISCONNECTING,
  };
 
  struct nvmet_tcp_queue {
  	struct socket		*sock;
+	struct file		*sock_file;
  	struct nvmet_tcp_port	*port;
  	struct work_struct	io_work;
  	struct nvmet_cq		nvme_cq;
@@ -1406,6 +1408,19 @@ static void nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)
  	write_unlock_bh(&sock->sk->sk_callback_lock);
  }
 
+static void nvmet_tcp_close_sock(struct nvmet_tcp_queue *queue)
+{
+	if (queue->sock_file) {
+		fput(queue->sock_file);
+		queue->sock_file = NULL;
+		queue->sock = NULL;
+	} else {
+		WARN_ON(!queue->sock->ops);
+		sock_release(queue->sock);
+		queue->sock = NULL;
+	}
+}
+
  static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue)
  {
  	struct nvmet_tcp_cmd *cmd = queue->cmds;
@@ -1455,12 +1470,11 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
  	nvmet_sq_destroy(&queue->nvme_sq);
  	cancel_work_sync(&queue->io_work);
  	nvmet_tcp_free_cmd_data_in_buffers(queue);
-	sock_release(queue->sock);
+	nvmet_tcp_close_sock(queue);
  	nvmet_tcp_free_cmds(queue);
  	if (queue->hdr_digest || queue->data_digest)
  		nvmet_tcp_free_crypto(queue);
  	ida_free(&nvmet_tcp_queue_ida, queue->idx);
-
  	page = virt_to_head_page(queue->pf_cache.va);
  	__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
  	kfree(queue);
@@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)
  	return ret;
  }
 
-static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
+static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
  		struct socket *newsock)
  {
  	struct nvmet_tcp_queue *queue;
@@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 
  	queue = kzalloc(sizeof(*queue), GFP_KERNEL);
  	if (!queue)
-		return -ENOMEM;
+		return;
 
  	INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work);
  	INIT_WORK(&queue->io_work, nvmet_tcp_io_work);
@@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
  	queue->port = port;
  	queue->nr_cmds = 0;
  	spin_lock_init(&queue->state_lock);
-	queue->state = NVMET_TCP_Q_CONNECTING;
+	if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
+	    NVMF_TCP_SECTYPE_TLS13)
+		queue->state = NVMET_TCP_Q_TLS_HANDSHAKE;
+	else
+		queue->state = NVMET_TCP_Q_CONNECTING;
  	INIT_LIST_HEAD(&queue->free_list);
  	init_llist_head(&queue->resp_list);

drivers/nvme/target/tcp.c static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, · line 1632

Quoted 10 lines
drivers/nvme/target/tcp.c +8 -0
  	INIT_LIST_HEAD(&queue->resp_send_list);
 
+	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
+		queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL);
+		if (IS_ERR(queue->sock_file)) {
+			ret = PTR_ERR(queue->sock_file);
+			queue->sock_file = NULL;
+			goto out_free_queue;
+		}
+	}

Why not always allocate a sock_file? Like in the host?

#53 lore raw

Re: [PATCH 14/18] nvmet-tcp: allocate socket file

Quoted 97 lines · Sagi Grimberg
Quoted 94 lines · Hannes Reinecke

When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.

---
  drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++----------

  1 file changed, 38 insertions(+), 13 deletions(-)
drivers/nvme/target/tcp.c +16 -2
index 66e8f9fd0ca7..5931971d715f 100644
@@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {
   enum nvmet_tcp_queue_state {
       NVMET_TCP_Q_CONNECTING,
+    NVMET_TCP_Q_TLS_HANDSHAKE,
       NVMET_TCP_Q_LIVE,
       NVMET_TCP_Q_DISCONNECTING,
   };
   struct nvmet_tcp_queue {
       struct socket        *sock;
+    struct file        *sock_file;
       struct nvmet_tcp_port    *port;
       struct work_struct    io_work;
       struct nvmet_cq        nvme_cq;
@@ -1406,6 +1408,19 @@ static void 
 nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)
       write_unlock_bh(&sock->sk->sk_callback_lock);
   }
+static void nvmet_tcp_close_sock(struct nvmet_tcp_queue *queue)
+{
+    if (queue->sock_file) {
+        fput(queue->sock_file);
+        queue->sock_file = NULL;
+        queue->sock = NULL;
+    } else {
+        WARN_ON(!queue->sock->ops);
+        sock_release(queue->sock);
+        queue->sock = NULL;
+    }
+}
+
   static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue)
   {
       struct nvmet_tcp_cmd *cmd = queue->cmds;
@@ -1455,12 +1470,11 @@ static void 
 nvmet_tcp_release_queue_work(struct work_struct *w)
       nvmet_sq_destroy(&queue->nvme_sq);
       cancel_work_sync(&queue->io_work);
       nvmet_tcp_free_cmd_data_in_buffers(queue);
-    sock_release(queue->sock);
+    nvmet_tcp_close_sock(queue);
       nvmet_tcp_free_cmds(queue);
       if (queue->hdr_digest || queue->data_digest)
           nvmet_tcp_free_crypto(queue);
       ida_free(&nvmet_tcp_queue_ida, queue->idx);
-
       page = virt_to_head_page(queue->pf_cache.va);
       __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);

      kfree(queue);

drivers/nvme/target/tcp.c +9 -3
@@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct 
 nvmet_tcp_queue *queue)
       return ret;
   }
-static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
+static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
           struct socket *newsock)
   {
       struct nvmet_tcp_queue *queue;
@@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct 
 nvmet_tcp_port *port,
       queue = kzalloc(sizeof(*queue), GFP_KERNEL);
       if (!queue)
-        return -ENOMEM;
+        return;
       INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work);
       INIT_WORK(&queue->io_work, nvmet_tcp_io_work);
@@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct 
 nvmet_tcp_port *port,
       queue->port = port;
       queue->nr_cmds = 0;
       spin_lock_init(&queue->state_lock);
-    queue->state = NVMET_TCP_Q_CONNECTING;
+    if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
+        NVMF_TCP_SECTYPE_TLS13)
+        queue->state = NVMET_TCP_Q_TLS_HANDSHAKE;
+    else
+        queue->state = NVMET_TCP_Q_CONNECTING;
       INIT_LIST_HEAD(&queue->free_list);
       init_llist_head(&queue->resp_list);
       INIT_LIST_HEAD(&queue->resp_send_list);
+    if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
+        queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, 
 NULL);
Quoted 8 lines

drivers/nvme/target/tcp.c static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, · line 1632

Quoted 6 lines
drivers/nvme/target/tcp.c +6 -0
+        if (IS_ERR(queue->sock_file)) {
+            ret = PTR_ERR(queue->sock_file);
+            queue->sock_file = NULL;
+            goto out_free_queue;
+        }
+    }

Why not always allocate a sock_file? Like in the host?

Wasn't sure if you'd go with it.
But sure.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
#54 lore raw

PATCH v2 15/18 nvmet-tcp: enable TLS handshake upcall

In reply to Hannes Reinecke

Add functions to start the TLS handshake upcall when the TCP RSAS sectype is set to 'tls1.3'.

---
 drivers/nvme/target/configfs.c |  32 +++++++-
 drivers/nvme/target/tcp.c      | 135 ++++++++++++++++++++++++++++++++-
 2 files changed, 163 insertions(+), 4 deletions(-)
drivers/nvme/target/configfs.c +30 -2
index ca66ee6dc153..36fbf6a22d09 100644
@@ -159,10 +159,12 @@ static const struct nvmet_type_name_map nvmet_addr_treq[] = {
 	{ NVMF_TREQ_NOT_REQUIRED,	"not required" },
 };
 
+#define NVMET_PORT_TREQ(port) ((port)->disc_addr.treq & NVME_TREQ_SECURE_CHANNEL_MASK)
+
 static ssize_t nvmet_addr_treq_show(struct config_item *item, char *page)
 {
-	u8 treq = to_nvmet_port(item)->disc_addr.treq &
-		NVME_TREQ_SECURE_CHANNEL_MASK;
+	struct nvmet_port *port = to_nvmet_port(item);
+	u8 treq = NVMET_PORT_TREQ(port);
 	int i;
 
 	for (i = 0; i < ARRAY_SIZE(nvmet_addr_treq); i++) {
@@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct config_item *item,
 	return -EINVAL;
 
 found:
+#ifdef CONFIG_NVME_TLS
+	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+		if (port->disc_addr.tsas.tcp.sectype != NVMF_TCP_SECTYPE_TLS13) {
+			pr_warn("cannot change TREQ when TLS is not enabled\n");
+			return -EINVAL;
+		} else if (nvmet_addr_treq[i].type == NVMF_TREQ_NOT_SPECIFIED) {
+			pr_warn("cannot set TREQ to 'not specified' when TLS is enabled\n");
+			return -EINVAL;
+		}
+	}
+#endif
 	treq |= nvmet_addr_treq[i].type;
 	port->disc_addr.treq = treq;
 	return count;
@@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 		const char *page, size_t count)
 {
 	struct nvmet_port *port = to_nvmet_port(item);
+	u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK;
 	int i;
 
 	if (nvmet_is_port_enabled(port, __func__))
@@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 
 found:
 	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
+	if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) {
+#ifdef CONFIG_NVME_TLS
+		if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED)
+			treq |= NVMF_TREQ_REQUIRED;
+		else
+			treq |= NVMET_PORT_TREQ(port);
+#else
+		pr_err("TLS not supported\n");
+		return -EINVAL;
+#endif
+	} else {
+		/* Set to 'not specified' if TLS is not enabled */
+		treq |= NVMF_TREQ_NOT_SPECIFIED;
+	}
 	return count;
 }
 
drivers/nvme/target/tcp.c +133 -2

195 lines not on this page · raw message

Signature · 1 line
2.35.3
maintainer
#55 lore raw

Re: [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall

Quoted 3 lines

Add functions to start the TLS handshake upcall when the TCP RSAS sectype is set to 'tls1.3'.

TSAS

Quoted 12 lines
---
  drivers/nvme/target/configfs.c |  32 +++++++-
  drivers/nvme/target/tcp.c      | 135 ++++++++++++++++++++++++++++++++-
  2 files changed, 163 insertions(+), 4 deletions(-)
drivers/nvme/target/configfs.c +0 -0
index ca66ee6dc153..36fbf6a22d09 100644

drivers/nvme/target/configfs.c static const struct nvmet_type_name_map nvmet_addr_treq[] = { · line 162

Quoted 4 lines
drivers/nvme/target/configfs.c +1 -0
@@ -159,10 +159,12 @@ static const struct nvmet_type_name_map nvmet_addr_treq[] = {
  	{ NVMF_TREQ_NOT_REQUIRED,	"not required" },
  };
 
+#define NVMET_PORT_TREQ(port) ((port)->disc_addr.treq & NVME_TREQ_SECURE_CHANNEL_MASK)

Can you make it a static inline?

Quoted 14 lines
drivers/nvme/target/configfs.c +3 -2
+
  static ssize_t nvmet_addr_treq_show(struct config_item *item, char *page)
  {
-	u8 treq = to_nvmet_port(item)->disc_addr.treq &
-		NVME_TREQ_SECURE_CHANNEL_MASK;
+	struct nvmet_port *port = to_nvmet_port(item);
+	u8 treq = NVMET_PORT_TREQ(port);
  	int i;
 
  	for (i = 0; i < ARRAY_SIZE(nvmet_addr_treq); i++) {
@@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct config_item *item,
  	return -EINVAL;
 
  found:

drivers/nvme/target/configfs.c static ssize_t nvmet_addr_treq_store(struct config_item *item, · line 207

Quoted 10 lines
drivers/nvme/target/configfs.c +10 -0
+#ifdef CONFIG_NVME_TLS
+	if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+		if (port->disc_addr.tsas.tcp.sectype != NVMF_TCP_SECTYPE_TLS13) {
+			pr_warn("cannot change TREQ when TLS is not enabled\n");
+			return -EINVAL;
+		} else if (nvmet_addr_treq[i].type == NVMF_TREQ_NOT_SPECIFIED) {
+			pr_warn("cannot set TREQ to 'not specified' when TLS is enabled\n");
+			return -EINVAL;
+		}
+	}

Is this code wrong if CONFIG_NVME_TLS is not enabled?

Quoted 13 lines
drivers/nvme/target/configfs.c +2 -0
+#endif
  	treq |= nvmet_addr_treq[i].type;
  	port->disc_addr.treq = treq;
  	return count;
@@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
  		const char *page, size_t count)
  {
  	struct nvmet_port *port = to_nvmet_port(item);
+	u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK;
  	int i;
 
  	if (nvmet_is_port_enabled(port, __func__))

drivers/nvme/target/configfs.c static ssize_t nvmet_addr_tsas_store(struct config_item *item, · line 409

Quoted 5 lines
drivers/nvme/target/configfs.c +2 -0
@@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 
  found:
  	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
+	if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) {
+#ifdef CONFIG_NVME_TLS

Maybe in the start of the function just do:

	if (!IS_ENABLED(CONFIG_NVME_TLS)) {
		pr_err("TLS not supported\n");
		return -EINVAL;
	}

Instead of incorporating it here.

Quoted 20 lines
drivers/nvme/target/configfs.c +12 -0
+		if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED)
+			treq |= NVMF_TREQ_REQUIRED;
+		else
+			treq |= NVMET_PORT_TREQ(port);
+#else
+		pr_err("TLS not supported\n");
+		return -EINVAL;
+#endif
+	} else {
+		/* Set to 'not specified' if TLS is not enabled */
+		treq |= NVMF_TREQ_NOT_SPECIFIED;
+	}
  	return count;
  }
 
drivers/nvme/target/tcp.c +0 -0
index 5931971d715f..ebec882120fd 100644

drivers/nvme/target/tcp.c line 15

Quoted 5 lines
drivers/nvme/target/tcp.c +2 -0
@@ -11,6 +11,10 @@
  #include <linux/nvme-tcp.h>
  #include <net/sock.h>
  #include <net/tcp.h>
+#ifdef CONFIG_NVME_TLS
+#include <net/handshake.h>

Is net/handshake.h under an ifdef? If so, CONFIG_NVME_TLS should select it.

drivers/nvme/target/tcp.c line 16

Quoted 1 line
drivers/nvme/target/tcp.c +1 -0
+#include <linux/nvme-keyring.h>

will this include not work if CONFIG_NVME_TLS not work? <linux/nvme-auth.h> is not under CONFIG_NVME_AUTH for example.

Quoted 22 lines
drivers/nvme/target/tcp.c +11 -0
+#endif
  #include <linux/inet.h>
  #include <linux/llist.h>
  #include <crypto/hash.h>
@@ -40,6 +44,16 @@ module_param(idle_poll_period_usecs, int, 0644);
  MODULE_PARM_DESC(idle_poll_period_usecs,
  		"nvmet tcp io_work poll till idle time period in usecs");
 
+#ifdef CONFIG_NVME_TLS
+/*
+ * TLS handshake timeout
+ */
+static int tls_handshake_timeout = 30;
+module_param(tls_handshake_timeout, int, 0644);
+MODULE_PARM_DESC(tls_handshake_timeout,
+		 "nvme TLS handshake timeout in seconds (default 30)");
+#endif
+
  #define NVMET_TCP_RECV_BUDGET		8
  #define NVMET_TCP_SEND_BUDGET		8
  #define NVMET_TCP_IO_WORK_BUDGET	64

drivers/nvme/target/tcp.c struct nvmet_tcp_queue { · line 150

Quoted 7 lines
drivers/nvme/target/tcp.c +4 -0
@@ -130,6 +144,10 @@ struct nvmet_tcp_queue {
  	bool			data_digest;
  	struct ahash_request	*snd_hash;
  	struct ahash_request	*rcv_hash;
+#ifdef CONFIG_NVME_TLS
+	struct key		*tls_psk;
+	struct delayed_work	tls_handshake_work;
+#endif

If these won't be under CONFIG_NVME_TLS will it save a lot of the other ifdefs in the code?

drivers/nvme/target/tcp.c static void nvmet_tcp_release_queue_work(struct work_struct *w) · line 1497

Quoted 10 lines
drivers/nvme/target/tcp.c +3 -0
  	unsigned long           poll_end;
 
@@ -1474,6 +1492,10 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)
  	nvmet_tcp_free_cmds(queue);
  	if (queue->hdr_digest || queue->data_digest)
  		nvmet_tcp_free_crypto(queue);
+#ifdef CONFIG_NVME_TLS
+	if (queue->tls_psk)
+		key_put(queue->tls_psk);

key_put is NULL safe.

Quoted 33 lines
drivers/nvme/target/tcp.c +17 -2
+#endif
  	ida_free(&nvmet_tcp_queue_ida, queue->idx);
  	page = virt_to_head_page(queue->pf_cache.va);
  	__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);
@@ -1488,8 +1510,12 @@ static void nvmet_tcp_data_ready(struct sock *sk)
 
  	read_lock_bh(&sk->sk_callback_lock);
  	queue = sk->sk_user_data;
-	if (likely(queue))
-		queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work);
+	if (queue->data_ready)
+		queue->data_ready(sk);
+	if (likely(queue) &&
+	    queue->state != NVMET_TCP_Q_TLS_HANDSHAKE)
+		queue_work_on(queue_cpu(queue), nvmet_tcp_wq,
+			      &queue->io_work);
  	read_unlock_bh(&sk->sk_callback_lock);
  }
 
@@ -1597,6 +1623,89 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)
  	return ret;
  }
 
+#ifdef CONFIG_NVME_TLS
+static void nvmet_tcp_tls_queue_restart(struct nvmet_tcp_queue *queue)
+{
+	spin_lock(&queue->state_lock);
+	if (queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) {
+		pr_warn("queue %d: TLS handshake already completed\n",
+			queue->idx);
+		spin_unlock(&queue->state_lock);
+		return;
+	}

drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1645

Quoted 10 lines
drivers/nvme/target/tcp.c +10 -0
+	queue->state = NVMET_TCP_Q_CONNECTING;
+	spin_unlock(&queue->state_lock);
+
+	pr_debug("queue %d: restarting queue after TLS handshake\n",
+		 queue->idx);
+	/*
+	 * Set callbacks after handshake; TLS implementation
+	 * might have changed the socket callbacks.
+	 */
+	nvmet_tcp_set_queue_sock(queue);

Maybe fold it into the caller? The name is confusing anyways. The queue is not restarted, it is post-configured for lack of a better term.

drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1649

Quoted 4 lines
drivers/nvme/target/tcp.c +4 -0
+}
+
+static void nvmet_tcp_tls_handshake_done(void *data, int status,
+					 key_serial_t peerid)

pskid.

drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1661

Quoted 12 lines
drivers/nvme/target/tcp.c +12 -0
+{
+	struct nvmet_tcp_queue *queue = data;
+
+	pr_debug("queue %d: TLS handshake done, key %x, status %d\n",
+		 queue->idx, peerid, status);
+	if (!status) {
+		spin_lock(&queue->state_lock);
+		queue->tls_psk = key_lookup(peerid);
+		if (IS_ERR(queue->tls_psk)) {
+			pr_warn("queue %d: TLS key %x not found\n",
+				queue->idx, peerid);
+			queue->tls_psk = NULL;

Here you let the timeout take care of it later?

#56 lore raw

Re: [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall

Quoted 6 lines · Sagi Grimberg
Quoted 3 lines

Add functions to start the TLS handshake upcall when the TCP RSAS sectype is set to 'tls1.3'.

TSAS

Ok.

Quoted 7 lines
Quoted 7 lines

---
  drivers/nvme/target/configfs.c |  32 +++++++- drivers/nvme/target/tcp.c      | 135 ++++++++++++++++++++++++++++++++-

  2 files changed, 163 insertions(+), 4 deletions(-)
Quoted 13 lines

drivers/nvme/target/configfs.c static const struct nvmet_type_name_map · line 163

Quoted 10 lines
drivers/nvme/target/configfs.c +1 -0
index ca66ee6dc153..36fbf6a22d09 100644
@@ -159,10 +159,12 @@ static const struct nvmet_type_name_map 
 nvmet_addr_treq[] = {
       { NVMF_TREQ_NOT_REQUIRED,    "not required" },
   };
+#define NVMET_PORT_TREQ(port) ((port)->disc_addr.treq & 
 NVME_TREQ_SECURE_CHANNEL_MASK)

Can you make it a static inline?

Sure.

Quoted 19 lines
Quoted 19 lines
drivers/nvme/target/configfs.c +3 -2
+
   static ssize_t nvmet_addr_treq_show(struct config_item *item, char *page)
   {
-    u8 treq = to_nvmet_port(item)->disc_addr.treq &
-        NVME_TREQ_SECURE_CHANNEL_MASK;
+    struct nvmet_port *port = to_nvmet_port(item);
+    u8 treq = NVMET_PORT_TREQ(port);
       int i;
       for (i = 0; i < ARRAY_SIZE(nvmet_addr_treq); i++) {
@@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct 
 config_item *item,
       return -EINVAL;

  found:
+#ifdef CONFIG_NVME_TLS
+    if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) {
+        if (port->disc_addr.tsas.tcp.sectype != NVMF_TCP_SECTYPE_TLS13) {
+            pr_warn("cannot change TREQ when TLS is not enabled\n"); +            return -EINVAL;

Quoted 9 lines
Quoted 6 lines

+        } else if (nvmet_addr_treq[i].type == NVMF_TREQ_NOT_SPECIFIED) { +            pr_warn("cannot set TREQ to 'not specified' when TLS is enabled\n");
+            return -EINVAL;
+        }
+    }

Is this code wrong if CONFIG_NVME_TLS is not enabled?

Strictly speaking, no; it just won't do anything except from having a different value in the discovery log page.

Quoted 21 lines
Quoted 14 lines

+#endif
      treq |= nvmet_addr_treq[i].type;
      port->disc_addr.treq = treq;
      return count;

drivers/nvme/target/configfs.c +1 -0
@@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct 
 config_item *item,
           const char *page, size_t count)
   {
       struct nvmet_port *port = to_nvmet_port(item);
+    u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK;
       int i;
       if (nvmet_is_port_enabled(port, __func__))
@@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct 
 config_item *item,
Quoted 4 lines

  found:
      nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type); +    if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) { +#ifdef CONFIG_NVME_TLS

Maybe in the start of the function just do:

Quoted 7 lines

    if (!IS_ENABLED(CONFIG_NVME_TLS)) {
        pr_err("TLS not supported\n");
        return -EINVAL;
    }

Instead of incorporating it here.

Ok.

Quoted 28 lines
Quoted 19 lines

+        if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED) +            treq |= NVMF_TREQ_REQUIRED;
+        else
+            treq |= NVMET_PORT_TREQ(port);

drivers/nvme/target/configfs.c +8 -0
+#else
+        pr_err("TLS not supported\n");
+        return -EINVAL;
+#endif
+    } else {
+        /* Set to 'not specified' if TLS is not enabled */
+        treq |= NVMF_TREQ_NOT_SPECIFIED;
+    }

      return count;
  }

drivers/nvme/target/tcp.c +0 -0
index 5931971d715f..ebec882120fd 100644

drivers/nvme/target/tcp.c line 15

Quoted 5 lines
drivers/nvme/target/tcp.c +2 -0
@@ -11,6 +11,10 @@
   #include <linux/nvme-tcp.h>
   #include <net/sock.h>
   #include <net/tcp.h>
+#ifdef CONFIG_NVME_TLS
+#include <net/handshake.h>

Is net/handshake.h under an ifdef? If so, CONFIG_NVME_TLS should select it.

Quoted 5 lines

drivers/nvme/target/tcp.c line 16

Quoted 1 line
drivers/nvme/target/tcp.c +1 -0
+#include <linux/nvme-keyring.h>

will this include not work if CONFIG_NVME_TLS not work? <linux/nvme-auth.h> is not under CONFIG_NVME_AUTH for example.

Hmm. Should. I can remove the ifdefs

Quoted 23 lines
Quoted 23 lines
drivers/nvme/target/tcp.c +11 -0
+#endif
   #include <linux/inet.h>
   #include <linux/llist.h>
   #include <crypto/hash.h>
@@ -40,6 +44,16 @@ module_param(idle_poll_period_usecs, int, 0644);
   MODULE_PARM_DESC(idle_poll_period_usecs,
           "nvmet tcp io_work poll till idle time period in usecs");
+#ifdef CONFIG_NVME_TLS
+/*
+ * TLS handshake timeout
+ */
+static int tls_handshake_timeout = 30;
+module_param(tls_handshake_timeout, int, 0644);
+MODULE_PARM_DESC(tls_handshake_timeout,
+         "nvme TLS handshake timeout in seconds (default 30)");
+#endif
+
   #define NVMET_TCP_RECV_BUDGET        8
   #define NVMET_TCP_SEND_BUDGET        8
   #define NVMET_TCP_IO_WORK_BUDGET    64
@@ -130,6 +144,10 @@ struct nvmet_tcp_queue {
       bool            data_digest;
       struct ahash_request    *snd_hash;
Quoted 9 lines

drivers/nvme/target/tcp.c struct nvmet_tcp_queue { · line 150

Quoted 5 lines
drivers/nvme/target/tcp.c +4 -0
       struct ahash_request    *rcv_hash;
+#ifdef CONFIG_NVME_TLS
+    struct key        *tls_psk;
+    struct delayed_work    tls_handshake_work;
+#endif

If these won't be under CONFIG_NVME_TLS will it save a lot of the other ifdefs in the code?

Wasn't sure if we always want to have it.
But if we do, sure, things will be easier.

Quoted 49 lines

drivers/nvme/target/tcp.c static void nvmet_tcp_release_queue_work(struct · line 1498

Quoted 9 lines
drivers/nvme/target/tcp.c +3 -0
       unsigned long           poll_end;
@@ -1474,6 +1492,10 @@ static void nvmet_tcp_release_queue_work(struct 
 work_struct *w)
       nvmet_tcp_free_cmds(queue);
       if (queue->hdr_digest || queue->data_digest)
           nvmet_tcp_free_crypto(queue);
+#ifdef CONFIG_NVME_TLS
+    if (queue->tls_psk)
+        key_put(queue->tls_psk);

key_put is NULL safe.

Quoted 37 lines
drivers/nvme/target/tcp.c +1 -0
+#endif
       ida_free(&nvmet_tcp_queue_ida, queue->idx);
       page = virt_to_head_page(queue->pf_cache.va);

      __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);

drivers/nvme/target/tcp.c +22 -2
@@ -1488,8 +1510,12 @@ static void nvmet_tcp_data_ready(struct sock *sk)
       read_lock_bh(&sk->sk_callback_lock);
       queue = sk->sk_user_data;
-    if (likely(queue))
-        queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work);
+    if (queue->data_ready)
+        queue->data_ready(sk);
+    if (likely(queue) &&
+        queue->state != NVMET_TCP_Q_TLS_HANDSHAKE)
+        queue_work_on(queue_cpu(queue), nvmet_tcp_wq,
+                  &queue->io_work);
       read_unlock_bh(&sk->sk_callback_lock);
   }
@@ -1597,6 +1623,89 @@ static int nvmet_tcp_set_queue_sock(struct 
 nvmet_tcp_queue *queue)
       return ret;
   }
+#ifdef CONFIG_NVME_TLS
+static void nvmet_tcp_tls_queue_restart(struct nvmet_tcp_queue *queue)
+{
+    spin_lock(&queue->state_lock);
+    if (queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) {
+        pr_warn("queue %d: TLS handshake already completed\n",
+            queue->idx);
+        spin_unlock(&queue->state_lock);
+        return;
+    }
+    queue->state = NVMET_TCP_Q_CONNECTING;
+    spin_unlock(&queue->state_lock);
+
+    pr_debug("queue %d: restarting queue after TLS handshake\n",
+         queue->idx);
+    /*
Quoted 9 lines

drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1645

Quoted 4 lines
drivers/nvme/target/tcp.c +4 -0
+     * Set callbacks after handshake; TLS implementation
+     * might have changed the socket callbacks.
+     */
+    nvmet_tcp_set_queue_sock(queue);

Maybe fold it into the caller? The name is confusing anyways. The queue is not restarted, it is post-configured for lack of a better term.

I see what I can do.

Quoted 7 lines

drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1649

Quoted 4 lines
drivers/nvme/target/tcp.c +4 -0
+}
+
+static void nvmet_tcp_tls_handshake_done(void *data, int status,
+                     key_serial_t peerid)

                    pskid.

Quoted 14 lines

drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1661

Quoted 12 lines
drivers/nvme/target/tcp.c +12 -0
+{
+    struct nvmet_tcp_queue *queue = data;
+
+    pr_debug("queue %d: TLS handshake done, key %x, status %d\n",
+         queue->idx, peerid, status);
+    if (!status) {
+        spin_lock(&queue->state_lock);
+        queue->tls_psk = key_lookup(peerid);
+        if (IS_ERR(queue->tls_psk)) {
+            pr_warn("queue %d: TLS key %x not found\n",
+                queue->idx, peerid);
+            queue->tls_psk = NULL;

Here you let the timeout take care of it later?

Well, this is a slightly odd case; we get a '0' status but failed to lookup the key.
_Technically_ we should be able to continue, but wasn't sure if I should.

But in the light of the key rotation discussion I probably should; new keys (and keyrings) might be provided at any time, so I might hit that case here.

Will be updating the code.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)
#57 lore raw

PATCH v2 16/18 nvmet-tcp: rework sendpage for kTLS

In reply to Hannes Reinecke

kTLS ->sendpage() doesn't support the MSG_EOR flag, so blank it out if kTLS is enabled.

---
 drivers/nvme/target/tcp.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/nvme/target/tcp.c +3 -3
index ebec882120fd..5daf16138471 100644
@@ -650,7 +650,7 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd,
 
 	if (!last_in_batch && cmd->queue->send_list_len)
 		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	else if (!cmd->queue->tls_psk)
 		flags |= MSG_EOR;
 
 	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu),
@@ -678,7 +678,7 @@ static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
 
 	if (!last_in_batch && cmd->queue->send_list_len)
 		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	else if (!cmd->queue->tls_psk)
 		flags |= MSG_EOR;
 
 	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu),
@@ -708,7 +708,7 @@ static int nvmet_try_send_ddgst(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
 
 	if (!last_in_batch && cmd->queue->send_list_len)
 		msg.msg_flags |= MSG_MORE;
-	else
+	else if (!cmd->queue->tls_psk)
 		msg.msg_flags |= MSG_EOR;
 
 	ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);
Signature · 1 line
2.35.3
maintainer
#58 lore raw

Re: [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS

Quoted 2 lines

kTLS ->sendpage() doesn't support the MSG_EOR flag, so blank it out if kTLS is enabled.

I'd prefer we don't do this, but lets wait for the verdict from Jakub.

Quoted 37 lines
---
  drivers/nvme/target/tcp.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
drivers/nvme/target/tcp.c +3 -3
index ebec882120fd..5daf16138471 100644
@@ -650,7 +650,7 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd,
 
  	if (!last_in_batch && cmd->queue->send_list_len)
  		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	else if (!cmd->queue->tls_psk)
  		flags |= MSG_EOR;
 
  	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu),
@@ -678,7 +678,7 @@ static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
 
  	if (!last_in_batch && cmd->queue->send_list_len)
  		flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST;
-	else
+	else if (!cmd->queue->tls_psk)
  		flags |= MSG_EOR;
 
  	ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu),
@@ -708,7 +708,7 @@ static int nvmet_try_send_ddgst(struct nvmet_tcp_cmd *cmd, bool last_in_batch)
 
  	if (!last_in_batch && cmd->queue->send_list_len)
  		msg.msg_flags |= MSG_MORE;
-	else
+	else if (!cmd->queue->tls_psk)
  		msg.msg_flags |= MSG_EOR;
 
  	ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);
#59 lore raw

PATCH v2 17/18 nvmet-tcp: control messages for recvmsg()

In reply to Hannes Reinecke

kTLS requires control messages for recvmsg() to relay any out-of-band TLS messages (eg TLS alerts) to the caller.

---
 drivers/nvme/target/tcp.c | 71 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
drivers/nvme/target/tcp.c +71 -0
index 5daf16138471..75017fd5da9f 100644
@@ -12,6 +12,7 @@
 #include <net/sock.h>
 #include <net/tcp.h>
 #ifdef CONFIG_NVME_TLS
+#include <net/tls.h>
 #include <net/handshake.h>
 #include <linux/nvme-keyring.h>
 #endif
@@ -92,6 +93,7 @@ struct nvmet_tcp_cmd {
 	u32				pdu_len;
 	u32				pdu_recv;
 	int				sg_idx;
+	char				recv_cbuf[CMSG_LEN(sizeof(char))];
 	struct msghdr			recv_msg;
 	struct bio_vec			*iov;
 	u32				flags;
@@ -1086,7 +1088,18 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;
 	int len;
 	struct kvec iov;
+#ifdef CONFIG_NVME_TLS
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	unsigned char ctype;
+	struct cmsghdr *cmsg;
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT
+	};
+#else
 	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+#endif
 
 recv:
 	iov.iov_base = (void *)&queue->pdu + queue->offset;
@@ -1095,6 +1108,19 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 			iov.iov_len, msg.msg_flags);
 	if (unlikely(len < 0))
 		return len;
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d unhandled TLS record %d\n",
+				queue->idx, ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif
 
 	queue->offset += len;
 	queue->left -= len;
@@ -1150,10 +1176,27 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 	int ret;
 
 	while (msg_data_left(&cmd->recv_msg)) {
+#ifdef CONFIG_NVME_TLS
+		struct cmsghdr *cmsg;
+		unsigned char ctype;
+#endif
 		ret = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
 			cmd->recv_msg.msg_flags);
 		if (ret <= 0)
 			return ret;
+#ifdef CONFIG_NVME_TLS
+		cmsg = (struct cmsghdr *)cmd->recv_cbuf;
+		if (CMSG_OK(&cmd->recv_msg, cmsg) &&
+		    cmsg->cmsg_level == SOL_TLS &&
+		    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+			ctype = *((unsigned char *)CMSG_DATA(cmsg));
+			if (ctype != TLS_RECORD_TYPE_DATA) {
+				pr_err("queue %d unhandled TLS record %d\n",
+				       queue->idx, ctype);
+				return -ENOTCONN;
+			}
+		}
+#endif
 
 		cmd->pdu_recv += ret;
 		cmd->rbytes_done += ret;
@@ -1175,7 +1218,18 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 {
 	struct nvmet_tcp_cmd *cmd = queue->cmd;
 	int ret;
+#ifdef CONFIG_NVME_TLS
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	unsigned char ctype;
+	struct cmsghdr *cmsg;
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT
+	};
+#else
 	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+#endif
 	struct kvec iov = {
 		.iov_base = (void *)&cmd->recv_ddgst + queue->offset,
 		.iov_len = queue->left
@@ -1185,6 +1239,19 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 			iov.iov_len, msg.msg_flags);
 	if (unlikely(ret < 0))
 		return ret;
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d unhandled TLS record %d\n",
+				queue->idx, ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif
 
 	queue->offset += ret;
 	queue->left -= ret;
@@ -1354,6 +1421,10 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
 	if (!c->r2t_pdu)
 		goto out_free_data;
 
+#ifdef CONFIG_NVME_TLS
+	c->recv_msg.msg_control = c->recv_cbuf;
+	c->recv_msg.msg_controllen = sizeof(c->recv_cbuf);
+#endif
 	c->recv_msg.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL;
 
 	list_add_tail(&c->entry, &queue->free_list);
Signature · 1 line
2.35.3
maintainer
#60 lore raw

Re: [PATCH 17/18] nvmet-tcp: control messages for recvmsg()

Quoted 29 lines · Hannes Reinecke

kTLS requires control messages for recvmsg() to relay any out-of-band TLS messages (eg TLS alerts) to the caller.

---
  drivers/nvme/target/tcp.c | 71 +++++++++++++++++++++++++++++++++++++++
  1 file changed, 71 insertions(+)
drivers/nvme/target/tcp.c +2 -0
index 5daf16138471..75017fd5da9f 100644
@@ -12,6 +12,7 @@
  #include <net/sock.h>
  #include <net/tcp.h>
  #ifdef CONFIG_NVME_TLS
+#include <net/tls.h>
  #include <net/handshake.h>
  #include <linux/nvme-keyring.h>
  #endif
@@ -92,6 +93,7 @@ struct nvmet_tcp_cmd {
  	u32				pdu_len;
  	u32				pdu_recv;
  	int				sg_idx;
+	char				recv_cbuf[CMSG_LEN(sizeof(char))];
  	struct msghdr			recv_msg;
  	struct bio_vec			*iov;
  	u32				flags;

drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue) · line 1091

Quoted 4 lines
drivers/nvme/target/tcp.c +1 -0
@@ -1086,7 +1088,18 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
  	struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;
  	int len;
  	struct kvec iov;
+#ifdef CONFIG_NVME_TLS

Do you need this ifdef? The recv_cbuf in the cmd is not under and ifdef.

Quoted 21 lines
drivers/nvme/target/tcp.c +13 -0
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	unsigned char ctype;
+	struct cmsghdr *cmsg;
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT
+	};
+#else
  	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+#endif
 
  recv:
  	iov.iov_base = (void *)&queue->pdu + queue->offset;
@@ -1095,6 +1108,19 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
  			iov.iov_len, msg.msg_flags);
  	if (unlikely(len < 0))
  		return len;
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&

drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue) · line 1123

Quoted 10 lines
drivers/nvme/target/tcp.c +10 -0
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d unhandled TLS record %d\n",
+				queue->idx, ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif

Can we consolidate this to a helper?

Something like:
static int nvmet_tcp_check_cmsg(struct nvmet_tcp_queue *queue, struct cmsghdr *cmsg)

{
	if (!IS_ENABLED(CONFIG_NVME_TLS))
		return 0;

	if (CMSG_OK(&msg, cmsg) &&
	    cmsg->cmsg_level == SOL_TLS &&
	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
		ctype = *((unsigned char *)CMSG_DATA(cmsg));
		if (ctype != TLS_RECORD_TYPE_DATA) {
			pr_err("queue %d unhandled TLS record %d\n",
				queue->idx, ctype);
			return -ENOTCONN;
		}
	}
	return 0;
}
Quoted 81 lines
drivers/nvme/target/tcp.c +45 -0
  	queue->offset += len;
  	queue->left -= len;
@@ -1150,10 +1176,27 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
  	int ret;
 
  	while (msg_data_left(&cmd->recv_msg)) {
+#ifdef CONFIG_NVME_TLS
+		struct cmsghdr *cmsg;
+		unsigned char ctype;
+#endif
  		ret = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
  			cmd->recv_msg.msg_flags);
  		if (ret <= 0)
  			return ret;
+#ifdef CONFIG_NVME_TLS
+		cmsg = (struct cmsghdr *)cmd->recv_cbuf;
+		if (CMSG_OK(&cmd->recv_msg, cmsg) &&
+		    cmsg->cmsg_level == SOL_TLS &&
+		    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+			ctype = *((unsigned char *)CMSG_DATA(cmsg));
+			if (ctype != TLS_RECORD_TYPE_DATA) {
+				pr_err("queue %d unhandled TLS record %d\n",
+				       queue->idx, ctype);
+				return -ENOTCONN;
+			}
+		}
+#endif
 
  		cmd->pdu_recv += ret;
  		cmd->rbytes_done += ret;
@@ -1175,7 +1218,18 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
  {
  	struct nvmet_tcp_cmd *cmd = queue->cmd;
  	int ret;
+#ifdef CONFIG_NVME_TLS
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	unsigned char ctype;
+	struct cmsghdr *cmsg;
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT
+	};
+#else
  	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+#endif
  	struct kvec iov = {
  		.iov_base = (void *)&cmd->recv_ddgst + queue->offset,
  		.iov_len = queue->left
@@ -1185,6 +1239,19 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
  			iov.iov_len, msg.msg_flags);
  	if (unlikely(ret < 0))
  		return ret;
+#ifdef CONFIG_NVME_TLS
+	cmsg = (struct cmsghdr *)cbuf;
+	if (CMSG_OK(&msg, cmsg) &&
+	    cmsg->cmsg_level == SOL_TLS &&
+	    cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
+		ctype = *((unsigned char *)CMSG_DATA(cmsg));
+		if (ctype != TLS_RECORD_TYPE_DATA) {
+			pr_err("queue %d unhandled TLS record %d\n",
+				queue->idx, ctype);
+			return -ENOTCONN;
+		}
+	}
+#endif
 
  	queue->offset += ret;
  	queue->left -= ret;
@@ -1354,6 +1421,10 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
  	if (!c->r2t_pdu)
  		goto out_free_data;
 
+#ifdef CONFIG_NVME_TLS
+	c->recv_msg.msg_control = c->recv_cbuf;
+	c->recv_msg.msg_controllen = sizeof(c->recv_cbuf);
+#endif
  	c->recv_msg.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL;
 
  	list_add_tail(&c->entry, &queue->free_list);
#61 lore raw

PATCH v2 18/18 nvmet-tcp: add configfs attribute 'param_keyring'

In reply to Hannes Reinecke

Add a configfs attribute to list and change the default keyring.

---
 drivers/nvme/target/configfs.c | 68 +++++++++++++++++++++++++++++++++-
 drivers/nvme/target/nvmet.h    |  1 +
 drivers/nvme/target/tcp.c      |  3 +-
 3 files changed, 70 insertions(+), 2 deletions(-)
drivers/nvme/target/configfs.c +67 -1
index 36fbf6a22d09..0a92bc8b25e5 100644
@@ -15,6 +15,10 @@
 #ifdef CONFIG_NVME_TARGET_AUTH
 #include <linux/nvme-auth.h>
 #endif
+#include <linux/key.h>
+#ifdef CONFIG_NVME_TLS
+#include <linux/nvme-keyring.h>
+#endif
 #include <crypto/hash.h>
 #include <crypto/kpp.h>
 
@@ -294,6 +298,48 @@ static ssize_t nvmet_param_pi_enable_store(struct config_item *item,
 CONFIGFS_ATTR(nvmet_, param_pi_enable);
 #endif
 
+#ifdef CONFIG_NVME_TLS
+static ssize_t nvmet_param_keyring_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+
+	if (!port->keyring)
+		return sprintf(page, "\n");
+	return snprintf(page, PAGE_SIZE, "%s\n",
+			port->keyring->description);
+}
+
+static ssize_t nvmet_param_keyring_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_port *port = to_nvmet_port(item);
+	struct key *keyring;
+	unsigned int keyring_id;
+	int ret;
+
+	if (nvmet_is_port_enabled(port, __func__))
+		return -EACCES;
+
+	ret = kstrtou32(page, 0, &keyring_id);
+	if (ret) {
+		pr_err("Invalid keyring id '%s'\n", page);
+		return ret;
+	}
+	keyring = key_lookup(keyring_id);
+	if (IS_ERR(keyring)) {
+		pr_err("Invalid keyring '%08x'\n", keyring_id);
+		return PTR_ERR(keyring);
+	}
+	if (port->keyring)
+		key_put(port->keyring);
+	port->keyring = keyring;
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_, param_keyring);
+#endif
+
 static ssize_t nvmet_addr_trtype_show(struct config_item *item,
 		char *page)
 {
@@ -404,9 +450,12 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 	return -EINVAL;
 
 found:
-	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
 	if (nvmet_addr_tsas_tcp[i].type == NVMF_TCP_SECTYPE_TLS13) {
 #ifdef CONFIG_NVME_TLS
+		if (!port->keyring) {
+			pr_err("NVMe keyring not available, cannot enable TLS 1.3\n");
+			return -ENOTSUPP;
+		}
 		if (NVMET_PORT_TREQ(port) == NVMF_TREQ_NOT_SPECIFIED)
 			treq |= NVMF_TREQ_REQUIRED;
 		else
@@ -419,6 +468,8 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,
 		/* Set to 'not specified' if TLS is not enabled */
 		treq |= NVMF_TREQ_NOT_SPECIFIED;
 	}
+	port->disc_addr.treq = treq;
+	nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type);
 	return count;
 }
 
@@ -1825,6 +1876,8 @@ static void nvmet_port_release(struct config_item *item)
 	flush_workqueue(nvmet_wq);
 	list_del(&port->global_entry);
 
+	if (port->keyring)
+		key_put(port->keyring);
 	kfree(port->ana_state);
 	kfree(port);
 }
@@ -1839,6 +1892,9 @@ static struct configfs_attribute *nvmet_port_attrs[] = {
 	&nvmet_attr_param_inline_data_size,
 #ifdef CONFIG_BLK_DEV_INTEGRITY
 	&nvmet_attr_param_pi_enable,
+#endif
+#ifdef CONFIG_NVME_TLS
+	&nvmet_attr_param_keyring,
 #endif
 	NULL,
 };
@@ -1857,6 +1913,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
 		const char *name)
 {
 	struct nvmet_port *port;
+	struct key *keyring = NULL;
 	u16 portid;
 	u32 i;
 
@@ -1874,6 +1931,15 @@ static struct config_group *nvmet_ports_make(struct config_group *group,
 		return ERR_PTR(-ENOMEM);
 	}
 
+#ifdef CONFIG_NVME_TLS
+	keyring = key_lookup(nvme_keyring_id());
+	if (IS_ERR(keyring)) {
+		pr_warn("NVMe keyring not available, disabling TLS\n");
+		keyring = NULL;
+	}
+#endif
+	port->keyring = keyring;
+
 	for (i = 1; i <= NVMET_MAX_ANAGRPS; i++) {
 		if (i == NVMET_DEFAULT_ANA_GRPID)
 			port->ana_state[1] = NVME_ANA_OPTIMIZED;
drivers/nvme/target/nvmet.h +1 -0
index ed3786140965..c31e310d269a 100644
@@ -158,6 +158,7 @@ struct nvmet_port {
 	struct config_group		ana_groups_group;
 	struct nvmet_ana_group		ana_default_group;
 	enum nvme_ana_state		*ana_state;
+	struct key			*keyring;
 	void				*priv;
 	bool				enabled;
 	int				inline_data_size;
drivers/nvme/target/tcp.c +2 -1
index 75017fd5da9f..5b1069854f15 100644
@@ -8,6 +8,7 @@
 #include <linux/init.h>
 #include <linux/slab.h>
 #include <linux/err.h>
+#include <linux/key.h>
 #include <linux/nvme-tcp.h>
 #include <net/sock.h>
 #include <net/tcp.h>
@@ -1763,7 +1764,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 	args.ta_sock = queue->sock;
 	args.ta_done = nvmet_tcp_tls_handshake_done;
 	args.ta_data = queue;
-	args.ta_keyring = nvme_keyring_id();
+	args.ta_keyring = key_serial(queue->port->nport->keyring);
 	args.ta_timeout_ms = tls_handshake_timeout * 2 * 1024;
 
 	ret = tls_server_hello_psk(&args, GFP_KERNEL);
Signature · 1 line
2.35.3
maintainer
#62 lore raw

Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'

Quoted 2 lines

Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?

#63 lore raw

Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'

Quoted 6 lines · Sagi Grimberg
Quoted 2 lines

Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?

As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg
Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman
(HRB 36809, AG Nürnberg)
maintainer
#64 lore raw

Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?

As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.

I think we should move it to the subsystem in Linux.

#65 lore raw

Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'

Quoted 11 lines · Sagi Grimberg
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?

As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.

I think we should move it to the subsystem in Linux.

Easier said than done.

We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.

So we'll have to keep it at the port level, I'm afraid.

Cheers,

Hannes

maintainer
#66 lore raw

Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'

Quoted 13 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?

As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.

I think we should move it to the subsystem in Linux.

Easier said than done.

Quoted 7 lines

We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.

So we'll have to keep it at the port level, I'm afraid.

Question, why does nvmet needs to support user defined keyrings? I understood your argument for the host, but why should the target support it?

#67 lore raw

Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'

Quoted 19 lines · Sagi Grimberg
Quoted 19 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?

As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.

I think we should move it to the subsystem in Linux.

Easier said than done.

We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.

Quoted 5 lines
Quoted 1 line

So we'll have to keep it at the port level, I'm afraid.

Question, why does nvmet needs to support user defined keyrings? I understood your argument for the host, but why should the target support it?

Hmm. Originally I implemented it to be similar to the host configuration, but from the implementation we don't _actually_ need it as we select the key from the entire key store. Having it is just a nice way of avoiding having to have a daemon configuration.
So if you insist we can drop it for the initial implementation.

Cheers,

Hannes

maintainer
#68 lore raw

Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Add a configfs attribute to list and change the default keyring.

is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?

As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.

I think we should move it to the subsystem in Linux.

Easier said than done.

We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

So we'll have to keep it at the port level, I'm afraid.

Question, why does nvmet needs to support user defined keyrings? I understood your argument for the host, but why should the target support it?

Hmm. Originally I implemented it to be similar to the host configuration, but from the implementation we don't _actually_ need it as we select the key from the entire key store.

Thanks for confirming.

Quoted 1 line

Having it is just a nice way of avoiding having to have a daemon configuration.

I'm sorry, but I don't understand this statement. Can you clarify?

Quoted 1 line

So if you insist we can drop it for the initial implementation.

If there is a use-case, we can add it. But I'm trying to understand what is it.