nvme: In-kernel TLS support for TCP
Subsystems (8)
v2 cover nvme: In-kernel TLS support for TCP
Hi all,
finally I've managed to put all things together and enable in-kernel TLS support for NVMe-over-TCP.
The patchset is based on the TLS upcall mechanism from Chuck Lever (cf '[PATCH v7 0/2] Another crack at a handshake upcall mechanism' posted to the linux netdev list), and requires the 'tlshd' userspace daemon (https://github.com/oracle/ktls-utils) for the actual TLS handshake. Changes for nvme-cli are already included in the upstream repository.
Theory of operation:
A dedicated '.nvme' keyring is created to hold the pre-shared keys (PSKs) for the TLS handshake. Keys will have to be provisioned before TLS handshake is attempted; that can be done with the 'nvme gen-tls-key' command for nvme-cli (patches are already merged upstream).
After connection to the remote TCP port the client side will use the 'best' PSK (as inferred from the NVMe TCP spec) or the PSK specified by the '--tls_key' option to nvme-cli and call the TLS userspace daemon to initiate a TLS handshake.
The server side will then invoke the TLS userspace daemon to run the TLS handshake.
If the TLS handshake succeeds the userspace daemon will be activating kTLS on the socket, and control is passed back to the kernel.
To make this work I had to implement the 'read_sock()' functionality for TLS; it seems to be holding up well enough (for me), but it really could do with reviews from persons with more network stack knowledge.
As usual, comments and reviews are welcome.
Changes to the original RFC:
- Add a CONFIG_NVME_TLS config option
- Use a single PSK for the TLS handshake
- Make TLS connections mandatory
- Do not peek messages for the server
- Simplify data_ready callback
- Implement read_sock() for TLS
Hannes Reinecke (18):
nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
nvme-keyring: define a 'psk' keytype
nvme: add TCP TSAS definitions
nvme-tcp: add definitions for TLS cipher suites net/tls: implement ->read_sock()
nvme/tcp: allocate socket file
nvme-keyring: implement nvme_tls_psk_default() security/keys: export key_lookup()
nvme-tcp: enable TLS handshake upcall
nvme-tcp: fixup send workflow for kTLS
nvme-tcp: control message handling for recvmsg()
nvme-fabrics: parse options 'keyring' and 'tls_key'
nvmet: make TCP sectype settable via configfs
nvmet-tcp: allocate socket file
nvmet-tcp: enable TLS handshake upcall
nvmet-tcp: rework sendpage for kTLS
nvmet-tcp: control messages for recvmsg()
nvmet-tcp: add configfs attribute 'param_keyring'
drivers/nvme/common/Kconfig | 9 ++ drivers/nvme/common/Makefile | 1 + drivers/nvme/common/keyring.c | 181 ++++++++++++++++++++++ drivers/nvme/host/core.c | 44 +++++- drivers/nvme/host/fabrics.c | 90 ++++++++++- drivers/nvme/host/fabrics.h | 9 ++ drivers/nvme/host/nvme.h | 4 +- drivers/nvme/host/tcp.c | 213 +++++++++++++++++++++++--- drivers/nvme/target/configfs.c | 165 ++++++++++++++++++++- drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/tcp.c | 264 ++++++++++++++++++++++++++++++--- include/linux/nvme-keyring.h | 20 +++ include/linux/nvme-tcp.h | 6 + include/linux/nvme.h | 10 ++ net/tls/tls.h | 2 + net/tls/tls_main.c | 2 + net/tls/tls_sw.c | 71 +++++++++ security/keys/key.c | 1 + 18 files changed, 1050 insertions(+), 43 deletions(-) create mode 100644 drivers/nvme/common/keyring.c create mode 100644 include/linux/nvme-keyring.h
Signature · 1 line
2.35.3
PATCH v2 1/18 nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option NVME_TLS to enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.
--- drivers/nvme/common/Kconfig | 9 +++++++++ drivers/nvme/common/Makefile | 1 + drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/core.c | 19 +++++++++++++++--- include/linux/nvme-keyring.h | 12 ++++++++++++ 5 files changed, 74 insertions(+), 3 deletions(-) create mode 100644 drivers/nvme/common/keyring.c create mode 100644 include/linux/nvme-keyring.h
drivers/nvme/common/Kconfig +9 -0
@@ -2,3 +2,12 @@ config NVME_COMMON tristate + +config NVME_TLS + bool "NVMe/TCP TLS encryption support" + depends on NVME_COMMON + select KEYS + help + Enables TLS encryption for NVMe/TCP using the netlink handshake API. + + If unsure, say N.
drivers/nvme/common/Makefile +1 -0
@@ -5,3 +5,4 @@ ccflags-y += -I$(src) obj-$(CONFIG_NVME_COMMON) += nvme-common.o nvme-common-y += auth.o +nvme-common-$(CONFIG_NVME_TLS) += keyring.o
drivers/nvme/common/keyring.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (c) 2020 Hannes Reinecke, SUSE Linux +*/ + +#include <linux/module.h> +#include <linux/nvme.h> +#include <linux/seq_file.h> +#include <linux/key-type.h> +#include <keys/user-type.h> + +static struct key *nvme_keyring; + +int nvme_keyring_init (void ) +{ +int err; + + nvme_keyring =keyring_alloc (".nvme" , +GLOBAL_ROOT_UID ,GLOBAL_ROOT_GID , +current_cred (), + (KEY_POS_ALL & ~KEY_POS_SETATTR ) | + (KEY_USR_ALL & ~KEY_USR_SETATTR ), +KEY_ALLOC_NOT_IN_QUOTA ,NULL ,NULL ); +if (IS_ERR (nvme_keyring)) +return PTR_ERR (nvme_keyring); + +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_keyring_init); + +void nvme_keyring_exit (void ) +{ +key_revoke (nvme_keyring); +key_put (nvme_keyring); +} +EXPORT_SYMBOL_GPL (nvme_keyring_exit);
drivers/nvme/host/core.c +16 -3
@@ -25,6 +25,9 @@#include "nvme.h" #include "fabrics.h" #include <linux/nvme-auth.h> +#ifdef CONFIG_NVME_TLS +#include <linux/nvme-keyring.h> +#endif #define CREATE_TRACE_POINTS #include "trace.h"
@@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)return 0 ;#endif -return a->mode; }
@@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void) result =PTR_ERR (nvme_ns_chr_class); goto unregister_generic_ns; } - - result =nvme_init_auth (); +#ifdef CONFIG_NVME_TLS + result =nvme_keyring_init ();if (result) goto destroy_ns_chr; +#endif + result =nvme_init_auth (); +if (result) + goto keyring_exit;return 0 ; +keyring_exit: +#ifdef CONFIG_NVME_TLS +nvme_keyring_exit (); destroy_ns_chr: +#endifclass_destroy (nvme_ns_chr_class); unregister_generic_ns:unregister_chrdev_region (nvme_ns_chr_devt,NVME_MINORS );
@@ -5443,6 +5453,9 @@ static int __init nvme_core_init(void) static void __exit nvme_core_exit(void) {nvme_exit_auth (); +#ifdef CONFIG_NVME_TLS +nvme_keyring_exit (); +#endif class_destroy (nvme_ns_chr_class);class_destroy (nvme_subsys_class);class_destroy (nvme_class);
include/linux/nvme-keyring.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions +*/ + +#ifndef _NVME_KEYRING_H +#define _NVME_KEYRING_H + +int nvme_keyring_init (void ); +void nvme_keyring_exit (void ); + +#endif
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
Quoted 23 lines
Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option
NVME_TLSto enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/common/Kconfig | 9 +++++++++ drivers/nvme/common/Makefile | 1 + drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/core.c | 19 +++++++++++++++--- include/linux/nvme-keyring.h | 12 ++++++++++++ 5 files changed, 74 insertions(+), 3 deletions(-)create mode 100644
drivers/nvme/common/keyring.ccreate mode 100644include/linux/nvme-keyring.hdrivers/nvme/common/Kconfig +0
-0
drivers/nvme/common/Kconfig line 8
Quoted 7 lines
drivers/nvme/common/Kconfig +4
-0@@ -2,3 +2,12 @@ config NVME_COMMON tristate + +config NVME_TLS + bool "NVMe/TCP TLS encryption support" + depends on NVME_COMMON
depends on TLS as well? Or maybe select would be more appropriate?
drivers/nvme/common/Kconfig line 13
Quoted 5 lines
drivers/nvme/common/Kconfig +5
-0+ select KEYS + help + Enables TLS encryption for NVMe/TCP using the netlink handshake API. + + If unsure, say N.
Would it not make sense to default it to y?
Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
Quoted 18 lines · Sagi Grimberg
Quoted 18 lines
Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option
NVME_TLSto enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/common/Kconfig | 9 +++++++++ drivers/nvme/common/Makefile | 1 +drivers/nvme/common/keyring.c| 36 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/core.c | 19 +++++++++++++++--- include/linux/nvme-keyring.h | 12 ++++++++++++5 files changed, 74 insertions(+), 3 deletions(-)create mode 100644
drivers/nvme/common/keyring.ccreate mode 100644include/linux/nvme-keyring.h
Quoted 14 lines
drivers/nvme/common/Kconfig line 8
Quoted 11 lines
drivers/nvme/common/Kconfig +0
-0@@ -2,3 +2,12 @@config
NVME_COMMON
tristatedrivers/nvme/common/Kconfig +4
-0+ +config NVME_TLS + bool "NVMe/TCP TLS encryption support" + depends on NVME_COMMONdepends on TLS as well? Or maybe select would be more appropriate?
I would argue 'depends', as we cannot use the standard session/process/thread keyrings, as they'll vanish after the initial 'connect', and then there's no keyring to lookup keys for subsection reconnects.
Quoted 8 lines
Quoted 6 lines
+ select KEYS
+ help
+ Enables TLS encryption for NVMe/TCP using the netlink handshake API.
+
+ If unsure, say N.Would it not make sense to default it to y?
Yeah.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerRe: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
In reply to Hannes ReineckeQuoted 23 lines
Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option
NVME_TLSto enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/common/Kconfig | 9 +++++++++ drivers/nvme/common/Makefile | 1 + drivers/nvme/common/keyring.c | 36 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/core.c | 19 +++++++++++++++--- include/linux/nvme-keyring.h | 12 ++++++++++++ 5 files changed, 74 insertions(+), 3 deletions(-)create mode 100644
drivers/nvme/common/keyring.ccreate mode 100644include/linux/nvme-keyring.hdrivers/nvme/common/Kconfig +0
-0
drivers/nvme/common/Kconfig line 6
Quoted 5 lines
drivers/nvme/common/Kconfig +2
-0@@ -2,3 +2,12 @@ config NVME_COMMON tristate + +config NVME_TLS
Better to call it NVME_TCP_TLS... although not a biggie, should be well understood this is tcp
Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
Quoted 18 lines · Sagi Grimberg
Quoted 18 lines
Register a '.nvme' keyring to hold keys for TLS and DH-HMAC-CHAP and add a new config option
NVME_TLSto enable support for NVMe-TCP/TLS. We need a separate keyring for NVMe as the configuration is done via individual commands (eg for configfs), and the usual per-session or per-process keyrings can't be used.Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/common/Kconfig | 9 +++++++++ drivers/nvme/common/Makefile | 1 +drivers/nvme/common/keyring.c| 36 +++++++++++++++++++++++++++++++++++ drivers/nvme/host/core.c | 19 +++++++++++++++--- include/linux/nvme-keyring.h | 12 ++++++++++++5 files changed, 74 insertions(+), 3 deletions(-)create mode 100644
drivers/nvme/common/keyring.ccreate mode 100644include/linux/nvme-keyring.h
Quoted 13 lines
drivers/nvme/common/Kconfig line 6
Quoted 9 lines
drivers/nvme/common/Kconfig +0
-0@@ -2,3 +2,12 @@config
NVME_COMMON
tristatedrivers/nvme/common/Kconfig +2
-0+ +config NVME_TLSBetter to call it
NVME_TCP_TLS... although not a biggie, should be well understood this is tcp
Sure.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Re: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
In reply to Hannes ReineckeQuoted 20 lines · Hannes Reinecke
+++
b/drivers/nvme/host/core.cdrivers/nvme/host/core.c +3
-1@@ -25,6 +25,9 @@#include "nvme.h" #include "fabrics.h" #include <linux/nvme-auth.h> +#ifdef CONFIG_NVME_TLS +#include <linux/nvme-keyring.h> +#endif #define CREATE_TRACE_POINTS #include "trace.h" @@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)return 0 ;#endif -return a->mode; }
drivers/nvme/host/core.c static int __init nvme_core_init(void) · line 5423
Quoted 10 lines
drivers/nvme/host/core.c +3
-2@@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void) result =PTR_ERR (nvme_ns_chr_class); goto unregister_generic_ns; } - - result =nvme_init_auth (); +#ifdef CONFIG_NVME_TLS + result =nvme_keyring_init ();if (result) goto destroy_ns_chr; +#endif
Would it make sense to introduce empty nvme_keyring_init() function for !CONFIG_NVME_TLS and avoid this ifdefery here?
Sagi Grimberg
maintainerRe: [PATCH 01/18] nvme-keyring: register '.nvme' keyring and add CONFIG_NVME_TLS
Quoted 20 lines · Daniel Wagner
Quoted 19 lines · Hannes Reinecke
+++
b/drivers/nvme/host/core.cdrivers/nvme/host/core.c +3
-1@@ -25,6 +25,9 @@#include "nvme.h" #include "fabrics.h" #include <linux/nvme-auth.h> +#ifdef CONFIG_NVME_TLS +#include <linux/nvme-keyring.h> +#endif #define CREATE_TRACE_POINTS #include "trace.h" @@ -3954,7 +3957,6 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)return 0 ;#endif -return a->mode; }
Quoted 14 lines
drivers/nvme/host/core.c static int __init nvme_core_init(void) · line 5423
Quoted 11 lines
drivers/nvme/host/core.c +3
-2@@ -5414,13 +5416,21 @@ static int __init nvme_core_init(void) result =PTR_ERR (nvme_ns_chr_class); goto unregister_generic_ns; } - - result =nvme_init_auth (); +#ifdef CONFIG_NVME_TLS + result =nvme_keyring_init ();if (result) goto destroy_ns_chr; +#endif Would it make sense to introduce empty
nvme_keyring_init()function for !CONFIG_NVME_TLS and avoid this ifdefery here?
Yes.
PATCH v2 2/18 nvme-keyring: define a 'psk' keytype
In reply to Hannes ReineckeDefine a 'psk' keytype to hold the NVMe TLS PSKs.
--- drivers/nvme/common/keyring.c | 98 +++++++++++++++++++++++++++++++++++ include/linux/nvme-keyring.h | 6 +++ 2 files changed, 104 insertions(+)
drivers/nvme/common/keyring.c +98 -0
@@ -5,12 +5,104 @@#include <linux/module.h> #include <linux/nvme.h> +#include <linux/nvme-tcp.h> #include <linux/seq_file.h> #include <linux/key-type.h> #include <keys/user-type.h> static struct key *nvme_keyring; +key_serial_t nvme_keyring_id (void ) +{ +return nvme_keyring->serial; +} +EXPORT_SYMBOL_GPL (nvme_keyring_id); + +static void nvme_tls_psk_describe (const struct key *key,struct seq_file *m) +{ +seq_puts (m, key->description); +seq_printf (m,": %u" , key->datalen); +} + +static bool nvme_tls_psk_match (const struct key *key, +const struct key_match_data *match_data) +{ +const char *match_id; +size_t match_len; + +if (!key->description) { +pr_debug ("%s: no key description\n" , __func__); +return false; + } + match_len =strlen (key->description); +pr_debug ("%s: id %s len %zd\n" , __func__, key->description, match_len); + +if (!match_data->raw_data) { +pr_debug ("%s: no match data\n" , __func__); +return false; + } + match_id = match_data->raw_data; +pr_debug ("%s: match '%s' '%s' len %zd\n" , + __func__, match_id, key->description, match_len); +return !memcmp (key->description, match_id, match_len); +} + +static int nvme_tls_psk_match_preparse (struct key_match_data *match_data) +{ + match_data->lookup_type =KEYRING_SEARCH_LOOKUP_ITERATE ; + match_data->cmp = nvme_tls_psk_match; +return 0 ; +} + +static struct key_type nvme_tls_psk_key_type = { + .name ="psk" , + .flags =KEY_TYPE_NET_DOMAIN , + .preparse = user_preparse, + .free_preparse = user_free_preparse, + .match_preparse = nvme_tls_psk_match_preparse, + .instantiate = generic_key_instantiate, + .revoke = user_revoke, + .destroy = user_destroy, + .describe = nvme_tls_psk_describe, + .read = user_read, +}; + +struct key *nvme_tls_psk_lookup (struct key *keyring, +const char *hostnqn,const char *subnqn, +int hmac,bool generated) +{ +char *identity; +size_t identity_len = (NVMF_NQN_SIZE ) *2 +11 ; +key_ref_t keyref; +key_serial_t keyring_id; + + identity =kzalloc (identity_len,GFP_KERNEL ); +if (!identity) +return ERR_PTR (-ENOMEM ); + +snprintf (identity, identity_len,"NVMe0%c%02d %s %s" , + generated ?'G' :'R' , hmac, hostnqn, subnqn); + +if (!keyring) + keyring = nvme_keyring; + keyring_id =key_serial (keyring); +pr_debug ("keyring %x lookup tls psk '%s'\n" , + keyring_id, identity); + keyref =keyring_search (make_key_ref (keyring, true), + &nvme_tls_psk_key_type, + identity, false); +if (IS_ERR (keyref)) { +pr_debug ("lookup tls psk '%s' failed, error %ld\n" , + identity,PTR_ERR (keyref)); +kfree (identity); +return ERR_PTR (-ENOKEY ); + } +kfree (identity); + +return key_ref_to_ptr (keyref); +} +EXPORT_SYMBOL_GPL (nvme_tls_psk_lookup); +int nvme_keyring_init (void ) {int err;
@@ -24,12 +116,18 @@ int nvme_keyring_init(void)if (IS_ERR (nvme_keyring))return PTR_ERR (nvme_keyring); + err =register_key_type (&nvme_tls_psk_key_type); +if (err) { +key_put (nvme_keyring); +return err; + }return 0 ; }EXPORT_SYMBOL_GPL (nvme_keyring_init);void nvme_keyring_exit (void ) { +unregister_key_type (&nvme_tls_psk_key_type);key_revoke (nvme_keyring);key_put (nvme_keyring); }
include/linux/nvme-keyring.h +6 -0
@@ -6,6 +6,12 @@#ifndef _NVME_KEYRING_H#define _NVME_KEYRING_H +struct key *nvme_tls_psk_lookup (struct key *keyring, +const char *hostnqn,const char *subnqn, +int hmac,bool generated); + +key_serial_t nvme_keyring_id (void ); +int nvme_keyring_init (void );void nvme_keyring_exit (void );
Signature · 1 line
2.35.3
PATCH v2 3/18 nvme: add TCP TSAS definitions
In reply to Hannes Reinecke--- include/linux/nvme.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
include/linux/nvme.h +10 -0
@@ -108,6 +108,13 @@ enum { NVMF_RDMA_CMS_RDMA_CM = 1, /* Sockets based endpoint addressing */ }; +/* TSAS SECTYPE for TCP transport */ +enum { +NVMF_TCP_SECTYPE_NONE =0 ,/* No Security */ +NVMF_TCP_SECTYPE_TLS12 =1 ,/* TLSv1.2, NVMe-oF 1.1 and NVMe-TCP 3.6.1.1 */ +NVMF_TCP_SECTYPE_TLS13 =2 ,/* TLSv1.3, NVMe-oF 1.1 and NVMe-TCP 3.6.1.1 */ +}; +#define NVME_AQ_DEPTH 32#define NVME_NR_AEN_COMMANDS 1#define NVME_AQ_BLK_MQ_DEPTH (NVME_AQ_DEPTH - NVME_NR_AEN_COMMANDS)
@@ -1458,6 +1465,9 @@ struct nvmf_disc_rsp_page_entry {__u16 pkey;__u8 resv10[246 ]; } rdma; +struct tcp { +__u8 sectype; + } tcp; } tsas; };
Signature · 1 line
2.35.3
PATCH v2 4/18 nvme-tcp: add definitions for TLS cipher suites
In reply to Hannes Reinecke--- include/linux/nvme-tcp.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/nvme-tcp.h +6 -0
@@ -18,6 +18,12 @@ enum nvme_tcp_pfv { NVME_TCP_PFV_1_0 = 0x0, }; +enum nvme_tcp_tls_cipher { +NVME_TCP_TLS_CIPHER_INVALID =0 , +NVME_TCP_TLS_CIPHER_SHA256 =1 , +NVME_TCP_TLS_CIPHER_SHA384 =2 , +}; +enum nvme_tcp_fatal_error_status {NVME_TCP_FES_INVALID_PDU_HDR =0x01 ,NVME_TCP_FES_PDU_SEQ_ERR =0x02 ,
Signature · 1 line
2.35.3
PATCH v2 5/18 net/tls: implement ->read_sock()
In reply to Hannes ReineckeImplement ->read_sock() function for use with nvme-tcp.
--- net/tls/tls.h | 2 ++ net/tls/tls_main.c | 2 ++ net/tls/tls_sw.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+)
net/tls/tls.h +2 -0
@@ -113,6 +113,8 @@ bool tls_sw_sock_is_readable(struct sock *sk);ssize_t tls_sw_splice_read (struct socket *sock,loff_t *ppos,struct pipe_inode_info *pipe,size_t len,unsigned int flags); +int tls_sw_read_sock (struct sock *sk,read_descriptor_t *desc, +sk_read_actor_t read_actor);int tls_device_sendmsg (struct sock *sk,struct msghdr *msg,size_t size); int tls_device_sendpage(struct sock *sk, struct page *page,
net/tls/tls_main.c +2 -0
@@ -934,9 +934,11 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG] ops[TLS_BASE ][TLS_SW ] = ops[TLS_BASE ][TLS_BASE ]; ops[TLS_BASE ][TLS_SW ].splice_read = tls_sw_splice_read; + ops[TLS_BASE ][TLS_SW ].read_sock = tls_sw_read_sock; ops[TLS_SW ][TLS_SW ] = ops[TLS_SW ][TLS_BASE ]; ops[TLS_SW ][TLS_SW ].splice_read = tls_sw_splice_read; + ops[TLS_SW ][TLS_SW ].read_sock = tls_sw_read_sock;#ifdef CONFIG_TLS_DEVICE ops[TLS_HW ][TLS_BASE ] = ops[TLS_BASE ][TLS_BASE ];
net/tls/tls_sw.c +71 -0
77 lines not on this page · raw message
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 05/18] net/tls: implement ->read_sock()
Quoted 2 lines
Implement ->read_sock() function for use with nvme-tcp.
Nice.
Quoted 2 lines
Signed-off-by Hannes Reinecke <[email protected]>Cc Boris Pismenny <[email protected]>
Would be great to see a tls_device implementation as well.
Re: [PATCH 05/18] net/tls: implement ->read_sock()
Quoted 9 lines · Sagi Grimberg
Quoted 2 lines
Implement ->read_sock() function for use with nvme-tcp.
Nice.
Quoted 2 lines
Signed-off-by Hannes Reinecke <[email protected]>Cc Boris Pismenny <[email protected]>Would be great to see a
tls_deviceimplementation as well.
Sheesh. But first we would need to update the tls_device implementation to TLS 1.3:
int tls_set_device_offload(struct sock *sk, struct tls_context *ctx) [ .. ]
crypto_info = &ctx->crypto_send.info;
if (crypto_info->version != TLS_1_2_VERSION) {
rc = -EOPNOTSUPP;
goto release_netdev;
}which I'd rather defer to Boris or whoever maintains the hardware.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerRe: [PATCH 05/18] net/tls: implement ->read_sock()
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
Implement ->read_sock() function for use with nvme-tcp.
Nice.
Quoted 2 lines
Signed-off-by Hannes Reinecke <[email protected]>Cc Boris Pismenny <[email protected]>Would be great to see a
tls_deviceimplementation as well.Sheesh. But first we would need to update the
tls_deviceimplementation to TLS 1.3:
;)
The comment was for the nvidia folks..
Quoted 9 lines
int
tls_set_device_offload(struct sock *sk, struct tls_context *ctx)[ .. ]
crypto_info=&ctx->crypto_send.info;
if (crypto_info->version!=TLS_1_2_VERSION) { rc =-EOPNOTSUPP;
gotorelease_netdev;
}which I'd rather defer to Boris or whoever maintains the hardware
Sagi Grimberg
maintainerRe: [PATCH 05/18] net/tls: implement ->read_sock()
In reply to Hannes ReineckeQuoted 78 lines
Implement ->read_sock() function for use with nvme-tcp.
Signed-off-by Hannes Reinecke <[email protected]>Cc Boris Pismenny <[email protected]>--- net/tls/tls.h | 2 ++ net/tls/tls_main.c | 2 ++ net/tls/tls_sw.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+)net/tls/tls.h +2
-0@@ -113,6 +113,8 @@ bool tls_sw_sock_is_readable(struct sock *sk);ssize_t tls_sw_splice_read (struct socket *sock,loff_t *ppos,struct pipe_inode_info *pipe,size_t len,unsigned int flags); +int tls_sw_read_sock (struct sock *sk,read_descriptor_t *desc, +sk_read_actor_t read_actor);int tls_device_sendmsg (struct sock *sk,struct msghdr *msg,size_t size); int tls_device_sendpage(struct sock *sk, struct page *page,net/tls/tls_main.c +2
-0@@ -934,9 +934,11 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG] ops[TLS_BASE ][TLS_SW ] = ops[TLS_BASE ][TLS_BASE ]; ops[TLS_BASE ][TLS_SW ].splice_read = tls_sw_splice_read; + ops[TLS_BASE ][TLS_SW ].read_sock = tls_sw_read_sock; ops[TLS_SW ][TLS_SW ] = ops[TLS_SW ][TLS_BASE ]; ops[TLS_SW ][TLS_SW ].splice_read = tls_sw_splice_read; + ops[TLS_SW ][TLS_SW ].read_sock = tls_sw_read_sock;#ifdef CONFIG_TLS_DEVICE ops[TLS_HW ][TLS_BASE ] = ops[TLS_BASE ][TLS_BASE ];net/tls/tls_sw.c +30
-0@@ -2210,6 +2210,77 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos, goto splice_read_end; } +int tls_sw_read_sock (struct sock *sk,read_descriptor_t *desc, +sk_read_actor_t read_actor) +{ +struct tls_context *tls_ctx =tls_get_ctx (sk); +struct tls_sw_context_rx *ctx =tls_sw_ctx_rx (tls_ctx); +struct strp_msg *rxm =NULL ; +struct tls_msg *tlm; +struct sk_buff *skb; +ssize_t copied =0 ; +int err, used; + +if (!skb_queue_empty (&ctx->rx_list)) { + skb =__skb_dequeue (&ctx->rx_list); + }else { +struct tls_decrypt_arg darg; + + err =tls_rx_rec_wait (sk,NULL , true, true); +if (err <=0 ) +return err; + +memset (&darg.inargs,0 ,sizeof (darg.inargs)); + + err =tls_rx_one_record (sk,NULL , &darg); +if (err <0 ) { +tls_err_abort (sk, -EBADMSG ); +return err; + } + +tls_rx_rec_done (ctx); + skb = darg.skb;
net/tls/tls_sw.c ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos, · line 2252
Quoted 10 lines
net/tls/tls_sw.c +10
-0+ } + + do { + rxm = strp_msg(skb); + tlm = tls_msg(skb); + + /* read_sock does not support reading control messages */ + if (tlm->control != TLS_RECORD_TYPE_DATA) { + err = -EINVAL; + goto read_sock_requeue;
In the general case, this could just pass in the skb to recv_actor and have it access the tlm and decide what to do with it... (will require moving tls_msg() to include/net/strparser.h)
I agree that for nvme-tcp this would not have anything useful other than breaking the connection and re-establish it later...
PATCH v2 6/18 nvme/tcp: allocate socket file
In reply to Hannes ReineckeWhen using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.
--- drivers/nvme/host/tcp.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/nvme/host/tcp.c +15 -2
@@ -115,6 +115,7 @@ enum nvme_tcp_recv_state {struct nvme_tcp_ctrl ;struct nvme_tcp_queue {struct socket *sock; +struct file *sock_file;struct work_struct io_work;int io_cpu;
@@ -1330,7 +1331,10 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) } noreclaim_flag =memalloc_noreclaim_save (); -sock_release (queue->sock); +/* ->sock will be released by fput() */ +fput (queue->sock_file); + queue->sock_file =NULL ; + queue->sock =NULL ;memalloc_noreclaim_restore (noreclaim_flag);kfree (queue->pdu);
@@ -1526,6 +1530,13 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) goto err_destroy_mutex; } + queue->sock_file =sock_alloc_file (queue->sock,O_CLOEXEC ,NULL ); +if (IS_ERR (queue->sock_file)) { +sock_release (queue->sock); + ret =PTR_ERR (queue->sock_file); + queue->sock_file =NULL ; + goto err_sock; + }nvme_tcp_reclassify_socket (queue->sock);/* Single syn retry */
@@ -1647,7 +1658,9 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)if (queue->hdr_digest || queue->data_digest)nvme_tcp_free_crypto (queue); err_sock: -sock_release (queue->sock); +/* ->sock will be released by fput() */ +fput (queue->sock_file); + queue->sock_file =NULL ; queue->sock =NULL ; err_destroy_mutex:mutex_destroy (&queue->send_mutex);
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 06/18] nvme/tcp: allocate socket file
Quoted 13 lines
When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/tcp.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c enum nvme_tcp_recv_state { · line 118
Quoted 4 lines
drivers/nvme/host/tcp.c +1
-0@@ -115,6 +115,7 @@ enum nvme_tcp_recv_state {struct nvme_tcp_ctrl ;struct nvme_tcp_queue {struct socket *sock; +struct file *sock_file;
I think that sock->file is storing the file, no need to store it as well.
drivers/nvme/host/tcp.c static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) · line 1337
Quoted 12 lines
drivers/nvme/host/tcp.c +4
-1struct work_struct io_work;int io_cpu;@@ -1330,7 +1331,10 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) } noreclaim_flag =memalloc_noreclaim_save (); -sock_release (queue->sock); +/* ->sock will be released by fput() */ +fput (queue->sock_file); + queue->sock_file =NULL ; + queue->sock =NULL ;
Is the queue->sock = NULL here required because we allocate a sock_file?
PATCH v2 7/18 nvme-keyring: implement nvme_tls_psk_default()
In reply to Hannes ReineckeImplement a function to select the 'best' PSK for TLS.
--- drivers/nvme/common/keyring.c | 47 +++++++++++++++++++++++++++++++++++ include/linux/nvme-keyring.h | 2 ++ 2 files changed, 49 insertions(+)
drivers/nvme/common/keyring.c +47 -0
@@ -103,6 +103,53 @@ struct key *nvme_tls_psk_lookup(struct key *keyring, }EXPORT_SYMBOL_GPL (nvme_tls_psk_lookup); +/* +* NVMe PSK priority list +* +* 'Retained' PSKs (ie 'generated == false') +* should be preferred to 'generated' PSKs, +* and SHA-384 should be preferred to SHA-256. +*/ +struct nvme_psk_priority_list { +bool generated; +enum nvme_tcp_tls_cipher cipher; +} nvme_psk_prio[] = { + { .generated = false, + .cipher =NVME_TCP_TLS_CIPHER_SHA384 , }, + { .generated = false, + .cipher =NVME_TCP_TLS_CIPHER_SHA256 , }, + { .generated = true, + .cipher =NVME_TCP_TLS_CIPHER_SHA384 , }, + { .generated = true, + .cipher =NVME_TCP_TLS_CIPHER_SHA256 , }, +}; + +/* +* nvme_tls_psk_default - Return 'best' PSK to use for TLS ClientHello +*/ +key_serial_t nvme_tls_psk_default (struct key *keyring, +const char *hostnqn,const char *subnqn) +{ +struct key *tls_key; +key_serial_t tls_key_id; +int prio; + +for (prio =0 ; prio <ARRAY_SIZE (nvme_psk_prio); prio++) { +bool generated = nvme_psk_prio[prio].generated; +enum nvme_tcp_tls_cipher cipher = nvme_psk_prio[prio].cipher; + + tls_key =nvme_tls_psk_lookup (keyring, hostnqn, subnqn, + cipher, generated); +if (!IS_ERR (tls_key)) { + tls_key_id = tls_key->serial; +key_put (tls_key); +return tls_key_id; + } + } +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_tls_psk_default); +int nvme_keyring_init (void ) {int err;
include/linux/nvme-keyring.h +2 -0
@@ -9,6 +9,8 @@struct key *nvme_tls_psk_lookup (struct key *keyring,const char *hostnqn,const char *subnqn,int hmac,bool generated); +key_serial_t nvme_tls_psk_default (struct key *keyring, +const char *hostnqn,const char *subnqn);key_serial_t nvme_keyring_id (void );
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 07/18] nvme-keyring: implement nvme_tls_psk_default()
Quoted 14 lines · Hannes Reinecke
Implement a function to select the 'best' PSK for TLS.
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/common/keyring.c | 47 +++++++++++++++++++++++++++++++++++ include/linux/nvme-keyring.h | 2 ++ 2 files changed, 49 insertions(+)drivers/nvme/common/keyring.c +0
-0@@ -103,6 +103,53 @@ struct key *nvme_tls_psk_lookup(struct key *keyring, }
drivers/nvme/common/keyring.c struct key *nvme_tls_psk_lookup(struct key *keyring, · line 113
Quoted 10 lines
drivers/nvme/common/keyring.c +8
-0EXPORT_SYMBOL_GPL (nvme_tls_psk_lookup); +/* +* NVMe PSK priority list +* +* 'Retained' PSKs (ie 'generated == false') +* should be preferred to 'generated' PSKs, +* and SHA-384 should be preferred to SHA-256. +*/ +struct nvme_psk_priority_list {
nvme_tls_psk_priority_list
drivers/nvme/common/keyring.c struct key *nvme_tls_psk_lookup(struct key *keyring, · line 116
Quoted 3 lines
drivers/nvme/common/keyring.c +3
-0+bool generated; +enum nvme_tcp_tls_cipher cipher; +} nvme_psk_prio[] = {
nvme_tls_psk_prio
drivers/nvme/common/keyring.c struct key *nvme_tls_psk_lookup(struct key *keyring, · line 128
Quoted 12 lines
drivers/nvme/common/keyring.c +12
-0+ { .generated = false, + .cipher = NVME_TCP_TLS_CIPHER_SHA384, }, + { .generated = false, + .cipher = NVME_TCP_TLS_CIPHER_SHA256, }, + { .generated = true, + .cipher = NVME_TCP_TLS_CIPHER_SHA384, }, + { .generated = true, + .cipher = NVME_TCP_TLS_CIPHER_SHA256, }, +}; + +/* + * nvme_tls_psk_default - Return 'best' PSK to use for TLS ClientHello
s/'best'/the preferred/...
PATCH v2 8/18 security/keys: export key_lookup()
In reply to Hannes ReineckeFor in-kernel consumers one cannot readily assign a user (eg when running from a workqueue), so the normal key search permissions cannot be applied.
This patch exports the 'key_lookup()' function for a simple lookup of keys without checking for permissions.
--- security/keys/key.c | 1 + 1 file changed, 1 insertion(+)
security/keys/key.c +1 -0
@@ -693,6 +693,7 @@ struct key *key_lookup(key_serial_t id)spin_unlock (&key_serial_lock);return key; } +EXPORT_SYMBOL_GPL (key_lookup);/* * Find and lock the specified key type against removal.
Signature · 1 line
2.35.3
PATCH v2 9/18 nvme-tcp: enable TLS handshake upcall
In reply to Hannes ReineckeAdd a fabrics option 'tls' and start the TLS handshake upcall with the default PSK. When TLS is started the PSK key serial number is displayed in the sysfs attribute 'tls_key'
--- drivers/nvme/host/core.c | 25 ++++++- drivers/nvme/host/fabrics.c | 11 +++ drivers/nvme/host/fabrics.h | 3 + drivers/nvme/host/nvme.h | 4 +- drivers/nvme/host/tcp.c | 141 ++++++++++++++++++++++++++++++++++-- 5 files changed, 174 insertions(+), 10 deletions(-)
drivers/nvme/host/core.c +24 -1
@@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR, nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);#endif +#ifdef CONFIG_NVME_TLS +static ssize_t tls_key_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +if (!ctrl->tls_key) +return 0 ; +return sysfs_emit (buf,"%08x" ,key_serial (ctrl->tls_key)); +} +static DEVICE_ATTR_RO (tls_key); +#endif +static struct attribute *nvme_dev_attrs[] = { &dev_attr_reset_controller.attr, &dev_attr_rescan_controller.attr,
@@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {#ifdef CONFIG_NVME_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr, +#endif +#ifdef CONFIG_NVME_TLS + &dev_attr_tls_key.attr, #endifNULL };
@@ -3956,6 +3972,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,return 0 ;if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)return 0 ; +#endif +#ifdef CONFIG_NVME_TLS +if (a == &dev_attr_tls_key.attr && !ctrl->opts) +return 0 ;#endif return a->mode; }
@@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)if (!subsys || ctrl->instance != subsys->instance)ida_free (&nvme_instance_ida, ctrl->instance); - +#ifdef CONFIG_NVME_TLS +if (ctrl->tls_key) +key_put (ctrl->tls_key); +#endif nvme_free_cels (ctrl);nvme_mpath_uninit (ctrl);nvme_auth_stop (ctrl);
drivers/nvme/host/fabrics.c +11 -0
@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" }, { NVMF_OPT_DHCHAP_CTRL_SECRET, "dhchap_ctrl_secret=%s" }, + { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_ERR, NULL } };
@@ -632,6 +633,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, opts->hdr_digest = false; opts->data_digest = false; opts->tos =-1 ;/* < 0 == use transport default */ + opts->tls = false; options = o =kstrdup (buf,GFP_KERNEL );if (!options)
@@ -918,6 +920,15 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,kfree (opts->dhchap_ctrl_secret); opts->dhchap_ctrl_secret = p;break ; +case NVMF_OPT_TLS : +#ifdef CONFIG_NVME_TLS + opts->tls = true; +break ; +#else +pr_err ("TLS is not supported\n" ); + ret = -EINVAL ; + goto out; +#endif default :pr_warn ("unknown parameter or missing value '%s' in ctrl creation request\n" , p);
drivers/nvme/host/fabrics.h +3 -0
@@ -70,6 +70,7 @@ enum {NVMF_OPT_DISCOVERY =1 <<22 ,NVMF_OPT_DHCHAP_SECRET =1 <<23 ,NVMF_OPT_DHCHAP_CTRL_SECRET =1 <<24 , + NVMF_OPT_TLS = 1 << 25, };/**
@@ -102,6 +103,7 @@ enum {* @dhchap_secret: DH-HMAC-CHAP secret * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional * authentication +* @tls: Start TLS encrypted connections (TCP) * @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP) * @data_digest: generate/verify data digest (TCP)
@@ -128,6 +130,7 @@ struct nvmf_ctrl_options {int max_reconnects;char *dhchap_secret;char *dhchap_ctrl_secret; +bool tls;bool disable_sqflow;bool hdr_digest;bool data_digest;
drivers/nvme/host/nvme.h +3 -1
@@ -347,7 +347,9 @@ struct nvme_ctrl {struct nvme_dhchap_key *ctrl_key;u16 transaction;#endif - +#ifdef CONFIG_NVME_TLS +struct key *tls_key; +#endif /* Power saving configuration */ u64 ps_max_latency_us;bool apst_enabled;
drivers/nvme/host/tcp.c +133 -8
194 lines not on this page · raw message
Signature · 1 line
2.35.3
Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
On Wed, Mar 29, 2023 at 03:59:29PM +0200, Hannes Reinecke wrote:
--- a/drivers/nvme/host/fabrics.c
drivers/nvme/host/fabrics.c static const match_table_t opt_tokens = { · line 614
Quoted 8 lines
+++
b/drivers/nvme/host/fabrics.cdrivers/nvme/host/fabrics.c +1
-0@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" }, { NVMF_OPT_DHCHAP_CTRL_SECRET, "dhchap_ctrl_secret=%s" }, + { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_ERR, NULL } };
This should be ifdef out for !CONFIG_NVME_TLS
Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
Quoted 12 lines · Daniel Wagner
On Wed, Mar 29, 2023 at 03:59:29PM +0200, Hannes Reinecke wrote:
---a/drivers/nvme/host/fabrics.cdrivers/nvme/host/fabrics.c static const match_table_t opt_tokens = { · line 614
Quoted 8 lines
+++
b/drivers/nvme/host/fabrics.cdrivers/nvme/host/fabrics.c +1
-0@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" }, { NVMF_OPT_DHCHAP_CTRL_SECRET, "dhchap_ctrl_secret=%s" }, + { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_ERR, NULL } };This should be ifdef out for !CONFIG_NVME_TLS
No, I did that on purpose.
Error handling in the fabrics option parsing is notoriously bad, and if we disable that nvme-cli will fail with annoying interfaces. As it stands, the 'tls' option will be accepted, but you'll get a kernel message telling you it's not compiled in, and the function returns -EINVAL.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman (HRB 36809, AG Nürnberg)
Sagi Grimberg
maintainerRe: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
In reply to Hannes ReineckeQuoted 93 lines · Hannes Reinecke
Add a fabrics option 'tls' and start the TLS handshake upcall with the default PSK. When TLS is started the PSK key serial number is displayed in the sysfs attribute '
tls_key'Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/core.c | 25 ++++++- drivers/nvme/host/fabrics.c | 11 +++ drivers/nvme/host/fabrics.h | 3 + drivers/nvme/host/nvme.h | 4 +- drivers/nvme/host/tcp.c | 141 ++++++++++++++++++++++++++++++++++-- 5 files changed, 174 insertions(+), 10 deletions(-)drivers/nvme/host/core.c +24
-1@@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR, nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store);#endif +#ifdef CONFIG_NVME_TLS +static ssize_t tls_key_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +if (!ctrl->tls_key) +return 0 ; +return sysfs_emit (buf,"%08x" ,key_serial (ctrl->tls_key)); +} +static DEVICE_ATTR_RO (tls_key); +#endif +static struct attribute *nvme_dev_attrs[] = { &dev_attr_reset_controller.attr, &dev_attr_rescan_controller.attr,@@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = {#ifdef CONFIG_NVME_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr, +#endif +#ifdef CONFIG_NVME_TLS + &dev_attr_tls_key.attr, #endifNULL };@@ -3956,6 +3972,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,return 0 ;if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts)return 0 ; +#endif +#ifdef CONFIG_NVME_TLS +if (a == &dev_attr_tls_key.attr && !ctrl->opts) +return 0 ;#endif return a->mode; }@@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev)if (!subsys || ctrl->instance != subsys->instance)ida_free (&nvme_instance_ida, ctrl->instance); - +#ifdef CONFIG_NVME_TLS +if (ctrl->tls_key) +key_put (ctrl->tls_key); +#endif nvme_free_cels (ctrl);nvme_mpath_uninit (ctrl);nvme_auth_stop (ctrl);drivers/nvme/host/fabrics.c +2
-0@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" }, { NVMF_OPT_DHCHAP_CTRL_SECRET, "dhchap_ctrl_secret=%s" }, + { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_ERR, NULL } };@@ -632,6 +633,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, opts->hdr_digest = false; opts->data_digest = false; opts->tos =-1 ;/* < 0 == use transport default */ + opts->tls = false; options = o =kstrdup (buf,GFP_KERNEL );if (!options)@@ -918,6 +920,15 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,kfree (opts->dhchap_ctrl_secret); opts->dhchap_ctrl_secret = p;
drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct nvmf_ctrl_options *opts, · line 931
Quoted 10 lines
drivers/nvme/host/fabrics.c +9
-0break ; +case NVMF_OPT_TLS : +#ifdef CONFIG_NVME_TLS + opts->tls = true; +break ; +#else +pr_err ("TLS is not supported\n" ); + ret = -EINVAL ; + goto out; +#endif
This can become:
if (!IS_ENABLED(CONFIG_NVME_TLS)) {
pr_err("TLS is not supported\n");
ret = -EINVAL;
goto out;
}
opts->tls = true;
break;Quoted 36 lines
drivers/nvme/host/fabrics.c +0
-0default :pr_warn ("unknown parameter or missing value '%s' in ctrl creation request\n" , p);drivers/nvme/host/fabrics.h +3
-0@@ -70,6 +70,7 @@ enum {NVMF_OPT_DISCOVERY =1 <<22 ,NVMF_OPT_DHCHAP_SECRET =1 <<23 ,NVMF_OPT_DHCHAP_CTRL_SECRET =1 <<24 , + NVMF_OPT_TLS = 1 << 25, };/** @@ -102,6 +103,7 @@ enum {* @dhchap_secret: DH-HMAC-CHAP secret * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional * authentication +* @tls: Start TLS encrypted connections (TCP) * @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP) * @data_digest: generate/verify data digest (TCP) @@ -128,6 +130,7 @@ struct nvmf_ctrl_options {int max_reconnects;char *dhchap_secret;char *dhchap_ctrl_secret; +bool tls;bool disable_sqflow;bool hdr_digest;bool data_digest;drivers/nvme/host/nvme.h +0
-0
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 352
Quoted 7 lines
drivers/nvme/host/nvme.h +3
-1@@ -347,7 +347,9 @@ struct nvme_ctrl {struct nvme_dhchap_key *ctrl_key;u16 transaction;#endif - +#ifdef CONFIG_NVME_TLS +struct key *tls_key; +#endif
I'd add a comment that this is nvme_tcp specific.
Quoted 41 lines
drivers/nvme/host/nvme.h +0
-0/* Power saving configuration */ u64 ps_max_latency_us;bool apst_enabled;drivers/nvme/host/tcp.c +16
-0@@ -8,9 +8,15 @@#include <linux/init.h> #include <linux/slab.h> #include <linux/err.h> +#include <linux/key.h> #include <linux/nvme-tcp.h> #include <net/sock.h> #include <net/tcp.h> +#ifdef CONFIG_NVME_TLS +#include <linux/nvme-keyring.h> +#include <net/tls.h> +#include <net/handshake.h> +#endif #include <linux/blk-mq.h> #include <crypto/hash.h> #include <net/busy_poll.h> @@ -31,6 +37,16 @@ static int so_priority;module_param (so_priority, int,0644 );MODULE_PARM_DESC (so_priority,"nvme tcp socket optimize priority" ); +#ifdef CONFIG_NVME_TLS +/* +* TLS handshake timeout +*/ +static int tls_handshake_timeout =10 ; +module_param (tls_handshake_timeout, int,0644 ); +MODULE_PARM_DESC (tls_handshake_timeout, +"nvme TLS handshake timeout in seconds (default 10)" ); +#endif +#ifdef CONFIG_DEBUG_LOCK_ALLOC /* lockdep can detect a circular dependency of the form * sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock
drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 167
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-1@@ -147,7 +163,10 @@ struct nvme_tcp_queue {struct ahash_request *snd_hash;__le32 exp_ddgst;__le32 recv_ddgst; - +#ifdef CONFIG_NVME_TLS +struct completion *tls_complete;
Why is this a pointer?
Quoted 9 lines
drivers/nvme/host/tcp.c +2
-0+int tls_err; +#endif struct page_frag_cache pf_cache;void (*state_change)(struct sock *);@@ -1503,7 +1522,80 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) queue->io_cpu =cpumask_next_wrap (n -1 , cpu_online_mask,-1 , false); }
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1533
Quoted 10 lines
drivers/nvme/host/tcp.c +9
-1-static int nvme_tcp_alloc_queue (struct nvme_ctrl *nctrl,int qid) +#ifdef CONFIG_NVME_TLS +static void nvme_tcp_tls_done (void *data,int status,key_serial_t peerid) +{ +struct nvme_tcp_queue *queue = data; +struct nvme_tcp_ctrl *ctrl = queue->ctrl; +int qid =nvme_tcp_queue_id (queue); + +dev_dbg (ctrl->ctrl.device,"queue %d: TLS handshake done, key %x, status %d\n" , + qid, peerid, status);
Can we avoid naming the key peerid? pskid perhaps.
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1536
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+ +if (status) + queue->tls_err = -status;
Maybe just return or goto err, instead of the else case?
Quoted 15 lines
drivers/nvme/host/tcp.c +15
-0+else { +struct key *tls_key =key_lookup (peerid); +if (IS_ERR (tls_key)) { +dev_warn (ctrl->ctrl.device,"queue %d: Invalid key %x\n" , + qid, peerid); + queue->tls_err = -ENOKEY ; + }else { + ctrl->ctrl.tls_key = tls_key; + queue->tls_err =0 ; + } + } + +if (queue->tls_complete) +complete (queue->tls_complete); +}
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1561
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ +static int nvme_tcp_start_tls (struct nvme_ctrl *nctrl, +struct nvme_tcp_queue *queue, +key_serial_t peerid) +{ +int qid =nvme_tcp_queue_id (queue); +int ret; +struct tls_handshake_args args; +unsigned long tmo = tls_handshake_timeout *HZ ; +DECLARE_COMPLETION_ONSTACK (tls_complete);
I think we can have this completion a queue member.
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1572
Quoted 11 lines
drivers/nvme/host/tcp.c +11
-0+key_serial_t keyring =nvme_keyring_id (); + +dev_dbg (nctrl->device,"queue %d: start TLS with key %x\n" , + qid, peerid); + args.ta_sock = queue->sock; + args.ta_done = nvme_tcp_tls_done; + args.ta_data = queue; + args.ta_my_peerids[0 ] = peerid; + args.ta_num_peerids =1 ; + args.ta_keyring = keyring; + args.ta_timeout_ms = tls_handshake_timeout *2 *1000 ;
remove the 2x.
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1578
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+ queue->tls_err = -EOPNOTSUPP ; + queue->tls_complete = &tls_complete; + ret =tls_client_hello_psk (&args,GFP_KERNEL ); +if (ret) { +dev_dbg (nctrl->device,"queue %d: failed to start TLS: %d\n" , + qid, ret);
dev_err
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1583
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+return ret; + } +if (wait_for_completion_timeout (queue->tls_complete, tmo) ==0 ) { +dev_dbg (nctrl->device, +"queue %d: TLS handshake timeout\n" , qid);
dev_err
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1592
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+ queue->tls_complete =NULL ; + ret = -ETIMEDOUT ; + }else { +dev_dbg (nctrl->device, +"queue %d: TLS handshake complete, error %d\n" , + qid, queue->tls_err); + ret = queue->tls_err; + } + queue->tls_complete =NULL ;
as mentioned, lets make the completion a queue member and not a pointer that we need to pay attention to.
Quoted 34 lines
drivers/nvme/host/tcp.c +18
-2+ return ret; +} +#endif + +static int nvme_tcp_alloc_queue (struct nvme_ctrl *nctrl,int qid, +key_serial_t peerid) {struct nvme_tcp_ctrl *ctrl =to_tcp_ctrl (nctrl);struct nvme_tcp_queue *queue = &ctrl->queues[qid];@@ -1627,6 +1719,14 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) goto err_rcv_pdu; } +#ifdef CONFIG_NVME_TLS +/* If PSKs are configured try to start TLS */ +if (peerid) { + ret =nvme_tcp_start_tls (nctrl, queue, peerid); +if (ret) + goto err_init_connect; + } +#endif ret =nvme_tcp_init_connection (queue);if (ret) goto err_init_connect;@@ -1771,10 +1871,23 @@ static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl,static int nvme_tcp_alloc_admin_queue (struct nvme_ctrl *ctrl) {int ret; - - ret =nvme_tcp_alloc_queue (ctrl,0 ); +key_serial_t psk_id =0 ; + +#ifdef CONFIG_NVME_TLS +if (ctrl->opts->tls) {
drivers/nvme/host/tcp.c static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl, · line 1887
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ psk_id =nvme_tls_psk_default (NULL , + ctrl->opts->host->nqn, + ctrl->opts->subsysnqn); +if (!psk_id) { +dev_err (ctrl->device,"no valid PSK found\n" ); + ret = -ENOKEY ; + goto out_free_queue; + } + } +#endif
ctrl->opts->tls == true guarantes that CONFIG_NVME_TLS is enabled. No need for it, as long as you give a nvme_tls_psk_default stub when it is not.
Quoted 15 lines
drivers/nvme/host/tcp.c +5
-2+ ret =nvme_tcp_alloc_queue (ctrl,0 , psk_id);if (ret) -return ret; + goto out_free_queue; ret =nvme_tcp_alloc_async_req (to_tcp_ctrl (ctrl));if (ret)@@ -1790,9 +1903,21 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)static int __nvme_tcp_alloc_io_queues (struct nvme_ctrl *ctrl) {int i, ret; - +key_serial_t psk_id =0 ; + +#ifdef CONFIG_NVME_TLS
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) · line 1918
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+if (ctrl->opts->tls) { + psk_id =nvme_tls_psk_default (NULL , + ctrl->opts->host->nqn, + ctrl->opts->subsysnqn); +if (!psk_id) { +dev_err (ctrl->device,"no valid PSK found\n" ); +return -ENOKEY ; + } + } +#endif
Do we need this for io queues? why not just store it on nvme_tcp_ctrl and just use it?
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-1for (i =1 ; i < ctrl->queue_count; i++) { - ret =nvme_tcp_alloc_queue (ctrl, i); + ret =nvme_tcp_alloc_queue (ctrl, i, psk_id);if (ret) goto out_free_queues; }
drivers/nvme/host/tcp.c static struct nvmf_transport_ops nvme_tcp_transport = { · line 2832
Quoted 8 lines
drivers/nvme/host/tcp.c +1
-1@@ -2701,7 +2826,7 @@ static struct nvmf_transport_ops nvme_tcp_transport = {NVMF_OPT_HOST_TRADDR |NVMF_OPT_CTRL_LOSS_TMO |NVMF_OPT_HDR_DIGEST |NVMF_OPT_DATA_DIGEST |NVMF_OPT_NR_WRITE_QUEUES |NVMF_OPT_NR_POLL_QUEUES | - NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE, + NVMF_OPT_TOS | NVMF_OPT_HOST_IFACE | NVMF_OPT_TLS, .create_ctrl = nvme_tcp_create_ctrl, };
Overall this looks much better Hannes!
Re: [PATCH 09/18] nvme-tcp: enable TLS handshake upcall
Quoted 107 lines · Sagi Grimberg
Quoted 93 lines · Hannes Reinecke
Add a fabrics option 'tls' and start the TLS handshake upcall with the default PSK. When TLS is started the PSK key serial number is displayed in the sysfs attribute '
tls_key'Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/core.c | 25 ++++++-drivers/nvme/host/fabrics.c| 11 +++drivers/nvme/host/fabrics.h| 3 + drivers/nvme/host/nvme.h | 4 +- drivers/nvme/host/tcp.c | 141 ++++++++++++++++++++++++++++++++++--5 files changed, 174 insertions(+), 10 deletions(-)drivers/nvme/host/core.c +16
-0@@ -3901,6 +3901,19 @@ static DEVICE_ATTR(dhchap_ctrl_secret, S_IRUGO | S_IWUSR, nvme_ctrl_dhchap_ctrl_secret_show, nvme_ctrl_dhchap_ctrl_secret_store); #endif +#ifdef CONFIG_NVME_TLS +static ssize_t tls_key_show (struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + if (!ctrl->tls_key) + return 0; + return sysfs_emit(buf, "%08x", key_serial(ctrl->tls_key)); +} +static DEVICE_ATTR_RO (tls_key); +#endif + static struct attribute *nvme_dev_attrs[] = { &dev_attr_reset_controller.attr, &dev_attr_rescan_controller.attr,@@ -3927,6 +3940,9 @@ static struct attribute *nvme_dev_attrs[] = { #ifdef CONFIG_NVME_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr, +#endif +#ifdef CONFIG_NVME_TLS + &dev_attr_tls_key.attr, #endifNULL
};drivers/nvme/host/core.c +4
-0@@ -3956,6 +3972,10 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj, return 0; if (a == &dev_attr_dhchap_ctrl_secret.attr && !ctrl->opts) return 0; +#endif +#ifdef CONFIG_NVME_TLS + if (a == &dev_attr_tls_key.attr && !ctrl->opts) + return 0; #endif return a->mode;}
drivers/nvme/host/core.c +4
-1@@ -5094,7 +5114,10 @@ static void nvme_free_ctrl(struct device *dev) if (!subsys || ctrl->instance != subsys->instance) ida_free(&nvme_instance_ida, ctrl->instance); - +#ifdef CONFIG_NVME_TLS + if (ctrl->tls_key) + key_put(ctrl->tls_key); +#endif nvme_free_cels(ctrl); nvme_mpath_uninit(ctrl); nvme_auth_stop(ctrl);drivers/nvme/host/fabrics.c +1
-0@@ -609,6 +609,7 @@ static const match_table_t opt_tokens = { { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" }, { NVMF_OPT_DHCHAP_CTRL_SECRET, "dhchap_ctrl_secret=%s" }, + { NVMF_OPT_TLS, "tls" }, { NVMF_OPT_ERR, NULL } };@@ -632,6 +633,7 @@ static int nvmf_parse_options(structnvmf_ctrl_options *opts, opts->hdr_digest = false; opts->data_digest = false;
opts->tos= -1; /* < 0 == use transport default */ +opts->tls= false;
options = o =kstrdup(buf, GFP_KERNEL); if (!options)drivers/nvme/host/fabrics.c +0
-0@@ -918,6 +920,15 @@ static int nvmf_parse_options(structnvmf_ctrl_options *opts, kfree(opts->dhchap_ctrl_secret); opts->dhchap_ctrl_secret = p;drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct · line 932
Quoted 10 lines
drivers/nvme/host/fabrics.c +9
-0break; + case NVMF_OPT_TLS: +#ifdef CONFIG_NVME_TLS + opts->tls = true; + break; +#else + pr_err("TLS is not supported\n"); + ret = -EINVAL; + goto out; +#endif
Quoted 9 lines
This can become:
if (!IS_ENABLED(CONFIG_NVME_TLS)) {pr_err("TLS is not supported\n"); ret =-EINVAL;
goto out;
}
opts->tls= true;
break;
Yes.
Quoted 36 lines
Quoted 36 lines
default:
pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n", p);drivers/nvme/host/fabrics.h +3
-0@@ -70,6 +70,7 @@ enum { NVMF_OPT_DISCOVERY = 1 << 22, NVMF_OPT_DHCHAP_SECRET = 1 << 23, NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24, + NVMF_OPT_TLS = 1 << 25, }; /**@@ -102,6 +103,7 @@ enum { * @dhchap_secret: DH-HMAC-CHAP secret * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional * authentication + * @tls: Start TLS encrypted connections (TCP) * @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP) * @data_digest: generate/verify data digest (TCP)@@ -128,6 +130,7 @@ struct nvmf_ctrl_options { int max_reconnects; char *dhchap_secret; char *dhchap_ctrl_secret; + bool tls; bool disable_sqflow; bool hdr_digest; bool data_digest;drivers/nvme/host/nvme.h +0
-0@@ -347,7 +347,9 @@ struct nvme_ctrl { struct nvme_dhchap_key *ctrl_key;
Quoted 9 lines
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 352
Quoted 6 lines
drivers/nvme/host/nvme.h +3
-1u16 transaction; #endif - +#ifdef CONFIG_NVME_TLS + struct key *tls_key; +#endif I'd add a comment that this is
nvme_tcpspecific.
Okay.
Quoted 40 lines
Quoted 40 lines
drivers/nvme/host/nvme.h +0
-0/* Power saving configuration */ u64 ps_max_latency_us; bool apst_enabled;drivers/nvme/host/tcp.c +16
-0@@ -8,9 +8,15 @@ #include <linux/init.h> #include <linux/slab.h> #include <linux/err.h> +#include <linux/key.h> #include <linux/nvme-tcp.h> #include <net/sock.h> #include <net/tcp.h> +#ifdef CONFIG_NVME_TLS +#include <linux/nvme-keyring.h> +#include <net/tls.h> +#include <net/handshake.h> +#endif #include <linux/blk-mq.h> #include <crypto/hash.h> #include <net/busy_poll.h>@@ -31,6 +37,16 @@ static int so_priority; module_param(so_priority, int, 0644); MODULE_PARM_DESC(so_priority, "nvme tcp socket optimize priority"); +#ifdef CONFIG_NVME_TLS +/* +* TLS handshake timeout +*/ +static int tls_handshake_timeout =10 ; +module_param (tls_handshake_timeout, int,0644 ); +MODULE_PARM_DESC (tls_handshake_timeout, + "nvme TLS handshake timeout in seconds (default 10)"); +#endif + #ifdef CONFIG_DEBUG_LOCK_ALLOC /* lockdep can detect a circular dependency of the form* sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock @@ -147,7 +163,10 @@ struct nvme_tcp_queue {
Quoted 9 lines
drivers/nvme/host/tcp.c struct nvme_tcp_queue { · line 167
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-1struct ahash_request *snd_hash; __le32 exp_ddgst; __le32 recv_ddgst; - +#ifdef CONFIG_NVME_TLS + struct completion *tls_complete;Why is this a pointer?
Hmm. I'll change it.
Quoted 14 lines
Quoted 14 lines
drivers/nvme/host/tcp.c +6
-1+ int tls_err; +#endif struct page_frag_cache pf_cache; void (*state_change)(struct sock *);@@ -1503,7 +1522,80 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false); } -static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) +#ifdef CONFIG_NVME_TLS +static void nvme_tcp_tls_done (void *data,int status,key_serial_t peerid) +{ + struct nvme_tcp_queue *queue = data;
Quoted 9 lines
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1533
Quoted 6 lines
drivers/nvme/host/tcp.c +5
-0+ struct nvme_tcp_ctrl *ctrl = queue->ctrl; + int qid = nvme_tcp_queue_id(queue); + + dev_dbg(ctrl->ctrl.device, "queue %d: TLS handshake done, key %x, status %d\n", + qid, peerid, status);Can we avoid naming the key peerid? pskid perhaps.
Okay.
Quoted 6 lines
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1536
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+ + if (status) + queue->tls_err = -status;Maybe just return or goto err, instead of the else case?
Okay.
Quoted 19 lines
Quoted 19 lines
drivers/nvme/host/tcp.c +19
-0+ else { + struct key *tls_key = key_lookup(peerid); + if (IS_ERR(tls_key)) { + dev_warn(ctrl->ctrl.device, "queue %d: Invalid key %x\n", + qid, peerid); + queue->tls_err = -ENOKEY; + } else { + ctrl->ctrl.tls_key = tls_key; + queue->tls_err = 0; + } + } + + if (queue->tls_complete) + complete(queue->tls_complete); +} + +static int nvme_tcp_start_tls (struct nvme_ctrl *nctrl, + struct nvme_tcp_queue *queue, + key_serial_t peerid)
Quoted 9 lines
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1561
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+{ + int qid = nvme_tcp_queue_id(queue); + int ret; + struct tls_handshake_args args; + unsigned long tmo = tls_handshake_timeout * HZ; + DECLARE_COMPLETION_ONSTACK(tls_complete);I think we can have this completion a queue member.
Let's see.
Quoted 12 lines
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1572
Quoted 11 lines
drivers/nvme/host/tcp.c +11
-0+ key_serial_t keyring = nvme_keyring_id(); + + dev_dbg(nctrl->device, "queue %d: start TLS with key %x\n", + qid, peerid); + args.ta_sock = queue->sock; + args.ta_done = nvme_tcp_tls_done; + args.ta_data = queue; + args.ta_my_peerids[0] = peerid; + args.ta_num_peerids = 1; + args.ta_keyring = keyring; + args.ta_timeout_ms = tls_handshake_timeout * 2 * 1000;
Quoted 2 lines
remove the 2x.
Leftover, indeed.
Quoted 27 lines
drivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1578
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+ queue->tls_err = -EOPNOTSUPP; + queue->tls_complete = &tls_complete; + ret = tls_client_hello_psk(&args, GFP_KERNEL); + if (ret) { + dev_dbg(nctrl->device, "queue %d: failed to start TLS: %d\n", + qid, ret);
dev_errdrivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1583
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ return ret; + } + if (wait_for_completion_timeout(queue->tls_complete, tmo) == 0) { + dev_dbg(nctrl->device, + "queue %d: TLS handshake timeout\n", qid);
dev_errdrivers/nvme/host/tcp.c static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) · line 1592
Quoted 9 lines
drivers/nvme/host/tcp.c +9
-0+ queue->tls_complete = NULL; + ret = -ETIMEDOUT; + } else { + dev_dbg(nctrl->device, + "queue %d: TLS handshake complete, error %d\n", + qid, queue->tls_err); + ret = queue->tls_err; + } + queue->tls_complete = NULL;
Quoted 3 lines
as mentioned, lets make the completion a queue member and not a pointer that we need to pay attention to.
Okay.
Quoted 41 lines
Quoted 41 lines
drivers/nvme/host/tcp.c +6
-0+ return ret; +} +#endif + +static int nvme_tcp_alloc_queue (struct nvme_ctrl *nctrl,int qid, + key_serial_t peerid) {struct
nvme_tcp_ctrl*ctrl =to_tcp_ctrl(nctrl); structnvme_tcp_queue*queue = &ctrl->queues[qid];drivers/nvme/host/tcp.c +18
-2@@ -1627,6 +1719,14 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) goto err_rcv_pdu; } +#ifdef CONFIG_NVME_TLS + /* If PSKs are configured try to start TLS */ + if (peerid) { + ret = nvme_tcp_start_tls(nctrl, queue, peerid); + if (ret) + goto err_init_connect; + } +#endif ret = nvme_tcp_init_connection(queue); if (ret) goto err_init_connect;@@ -1771,10 +1871,23 @@ static int nvme_tcp_start_io_queues(structnvme_ctrl *ctrl, static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) { int ret; - - ret = nvme_tcp_alloc_queue(ctrl, 0); + key_serial_t psk_id = 0; + +#ifdef CONFIG_NVME_TLS + if (ctrl->opts->tls) { + psk_id = nvme_tls_psk_default(NULL, + ctrl->opts->host->nqn, + ctrl->opts->subsysnqn); + if (!psk_id) { + dev_err(ctrl->device, "no valid PSK found\n"); + ret = -ENOKEY;
Quoted 9 lines
drivers/nvme/host/tcp.c static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl, · line 1887
Quoted 4 lines
drivers/nvme/host/tcp.c +4
-0+ goto out_free_queue; + } + } +#endif
ctrl->opts->tls== true guarantes thatCONFIG_NVME_TLSis enabled. No need for it, as long as you give anvme_tls_psk_defaultstub when it is not.
Hmm. I deliberately had the 'tls' option always enabled to avoid parsing errors. But sure, if we make that a compile condition this can be simplified.
Quoted 20 lines
Quoted 20 lines
drivers/nvme/host/tcp.c +10
-2+ ret = nvme_tcp_alloc_queue(ctrl, 0, psk_id); if (ret) - return ret; + goto out_free_queue; ret = nvme_tcp_alloc_async_req(to_tcp_ctrl(ctrl)); if (ret)@@ -1790,9 +1903,21 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) { int i, ret; - + key_serial_t psk_id = 0; + +#ifdef CONFIG_NVME_TLS + if (ctrl->opts->tls) { + psk_id = nvme_tls_psk_default(NULL, + ctrl->opts->host->nqn, + ctrl->opts->subsysnqn); + if (!psk_id) {
Quoted 9 lines
drivers/nvme/host/tcp.c static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) · line 1918
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+ dev_err(ctrl->device, "no valid PSK found\n"); + return -ENOKEY; + } + } +#endif Do we need this for io queues? why not just store it on
nvme_tcp_ctrland just use it?
Guess we can; current debate at fmds is heading to having just one PSK for both admin and I/O queues anyway.
Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/tcp.c +1
-1for (i = 1; i < ctrl->queue_count; i++) { - ret = nvme_tcp_alloc_queue(ctrl, i); + ret = nvme_tcp_alloc_queue(ctrl, i, psk_id); if (ret) goto out_free_queues;}
drivers/nvme/host/tcp.c +0
-0@@ -2701,7 +2826,7 @@ static struct nvmf_transport_ops nvme_tcp_transport = {
NVMF_OPT_HOST_TRADDR|NVMF_OPT_CTRL_LOSS_TMO|NVMF_OPT_HDR_DIGEST|NVMF_OPT_DATA_DIGEST|
Quoted 8 lines
Quoted 5 lines
NVMF_OPT_NR_WRITE_QUEUES|NVMF_OPT_NR_POLL_QUEUES| -NVMF_OPT_TOS|NVMF_OPT_HOST_IFACE,
+NVMF_OPT_TOS|NVMF_OPT_HOST_IFACE|NVMF_OPT_TLS, .create_ctrl =nvme_tcp_create_ctrl,
};Overall this looks much better Hannes!
Thanks.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
PATCH v2 10/18 nvme-tcp: fixup send workflow for kTLS
In reply to Hannes ReineckekTLS does not support MSG_EOR flag for sendmsg(), and in general is really picky about invalid MSG_XXX flags.
So ensure that the MSG_EOR flags is blanked out for TLS, and that the MSG_SENDPAGE_LAST is only set if we actually do sendpage().
--- drivers/nvme/host/tcp.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +20 -8
@@ -218,6 +218,15 @@ static inline struct blk_mq_tags *nvme_tcp_tagset(struct nvme_tcp_queue *queue)return queue->ctrl->tag_set.tags[queue_idx -1 ]; } +static inline bool nvme_tcp_tls_enabled (struct nvme_tcp_queue *queue) +{ +#ifdef CONFIG_NVME_TLS +return (queue->ctrl->ctrl.tls_key !=NULL ); +#else +return false; +#endif +} +static inline u8 nvme_tcp_hdgst_len (struct nvme_tcp_queue *queue) {return queue->hdr_digest ?NVME_TCP_DIGEST_LENGTH :0 ;
@@ -1021,12 +1030,14 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)int req_data_sent = req->data_sent;int ret, flags =MSG_DONTWAIT ; -if (last && !queue->data_digest && !nvme_tcp_queue_more (queue)) +if (!last || queue->data_digest ||nvme_tcp_queue_more (queue)) + flags |=MSG_MORE ; +else if (!nvme_tcp_tls_enabled (queue)) flags |=MSG_EOR ; -else - flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ;if (sendpage_ok (page)) { +if (flags &MSG_MORE ) + flags |=MSG_SENDPAGE_NOTLAST ; ret =kernel_sendpage (queue->sock, page, offset, len, flags); }else {
@@ -1077,9 +1088,11 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)int flags =MSG_DONTWAIT ;int ret; -if (inline_data ||nvme_tcp_queue_more (queue)) - flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; -else +if (inline_data ||nvme_tcp_queue_more (queue)) { + flags |=MSG_MORE ; +if (!nvme_tcp_tls_enabled (queue)) + flags |=MSG_SENDPAGE_NOTLAST ; + }else if (!nvme_tcp_tls_enabled (queue)) flags |=MSG_EOR ;if (queue->hdr_digest && !req->offset)
@@ -1154,9 +1167,8 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)if (nvme_tcp_queue_more (queue)) msg.msg_flags |=MSG_MORE ; -else +else if (!nvme_tcp_tls_enabled (queue)) msg.msg_flags |=MSG_EOR ; - ret =kernel_sendmsg (queue->sock, &msg, &iov,1 , iov.iov_len);if (unlikely (ret <=0 ))return ret;
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Quoted 3 lines
kTLS does not support
MSG_EORflag forsendmsg(), and in general is really picky about invalidMSG_XXXflags.
CC'ing TLS folks.
Can't tls simply ignore MSG_EOR instead of consumers having to be careful over it?
Quoted 2 lines
So ensure that the
MSG_EORflags is blanked out for TLS, and that theMSG_SENDPAGE_LASTis only set if we actually dosendpage().
You mean MSG_SENDPAGE_NOTLAST.
It is also a bit annoying that a tls socket dictates different behavior than a normal socket.
The current logic is rather simple:
if more data comming:
flags = MSG_MORE | MSG_SENDPAGE_NOTLAST
else:
flags = MSG_EOR
Would like to keep it that way for tls as well. Can someone explain why this is a problem with tls?
Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Quoted 16 lines · Sagi Grimberg
Quoted 3 lines
kTLS does not support
MSG_EORflag forsendmsg(), and in general is really picky about invalidMSG_XXXflags.CC'ing TLS folks.
Can't tls simply ignore
MSG_EORinstead of consumers having to be careful over it?Quoted 2 lines
So ensure that the
MSG_EORflags is blanked out for TLS, and that theMSG_SENDPAGE_LASTis only set if we actually dosendpage().You mean
MSG_SENDPAGE_NOTLAST.It is also a bit annoying that a tls socket dictates different behavior than a normal socket.
Quoted 8 lines
The current logic is rather simple:
if more data comming:
flags =MSG_MORE|MSG_SENDPAGE_NOTLAST
else:
flags =MSG_EORWould like to keep it that way for tls as well. Can someone explain why this is a problem with tls?
TLS is using MSG_EOR internally (to control the flow of the underlying tcp stream), so it's meaningless for the ULP. I've no idea about SENDPAGE_NOTLAST; that one is particularly annoying
as we have to check whether we do sendpage() or sendmsg().
But TLS really should blank out invalid flags, as it has different rules for them than anyone else. And the caller really shouldn't be burdened with checking whether TLS is enabled or not.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Jakub Kicinski
maintainerRe: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
In reply to Sagi GrimbergQuoted 7 lines · On Thu, 30 Mar 2023 18:24:04 +0300 Sagi Grimberg wrote:
Quoted 2 lines
kTLS does not support
MSG_EORflag forsendmsg(), and in general is really picky about invalidMSG_XXXflags.CC'ing TLS folks.
Can't tls simply ignore
MSG_EORinstead of consumers having to be careful over it?
I think we can support EOR, I don't see any fundamental problem there.
Quoted 8 lines
Quoted 2 lines
So ensure that the
MSG_EORflags is blanked out for TLS, and that theMSG_SENDPAGE_LASTis only set if we actually dosendpage().You mean
MSG_SENDPAGE_NOTLAST.It is also a bit annoying that a tls socket dictates different behavior than a normal socket.
Quoted 8 lines
The current logic is rather simple:
if more data comming:
flags =MSG_MORE|MSG_SENDPAGE_NOTLAST
else:
flags =MSG_EORWould like to keep it that way for tls as well. Can someone explain why this is a problem with tls?
Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(
Re: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Quoted 25 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 30 Mar 2023 18:24:04 +0300 Sagi Grimberg wrote:
Quoted 2 lines
kTLS does not support
MSG_EORflag forsendmsg(), and in general is really picky about invalidMSG_XXXflags.CC'ing TLS folks.
Can't tls simply ignore
MSG_EORinstead of consumers having to be careful over it?I think we can support EOR, I don't see any fundamental problem there.
Quoted 14 lines
Quoted 2 lines
So ensure that the
MSG_EORflags is blanked out for TLS, and that theMSG_SENDPAGE_LASTis only set if we actually dosendpage().You mean
MSG_SENDPAGE_NOTLAST.It is also a bit annoying that a tls socket dictates different behavior than a normal socket.
The current logic is rather simple:
if more data comming:
flags =MSG_MORE|MSG_SENDPAGE_NOTLAST
else:
flags =MSG_EOR
Quoted 7 lines
Quoted 2 lines
Would like to keep it that way for tls as well. Can someone explain why this is a problem with tls?
Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(
A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerRe: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Hey Jakub, Hannes.
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
kTLS does not support
MSG_EORflag forsendmsg(), and in general is really picky about invalidMSG_XXXflags.CC'ing TLS folks.
Can't tls simply ignore
MSG_EORinstead of consumers having to be careful over it?I think we can support EOR, I don't see any fundamental problem there.
It would help at least one consumer (nvme-tcp) to not change its behavior between tls and non-tls. At the very minimum don't fail the send operation (just do the same as if it wasn't passed).
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 2 lines
So ensure that the
MSG_EORflags is blanked out for TLS, and that theMSG_SENDPAGE_LASTis only set if we actually dosendpage().You mean
MSG_SENDPAGE_NOTLAST.It is also a bit annoying that a tls socket dictates different behavior than a normal socket.
The current logic is rather simple:
if more data comming:
flags =MSG_MORE|MSG_SENDPAGE_NOTLAST
else:
flags =MSG_EORWould like to keep it that way for tls as well. Can someone explain why this is a problem with tls?
Quoted 8 lines
Quoted 4 lines
Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(
A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?
MSG_SENDPAGE_NOTLAST is not an internal flag, I thought it was essentially similar semantics to MSG_MORE but for sendpage. It'd be great if this can be allowed in tls (again, at the very least don't fail but continue as if it wasn't passed).
If this turns out to be a big project, I would prefer to change nvme-tcp for now in order not to block nvme tls support (although it is a hidden capability interface, which is always bad).
Jakub Kicinski
maintainerRe: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Quoted 5 lines · On Mon, 3 Apr 2023 15:20:13 +0300 Sagi Grimberg wrote:
Quoted 5 lines
Quoted 4 lines
Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(
Quoted 8 lines
Quoted 3 lines
A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?
MSG_SENDPAGE_NOTLASTis not an internal flag, I thought it was essentially similar semantics toMSG_MOREbut for sendpage. It'd be great if this can be allowed in tls (again, at the very least don't fail but continue as if it wasn't passed).
.. but.. MSG_SENDPAGE_NOTLAST is supported in TLS, isn't it? Why are we talking about it?
Sagi Grimberg
maintainerRe: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
Some of the flags are call specific, others may be internal to the networking stack (e.g. the DECRYPTED flag). Old protocols didn't do any validation because people coded more haphazardly in the 90s. This lack of validation is a major source of technical debt :(
A-ha. So what is the plan?
Should the stack validate flags?
And should the rules for validating be the same for all protocols?
Quoted 7 lines
Quoted 4 lines
MSG_SENDPAGE_NOTLASTis not an internal flag, I thought it was essentially similar semantics toMSG_MOREbut for sendpage. It'd be great if this can be allowed in tls (again, at the very least don't fail but continue as if it wasn't passed)... but..
MSG_SENDPAGE_NOTLASTis supported in TLS, isn't it? Why are we talking about it?
Ah, right.
What I'm assuming that Hannes is tripping on is that tls does not accept when this flag is sent to sock_no_sendpage, which is simply calling sendmsg. TLS will not accept this flag when passed to sendmsg IIUC.
Today the rough logic in nvme send path is:
if (more_coming(queue)) {
flags = MSG_MORE | MSG_SENDPAGE_NOTLAST;
} else {
flags = MSG_EOR;
}
if (!sendpage_ok(page)) {
kernel_sendpage();
} else {
sock_no_sendpage();
}This pattern (note that sock_no_sednpage was added later following bug reports where nvme attempted to sendpage a slab allocated page), is perfectly acceptable with normal sockets, but not with TLS.
So there are two options:
1. have tls accept MSG_SENDPAGE_NOTLAST in sendmsg (called from sock_no_sendpage)
2. Make nvme set MSG_SENDPAGE_NOTLAST only when calling kernel_sendpage and clear it when calling sock_no_sendpage
If you say that MSG_SENDPAGE_NOTLAST must be cleared when calling sock_no_sendpage and it is a bug that it isn't enforced for normal tcp sockets, then we need to change nvme, but I did not find any documentation that indicates it, and right now, normal sockets behave differently than tls sockets (wrt this flag in particular).
Hope this clarifies.
Jakub Kicinski
maintainerRe: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Quoted 29 lines · On Mon, 3 Apr 2023 18:51:09 +0300 Sagi Grimberg wrote:
What I'm assuming that Hannes is tripping on is that tls does not accept when this flag is sent to
sock_no_sendpage, which is simply calling sendmsg. TLS will not accept this flag when passed to sendmsg IIUC.Today the rough logic in nvme send path is:
if (more_coming(queue)) { flags = MSG_MORE | MSG_SENDPAGE_NOTLAST; } else { flags = MSG_EOR; } if (!sendpage_ok(page)) { kernel_sendpage(); } else { sock_no_sendpage(); }This pattern (note that
sock_no_sednpagewas added later following bug reports where nvme attempted to sendpage a slab allocated page), is perfectly acceptable with normal sockets, but not with TLS.So there are two options:
1. have tls acceptMSG_SENDPAGE_NOTLASTin sendmsg (called fromsock_no_sendpage)
2. Make nvme setMSG_SENDPAGE_NOTLASTonly when callingkernel_sendpageand clear it when callingsock_no_sendpage
Quoted 7 lines
If you say that
MSG_SENDPAGE_NOTLASTmust be cleared when callingsock_no_sendpageand it is a bug that it isn't enforced for normal tcp sockets, then we need to change nvme, but I did not find any documentation that indicates it, and right now, normal sockets behave differently than tls sockets (wrt this flag in particular).Hope this clarifies.
Oh right, it does, the context evaporated from my head over the weekend.
IMHO it's best if the caller passes the right flags. The semantics of MSG_MORE vs NOTLAST are quite murky and had already caused bugs in the past :(
See commit d452d48b9f8b ("tls: prevent oversized sendfile() hangs by ignoring MSG_MORE")
Alternatively we could have sock_no_sendpage drop NOTLAST to help all protos. But if we consider sendfile behavior as the standard simply clearing it isn't right, it should be a:
more = (flags & (MORE | NOTLAST)) == MORE | NOTLAST flags &= ~(MORE | NOTLAST) if (more) flags |= MORE
Sagi Grimberg
maintainerRe: [PATCH 10/18] nvme-tcp: fixup send workflow for kTLS
Quoted 38 lines · Jakub Kicinski
Quoted 29 lines · On Mon, 3 Apr 2023 18:51:09 +0300 Sagi Grimberg wrote:
What I'm assuming that Hannes is tripping on is that tls does not accept when this flag is sent to
sock_no_sendpage, which is simply calling sendmsg. TLS will not accept this flag when passed to sendmsg IIUC.Today the rough logic in nvme send path is:
if (more_coming(queue)) { flags = MSG_MORE | MSG_SENDPAGE_NOTLAST; } else { flags = MSG_EOR; } if (!sendpage_ok(page)) { kernel_sendpage(); } else { sock_no_sendpage(); }This pattern (note that
sock_no_sednpagewas added later following bug reports where nvme attempted to sendpage a slab allocated page), is perfectly acceptable with normal sockets, but not with TLS.So there are two options:
1. have tls acceptMSG_SENDPAGE_NOTLASTin sendmsg (called fromsock_no_sendpage)
2. Make nvme setMSG_SENDPAGE_NOTLASTonly when callingkernel_sendpageand clear it when callingsock_no_sendpageQuoted 7 lines
If you say that
MSG_SENDPAGE_NOTLASTmust be cleared when callingsock_no_sendpageand it is a bug that it isn't enforced for normal tcp sockets, then we need to change nvme, but I did not find any documentation that indicates it, and right now, normal sockets behave differently than tls sockets (wrt this flag in particular).Hope this clarifies.
Quoted 8 lines
Oh right, it does, the context evaporated from my head over the weekend.
IMHO it's best if the caller passes the right flags. The semantics of
MSG_MOREvs NOTLAST are quite murky and had already caused bugs in the past :(See commit
d452d48b9f8b("tls: prevent oversizedsendfile()hangs by ignoringMSG_MORE")
Well, that is fine with me. This may change anyways with the new MSG_SPLICE_PAGES from David.
Quoted 8 lines
Alternatively we could have
sock_no_sendpagedrop NOTLAST to help all protos. But if we consider sendfile behavior as the standard simply clearing it isn't right, it should be a:more = (flags & (MORE | NOTLAST)) == MORE | NOTLAST flags &= ~(MORE | NOTLAST) if (more) flags |= MORE
I don't think this would be the best option. Requiring callers to clear NOTLAST if not calling sendpages is reasonable, but we need to have this consistent. And also fix this pattern for the rest of the kernel socket consumers that use this flag.
PATCH v2 11/18 nvme-tcp: control message handling for recvmsg()
In reply to Hannes ReineckekTLS is sending TLS ALERT messages as control messages for recvmsg(). As we can't do anything sensible with it just abort the connection and let the userspace agent to a re-negotiation.
--- drivers/nvme/host/tcp.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +22 -1
@@ -1377,6 +1377,11 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue) {struct nvme_tcp_icreq_pdu *icreq;struct nvme_tcp_icresp_pdu *icresp; +#ifdef CONFIG_NVME_TLS +char cbuf[CMSG_LEN (sizeof (char ))] = {}; +struct cmsghdr *cmsg; +unsigned char ctype; +#endif struct msghdr msg = {};struct kvec iov;bool ctrl_hdgst, ctrl_ddgst;
@@ -1414,11 +1419,27 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)memset (&msg,0 ,sizeof (msg)); iov.iov_base = icresp; iov.iov_len =sizeof (*icresp); +#ifdef CONFIG_NVME_TLS + msg.msg_control = cbuf; + msg.msg_controllen =sizeof (cbuf); +#endif ret =kernel_recvmsg (queue->sock, &msg, &iov,1 , iov.iov_len, msg.msg_flags);if (ret <0 ) goto free_icresp; - +#ifdef CONFIG_NVME_TLS + cmsg = (struct cmsghdr *)cbuf; +if (CMSG_OK (&msg, cmsg) && + cmsg->cmsg_level ==SOL_TLS && + cmsg->cmsg_type ==TLS_GET_RECORD_TYPE ) { + ctype = *((unsigned char *)CMSG_DATA (cmsg)); +if (ctype !=TLS_RECORD_TYPE_DATA ) { +pr_err ("queue %d: unhandled TLS record %d\n" , +nvme_tcp_queue_id (queue), ctype); +return -ENOTCONN ; + } + } +#endif ret = -EINVAL ;if (icresp->hdr.type != nvme_tcp_icresp) { pr_err("queue %d: bad type returned %d\n",
Signature · 1 line
2.35.3
Re: [PATCH 11/18] nvme-tcp: control message handling for recvmsg()
This looks fine to me,
PATCH v2 12/18 nvme-fabrics: parse options 'keyring' and 'tls_key'
In reply to Hannes ReineckeParse the fabrics options 'keyring' and 'tls_key' and store the referenced keys in the options structure.
--- drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++- drivers/nvme/host/fabrics.h | 6 +++ drivers/nvme/host/tcp.c | 20 +++++++--- 3 files changed, 98 insertions(+), 7 deletions(-)
drivers/nvme/host/fabrics.c +78 -1
@@ -605,6 +605,8 @@ static const match_table_t opt_tokens = { { NVMF_OPT_NR_WRITE_QUEUES, "nr_write_queues=%d" }, { NVMF_OPT_NR_POLL_QUEUES, "nr_poll_queues=%d" }, { NVMF_OPT_TOS, "tos=%d" }, + { NVMF_OPT_KEYRING, "keyring=%d" }, + { NVMF_OPT_TLS_KEY, "tls_key=%d" }, { NVMF_OPT_FAIL_FAST_TMO, "fast_io_fail_tmo=%d" }, { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" },
@@ -620,8 +622,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,char *options, *o, *p;int token, ret =0 ;size_t nqnlen =0 ; -int ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO ; +int ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO , key_id;uuid_t hostid; +struct key *key =NULL ;/* Set defaults */ opts->queue_size =NVMF_DEF_QUEUE_SIZE ;
@@ -889,6 +892,74 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } opts->tos = token;break ; +case NVMF_OPT_KEYRING : +#ifdef CONFIG_NVME_TLS +if (match_int (args, &key_id)) { + ret = -EINVAL ; + goto out; + } +if (key_id <0 ) { +pr_err ("Invalid keyring id %d\n" , key_id); + ret = -EINVAL ; + goto out; + } +if (!key_id) { +pr_debug ("Using default keyring\n" ); +if (opts->keyring) { +key_put (opts->keyring); + opts->keyring =NULL ; + } +break ; + } + key =key_lookup (key_id); +if (!key) { +pr_err ("Keyring id %08x not found\n" , key_id); + ret = -ENOKEY ; + goto out; + } +if (opts->keyring) +key_put (opts->keyring); + opts->keyring = key; +break ; +#else +pr_err ("TLS is not supported\n" ); + ret = -EINVAL ; + goto out; +#endif +case NVMF_OPT_TLS_KEY : +#ifdef CONFIG_NVME_TLS +if (match_int (args, &key_id)) { + ret = -EINVAL ; + goto out; + } +if (key_id <0 ) { +pr_err ("Invalid key id %d\n" , key_id); + ret = -EINVAL ; + goto out; + } +if (!key_id) { +pr_debug ("Using 'best' PSK\n" ); +if (opts->tls_key) { +key_put (opts->tls_key); + opts->tls_key =NULL ; + } +break ; + } + key =key_lookup (key_id); +if (!key) { +pr_err ("Key id %08x not found\n" , key_id); + ret = -ENOKEY ; + goto out; + } +if (opts->tls_key) +key_put (opts->tls_key); + opts->tls_key = key; +#else +pr_err ("TLS is not supported\n" ); + ret = -EINVAL ; + goto out; +#endif +break ;case NVMF_OPT_DISCOVERY : opts->discovery_nqn = true;break ;
@@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts,void nvmf_free_options (struct nvmf_ctrl_options *opts) {nvmf_host_put (opts->host); +#ifdef CONFIG_NVME_TLS +if (opts->keyring) +key_put (opts->keyring); +if (opts->tls_key) +key_put (opts->tls_key); +#endif kfree (opts->transport);kfree (opts->traddr);kfree (opts->trsvcid);
drivers/nvme/host/fabrics.h +6 -0
@@ -71,6 +71,8 @@ enum {NVMF_OPT_DHCHAP_SECRET =1 <<23 ,NVMF_OPT_DHCHAP_CTRL_SECRET =1 <<24 ,NVMF_OPT_TLS =1 <<25 , +NVMF_OPT_KEYRING =1 <<26 , + NVMF_OPT_TLS_KEY = 1 << 27, };/**
@@ -103,6 +105,8 @@ enum {* @dhchap_secret: DH-HMAC-CHAP secret * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional * authentication +* @keyring: Keyring to use for key lookups +* @tls_key: TLS key for encrypted connections (TCP) * @tls: Start TLS encrypted connections (TCP) * @disable_sqflow: disable controller sq flow control * @hdr_digest: generate/verify header digest (TCP)
@@ -130,6 +134,8 @@ struct nvmf_ctrl_options {int max_reconnects;char *dhchap_secret;char *dhchap_ctrl_secret; +struct key *keyring; +struct key *tls_key;bool tls;bool disable_sqflow;bool hdr_digest;
drivers/nvme/host/tcp.c +14 -6
@@ -1596,6 +1596,8 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,dev_dbg (nctrl->device,"queue %d: start TLS with key %x\n" , qid, peerid); +if (nctrl->opts->keyring) + keyring =key_serial (nctrl->opts->keyring); args.ta_sock = queue->sock; args.ta_done = nvme_tcp_tls_done; args.ta_data = queue;
@@ -1908,9 +1910,12 @@ static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl)#ifdef CONFIG_NVME_TLS if (ctrl->opts->tls) { - psk_id =nvme_tls_psk_default (NULL , - ctrl->opts->host->nqn, - ctrl->opts->subsysnqn); +if (ctrl->opts->tls_key) + psk_id =key_serial (ctrl->opts->tls_key); +else + psk_id =nvme_tls_psk_default (ctrl->opts->keyring, + ctrl->opts->host->nqn, + ctrl->opts->subsysnqn);if (!psk_id) {dev_err (ctrl->device,"no valid PSK found\n" ); ret = -ENOKEY ;
@@ -1940,9 +1945,12 @@ static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)#ifdef CONFIG_NVME_TLS if (ctrl->opts->tls) { - psk_id =nvme_tls_psk_default (NULL , - ctrl->opts->host->nqn, - ctrl->opts->subsysnqn); +if (ctrl->opts->tls_key) + psk_id =key_serial (ctrl->opts->tls_key); +else + psk_id =nvme_tls_psk_default (ctrl->opts->keyring, + ctrl->opts->host->nqn, + ctrl->opts->subsysnqn);if (!psk_id) {dev_err (ctrl->device,"no valid PSK found\n" );return -ENOKEY ;
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
Quoted 2 lines
Parse the fabrics options 'keyring' and '
tls_key' and store the referenced keys in the options structure.
Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?
Quoted 33 lines
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/fabrics.c | 79 ++++++++++++++++++++++++++++++++++++- drivers/nvme/host/fabrics.h | 6 +++ drivers/nvme/host/tcp.c | 20 +++++++--- 3 files changed, 98 insertions(+), 7 deletions(-)drivers/nvme/host/fabrics.c +4
-1@@ -605,6 +605,8 @@ static const match_table_t opt_tokens = { { NVMF_OPT_NR_WRITE_QUEUES, "nr_write_queues=%d" }, { NVMF_OPT_NR_POLL_QUEUES, "nr_poll_queues=%d" }, { NVMF_OPT_TOS, "tos=%d" }, + { NVMF_OPT_KEYRING, "keyring=%d" }, + { NVMF_OPT_TLS_KEY, "tls_key=%d" }, { NVMF_OPT_FAIL_FAST_TMO, "fast_io_fail_tmo=%d" }, { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" },@@ -620,8 +622,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,char *options, *o, *p;int token, ret =0 ;size_t nqnlen =0 ; -int ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO ; +int ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO , key_id;uuid_t hostid; +struct key *key =NULL ;/* Set defaults */ opts->queue_size =NVMF_DEF_QUEUE_SIZE ;
drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct nvmf_ctrl_options *opts, · line 896
Quoted 5 lines
drivers/nvme/host/fabrics.c +2
-0@@ -889,6 +892,74 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } opts->tos = token;break ; +case NVMF_OPT_KEYRING : +#ifdef CONFIG_NVME_TLS
Same comment as before:
if (!IS_ENABLED(CONFIG_NVME_TLS)) {
pr_err("TLS is not supported\n");
ret = -EINVAL;
goto out;
}Quoted 70 lines
drivers/nvme/host/fabrics.c +66
-0+if (match_int (args, &key_id)) { + ret = -EINVAL ; + goto out; + } +if (key_id <0 ) { +pr_err ("Invalid keyring id %d\n" , key_id); + ret = -EINVAL ; + goto out; + } +if (!key_id) { +pr_debug ("Using default keyring\n" ); +if (opts->keyring) { +key_put (opts->keyring); + opts->keyring =NULL ; + } +break ; + } + key =key_lookup (key_id); +if (!key) { +pr_err ("Keyring id %08x not found\n" , key_id); + ret = -ENOKEY ; + goto out; + } +if (opts->keyring) +key_put (opts->keyring); + opts->keyring = key; +break ; +#else +pr_err ("TLS is not supported\n" ); + ret = -EINVAL ; + goto out; +#endif +case NVMF_OPT_TLS_KEY : +#ifdef CONFIG_NVME_TLS +if (match_int (args, &key_id)) { + ret = -EINVAL ; + goto out; + } +if (key_id <0 ) { +pr_err ("Invalid key id %d\n" , key_id); + ret = -EINVAL ; + goto out; + } +if (!key_id) { +pr_debug ("Using 'best' PSK\n" ); +if (opts->tls_key) { +key_put (opts->tls_key); + opts->tls_key =NULL ; + } +break ; + } + key =key_lookup (key_id); +if (!key) { +pr_err ("Key id %08x not found\n" , key_id); + ret = -ENOKEY ; + goto out; + } +if (opts->tls_key) +key_put (opts->tls_key); + opts->tls_key = key; +#else +pr_err ("TLS is not supported\n" ); + ret = -EINVAL ; + goto out; +#endif +break ;case NVMF_OPT_DISCOVERY : opts->discovery_nqn = true;break ;
drivers/nvme/host/fabrics.c static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts, · line 1132
Quoted 8 lines
drivers/nvme/host/fabrics.c +5
-0@@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts,void nvmf_free_options (struct nvmf_ctrl_options *opts) {nvmf_host_put (opts->host); +#ifdef CONFIG_NVME_TLS +if (opts->keyring) +key_put (opts->keyring); +if (opts->tls_key) +key_put (opts->tls_key);
I think key_put is null safe.
Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
Quoted 6 lines · Sagi Grimberg
Quoted 2 lines
Parse the fabrics options 'keyring' and '
tls_key' and store the referenced keys in the options structure.Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?
Choice.
With a single keyring we can only have a single identity.
But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).
With this option we can prepare a new keyring, and use that instead of the old one.
(And it really doesn't add much complexity...)
Quoted 38 lines
Quoted 38 lines
Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/fabrics.c| 79 ++++++++++++++++++++++++++++++++++++-drivers/nvme/host/fabrics.h| 6 +++ drivers/nvme/host/tcp.c | 20 +++++++---3 files changed, 98 insertions(+), 7 deletions(-)drivers/nvme/host/fabrics.c +5
-1@@ -605,6 +605,8 @@ static const match_table_t opt_tokens = { { NVMF_OPT_NR_WRITE_QUEUES, "nr_write_queues=%d" }, { NVMF_OPT_NR_POLL_QUEUES, "nr_poll_queues=%d" }, { NVMF_OPT_TOS, "tos=%d" }, + { NVMF_OPT_KEYRING, "keyring=%d" }, + { NVMF_OPT_TLS_KEY, "tls_key=%d" }, { NVMF_OPT_FAIL_FAST_TMO, "fast_io_fail_tmo=%d" }, { NVMF_OPT_DISCOVERY, "discovery" }, { NVMF_OPT_DHCHAP_SECRET, "dhchap_secret=%s" },@@ -620,8 +622,9 @@ static int nvmf_parse_options(structnvmf_ctrl_options *opts, char *options, *o, *p; int token, ret = 0; size_t nqnlen = 0; - int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO; + int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO, key_id; uuid_t hostid; + struct key *key = NULL; /* Set defaults */ opts->queue_size = NVMF_DEF_QUEUE_SIZE;@@ -889,6 +892,74 @@ static int nvmf_parse_options(structnvmf_ctrl_options *opts, } opts->tos = token; break; + case NVMF_OPT_KEYRING:
Quoted 9 lines
drivers/nvme/host/fabrics.c static int nvmf_parse_options(struct · line 897
Quoted 1 line
drivers/nvme/host/fabrics.c +1
-0+#ifdef CONFIG_NVME_TLS Same comment as before:
if (!IS_ENABLED(CONFIG_NVME_TLS)) {pr_err("TLS is not supported\n"); ret =-EINVAL;
goto out;
}
OK.
Quoted 73 lines
Quoted 73 lines
drivers/nvme/host/fabrics.c +66
-0+ if (match_int(args, &key_id)) { + ret = -EINVAL; + goto out; + } + if (key_id < 0) { + pr_err("Invalid keyring id %d\n", key_id); + ret = -EINVAL; + goto out; + } + if (!key_id) { + pr_debug("Using default keyring\n"); + if (opts->keyring) { + key_put(opts->keyring); + opts->keyring = NULL; + } + break; + } + key = key_lookup(key_id); + if (!key) { + pr_err("Keyring id %08x not found\n", key_id); + ret = -ENOKEY; + goto out; + } + if (opts->keyring) + key_put(opts->keyring); + opts->keyring = key; + break; +#else + pr_err("TLS is not supported\n"); + ret = -EINVAL; + goto out; +#endif + case NVMF_OPT_TLS_KEY: +#ifdef CONFIG_NVME_TLS + if (match_int(args, &key_id)) { + ret = -EINVAL; + goto out; + } + if (key_id < 0) { + pr_err("Invalid key id %d\n", key_id); + ret = -EINVAL; + goto out; + } + if (!key_id) { + pr_debug("Using 'best' PSK\n"); + if (opts->tls_key) { + key_put(opts->tls_key); + opts->tls_key = NULL; + } + break; + } + key = key_lookup(key_id); + if (!key) { + pr_err("Key id %08x not found\n", key_id); + ret = -ENOKEY; + goto out; + } + if (opts->tls_key) + key_put(opts->tls_key); + opts->tls_key = key; +#else + pr_err("TLS is not supported\n"); + ret = -EINVAL; + goto out; +#endif + break;case
NVMF_OPT_DISCOVERY:
opts->discovery_nqn= true;
break;drivers/nvme/host/fabrics.c +0
-0@@ -1054,6 +1125,12 @@ static int nvmf_check_allowed_opts(structnvmf_ctrl_options *opts, void nvmf_free_options(struct nvmf_ctrl_options *opts) {
Quoted 8 lines
drivers/nvme/host/fabrics.c static int nvmf_check_allowed_opts(struct nvmf_ctrl_options *opts, · line 1132
Quoted 6 lines
drivers/nvme/host/fabrics.c +5
-0nvmf_host_put(opts->host); +#ifdef CONFIG_NVME_TLS + if (opts->keyring) + key_put(opts->keyring); + if (opts->tls_key) + key_put(opts->tls_key);I think
key_putis null safe.
Possibly. I'll check.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerRe: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
Parse the fabrics options 'keyring' and '
tls_key' and store the referenced keys in the options structure.Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?
Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).
How do you expect that rotation would work with this?
How does nvmet handle a non-nvme keyring?
Quoted 1 line
With this option we can prepare a new keyring, and use that instead of the old one.
On an existing controller?
Quoted 1 line
(And it really doesn't add much complexity...)
I know, it just adds one more argument, and I want to understand if it is really needed.
Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
Quoted 6 lines · Sagi Grimberg
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Parse the fabrics options 'keyring' and '
tls_key' and store the referenced keys in the options structure.Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?
Quoted 7 lines
Quoted 4 lines
Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).How do you expect that rotation would work with this?
The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyring
Quoted 2 lines
How does nvmet handle a non-nvme keyring?
There is a configfs attribute 'param_keyring', allowing you to use a different keyring (the nvme one is just the default).
Quoted 4 lines
Quoted 1 line
With this option we can prepare a new keyring, and use that instead of the old one.
On an existing controller?
Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.
Quoted 4 lines
Quoted 1 line
(And it really doesn't add much complexity...)
I know, it just adds one more argument, and I want to understand if it is really needed.
I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.
So I'd prefer to keep it.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman (HRB 36809, AG Nürnberg)
Sagi Grimberg
maintainerRe: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines
Parse the fabrics options 'keyring' and '
tls_key' and store the referenced keys in the options structure.Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?
Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).
Quoted 7 lines
Quoted 2 lines
How do you expect that rotation would work with this?
The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyring
And the user cannot update the nvme keyring?
Quoted 3 lines
Quoted 2 lines
How does nvmet handle a non-nvme keyring?
There is a configfs attribute '
param_keyring', allowing you to use a different keyring (the nvme one is just the default).
OK.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
With this option we can prepare a new keyring, and use that instead of the old one.
On an existing controller?
Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
(And it really doesn't add much complexity...)
I know, it just adds one more argument, and I want to understand if it is really needed.
I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.
I'm not sure I understand why authentication move to keyrings requires that the user is able to bring its own keyring.
Quoted 1 line
So I'd prefer to keep it.
I'm not against it, but I failing to understand in what situation a user _must_ send a different keyring to the driver (and not use the nvme keyring).
Re: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
Quoted 14 lines · Sagi Grimberg
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines
Parse the fabrics options 'keyring' and '
tls_key' and store the referenced keys in the options structure.Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?
Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).How do you expect that rotation would work with this?
Quoted 8 lines
Quoted 5 lines
The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyringAnd the user cannot update the nvme keyring?
Sure, he can. All I'm saying is that he might choose to create a different one (Different tenants, setups, what do I know).
Quoted 20 lines
Quoted 3 lines
Quoted 2 lines
How does nvmet handle a non-nvme keyring?
There is a configfs attribute '
param_keyring', allowing you to use a different keyring (the nvme one is just the default).OK.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
With this option we can prepare a new keyring, and use that instead of the old one.
On an existing controller?
Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
(And it really doesn't add much complexity...)
I know, it just adds one more argument, and I want to understand if it is really needed.
I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.
Quoted 8 lines
I'm not sure I understand why authentication move to keyrings requires that the user is able to bring its own keyring.
Quoted 1 line
So I'd prefer to keep it.
I'm not against it, but I failing to understand in what situation a user _must_ send a different keyring to the driver (and not use the nvme keyring).
Oh, there is no 'must'. He _can_. And under certain circumstances
(in a multi-tenant environment, say) there are reasons to keep the keys separate into several keyrings.
Bit like the '--key' parameter; the code will try to select the 'default' key, but if the user wants to use a different one by all means, do. Same for the keyring; we do have the default keyring, but if the user for whatever reason doesn't want to use it we're giving him the possibility for that.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman (HRB 36809, AG Nürnberg)
Sagi Grimberg
maintainerRe: [PATCH 12/18] nvme-fabrics: parse options 'keyring' and 'tls_key'
Quoted 50 lines · Hannes Reinecke
Quoted 14 lines · Sagi Grimberg
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines
Parse the fabrics options 'keyring' and '
tls_key' and store the referenced keys in the options structure.Can you explain the reasoning to why a user need to pass a keyring given that we already set up one?
Choice.
With a single keyring we can only have a single identity. But there might be configurations where we want to have different PSKs for the same identity (eg for key rotation).How do you expect that rotation would work with this?
Quoted 8 lines
Quoted 5 lines
The user creates a new keyring, adds the necessary keys to it, and then either
- updates the keyring for nvmet (via configfs)
or
- reconnects using the new keyringAnd the user cannot update the nvme keyring?
Sure, he can. All I'm saying is that he might choose to create a different one (Different tenants, setups, what do I know).
Quoted 25 lines
Quoted 3 lines
Quoted 2 lines
How does nvmet handle a non-nvme keyring?
There is a configfs attribute '
param_keyring', allowing you to use a different keyring (the nvme one is just the default).OK.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
With this option we can prepare a new keyring, and use that instead of the old one.
On an existing controller?
Sure; when we are updating the keyring and the key id the controller will use that after the next reset. Much like we do for DH-CHAP nowadays.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
(And it really doesn't add much complexity...)
I know, it just adds one more argument, and I want to understand if it is really needed.
I do agree that the keyring argument would not necessarily required if we pass in the key id, but I'll be needing the keyring for secure concatenation. And plan is to move DH-HMAC-CHAP over to keyrings, too.
I'm not sure I understand why authentication move to keyrings requires that the user is able to bring its own keyring.
Quoted 1 line
So I'd prefer to keep it.
Quoted 8 lines
Quoted 3 lines
I'm not against it, but I failing to understand in what situation a user _must_ send a different keyring to the driver (and not use the nvme keyring).
Oh, there is no 'must'. He _can_. And under certain circumstances
(in a multi-tenant environment, say) there are reasons to keep the keys separate into several keyrings.
Bit like the '--key' parameter; the code will try to select the 'default' key, but if the user wants to use a different one by all means, do. Same for the keyring; we do have the default keyring, but if the user for whatever reason doesn't want to use it we're giving him the possibility for that.
I'm not overly keen to overload the argument space without proper real-life justification. But maybe it does make sense to allow users to store their secrets in separate keyrings...
So I'm not really opposed to this one.
PATCH v2 13/18 nvmet: make TCP sectype settable via configfs
In reply to Hannes ReineckeAdd a new configfs attribute 'addr_tsas' to make the TCP sectype settable via configfs.
--- drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
drivers/nvme/target/configfs.c +67 -0
@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port) port->disc_addr.tsas.rdma.cms =NVMF_RDMA_CMS_RDMA_CM ; } +static void nvmet_port_init_tsas_tcp (struct nvmet_port *port,int sectype) +{ + port->disc_addr.tsas.tcp.sectype = sectype; +} +static ssize_t nvmet_addr_trtype_store (struct config_item *item,const char *page,size_t count) {
@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct config_item *item, port->disc_addr.trtype = nvmet_transport[i].type;if (port->disc_addr.trtype ==NVMF_TRTYPE_RDMA )nvmet_port_init_tsas_rdma (port); +else if (port->disc_addr.trtype ==NVMF_TRTYPE_TCP ) +nvmet_port_init_tsas_tcp (port,NVMF_TCP_SECTYPE_NONE );return count; }CONFIGFS_ATTR (nvmet_, addr_trtype); +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = { + {NVMF_TCP_SECTYPE_NONE ,"none" }, + {NVMF_TCP_SECTYPE_TLS13 ,"tls1.3" }, +}; + +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = { + {NVMF_RDMA_QPTYPE_CONNECTED ,"connected" }, + {NVMF_RDMA_QPTYPE_DATAGRAM ,"datagram" }, +}; + +static ssize_t nvmet_addr_tsas_show (struct config_item *item, +char *page) +{ +struct nvmet_port *port =to_nvmet_port (item); +int i; + +if (port->disc_addr.trtype ==NVMF_TRTYPE_TCP ) { +for (i =0 ; i <ARRAY_SIZE (nvmet_addr_tsas_tcp); i++) { +if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type) +return sprintf (page,"%s\n" , nvmet_addr_tsas_tcp[i].name); + } +return sprintf (page,"reserved\n" ); + }else if (port->disc_addr.trtype ==NVMF_TRTYPE_RDMA ) { +for (i =0 ; i <ARRAY_SIZE (nvmet_addr_tsas_rdma); i++) { +if (port->disc_addr.tsas.rdma.qptype == nvmet_addr_tsas_rdma[i].type) +return sprintf (page,"%s\n" , nvmet_addr_tsas_rdma[i].name); + } +return sprintf (page,"reserved\n" ); + } +return sprintf (page,"not required\n" ); +} + +static ssize_t nvmet_addr_tsas_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_port *port =to_nvmet_port (item); +int i; + +if (nvmet_is_port_enabled (port, __func__)) +return -EACCES ; + +if (port->disc_addr.trtype !=NVMF_TRTYPE_TCP ) +return -EINVAL ; + +for (i =0 ; i <ARRAY_SIZE (nvmet_addr_tsas_tcp); i++) { +if (sysfs_streq (page, nvmet_addr_tsas_tcp[i].name)) + goto found; + } + +pr_err ("Invalid value '%s' for tsas\n" , page); +return -EINVAL ; + +found: +nvmet_port_init_tsas_tcp (port, nvmet_addr_tsas_tcp[i].type); +return count; +} + +CONFIGFS_ATTR (nvmet_, addr_tsas); +/* * Namespace structures & file operation functions below */
@@ -1741,6 +1807,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_traddr, &nvmet_attr_addr_trsvcid, &nvmet_attr_addr_trtype, + &nvmet_attr_addr_tsas, &nvmet_attr_param_inline_data_size, #ifdef CONFIG_BLK_DEV_INTEGRITY &nvmet_attr_param_pi_enable,
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
Quoted 48 lines
Add a new configfs attribute '
addr_tsas' to make the TCP sectype settable via configfs.Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/target/configfs.c | 67 ++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)drivers/nvme/target/configfs.c +19
-0@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port) port->disc_addr.tsas.rdma.cms =NVMF_RDMA_CMS_RDMA_CM ; } +static void nvmet_port_init_tsas_tcp (struct nvmet_port *port,int sectype) +{ + port->disc_addr.tsas.tcp.sectype = sectype; +} +static ssize_t nvmet_addr_trtype_store (struct config_item *item,const char *page,size_t count) {@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(struct config_item *item, port->disc_addr.trtype = nvmet_transport[i].type;if (port->disc_addr.trtype ==NVMF_TRTYPE_RDMA )nvmet_port_init_tsas_rdma (port); +else if (port->disc_addr.trtype ==NVMF_TRTYPE_TCP ) +nvmet_port_init_tsas_tcp (port,NVMF_TCP_SECTYPE_NONE );return count; }CONFIGFS_ATTR (nvmet_, addr_trtype); +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = { + {NVMF_TCP_SECTYPE_NONE ,"none" }, + {NVMF_TCP_SECTYPE_TLS13 ,"tls1.3" }, +}; + +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = { + {NVMF_RDMA_QPTYPE_CONNECTED ,"connected" }, + {NVMF_RDMA_QPTYPE_DATAGRAM ,"datagram" }, +}; + +static ssize_t nvmet_addr_tsas_show (struct config_item *item, +char *page)
drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct config_item *item, · line 361
Quoted 10 lines
drivers/nvme/target/configfs.c +10
-0+{ +struct nvmet_port *port =to_nvmet_port (item); +int i; + +if (port->disc_addr.trtype ==NVMF_TRTYPE_TCP ) { +for (i =0 ; i <ARRAY_SIZE (nvmet_addr_tsas_tcp); i++) { +if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type) +return sprintf (page,"%s\n" , nvmet_addr_tsas_tcp[i].name); + } +return sprintf (page,"reserved\n" );
What is reserved?
Quoted 48 lines
drivers/nvme/target/configfs.c +38
-0+ }else if (port->disc_addr.trtype ==NVMF_TRTYPE_RDMA ) { +for (i =0 ; i <ARRAY_SIZE (nvmet_addr_tsas_rdma); i++) { +if (port->disc_addr.tsas.rdma.qptype == nvmet_addr_tsas_rdma[i].type) +return sprintf (page,"%s\n" , nvmet_addr_tsas_rdma[i].name); + } +return sprintf (page,"reserved\n" ); + } +return sprintf (page,"not required\n" ); +} + +static ssize_t nvmet_addr_tsas_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_port *port =to_nvmet_port (item); +int i; + +if (nvmet_is_port_enabled (port, __func__)) +return -EACCES ; + +if (port->disc_addr.trtype !=NVMF_TRTYPE_TCP ) +return -EINVAL ; + +for (i =0 ; i <ARRAY_SIZE (nvmet_addr_tsas_tcp); i++) { +if (sysfs_streq (page, nvmet_addr_tsas_tcp[i].name)) + goto found; + } + +pr_err ("Invalid value '%s' for tsas\n" , page); +return -EINVAL ; + +found: +nvmet_port_init_tsas_tcp (port, nvmet_addr_tsas_tcp[i].type); +return count; +} + +CONFIGFS_ATTR (nvmet_, addr_tsas); +/* * Namespace structures & file operation functions below */ @@ -1741,6 +1807,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_traddr, &nvmet_attr_addr_trsvcid, &nvmet_attr_addr_trtype, + &nvmet_attr_addr_tsas, &nvmet_attr_param_inline_data_size, #ifdef CONFIG_BLK_DEV_INTEGRITY &nvmet_attr_param_pi_enable,
Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
Quoted 54 lines · Sagi Grimberg
Quoted 54 lines
Add a new configfs attribute '
addr_tsas' to make the TCP sectype settable via configfs.Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/configfs.c| 67 ++++++++++++++++++++++++++++++++++1 file changed, 67 insertions(+)drivers/nvme/target/configfs.c +5
-0@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port) port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM; } +static void nvmet_port_init_tsas_tcp (struct nvmet_port *port,int sectype) +{ + port->disc_addr.tsas.tcp.sectype = sectype; +} + static ssize_t nvmet_addr_trtype_store(struct config_item *item, const char *page, size_t count){
drivers/nvme/target/configfs.c +20
-0@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(structconfig_item *item, port->disc_addr.trtype = nvmet_transport[i].type; if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) nvmet_port_init_tsas_rdma(port); + else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) + nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE); return count; } CONFIGFS_ATTR(nvmet_, addr_trtype); +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = { + { NVMF_TCP_SECTYPE_NONE, "none" }, + { NVMF_TCP_SECTYPE_TLS13, "tls1.3" }, +}; + +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = { + { NVMF_RDMA_QPTYPE_CONNECTED, "connected" }, + { NVMF_RDMA_QPTYPE_DATAGRAM, "datagram" }, +}; + +static ssize_t nvmet_addr_tsas_show (struct config_item *item, + char *page) +{ + struct nvmet_port *port = to_nvmet_port(item); + int i; + + if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) { + for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {
Quoted 9 lines
drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct · line 362
Quoted 6 lines
drivers/nvme/target/configfs.c +4
-0+ if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type) + return sprintf(page, "%s\n", nvmet_addr_tsas_tcp[i].name); + } + return sprintf(page, "reserved\n");What is reserved?
All other values?
Quoted 43 lines
Quoted 43 lines
drivers/nvme/target/configfs.c +37
-0+ } else if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) { + for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_rdma); i++) { + if (port->disc_addr.tsas.rdma.qptype == nvmet_addr_tsas_rdma[i].type) + return sprintf(page, "%s\n", nvmet_addr_tsas_rdma[i].name); + } + return sprintf(page, "reserved\n"); + } + return sprintf(page, "not required\n"); +} + +static ssize_t nvmet_addr_tsas_store (struct config_item *item, + const char *page, size_t count) +{ + struct nvmet_port *port = to_nvmet_port(item); + int i; + + if (nvmet_is_port_enabled(port, __func__)) + return -EACCES; + + if (port->disc_addr.trtype != NVMF_TRTYPE_TCP) + return -EINVAL; + + for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) { + if (sysfs_streq(page, nvmet_addr_tsas_tcp[i].name)) + goto found; + } + + pr_err("Invalid value '%s' for tsas\n", page); + return -EINVAL; + +found: + nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type); + return count; +} + +CONFIGFS_ATTR (nvmet_, addr_tsas); +/*
* Namespace structures & file operation functions below */drivers/nvme/target/configfs.c +0
-0@@ -1741,6 +1807,7 @@ static struct configfs_attribute
Quoted 8 lines
Quoted 8 lines
drivers/nvme/target/configfs.c +1
-0*nvmet_port_attrs[] = { &nvmet_attr_addr_traddr, &nvmet_attr_addr_trsvcid, &nvmet_attr_addr_trtype, + &nvmet_attr_addr_tsas, &nvmet_attr_param_inline_data_size, #ifdef CONFIG_BLK_DEV_INTEGRITY
&nvmet_attr_param_pi_enable,
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerRe: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
Quoted 63 lines
Quoted 54 lines
Quoted 54 lines
Add a new configfs attribute '
addr_tsas' to make the TCP sectype settable via configfs.Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/configfs.c| 67 ++++++++++++++++++++++++++++++++++1 file changed, 67 insertions(+)drivers/nvme/target/configfs.c +5
-0@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port) port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM; } +static void nvmet_port_init_tsas_tcp (struct nvmet_port *port,int sectype) +{ + port->disc_addr.tsas.tcp.sectype = sectype; +} + static ssize_t nvmet_addr_trtype_store(struct config_item *item, const char *page, size_t count){
drivers/nvme/target/configfs.c +20
-0@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(structconfig_item *item, port->disc_addr.trtype = nvmet_transport[i].type; if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) nvmet_port_init_tsas_rdma(port); + else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) + nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE); return count; } CONFIGFS_ATTR(nvmet_, addr_trtype); +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = { + { NVMF_TCP_SECTYPE_NONE, "none" }, + { NVMF_TCP_SECTYPE_TLS13, "tls1.3" }, +}; + +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = { + { NVMF_RDMA_QPTYPE_CONNECTED, "connected" }, + { NVMF_RDMA_QPTYPE_DATAGRAM, "datagram" }, +}; + +static ssize_t nvmet_addr_tsas_show (struct config_item *item, + char *page) +{ + struct nvmet_port *port = to_nvmet_port(item); + int i; + + if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) { + for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) {Quoted 9 lines
drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct · line 362
Quoted 6 lines
drivers/nvme/target/configfs.c +4
-0+ if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type) + return sprintf(page, "%s\n", nvmet_addr_tsas_tcp[i].name); + } + return sprintf(page, "reserved\n");What is reserved?
Quoted 1 line
All other values?
Yes, you should just consolidate it outside the if-else instead of "not required" (which is a treq value iirc).
Re: [PATCH 13/18] nvmet: make TCP sectype settable via configfs
Quoted 64 lines · Sagi Grimberg
Quoted 64 lines
Quoted 64 lines
Quoted 51 lines
Add a new configfs attribute '
addr_tsas' to make the TCP sectype settable via configfs.Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/configfs.c| 67
++++++++++++++++++++++++++++++++++1 file changed, 67 insertions(+)drivers/nvme/target/configfs.c +5
-0@@ -303,6 +303,11 @@ static void nvmet_port_init_tsas_rdma(struct nvmet_port *port) port->disc_addr.tsas.rdma.cms = NVMF_RDMA_CMS_RDMA_CM; } +static void nvmet_port_init_tsas_tcp (struct nvmet_port *port,int sectype) +{ + port->disc_addr.tsas.tcp.sectype = sectype; +} + static ssize_t nvmet_addr_trtype_store(struct config_item *item, const char *page, size_t count){
drivers/nvme/target/configfs.c +16
-0@@ -325,11 +330,72 @@ static ssize_t nvmet_addr_trtype_store(structconfig_item *item, port->disc_addr.trtype = nvmet_transport[i].type; if (port->disc_addr.trtype == NVMF_TRTYPE_RDMA) nvmet_port_init_tsas_rdma(port); + else if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) + nvmet_port_init_tsas_tcp(port, NVMF_TCP_SECTYPE_NONE); return count; } CONFIGFS_ATTR(nvmet_, addr_trtype); +static const struct nvmet_type_name_map nvmet_addr_tsas_tcp[] = { + { NVMF_TCP_SECTYPE_NONE, "none" }, + { NVMF_TCP_SECTYPE_TLS13, "tls1.3" }, +}; + +static const struct nvmet_type_name_map nvmet_addr_tsas_rdma[] = { + { NVMF_RDMA_QPTYPE_CONNECTED, "connected" }, + { NVMF_RDMA_QPTYPE_DATAGRAM, "datagram" }, +}; + +static ssize_t nvmet_addr_tsas_show (struct config_item *item, + char *page) +{ + struct nvmet_port *port = to_nvmet_port(item);drivers/nvme/target/configfs.c static ssize_t nvmet_addr_trtype_store(struct · line 362
Quoted 10 lines
drivers/nvme/target/configfs.c +8
-0+ int i; + + if (port->disc_addr.trtype == NVMF_TRTYPE_TCP) { + for (i = 0; i < ARRAY_SIZE(nvmet_addr_tsas_tcp); i++) { + if (port->disc_addr.tsas.tcp.sectype == nvmet_addr_tsas_tcp[i].type) + return sprintf(page, "%s\n", nvmet_addr_tsas_tcp[i].name); + } + return sprintf(page, "reserved\n");What is reserved?
Quoted 5 lines
Quoted 1 line
All other values?
Yes, you should just consolidate it outside the if-else instead of "not required" (which is a treq value iirc).
Okay, will do.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman (HRB 36809, AG Nürnberg)
PATCH v2 14/18 nvmet-tcp: allocate socket file
In reply to Hannes ReineckeWhen using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.
--- drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 13 deletions(-)
drivers/nvme/target/tcp.c +38 -13
@@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {enum nvmet_tcp_queue_state {NVMET_TCP_Q_CONNECTING , +NVMET_TCP_Q_TLS_HANDSHAKE ,NVMET_TCP_Q_LIVE ,NVMET_TCP_Q_DISCONNECTING , };struct nvmet_tcp_queue {struct socket *sock; +struct file *sock_file;struct nvmet_tcp_port *port;struct work_struct io_work;struct nvmet_cq nvme_cq;
@@ -1406,6 +1408,19 @@ static void nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)write_unlock_bh (&sock->sk->sk_callback_lock); } +static void nvmet_tcp_close_sock (struct nvmet_tcp_queue *queue) +{ +if (queue->sock_file) { +fput (queue->sock_file); + queue->sock_file =NULL ; + queue->sock =NULL ; + }else { +WARN_ON (!queue->sock->ops); +sock_release (queue->sock); + queue->sock =NULL ; + } +} +static void nvmet_tcp_uninit_data_in_cmds (struct nvmet_tcp_queue *queue) {struct nvmet_tcp_cmd *cmd = queue->cmds;
@@ -1455,12 +1470,11 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)nvmet_sq_destroy (&queue->nvme_sq);cancel_work_sync (&queue->io_work);nvmet_tcp_free_cmd_data_in_buffers (queue); -sock_release (queue->sock); +nvmet_tcp_close_sock (queue);nvmet_tcp_free_cmds (queue);if (queue->hdr_digest || queue->data_digest)nvmet_tcp_free_crypto (queue);ida_free (&nvmet_tcp_queue_ida, queue->idx); - page =virt_to_head_page (queue->pf_cache.va);__page_frag_cache_drain (page, queue->pf_cache.pagecnt_bias);kfree (queue);
@@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)return ret; } -static int nvmet_tcp_alloc_queue (struct nvmet_tcp_port *port, +static void nvmet_tcp_alloc_queue (struct nvmet_tcp_port *port,struct socket *newsock) {struct nvmet_tcp_queue *queue;
@@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, queue =kzalloc (sizeof (*queue),GFP_KERNEL );if (!queue) -return -ENOMEM ; +return ;INIT_WORK (&queue->release_work, nvmet_tcp_release_queue_work);INIT_WORK (&queue->io_work, nvmet_tcp_io_work);
@@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, queue->port = port; queue->nr_cmds =0 ;spin_lock_init (&queue->state_lock); - queue->state =NVMET_TCP_Q_CONNECTING ; +if (queue->port->nport->disc_addr.tsas.tcp.sectype == +NVMF_TCP_SECTYPE_TLS13 ) + queue->state =NVMET_TCP_Q_TLS_HANDSHAKE ; +else + queue->state =NVMET_TCP_Q_CONNECTING ;INIT_LIST_HEAD (&queue->free_list);init_llist_head (&queue->resp_list);INIT_LIST_HEAD (&queue->resp_send_list); +if (queue->state ==NVMET_TCP_Q_TLS_HANDSHAKE ) { + queue->sock_file =sock_alloc_file (queue->sock,O_CLOEXEC ,NULL ); +if (IS_ERR (queue->sock_file)) { + ret =PTR_ERR (queue->sock_file); + queue->sock_file =NULL ; + goto out_free_queue; + } + } + queue->idx =ida_alloc (&nvmet_tcp_queue_ida,GFP_KERNEL );if (queue->idx <0 ) { ret = queue->idx; - goto out_free_queue; + goto out_sock; } ret =nvmet_tcp_alloc_cmd (queue, &queue->connect);
@@ -1628,7 +1655,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,if (ret) goto out_destroy_sq; -return 0 ; +return ; out_destroy_sq:mutex_lock (&nvmet_tcp_queue_mutex);list_del_init (&queue->queue_list);
@@ -1638,9 +1665,11 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,nvmet_tcp_free_cmd (&queue->connect); out_ida_remove:ida_free (&nvmet_tcp_queue_ida, queue->idx); +out_sock: +nvmet_tcp_close_sock (queue); out_free_queue:kfree (queue); -return ret; +pr_err ("failed to allocate queue" ); }static void nvmet_tcp_accept_work (struct work_struct *w)
@@ -1657,11 +1686,7 @@ static void nvmet_tcp_accept_work(struct work_struct *w)pr_warn ("failed to accept err=%d\n" , ret);return ; } - ret =nvmet_tcp_alloc_queue (port, newsock); -if (ret) { -pr_err ("failed to allocate queue\n" ); -sock_release (newsock); - } +nvmet_tcp_alloc_queue (port, newsock); } }
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 14/18] nvmet-tcp: allocate socket file
Quoted 91 lines · Hannes Reinecke
When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/target/tcp.c | 51 +++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 13 deletions(-)drivers/nvme/target/tcp.c +23
-5@@ -96,12 +96,14 @@ struct nvmet_tcp_cmd {enum nvmet_tcp_queue_state {NVMET_TCP_Q_CONNECTING , +NVMET_TCP_Q_TLS_HANDSHAKE ,NVMET_TCP_Q_LIVE ,NVMET_TCP_Q_DISCONNECTING , };struct nvmet_tcp_queue {struct socket *sock; +struct file *sock_file;struct nvmet_tcp_port *port;struct work_struct io_work;struct nvmet_cq nvme_cq;@@ -1406,6 +1408,19 @@ static void nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue)write_unlock_bh (&sock->sk->sk_callback_lock); } +static void nvmet_tcp_close_sock (struct nvmet_tcp_queue *queue) +{ +if (queue->sock_file) { +fput (queue->sock_file); + queue->sock_file =NULL ; + queue->sock =NULL ; + }else { +WARN_ON (!queue->sock->ops); +sock_release (queue->sock); + queue->sock =NULL ; + } +} +static void nvmet_tcp_uninit_data_in_cmds (struct nvmet_tcp_queue *queue) {struct nvmet_tcp_cmd *cmd = queue->cmds;@@ -1455,12 +1470,11 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)nvmet_sq_destroy (&queue->nvme_sq);cancel_work_sync (&queue->io_work);nvmet_tcp_free_cmd_data_in_buffers (queue); -sock_release (queue->sock); +nvmet_tcp_close_sock (queue);nvmet_tcp_free_cmds (queue);if (queue->hdr_digest || queue->data_digest)nvmet_tcp_free_crypto (queue);ida_free (&nvmet_tcp_queue_ida, queue->idx); - page =virt_to_head_page (queue->pf_cache.va);__page_frag_cache_drain (page, queue->pf_cache.pagecnt_bias);kfree (queue);@@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)return ret; } -static int nvmet_tcp_alloc_queue (struct nvmet_tcp_port *port, +static void nvmet_tcp_alloc_queue (struct nvmet_tcp_port *port,struct socket *newsock) {struct nvmet_tcp_queue *queue;@@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, queue =kzalloc (sizeof (*queue),GFP_KERNEL );if (!queue) -return -ENOMEM ; +return ;INIT_WORK (&queue->release_work, nvmet_tcp_release_queue_work);INIT_WORK (&queue->io_work, nvmet_tcp_io_work);@@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, queue->port = port; queue->nr_cmds =0 ;spin_lock_init (&queue->state_lock); - queue->state =NVMET_TCP_Q_CONNECTING ; +if (queue->port->nport->disc_addr.tsas.tcp.sectype == +NVMF_TCP_SECTYPE_TLS13 ) + queue->state =NVMET_TCP_Q_TLS_HANDSHAKE ; +else + queue->state =NVMET_TCP_Q_CONNECTING ;INIT_LIST_HEAD (&queue->free_list);init_llist_head (&queue->resp_list);
drivers/nvme/target/tcp.c static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, · line 1632
Quoted 10 lines
drivers/nvme/target/tcp.c +8
-0INIT_LIST_HEAD (&queue->resp_send_list); +if (queue->state ==NVMET_TCP_Q_TLS_HANDSHAKE ) { + queue->sock_file =sock_alloc_file (queue->sock,O_CLOEXEC ,NULL ); +if (IS_ERR (queue->sock_file)) { + ret =PTR_ERR (queue->sock_file); + queue->sock_file =NULL ; + goto out_free_queue; + } + }
Why not always allocate a sock_file? Like in the host?
Re: [PATCH 14/18] nvmet-tcp: allocate socket file
Quoted 97 lines · Sagi Grimberg
Quoted 94 lines · Hannes Reinecke
When using the TLS upcall we need to allocate a socket file such that the userspace daemon is able to use the socket.
Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/target/tcp.c| 51 +++++++++++++++++++++++++++++----------1 file changed, 38 insertions(+), 13 deletions(-)drivers/nvme/target/tcp.c +16
-2@@ -96,12 +96,14 @@ struct nvmet_tcp_cmd { enum nvmet_tcp_queue_state { NVMET_TCP_Q_CONNECTING, + NVMET_TCP_Q_TLS_HANDSHAKE, NVMET_TCP_Q_LIVE, NVMET_TCP_Q_DISCONNECTING, }; struct nvmet_tcp_queue { struct socket *sock; + struct file *sock_file; struct nvmet_tcp_port *port; struct work_struct io_work; struct nvmet_cq nvme_cq;@@ -1406,6 +1408,19 @@ static void nvmet_tcp_restore_socket_callbacks(struct nvmet_tcp_queue *queue) write_unlock_bh(&sock->sk->sk_callback_lock); } +static void nvmet_tcp_close_sock (struct nvmet_tcp_queue *queue) +{ + if (queue->sock_file) { + fput(queue->sock_file); + queue->sock_file = NULL; + queue->sock = NULL; + } else { + WARN_ON(!queue->sock->ops); + sock_release(queue->sock); + queue->sock = NULL; + } +} + static void nvmet_tcp_uninit_data_in_cmds(struct nvmet_tcp_queue *queue) { struct nvmet_tcp_cmd *cmd = queue->cmds;@@ -1455,12 +1470,11 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) nvmet_sq_destroy(&queue->nvme_sq); cancel_work_sync(&queue->io_work); nvmet_tcp_free_cmd_data_in_buffers(queue); - sock_release(queue->sock); + nvmet_tcp_close_sock(queue); nvmet_tcp_free_cmds(queue); if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - page = virt_to_head_page(queue->pf_cache.va); __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);kfree(queue);
drivers/nvme/target/tcp.c +9
-3@@ -1583,7 +1597,7 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) return ret; } -static int nvmet_tcp_alloc_queue (struct nvmet_tcp_port *port, +static void nvmet_tcp_alloc_queue (struct nvmet_tcp_port *port, struct socket *newsock) { struct nvmet_tcp_queue *queue;@@ -1591,7 +1605,7 @@ static int nvmet_tcp_alloc_queue(structnvmet_tcp_port *port, queue = kzalloc(sizeof(*queue), GFP_KERNEL); if (!queue) - return -ENOMEM; + return; INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work); INIT_WORK(&queue->io_work, nvmet_tcp_io_work);@@ -1599,15 +1613,28 @@ static int nvmet_tcp_alloc_queue(structnvmet_tcp_port *port, queue->port = port; queue->nr_cmds = 0; spin_lock_init(&queue->state_lock); - queue->state = NVMET_TCP_Q_CONNECTING; + if (queue->port->nport->disc_addr.tsas.tcp.sectype == + NVMF_TCP_SECTYPE_TLS13) + queue->state = NVMET_TCP_Q_TLS_HANDSHAKE; + else + queue->state = NVMET_TCP_Q_CONNECTING; INIT_LIST_HEAD(&queue->free_list); init_llist_head(&queue->resp_list); INIT_LIST_HEAD(&queue->resp_send_list); + if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) { + queue->sock_file = sock_alloc_file(queue->sock, O_CLOEXEC,NULL );
Quoted 8 lines
drivers/nvme/target/tcp.c static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, · line 1632
Quoted 6 lines
drivers/nvme/target/tcp.c +6
-0+ if (IS_ERR(queue->sock_file)) { + ret = PTR_ERR(queue->sock_file); + queue->sock_file = NULL; + goto out_free_queue; + } + }Why not always allocate a
sock_file? Like in the host?
Wasn't sure if you'd go with it.
But sure.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
PATCH v2 15/18 nvmet-tcp: enable TLS handshake upcall
In reply to Hannes ReineckeAdd functions to start the TLS handshake upcall when the TCP RSAS sectype is set to 'tls1.3'.
--- drivers/nvme/target/configfs.c | 32 +++++++- drivers/nvme/target/tcp.c | 135 ++++++++++++++++++++++++++++++++- 2 files changed, 163 insertions(+), 4 deletions(-)
drivers/nvme/target/configfs.c +30 -2
@@ -159,10 +159,12 @@ static const struct nvmet_type_name_map nvmet_addr_treq[] = { { NVMF_TREQ_NOT_REQUIRED, "not required" }, }; +#define NVMET_PORT_TREQ (port) ((port)->disc_addr.treq & NVME_TREQ_SECURE_CHANNEL_MASK) +static ssize_t nvmet_addr_treq_show (struct config_item *item,char *page) { -u8 treq =to_nvmet_port (item)->disc_addr.treq & -NVME_TREQ_SECURE_CHANNEL_MASK ; +struct nvmet_port *port =to_nvmet_port (item); +u8 treq =NVMET_PORT_TREQ (port);int i;for (i =0 ; i <ARRAY_SIZE (nvmet_addr_treq); i++) {
@@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct config_item *item,return -EINVAL ; found: +#ifdef CONFIG_NVME_TLS +if (port->disc_addr.trtype ==NVMF_TRTYPE_TCP ) { +if (port->disc_addr.tsas.tcp.sectype !=NVMF_TCP_SECTYPE_TLS13 ) { +pr_warn ("cannot change TREQ when TLS is not enabled\n" ); +return -EINVAL ; + }else if (nvmet_addr_treq[i].type ==NVMF_TREQ_NOT_SPECIFIED ) { +pr_warn ("cannot set TREQ to 'not specified' when TLS is enabled\n" ); +return -EINVAL ; + } + } +#endif treq |= nvmet_addr_treq[i].type; port->disc_addr.treq = treq;return count;
@@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item, const char *page, size_t count) {struct nvmet_port *port =to_nvmet_port (item); +u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK ;int i;if (nvmet_is_port_enabled (port, __func__))
@@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item, found:nvmet_port_init_tsas_tcp (port, nvmet_addr_tsas_tcp[i].type); +if (nvmet_addr_tsas_tcp[i].type ==NVMF_TCP_SECTYPE_TLS13 ) { +#ifdef CONFIG_NVME_TLS +if (NVMET_PORT_TREQ (port) ==NVMF_TREQ_NOT_SPECIFIED ) + treq |=NVMF_TREQ_REQUIRED ; +else + treq |=NVMET_PORT_TREQ (port); +#else +pr_err ("TLS not supported\n" ); +return -EINVAL ; +#endif + }else { +/* Set to 'not specified' if TLS is not enabled */ + treq |=NVMF_TREQ_NOT_SPECIFIED ; + }return count; }
drivers/nvme/target/tcp.c +133 -2
195 lines not on this page · raw message
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall
Quoted 3 lines
Add functions to start the TLS handshake upcall when the TCP RSAS sectype is set to 'tls1.3'.
TSAS
Quoted 12 lines
Signed-off-by Hannes Reincke <[email protected]>--- drivers/nvme/target/configfs.c | 32 +++++++- drivers/nvme/target/tcp.c | 135 ++++++++++++++++++++++++++++++++- 2 files changed, 163 insertions(+), 4 deletions(-)drivers/nvme/target/configfs.c +0
-0
drivers/nvme/target/configfs.c static const struct nvmet_type_name_map nvmet_addr_treq[] = { · line 162
Quoted 4 lines
drivers/nvme/target/configfs.c +1
-0@@ -159,10 +159,12 @@ static const struct nvmet_type_name_map nvmet_addr_treq[] = { { NVMF_TREQ_NOT_REQUIRED, "not required" }, }; +#define NVMET_PORT_TREQ (port) ((port)->disc_addr.treq & NVME_TREQ_SECURE_CHANNEL_MASK)
Can you make it a static inline?
Quoted 14 lines
drivers/nvme/target/configfs.c +3
-2+static ssize_t nvmet_addr_treq_show (struct config_item *item,char *page) { -u8 treq =to_nvmet_port (item)->disc_addr.treq & -NVME_TREQ_SECURE_CHANNEL_MASK ; +struct nvmet_port *port =to_nvmet_port (item); +u8 treq =NVMET_PORT_TREQ (port);int i;for (i =0 ; i <ARRAY_SIZE (nvmet_addr_treq); i++) {@@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(struct config_item *item,return -EINVAL ; found:
drivers/nvme/target/configfs.c static ssize_t nvmet_addr_treq_store(struct config_item *item, · line 207
Quoted 10 lines
drivers/nvme/target/configfs.c +10
-0+#ifdef CONFIG_NVME_TLS +if (port->disc_addr.trtype ==NVMF_TRTYPE_TCP ) { +if (port->disc_addr.tsas.tcp.sectype !=NVMF_TCP_SECTYPE_TLS13 ) { +pr_warn ("cannot change TREQ when TLS is not enabled\n" ); +return -EINVAL ; + }else if (nvmet_addr_treq[i].type ==NVMF_TREQ_NOT_SPECIFIED ) { +pr_warn ("cannot set TREQ to 'not specified' when TLS is enabled\n" ); +return -EINVAL ; + } + }
Is this code wrong if CONFIG_NVME_TLS is not enabled?
Quoted 13 lines
drivers/nvme/target/configfs.c +2
-0+#endif treq |= nvmet_addr_treq[i].type; port->disc_addr.treq = treq;return count;@@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item, const char *page, size_t count) {struct nvmet_port *port =to_nvmet_port (item); +u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK ;int i;if (nvmet_is_port_enabled (port, __func__))
drivers/nvme/target/configfs.c static ssize_t nvmet_addr_tsas_store(struct config_item *item, · line 409
Quoted 5 lines
drivers/nvme/target/configfs.c +2
-0@@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item, found:nvmet_port_init_tsas_tcp (port, nvmet_addr_tsas_tcp[i].type); +if (nvmet_addr_tsas_tcp[i].type ==NVMF_TCP_SECTYPE_TLS13 ) { +#ifdef CONFIG_NVME_TLS
Maybe in the start of the function just do:
if (!IS_ENABLED(CONFIG_NVME_TLS)) {
pr_err("TLS not supported\n");
return -EINVAL;
}Instead of incorporating it here.
Quoted 20 lines
drivers/nvme/target/configfs.c +12
-0+if (NVMET_PORT_TREQ (port) ==NVMF_TREQ_NOT_SPECIFIED ) + treq |=NVMF_TREQ_REQUIRED ; +else + treq |=NVMET_PORT_TREQ (port); +#else +pr_err ("TLS not supported\n" ); +return -EINVAL ; +#endif + }else { +/* Set to 'not specified' if TLS is not enabled */ + treq |=NVMF_TREQ_NOT_SPECIFIED ; + }return count; }drivers/nvme/target/tcp.c +0
-0
drivers/nvme/target/tcp.c line 15
Quoted 5 lines
drivers/nvme/target/tcp.c +2
-0@@ -11,6 +11,10 @@#include <linux/nvme-tcp.h> #include <net/sock.h> #include <net/tcp.h> +#ifdef CONFIG_NVME_TLS +#include <net/handshake.h>
Is net/handshake.h under an ifdef? If so, CONFIG_NVME_TLS should select it.
drivers/nvme/target/tcp.c line 16
Quoted 1 line
drivers/nvme/target/tcp.c +1
-0+#include <linux/nvme-keyring.h>
will this include not work if CONFIG_NVME_TLS not work? <linux/nvme-auth.h> is not under CONFIG_NVME_AUTH for example.
Quoted 22 lines
drivers/nvme/target/tcp.c +11
-0+#endif #include <linux/inet.h> #include <linux/llist.h> #include <crypto/hash.h> @@ -40,6 +44,16 @@ module_param(idle_poll_period_usecs, int, 0644);MODULE_PARM_DESC (idle_poll_period_usecs,"nvmet tcp io_work poll till idle time period in usecs" ); +#ifdef CONFIG_NVME_TLS +/* +* TLS handshake timeout +*/ +static int tls_handshake_timeout =30 ; +module_param (tls_handshake_timeout, int,0644 ); +MODULE_PARM_DESC (tls_handshake_timeout, +"nvme TLS handshake timeout in seconds (default 30)" ); +#endif +#define NVMET_TCP_RECV_BUDGET 8#define NVMET_TCP_SEND_BUDGET 8#define NVMET_TCP_IO_WORK_BUDGET 64
drivers/nvme/target/tcp.c struct nvmet_tcp_queue { · line 150
Quoted 7 lines
drivers/nvme/target/tcp.c +4
-0@@ -130,6 +144,10 @@ struct nvmet_tcp_queue {bool data_digest;struct ahash_request *snd_hash;struct ahash_request *rcv_hash; +#ifdef CONFIG_NVME_TLS +struct key *tls_psk; +struct delayed_work tls_handshake_work; +#endif
If these won't be under CONFIG_NVME_TLS will it save a lot of the other ifdefs in the code?
drivers/nvme/target/tcp.c static void nvmet_tcp_release_queue_work(struct work_struct *w) · line 1497
Quoted 10 lines
drivers/nvme/target/tcp.c +3
-0unsigned long poll_end;@@ -1474,6 +1492,10 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w)nvmet_tcp_free_cmds (queue);if (queue->hdr_digest || queue->data_digest)nvmet_tcp_free_crypto (queue); +#ifdef CONFIG_NVME_TLS +if (queue->tls_psk) +key_put (queue->tls_psk);
key_put is NULL safe.
Quoted 33 lines
drivers/nvme/target/tcp.c +17
-2+#endif ida_free (&nvmet_tcp_queue_ida, queue->idx); page =virt_to_head_page (queue->pf_cache.va);__page_frag_cache_drain (page, queue->pf_cache.pagecnt_bias);@@ -1488,8 +1510,12 @@ static void nvmet_tcp_data_ready(struct sock *sk)read_lock_bh (&sk->sk_callback_lock); queue = sk->sk_user_data; -if (likely (queue)) -queue_work_on (queue_cpu (queue), nvmet_tcp_wq, &queue->io_work); +if (queue->data_ready) + queue->data_ready (sk); +if (likely (queue) && + queue->state !=NVMET_TCP_Q_TLS_HANDSHAKE ) +queue_work_on (queue_cpu (queue), nvmet_tcp_wq, + &queue->io_work);read_unlock_bh (&sk->sk_callback_lock); }@@ -1597,6 +1623,89 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue)return ret; } +#ifdef CONFIG_NVME_TLS +static void nvmet_tcp_tls_queue_restart (struct nvmet_tcp_queue *queue) +{ +spin_lock (&queue->state_lock); +if (queue->state !=NVMET_TCP_Q_TLS_HANDSHAKE ) { +pr_warn ("queue %d: TLS handshake already completed\n" , + queue->idx); +spin_unlock (&queue->state_lock); +return ; + }
drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1645
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+ queue->state =NVMET_TCP_Q_CONNECTING ; +spin_unlock (&queue->state_lock); + +pr_debug ("queue %d: restarting queue after TLS handshake\n" , + queue->idx); +/* +* Set callbacks after handshake; TLS implementation +* might have changed the socket callbacks. +*/ +nvmet_tcp_set_queue_sock (queue);
Maybe fold it into the caller? The name is confusing anyways. The queue is not restarted, it is post-configured for lack of a better term.
drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1649
Quoted 4 lines
drivers/nvme/target/tcp.c +4
-0+} + +static void nvmet_tcp_tls_handshake_done (void *data,int status, +key_serial_t peerid)
pskid.
drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1661
Quoted 12 lines
drivers/nvme/target/tcp.c +12
-0+{ +struct nvmet_tcp_queue *queue = data; + +pr_debug ("queue %d: TLS handshake done, key %x, status %d\n" , + queue->idx, peerid, status); +if (!status) { +spin_lock (&queue->state_lock); + queue->tls_psk =key_lookup (peerid); +if (IS_ERR (queue->tls_psk)) { +pr_warn ("queue %d: TLS key %x not found\n" , + queue->idx, peerid); + queue->tls_psk =NULL ;
Here you let the timeout take care of it later?
Re: [PATCH 15/18] nvmet-tcp: enable TLS handshake upcall
Quoted 6 lines · Sagi Grimberg
Quoted 3 lines
Add functions to start the TLS handshake upcall when the TCP RSAS sectype is set to 'tls1.3'.
TSAS
Ok.
Quoted 7 lines
Quoted 7 lines
Signed-off-by Hannes Reincke <[email protected]>---
drivers/nvme/target/configfs.c| 32 +++++++- drivers/nvme/target/tcp.c | 135 ++++++++++++++++++++++++++++++++-2 files changed, 163 insertions(+), 4 deletions(-)
Quoted 13 lines
drivers/nvme/target/configfs.c static const struct nvmet_type_name_map · line 163
Quoted 10 lines
drivers/nvme/target/configfs.c +1
-0@@ -159,10 +159,12 @@ static const struct nvmet_type_name_map nvmet_addr_treq[] = { { NVMF_TREQ_NOT_REQUIRED, "not required" }, }; +#define NVMET_PORT_TREQ (port) ((port)->disc_addr.treq & NVME_TREQ_SECURE_CHANNEL_MASK)Can you make it a static inline?
Sure.
Quoted 19 lines
Quoted 19 lines
drivers/nvme/target/configfs.c +3
-2+ static ssize_t nvmet_addr_treq_show(struct config_item *item, char *page) { - u8 treq = to_nvmet_port(item)->disc_addr.treq & - NVME_TREQ_SECURE_CHANNEL_MASK; + struct nvmet_port *port = to_nvmet_port(item); + u8 treq = NVMET_PORT_TREQ(port); int i; for (i = 0; i < ARRAY_SIZE(nvmet_addr_treq); i++) {@@ -193,6 +195,17 @@ static ssize_t nvmet_addr_treq_store(structconfig_item *item, return -EINVAL;found:
+#ifdefCONFIG_NVME_TLS
+ if (port->disc_addr.trtype==NVMF_TRTYPE_TCP) {
+ if (port->disc_addr.tsas.tcp.sectype!=NVMF_TCP_SECTYPE_TLS13) {
+pr_warn("cannot change TREQ when TLS is not enabled\n"); + return-EINVAL;
Quoted 9 lines
Quoted 6 lines
+ } else if (nvmet_addr_treq[i].type ==
NVMF_TREQ_NOT_SPECIFIED) { +pr_warn("cannot set TREQ to 'not specified' when TLS is enabled\n");
+ return-EINVAL;
+ }
+ }Is this code wrong if
CONFIG_NVME_TLSis not enabled?
Strictly speaking, no; it just won't do anything except from having a different value in the discovery log page.
Quoted 21 lines
Quoted 14 lines
+#endif
treq |= nvmet_addr_treq[i].type;
port->disc_addr.treq= treq;
return count;drivers/nvme/target/configfs.c +1
-0@@ -373,6 +386,7 @@ static ssize_t nvmet_addr_tsas_store(structconfig_item *item, const char *page, size_t count) { struct nvmet_port *port = to_nvmet_port(item); + u8 treq = port->disc_addr.treq & ~NVME_TREQ_SECURE_CHANNEL_MASK; int i; if (nvmet_is_port_enabled(port, __func__))@@ -391,6 +405,20 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,Quoted 4 lines
found:
nvmet_port_init_tsas_tcp(port, nvmet_addr_tsas_tcp[i].type); + if (nvmet_addr_tsas_tcp[i].type ==NVMF_TCP_SECTYPE_TLS13) { +#ifdefCONFIG_NVME_TLSMaybe in the start of the function just do:
Quoted 7 lines
if (!IS_ENABLED(CONFIG_NVME_TLS)) {
pr_err("TLS not supported\n");
return-EINVAL;
}Instead of incorporating it here.
Ok.
Quoted 28 lines
Quoted 19 lines
+ if (
NVMET_PORT_TREQ(port)==NVMF_TREQ_NOT_SPECIFIED) + treq |=NVMF_TREQ_REQUIRED;
+ else
+ treq |=NVMET_PORT_TREQ(port);drivers/nvme/target/configfs.c +8
-0+#else + pr_err("TLS not supported\n"); + return -EINVAL; +#endif + } else { + /* Set to 'not specified' if TLS is not enabled */ + treq |= NVMF_TREQ_NOT_SPECIFIED; + }return count;
}drivers/nvme/target/tcp.c +0
-0drivers/nvme/target/tcp.c line 15
Quoted 5 lines
drivers/nvme/target/tcp.c +2
-0@@ -11,6 +11,10 @@ #include <linux/nvme-tcp.h> #include <net/sock.h> #include <net/tcp.h> +#ifdef CONFIG_NVME_TLS +#include <net/handshake.h> Is
net/handshake.hunder an ifdef? If so,CONFIG_NVME_TLSshould select it.
Quoted 5 lines
drivers/nvme/target/tcp.c line 16
Quoted 1 line
drivers/nvme/target/tcp.c +1
-0+#include <linux/nvme-keyring.h> will this include not work if
CONFIG_NVME_TLSnot work? <linux/nvme-auth.h> is not underCONFIG_NVME_AUTHfor example.
Hmm. Should. I can remove the ifdefs
Quoted 23 lines
Quoted 23 lines
drivers/nvme/target/tcp.c +11
-0+#endif #include <linux/inet.h> #include <linux/llist.h> #include <crypto/hash.h>@@ -40,6 +44,16 @@ module_param(idle_poll_period_usecs, int, 0644); MODULE_PARM_DESC(idle_poll_period_usecs, "nvmet tcp io_work poll till idle time period in usecs"); +#ifdef CONFIG_NVME_TLS +/* +* TLS handshake timeout +*/ +static int tls_handshake_timeout =30 ; +module_param (tls_handshake_timeout, int,0644 ); +MODULE_PARM_DESC (tls_handshake_timeout, + "nvme TLS handshake timeout in seconds (default 30)"); +#endif + #define NVMET_TCP_RECV_BUDGET 8 #define NVMET_TCP_SEND_BUDGET 8 #define NVMET_TCP_IO_WORK_BUDGET 64@@ -130,6 +144,10 @@ struct nvmet_tcp_queue { bool data_digest; struct ahash_request *snd_hash;
Quoted 9 lines
drivers/nvme/target/tcp.c struct nvmet_tcp_queue { · line 150
Quoted 5 lines
drivers/nvme/target/tcp.c +4
-0struct ahash_request *rcv_hash; +#ifdef CONFIG_NVME_TLS + struct key *tls_psk; + struct delayed_work tls_handshake_work; +#endif If these won't be under
CONFIG_NVME_TLSwill it save a lot of the other ifdefs in the code?
Wasn't sure if we always want to have it.
But if we do, sure, things will be easier.
Quoted 49 lines
drivers/nvme/target/tcp.c static void nvmet_tcp_release_queue_work(struct · line 1498
Quoted 9 lines
drivers/nvme/target/tcp.c +3
-0unsigned long poll_end;@@ -1474,6 +1492,10 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) nvmet_tcp_free_cmds(queue); if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); +#ifdef CONFIG_NVME_TLS + if (queue->tls_psk) + key_put(queue->tls_psk);
key_putis NULL safe.Quoted 37 lines
drivers/nvme/target/tcp.c +1
-0+#endif ida_free(&nvmet_tcp_queue_ida, queue->idx); page = virt_to_head_page(queue->pf_cache.va);
__page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias);drivers/nvme/target/tcp.c +22
-2@@ -1488,8 +1510,12 @@ static void nvmet_tcp_data_ready(struct sock *sk) read_lock_bh(&sk->sk_callback_lock); queue = sk->sk_user_data; - if (likely(queue)) - queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work); + if (queue->data_ready) + queue->data_ready(sk); + if (likely(queue) && + queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) + queue_work_on(queue_cpu(queue), nvmet_tcp_wq, + &queue->io_work); read_unlock_bh(&sk->sk_callback_lock); }@@ -1597,6 +1623,89 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) return ret; } +#ifdef CONFIG_NVME_TLS +static void nvmet_tcp_tls_queue_restart (struct nvmet_tcp_queue *queue) +{ + spin_lock(&queue->state_lock); + if (queue->state != NVMET_TCP_Q_TLS_HANDSHAKE) { + pr_warn("queue %d: TLS handshake already completed\n", + queue->idx); + spin_unlock(&queue->state_lock); + return; + } + queue->state = NVMET_TCP_Q_CONNECTING; + spin_unlock(&queue->state_lock); + + pr_debug("queue %d: restarting queue after TLS handshake\n", + queue->idx); + /*
Quoted 9 lines
drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1645
Quoted 4 lines
drivers/nvme/target/tcp.c +4
-0+* Set callbacks after handshake; TLS implementation +* might have changed the socket callbacks. +*/ + nvmet_tcp_set_queue_sock(queue);Maybe fold it into the caller? The name is confusing anyways. The queue is not restarted, it is post-configured for lack of a better term.
I see what I can do.
Quoted 7 lines
drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1649
Quoted 4 lines
drivers/nvme/target/tcp.c +4
-0+} + +static void nvmet_tcp_tls_handshake_done (void *data,int status, + key_serial_t peerid)pskid.
Quoted 14 lines
drivers/nvme/target/tcp.c static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) · line 1661
Quoted 12 lines
drivers/nvme/target/tcp.c +12
-0+{ + struct nvmet_tcp_queue *queue = data; + + pr_debug("queue %d: TLS handshake done, key %x, status %d\n", + queue->idx, peerid, status); + if (!status) { + spin_lock(&queue->state_lock); + queue->tls_psk = key_lookup(peerid); + if (IS_ERR(queue->tls_psk)) { + pr_warn("queue %d: TLS key %x not found\n", + queue->idx, peerid); + queue->tls_psk = NULL;Here you let the timeout take care of it later?
Well, this is a slightly odd case; we get a '0' status but failed to lookup the key.
_Technically_ we should be able to continue, but wasn't sure if I should.
But in the light of the key rotation discussion I probably should; new keys (and keyrings) might be provided at any time, so I might hit that case here.
Will be updating the code.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman (HRB 36809, AG Nürnberg)
PATCH v2 16/18 nvmet-tcp: rework sendpage for kTLS
In reply to Hannes ReineckekTLS ->sendpage() doesn't support the MSG_EOR flag, so blank it out if kTLS is enabled.
--- drivers/nvme/target/tcp.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/nvme/target/tcp.c +3 -3
@@ -650,7 +650,7 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd,if (!last_in_batch && cmd->queue->send_list_len) flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; -else +else if (!cmd->queue->tls_psk) flags |=MSG_EOR ; ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu),
@@ -678,7 +678,7 @@ static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch)if (!last_in_batch && cmd->queue->send_list_len) flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; -else +else if (!cmd->queue->tls_psk) flags |=MSG_EOR ; ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu),
@@ -708,7 +708,7 @@ static int nvmet_try_send_ddgst(struct nvmet_tcp_cmd *cmd, bool last_in_batch)if (!last_in_batch && cmd->queue->send_list_len) msg.msg_flags |=MSG_MORE ; -else +else if (!cmd->queue->tls_psk) msg.msg_flags |=MSG_EOR ; ret =kernel_sendmsg (queue->sock, &msg, &iov,1 , iov.iov_len);
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 16/18] nvmet-tcp: rework sendpage for kTLS
Quoted 2 lines
kTLS ->sendpage() doesn't support the
MSG_EORflag, so blank it out if kTLS is enabled.
I'd prefer we don't do this, but lets wait for the verdict from Jakub.
Quoted 37 lines
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/target/tcp.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)drivers/nvme/target/tcp.c +3
-3@@ -650,7 +650,7 @@ static int nvmet_try_send_response(struct nvmet_tcp_cmd *cmd,if (!last_in_batch && cmd->queue->send_list_len) flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; -else +else if (!cmd->queue->tls_psk) flags |=MSG_EOR ; ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->rsp_pdu),@@ -678,7 +678,7 @@ static int nvmet_try_send_r2t(struct nvmet_tcp_cmd *cmd, bool last_in_batch)if (!last_in_batch && cmd->queue->send_list_len) flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; -else +else if (!cmd->queue->tls_psk) flags |=MSG_EOR ; ret = kernel_sendpage(cmd->queue->sock, virt_to_page(cmd->r2t_pdu),@@ -708,7 +708,7 @@ static int nvmet_try_send_ddgst(struct nvmet_tcp_cmd *cmd, bool last_in_batch)if (!last_in_batch && cmd->queue->send_list_len) msg.msg_flags |=MSG_MORE ; -else +else if (!cmd->queue->tls_psk) msg.msg_flags |=MSG_EOR ; ret =kernel_sendmsg (queue->sock, &msg, &iov,1 , iov.iov_len);
PATCH v2 17/18 nvmet-tcp: control messages for recvmsg()
In reply to Hannes ReineckekTLS requires control messages for recvmsg() to relay any out-of-band TLS messages (eg TLS alerts) to the caller.
--- drivers/nvme/target/tcp.c | 71 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+)
drivers/nvme/target/tcp.c +71 -0
@@ -12,6 +12,7 @@#include <net/sock.h> #include <net/tcp.h> #ifdef CONFIG_NVME_TLS +#include <net/tls.h> #include <net/handshake.h> #include <linux/nvme-keyring.h> #endif
@@ -92,6 +93,7 @@ struct nvmet_tcp_cmd {u32 pdu_len;u32 pdu_recv;int sg_idx; +char recv_cbuf[CMSG_LEN (sizeof (char ))];struct msghdr recv_msg;struct bio_vec *iov;u32 flags;
@@ -1086,7 +1088,18 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;int len;struct kvec iov; +#ifdef CONFIG_NVME_TLS +char cbuf[CMSG_LEN (sizeof (char ))] = {}; +unsigned char ctype; +struct cmsghdr *cmsg; +struct msghdr msg = { + .msg_control = cbuf, + .msg_controllen =sizeof (cbuf), + .msg_flags =MSG_DONTWAIT + }; +#else struct msghdr msg = { .msg_flags =MSG_DONTWAIT }; +#endif recv: iov.iov_base = (void *)&queue->pdu + queue->offset;
@@ -1095,6 +1108,19 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue) iov.iov_len, msg.msg_flags);if (unlikely (len <0 ))return len; +#ifdef CONFIG_NVME_TLS + cmsg = (struct cmsghdr *)cbuf; +if (CMSG_OK (&msg, cmsg) && + cmsg->cmsg_level ==SOL_TLS && + cmsg->cmsg_type ==TLS_GET_RECORD_TYPE ) { + ctype = *((unsigned char *)CMSG_DATA (cmsg)); +if (ctype !=TLS_RECORD_TYPE_DATA ) { +pr_err ("queue %d unhandled TLS record %d\n" , + queue->idx, ctype); +return -ENOTCONN ; + } + } +#endif queue->offset += len; queue->left -= len;
@@ -1150,10 +1176,27 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)int ret;while (msg_data_left (&cmd->recv_msg)) { +#ifdef CONFIG_NVME_TLS +struct cmsghdr *cmsg; +unsigned char ctype; +#endif ret =sock_recvmsg (cmd->queue->sock, &cmd->recv_msg, cmd->recv_msg.msg_flags);if (ret <=0 )return ret; +#ifdef CONFIG_NVME_TLS + cmsg = (struct cmsghdr *)cmd->recv_cbuf; +if (CMSG_OK (&cmd->recv_msg, cmsg) && + cmsg->cmsg_level ==SOL_TLS && + cmsg->cmsg_type ==TLS_GET_RECORD_TYPE ) { + ctype = *((unsigned char *)CMSG_DATA (cmsg)); +if (ctype !=TLS_RECORD_TYPE_DATA ) { +pr_err ("queue %d unhandled TLS record %d\n" , + queue->idx, ctype); +return -ENOTCONN ; + } + } +#endif cmd->pdu_recv += ret; cmd->rbytes_done += ret;
@@ -1175,7 +1218,18 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue) {struct nvmet_tcp_cmd *cmd = queue->cmd;int ret; +#ifdef CONFIG_NVME_TLS +char cbuf[CMSG_LEN (sizeof (char ))] = {}; +unsigned char ctype; +struct cmsghdr *cmsg; +struct msghdr msg = { + .msg_control = cbuf, + .msg_controllen =sizeof (cbuf), + .msg_flags =MSG_DONTWAIT + }; +#else struct msghdr msg = { .msg_flags =MSG_DONTWAIT }; +#endif struct kvec iov = { .iov_base = (void *)&cmd->recv_ddgst + queue->offset, .iov_len = queue->left
@@ -1185,6 +1239,19 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue) iov.iov_len, msg.msg_flags);if (unlikely (ret <0 ))return ret; +#ifdef CONFIG_NVME_TLS + cmsg = (struct cmsghdr *)cbuf; +if (CMSG_OK (&msg, cmsg) && + cmsg->cmsg_level ==SOL_TLS && + cmsg->cmsg_type ==TLS_GET_RECORD_TYPE ) { + ctype = *((unsigned char *)CMSG_DATA (cmsg)); +if (ctype !=TLS_RECORD_TYPE_DATA ) { +pr_err ("queue %d unhandled TLS record %d\n" , + queue->idx, ctype); +return -ENOTCONN ; + } + } +#endif queue->offset += ret; queue->left -= ret;
@@ -1354,6 +1421,10 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,if (!c->r2t_pdu) goto out_free_data; +#ifdef CONFIG_NVME_TLS + c->recv_msg.msg_control = c->recv_cbuf; + c->recv_msg.msg_controllen =sizeof (c->recv_cbuf); +#endif c->recv_msg.msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL ;list_add_tail (&c->entry, &queue->free_list);
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 17/18] nvmet-tcp: control messages for recvmsg()
Quoted 29 lines · Hannes Reinecke
kTLS requires control messages for
recvmsg()to relay any out-of-band TLS messages (eg TLS alerts) to the caller.Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/target/tcp.c | 71 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+)drivers/nvme/target/tcp.c +2
-0@@ -12,6 +12,7 @@#include <net/sock.h> #include <net/tcp.h> #ifdef CONFIG_NVME_TLS +#include <net/tls.h> #include <net/handshake.h> #include <linux/nvme-keyring.h> #endif @@ -92,6 +93,7 @@ struct nvmet_tcp_cmd {u32 pdu_len;u32 pdu_recv;int sg_idx; +char recv_cbuf[CMSG_LEN (sizeof (char ))];struct msghdr recv_msg;struct bio_vec *iov;u32 flags;
drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue) · line 1091
Quoted 4 lines
drivers/nvme/target/tcp.c +1
-0@@ -1086,7 +1088,18 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;int len;struct kvec iov; +#ifdef CONFIG_NVME_TLS
Do you need this ifdef? The recv_cbuf in the cmd is not under and ifdef.
Quoted 21 lines
drivers/nvme/target/tcp.c +13
-0+char cbuf[CMSG_LEN (sizeof (char ))] = {}; +unsigned char ctype; +struct cmsghdr *cmsg; +struct msghdr msg = { + .msg_control = cbuf, + .msg_controllen =sizeof (cbuf), + .msg_flags =MSG_DONTWAIT + }; +#else struct msghdr msg = { .msg_flags =MSG_DONTWAIT }; +#endif recv: iov.iov_base = (void *)&queue->pdu + queue->offset;@@ -1095,6 +1108,19 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue) iov.iov_len, msg.msg_flags);if (unlikely (len <0 ))return len; +#ifdef CONFIG_NVME_TLS + cmsg = (struct cmsghdr *)cbuf; + if (CMSG_OK(&msg, cmsg) &&
drivers/nvme/target/tcp.c static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue) · line 1123
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+ cmsg->cmsg_level == SOL_TLS && + cmsg->cmsg_type == TLS_GET_RECORD_TYPE) { + ctype = *((unsigned char *)CMSG_DATA (cmsg)); +if (ctype !=TLS_RECORD_TYPE_DATA ) { +pr_err ("queue %d unhandled TLS record %d\n" , + queue->idx, ctype); +return -ENOTCONN ; + } + } +#endif
Can we consolidate this to a helper?
Something like:
static int nvmet_tcp_check_cmsg(struct nvmet_tcp_queue *queue, struct cmsghdr *cmsg)
{
if (!IS_ENABLED(CONFIG_NVME_TLS))
return 0;
if (CMSG_OK(&msg, cmsg) &&
cmsg->cmsg_level == SOL_TLS &&
cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {
ctype = *((unsigned char *)CMSG_DATA(cmsg));
if (ctype != TLS_RECORD_TYPE_DATA) {
pr_err("queue %d unhandled TLS record %d\n",
queue->idx, ctype);
return -ENOTCONN;
}
}
return 0;
}Quoted 81 lines
drivers/nvme/target/tcp.c +45
-0queue->offset += len; queue->left -= len;@@ -1150,10 +1176,27 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)int ret;while (msg_data_left (&cmd->recv_msg)) { +#ifdef CONFIG_NVME_TLS +struct cmsghdr *cmsg; +unsigned char ctype; +#endif ret =sock_recvmsg (cmd->queue->sock, &cmd->recv_msg, cmd->recv_msg.msg_flags);if (ret <=0 )return ret; +#ifdef CONFIG_NVME_TLS + cmsg = (struct cmsghdr *)cmd->recv_cbuf; +if (CMSG_OK (&cmd->recv_msg, cmsg) && + cmsg->cmsg_level ==SOL_TLS && + cmsg->cmsg_type ==TLS_GET_RECORD_TYPE ) { + ctype = *((unsigned char *)CMSG_DATA (cmsg)); +if (ctype !=TLS_RECORD_TYPE_DATA ) { +pr_err ("queue %d unhandled TLS record %d\n" , + queue->idx, ctype); +return -ENOTCONN ; + } + } +#endif cmd->pdu_recv += ret; cmd->rbytes_done += ret;@@ -1175,7 +1218,18 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue) {struct nvmet_tcp_cmd *cmd = queue->cmd;int ret; +#ifdef CONFIG_NVME_TLS +char cbuf[CMSG_LEN (sizeof (char ))] = {}; +unsigned char ctype; +struct cmsghdr *cmsg; +struct msghdr msg = { + .msg_control = cbuf, + .msg_controllen =sizeof (cbuf), + .msg_flags =MSG_DONTWAIT + }; +#else struct msghdr msg = { .msg_flags =MSG_DONTWAIT }; +#endif struct kvec iov = { .iov_base = (void *)&cmd->recv_ddgst + queue->offset, .iov_len = queue->left@@ -1185,6 +1239,19 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue) iov.iov_len, msg.msg_flags);if (unlikely (ret <0 ))return ret; +#ifdef CONFIG_NVME_TLS + cmsg = (struct cmsghdr *)cbuf; +if (CMSG_OK (&msg, cmsg) && + cmsg->cmsg_level ==SOL_TLS && + cmsg->cmsg_type ==TLS_GET_RECORD_TYPE ) { + ctype = *((unsigned char *)CMSG_DATA (cmsg)); +if (ctype !=TLS_RECORD_TYPE_DATA ) { +pr_err ("queue %d unhandled TLS record %d\n" , + queue->idx, ctype); +return -ENOTCONN ; + } + } +#endif queue->offset += ret; queue->left -= ret;@@ -1354,6 +1421,10 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,if (!c->r2t_pdu) goto out_free_data; +#ifdef CONFIG_NVME_TLS + c->recv_msg.msg_control = c->recv_cbuf; + c->recv_msg.msg_controllen =sizeof (c->recv_cbuf); +#endif c->recv_msg.msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL ;list_add_tail (&c->entry, &queue->free_list);
PATCH v2 18/18 nvmet-tcp: add configfs attribute 'param_keyring'
In reply to Hannes ReineckeAdd a configfs attribute to list and change the default keyring.
--- drivers/nvme/target/configfs.c | 68 +++++++++++++++++++++++++++++++++- drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/tcp.c | 3 +- 3 files changed, 70 insertions(+), 2 deletions(-)
drivers/nvme/target/configfs.c +67 -1
@@ -15,6 +15,10 @@#ifdef CONFIG_NVME_TARGET_AUTH #include <linux/nvme-auth.h> #endif +#include <linux/key.h> +#ifdef CONFIG_NVME_TLS +#include <linux/nvme-keyring.h> +#endif #include <crypto/hash.h> #include <crypto/kpp.h>
@@ -294,6 +298,48 @@ static ssize_t nvmet_param_pi_enable_store(struct config_item *item,CONFIGFS_ATTR (nvmet_, param_pi_enable);#endif +#ifdef CONFIG_NVME_TLS +static ssize_t nvmet_param_keyring_show (struct config_item *item, +char *page) +{ +struct nvmet_port *port =to_nvmet_port (item); + +if (!port->keyring) +return sprintf (page,"\n" ); +return snprintf (page,PAGE_SIZE ,"%s\n" , + port->keyring->description); +} + +static ssize_t nvmet_param_keyring_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_port *port =to_nvmet_port (item); +struct key *keyring; +unsigned int keyring_id; +int ret; + +if (nvmet_is_port_enabled (port, __func__)) +return -EACCES ; + + ret =kstrtou32 (page,0 , &keyring_id); +if (ret) { +pr_err ("Invalid keyring id '%s'\n" , page); +return ret; + } + keyring =key_lookup (keyring_id); +if (IS_ERR (keyring)) { +pr_err ("Invalid keyring '%08x'\n" , keyring_id); +return PTR_ERR (keyring); + } +if (port->keyring) +key_put (port->keyring); + port->keyring = keyring; +return count; +} + +CONFIGFS_ATTR (nvmet_, param_keyring); +#endif +static ssize_t nvmet_addr_trtype_show (struct config_item *item,char *page) {
@@ -404,9 +450,12 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,return -EINVAL ; found: -nvmet_port_init_tsas_tcp (port, nvmet_addr_tsas_tcp[i].type);if (nvmet_addr_tsas_tcp[i].type ==NVMF_TCP_SECTYPE_TLS13 ) {#ifdef CONFIG_NVME_TLS +if (!port->keyring) { +pr_err ("NVMe keyring not available, cannot enable TLS 1.3\n" ); +return -ENOTSUPP ; + }if (NVMET_PORT_TREQ (port) ==NVMF_TREQ_NOT_SPECIFIED ) treq |=NVMF_TREQ_REQUIRED ;else
@@ -419,6 +468,8 @@ static ssize_t nvmet_addr_tsas_store(struct config_item *item,/* Set to 'not specified' if TLS is not enabled */ treq |=NVMF_TREQ_NOT_SPECIFIED ; } + port->disc_addr.treq = treq; +nvmet_port_init_tsas_tcp (port, nvmet_addr_tsas_tcp[i].type);return count; }
@@ -1825,6 +1876,8 @@ static void nvmet_port_release(struct config_item *item)flush_workqueue (nvmet_wq);list_del (&port->global_entry); +if (port->keyring) +key_put (port->keyring);kfree (port->ana_state);kfree (port); }
@@ -1839,6 +1892,9 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_param_inline_data_size,#ifdef CONFIG_BLK_DEV_INTEGRITY &nvmet_attr_param_pi_enable, +#endif +#ifdef CONFIG_NVME_TLS + &nvmet_attr_param_keyring, #endif NULL, };
@@ -1857,6 +1913,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group, const char *name) {struct nvmet_port *port; +struct key *keyring =NULL ;u16 portid;u32 i;
@@ -1874,6 +1931,15 @@ static struct config_group *nvmet_ports_make(struct config_group *group,return ERR_PTR (-ENOMEM ); } +#ifdef CONFIG_NVME_TLS + keyring =key_lookup (nvme_keyring_id ()); +if (IS_ERR (keyring)) { +pr_warn ("NVMe keyring not available, disabling TLS\n" ); + keyring =NULL ; + } +#endif + port->keyring = keyring; +for (i =1 ; i <=NVMET_MAX_ANAGRPS ; i++) {if (i ==NVMET_DEFAULT_ANA_GRPID ) port->ana_state[1 ] =NVME_ANA_OPTIMIZED ;
drivers/nvme/target/nvmet.h +1 -0
@@ -158,6 +158,7 @@ struct nvmet_port {struct config_group ana_groups_group;struct nvmet_ana_group ana_default_group;enum nvme_ana_state *ana_state; +struct key *keyring;void *priv;bool enabled;int inline_data_size;
drivers/nvme/target/tcp.c +2 -1
@@ -8,6 +8,7 @@#include <linux/init.h> #include <linux/slab.h> #include <linux/err.h> +#include <linux/key.h> #include <linux/nvme-tcp.h> #include <net/sock.h> #include <net/tcp.h>
@@ -1763,7 +1764,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue) args.ta_sock = queue->sock; args.ta_done = nvmet_tcp_tls_handshake_done; args.ta_data = queue; - args.ta_keyring =nvme_keyring_id (); + args.ta_keyring =key_serial (queue->port->nport->keyring); args.ta_timeout_ms = tls_handshake_timeout *2 *1024 ; ret =tls_server_hello_psk (&args,GFP_KERNEL );
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerRe: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
Quoted 2 lines
Add a configfs attribute to list and change the default keyring.
is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?
Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
Quoted 6 lines · Sagi Grimberg
Quoted 2 lines
Add a configfs attribute to list and change the default keyring.
is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?
As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg Managing Directors: I. Totev, A. Myers, A. McDonald, M. B. Moerman (HRB 36809, AG Nürnberg)
Sagi Grimberg
maintainerRe: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Add a configfs attribute to list and change the default keyring.
is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?
As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.
I think we should move it to the subsystem in Linux.
Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
Quoted 11 lines · Sagi Grimberg
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Add a configfs attribute to list and change the default keyring.
is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?
As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.
I think we should move it to the subsystem in Linux.
Easier said than done.
We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.
So we'll have to keep it at the port level, I'm afraid.
Cheers,
Hannes
Sagi Grimberg
maintainerRe: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
Quoted 13 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Add a configfs attribute to list and change the default keyring.
is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?
As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.
I think we should move it to the subsystem in Linux.
Easier said than done.
Quoted 7 lines
We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.So we'll have to keep it at the port level, I'm afraid.
Question, why does nvmet needs to support user defined keyrings? I understood your argument for the host, but why should the target support it?
Re: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
Quoted 19 lines · Sagi Grimberg
Quoted 19 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Add a configfs attribute to list and change the default keyring.
is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?
As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.
I think we should move it to the subsystem in Linux.
Easier said than done.
We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.
Quoted 5 lines
Quoted 1 line
So we'll have to keep it at the port level, I'm afraid.
Question, why does nvmet needs to support user defined keyrings? I understood your argument for the host, but why should the target support it?
Hmm. Originally I implemented it to be similar to the host configuration, but from the implementation we don't _actually_ need it as we select the key from the entire key store. Having it is just a nice way of avoiding having to have a daemon configuration.
So if you insist we can drop it for the initial implementation.
Cheers,
Hannes
Sagi Grimberg
maintainerRe: [PATCH 18/18] nvmet-tcp: add configfs attribute 'param_keyring'
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Add a configfs attribute to list and change the default keyring.
is the keyring an attribute of a port? or a subsystem? Why the choice of making it per port?
As per discussion at fmds each port might want to use different credentials. We can move it to subsystems if we decide we don't want to support that use-case.
I think we should move it to the subsystem in Linux.
Easier said than done.
We can only fetch the subsystem once we have the NQN, which we will know after the connect command.
But out of necessity the connect command will only be transferred after TLS handshake completed, so there is not subsystem NQN allowing us to figure out the correct subsystem.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
So we'll have to keep it at the port level, I'm afraid.
Question, why does nvmet needs to support user defined keyrings? I understood your argument for the host, but why should the target support it?
Hmm. Originally I implemented it to be similar to the host configuration, but from the implementation we don't _actually_ need it as we select the key from the entire key store.
Thanks for confirming.
Quoted 1 line
Having it is just a nice way of avoiding having to have a daemon configuration.
I'm sorry, but I don't understand this statement. Can you clarify?
Quoted 1 line
So if you insist we can drop it for the initial implementation.
If there is a use-case, we can add it. But I'm trying to understand what is it.