nvme: In-band authentication support

Hi all,

recent updates to the NVMe spec have added definitions for in-band authentication, and seeing that it provides some real benefit especially for NVMe-TCP here's an attempt to implement it.

Thanks to Nicolai Stange the crypto DH framework has been upgraded to provide us with a FFDHE implementation; I've updated the patchset to use the ephemeral key generation provided there.

Note that this is just for in-band authentication. Secure concatenation (ie starting TLS with the negotiated parameters) requires a TLS handshake, which the in-kernel TLS implementation does not provide. This is being worked on with a different patchset which is still WIP.

The nvme-cli support has already been merged; please use the latest nvme-cli git repository to build the most recent version.

A copy of this patchset can be found at
git://git.kernel.org/pub/scm/linux/kernel/git/hare/scsi-devel
branch auth.v14

The patchset is being cut against v5.18.

As usual, comments and reviews are welcome.

Changes to v13:
- Fixup sysfs attribute visibility (reported by Chaitanya)
- Fixup SUCCESS1 handling
- Re-introduce crypto_has_shash() and crypto_has_kpp() helper
- Unse 'authenticated' flag on failure

Changes to v12:
- Replaces crypto_has_shash() and crypto_has_kpp() helper with CONFIG_ checks
- Fixed kbuild robot warning in pr_fmt()

Changes to v11:
- Fixup type for FAILURE2 message (Prashant Nayak)
- Do not sent SUCCESS2 if bi-directional authentication is not requested (Martin George)

Changes to v10:
- Fixup error return value when authentication failed

Changes to v9:
- Include review from Chaitanya
- Use sparse array for dhgroup and hash lookup
- Common function for auth_send and auth_receive

Changes to v8:
- Rebased to Nicolais crypto DH rework
- Fixed oops on non-fabrics devices

Changes to v7:
- Space out hash list and dhgroup list in nvme negotiate data to be conformant with the spec
- Update sequence number handling to start with a random value and ignore '0' as mandated by the spec
- Update nvme_auth_generate_key to return the key as suggested by Sagi
- Add nvmet_parse_fabrics_io_cmd() as suggested by hch

Changes to v6:
- Use 'u8' for DH group id and hash id
- Use 'struct nvme_dhchap_key'
- Rename variables to drop 'DHCHAP'
- Include reviews from Chaitanya

Changes to v5:
- Unify nvme_auth_generate_key()
- Unify nvme_auth_extract_key()
- Fixed bug where re-authentication with wrong controller key would not fail
- Include reviews from Sagi

Changes to v4:
- Validate against blktest suite
- Fixup base64 decoding
- Transform secret with correct hmac algorithm

Changes to v3:
- Renamed parameter to 'dhchap_ctrl_key'
- Fixed bi-directional authentication
- Included reviews from Sagi
- Fixed base64 algorithm for transport encoding

Changes to v2:
- Dropped non-standard algorithms
- Reworked base64 based on fs/crypto/fname.c
- Fixup crash with no keys

Changes to the original submission:
- Included reviews from Vladislav
- Included reviews from Sagi
- Implemented re-authentication support
- Fixed up key handling

Hannes Reinecke (11):
crypto: add crypto_has_shash()
crypto: add crypto_has_kpp()
lib/base64: RFC4648-compliant base64 encoding
nvme: add definitions for NVMe In-Band authentication
nvme-fabrics: decode 'authentication required' connect error
nvme: Implement In-Band authentication
nvme-auth: Diffie-Hellman key exchange support
nvmet: parse fabrics commands on io queues
nvmet: Implement basic In-Band Authentication
nvmet-auth: Diffie-Hellman key exchange support
nvmet-auth: expire authentication sessions

 crypto/kpp.c                           |    6 +
 crypto/shash.c                         |    6 +
 drivers/nvme/Kconfig                   |    1 +
 drivers/nvme/Makefile                  |    1 +
 drivers/nvme/common/Kconfig            |    4 +
 drivers/nvme/common/Makefile           |    7 +
 drivers/nvme/common/auth.c             |  481 ++++++++++++
 drivers/nvme/host/Kconfig              |   13 +
 drivers/nvme/host/Makefile             |    1 +
 drivers/nvme/host/auth.c               | 1000 ++++++++++++++++++++++++
 drivers/nvme/host/core.c               |  143 +++-
 drivers/nvme/host/fabrics.c            |   83 +-
 drivers/nvme/host/fabrics.h            |    7 +
 drivers/nvme/host/nvme.h               |   30 +
 drivers/nvme/host/rdma.c               |    1 +
 drivers/nvme/host/tcp.c                |    1 +
 drivers/nvme/host/trace.c              |   32 +
 drivers/nvme/target/Kconfig            |   13 +
 drivers/nvme/target/Makefile           |    1 +
 drivers/nvme/target/admin-cmd.c        |    4 +-
 drivers/nvme/target/auth.c             |  525 +++++++++++++
 drivers/nvme/target/configfs.c         |  138 +++-
 drivers/nvme/target/core.c             |   15 +
 drivers/nvme/target/fabrics-cmd-auth.c |  542 +++++++++++++
 drivers/nvme/target/fabrics-cmd.c      |   55 +-
 drivers/nvme/target/nvmet.h            |   75 +-
 include/crypto/hash.h                  |    2 +
 include/crypto/kpp.h                   |    2 +
 include/linux/base64.h                 |   16 +
 include/linux/nvme-auth.h              |   41 +
 include/linux/nvme.h                   |  204 ++++-
 lib/Makefile                           |    2 +-
 lib/base64.c                           |  103 +++
 33 files changed, 3540 insertions(+), 15 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 drivers/nvme/target/auth.c
 create mode 100644 drivers/nvme/target/fabrics-cmd-auth.c
 create mode 100644 include/linux/base64.h
 create mode 100644 include/linux/nvme-auth.h
 create mode 100644 lib/base64.c

2.29.2

Add helper function to determine if a given synchronous hash is supported.

---
 crypto/shash.c        | 6 ++++++
 include/crypto/hash.h | 2 ++
 2 files changed, 8 insertions(+)

index 0a0a50cb694f..4c88e63b3350 100644
@@ -521,6 +521,12 @@ struct crypto_shash *crypto_alloc_shash(const char *alg_name, u32 type,
 }
 EXPORT_SYMBOL_GPL(crypto_alloc_shash);
 
+int crypto_has_shash(const char *alg_name, u32 type, u32 mask)
+{
+	return crypto_type_has_alg(alg_name, &crypto_shash_type, type, mask);
+}
+EXPORT_SYMBOL_GPL(crypto_has_shash);
+
 static int shash_prepare_alg(struct shash_alg *alg)
 {
 	struct crypto_alg *base = &alg->base;

index f140e4643949..f5841992dc9b 100644
@@ -718,6 +718,8 @@ static inline void ahash_request_set_crypt(struct ahash_request *req,
 struct crypto_shash *crypto_alloc_shash(const char *alg_name, u32 type,
 					u32 mask);
 
+int crypto_has_shash(const char *alg_name, u32 type, u32 mask);
+
 static inline struct crypto_tfm *crypto_shash_tfm(struct crypto_shash *tfm)
 {
 	return &tfm->base;

2.29.2

Add helper function to determine if a given key-agreement protocol primitive is supported.

---
 crypto/kpp.c         | 6 ++++++
 include/crypto/kpp.h | 2 ++
 2 files changed, 8 insertions(+)

index 7aa6ba4b60a4..678e871ce418 100644
@@ -104,6 +104,12 @@ int crypto_grab_kpp(struct crypto_kpp_spawn *spawn,
 }
 EXPORT_SYMBOL_GPL(crypto_grab_kpp);
 
+int crypto_has_kpp(const char *alg_name, u32 type, u32 mask)
+{
+	return crypto_type_has_alg(alg_name, &crypto_kpp_type, type, mask);
+}
+EXPORT_SYMBOL_GPL(crypto_has_kpp);
+
 static void kpp_prepare_alg(struct kpp_alg *alg)
 {
 	struct crypto_alg *base = &alg->base;

index cccceadc164b..24d01e9877c1 100644
@@ -104,6 +104,8 @@ struct kpp_alg {
  */
 struct crypto_kpp *crypto_alloc_kpp(const char *alg_name, u32 type, u32 mask);
 
+int crypto_has_kpp(const char *alg_name, u32 type, u32 mask);
+
 static inline struct crypto_tfm *crypto_kpp_tfm(struct crypto_kpp *tfm)
 {
 	return &tfm->base;

2.29.2

Add RFC4648-compliant base64 encoding and decoding routines, based on the base64url encoding in fs/crypto/fname.c.

---
 include/linux/base64.h |  16 +++++++
 lib/Makefile           |   2 +-
 lib/base64.c           | 103 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/base64.h
 create mode 100644 lib/base64.c

new file mode 100644
index 000000000000..660d4cb1ef31
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * base64 encoding, lifted from fs/crypto/fname.c.
+ */
+
+#ifndef _LINUX_BASE64_H
+#define _LINUX_BASE64_H
+
+#include <linux/types.h>
+
+#define BASE64_CHARS(nbytes)   DIV_ROUND_UP((nbytes) * 4, 3)
+
+int base64_encode(const u8 *src, int len, char *dst);
+int base64_decode(const char *src, int len, u8 *dst);
+
+#endif /* _LINUX_BASE64_H */

index 6b9ffc1bd1ee..81a8c8230148 100644
@@ -46,7 +46,7 @@ obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \
 	 bust_spinlocks.o kasprintf.o bitmap.o scatterlist.o \
 	 list_sort.o uuid.o iov_iter.o clz_ctz.o \
 	 bsearch.o find_bit.o llist.o memweight.o kfifo.o \
-	 percpu-refcount.o rhashtable.o \
+	 percpu-refcount.o rhashtable.o base64.o \
 	 once.o refcount.o usercopy.o errseq.o bucket_locks.o \
 	 generic-radix-tree.o
 obj-$(CONFIG_STRING_SELFTEST) += test_string.o

2.29.2

Add new definitions for NVMe In-band authentication as defined in the NVMe Base Specification v2.0.

---
 include/linux/nvme.h | 204 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 203 insertions(+), 1 deletion(-)

index f626a445d1a8..dc3c7fcaa730 100644
@@ -19,6 +19,7 @@
 #define NVMF_TRSVCID_SIZE	32
 #define NVMF_TRADDR_SIZE	256
 #define NVMF_TSAS_SIZE		256
+#define NVMF_AUTH_HASH_LEN	64
 
 #define NVME_DISC_SUBSYS_NAME	"nqn.2014-08.org.nvmexpress.discovery"
 

@@ -1328,6 +1329,8 @@ enum nvmf_capsule_command {
 	nvme_fabrics_type_property_set	= 0x00,
 	nvme_fabrics_type_connect	= 0x01,
 	nvme_fabrics_type_property_get	= 0x04,
+	nvme_fabrics_type_auth_send	= 0x05,
+	nvme_fabrics_type_auth_receive	= 0x06,
 };
 
 #define nvme_fabrics_type_name(type)   { type, #type }

@@ -1335,7 +1338,9 @@ enum nvmf_capsule_command {
 	__print_symbolic(type,						\
 		nvme_fabrics_type_name(nvme_fabrics_type_property_set),	\
 		nvme_fabrics_type_name(nvme_fabrics_type_connect),	\
-		nvme_fabrics_type_name(nvme_fabrics_type_property_get))
+		nvme_fabrics_type_name(nvme_fabrics_type_property_get), \
+		nvme_fabrics_type_name(nvme_fabrics_type_auth_send),	\
+		nvme_fabrics_type_name(nvme_fabrics_type_auth_receive))
 
 /*
  * If not fabrics command, fctype will be ignored.

@@ -1465,6 +1470,200 @@ struct nvmf_property_get_command {
 	__u8		resv4[16];
 };
 
+struct nvmf_auth_common_command {
+	__u8		opcode;
+	__u8		resv1;
+	__u16		command_id;
+	__u8		fctype;
+	__u8		resv2[19];
+	union nvme_data_ptr dptr;
+	__u8		resv3;
+	__u8		spsp0;
+	__u8		spsp1;
+	__u8		secp;
+	__le32		al_tl;
+	__u8		resv4[16];
+};
+
+struct nvmf_auth_send_command {
+	__u8		opcode;
+	__u8		resv1;
+	__u16		command_id;
+	__u8		fctype;
+	__u8		resv2[19];
+	union nvme_data_ptr dptr;
+	__u8		resv3;
+	__u8		spsp0;
+	__u8		spsp1;
+	__u8		secp;
+	__le32		tl;
+	__u8		resv4[16];
+};
+
+struct nvmf_auth_receive_command {
+	__u8		opcode;
+	__u8		resv1;
+	__u16		command_id;
+	__u8		fctype;
+	__u8		resv2[19];
+	union nvme_data_ptr dptr;
+	__u8		resv3;
+	__u8		spsp0;
+	__u8		spsp1;
+	__u8		secp;
+	__le32		al;
+	__u8		resv4[16];
+};
+
+/* Value for secp */
+enum {
+	NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER	= 0xe9,
+};
+
+/* Defined value for auth_type */
+enum {
+	NVME_AUTH_COMMON_MESSAGES	= 0x00,
+	NVME_AUTH_DHCHAP_MESSAGES	= 0x01,
+};
+
+/* Defined messages for auth_id */
+enum {
+	NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE	= 0x00,
+	NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE	= 0x01,
+	NVME_AUTH_DHCHAP_MESSAGE_REPLY		= 0x02,
+	NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1	= 0x03,
+	NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2	= 0x04,
+	NVME_AUTH_DHCHAP_MESSAGE_FAILURE2	= 0xf0,
+	NVME_AUTH_DHCHAP_MESSAGE_FAILURE1	= 0xf1,
+};
+
+struct nvmf_auth_dhchap_protocol_descriptor {
+	__u8		authid;
+	__u8		rsvd;
+	__u8		halen;
+	__u8		dhlen;
+	__u8		idlist[60];
+};
+
+enum {
+	NVME_AUTH_DHCHAP_AUTH_ID	= 0x01,
+};
+
+/* Defined hash functions for DH-HMAC-CHAP authentication */
+enum {
+	NVME_AUTH_HASH_SHA256	= 0x01,
+	NVME_AUTH_HASH_SHA384	= 0x02,
+	NVME_AUTH_HASH_SHA512	= 0x03,
+	NVME_AUTH_HASH_INVALID	= 0xff,
+};
+
+/* Defined Diffie-Hellman group identifiers for DH-HMAC-CHAP authentication */
+enum {
+	NVME_AUTH_DHGROUP_NULL		= 0x00,
+	NVME_AUTH_DHGROUP_2048		= 0x01,
+	NVME_AUTH_DHGROUP_3072		= 0x02,
+	NVME_AUTH_DHGROUP_4096		= 0x03,
+	NVME_AUTH_DHGROUP_6144		= 0x04,
+	NVME_AUTH_DHGROUP_8192		= 0x05,
+	NVME_AUTH_DHGROUP_INVALID	= 0xff,
+};
+
+union nvmf_auth_protocol {
+	struct nvmf_auth_dhchap_protocol_descriptor dhchap;
+};
+
+struct nvmf_auth_dhchap_negotiate_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd;
+	__le16		t_id;
+	__u8		sc_c;
+	__u8		napd;
+	union nvmf_auth_protocol auth_protocol[];
+};
+
+struct nvmf_auth_dhchap_challenge_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__u16		rsvd1;
+	__le16		t_id;
+	__u8		hl;
+	__u8		rsvd2;
+	__u8		hashid;
+	__u8		dhgid;
+	__le16		dhvlen;
+	__le32		seqnum;
+	/* 'hl' bytes of challenge value */
+	__u8		cval[];
+	/* followed by 'dhvlen' bytes of DH value */
+};
+
+struct nvmf_auth_dhchap_reply_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		hl;
+	__u8		rsvd2;
+	__u8		cvalid;
+	__u8		rsvd3;
+	__le16		dhvlen;
+	__le32		seqnum;
+	/* 'hl' bytes of response data */
+	__u8		rval[];
+	/* followed by 'hl' bytes of Challenge value */
+	/* followed by 'dhvlen' bytes of DH value */
+};
+
+enum {
+	NVME_AUTH_DHCHAP_RESPONSE_VALID	= (1 << 0),
+};
+
+struct nvmf_auth_dhchap_success1_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		hl;
+	__u8		rsvd2;
+	__u8		rvalid;
+	__u8		rsvd3[7];
+	/* 'hl' bytes of response value if 'rvalid' is set */
+	__u8		rval[];
+};
+
+struct nvmf_auth_dhchap_success2_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		rsvd2[10];
+};
+
+struct nvmf_auth_dhchap_failure_data {
+	__u8		auth_type;
+	__u8		auth_id;
+	__le16		rsvd1;
+	__le16		t_id;
+	__u8		rescode;
+	__u8		rescode_exp;
+};
+
+enum {
+	NVME_AUTH_DHCHAP_FAILURE_REASON_FAILED	= 0x01,
+};
+
+enum {
+	NVME_AUTH_DHCHAP_FAILURE_FAILED			= 0x01,
+	NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE		= 0x02,
+	NVME_AUTH_DHCHAP_FAILURE_CONCAT_MISMATCH	= 0x03,
+	NVME_AUTH_DHCHAP_FAILURE_HASH_UNUSABLE		= 0x04,
+	NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE	= 0x05,
+	NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD	= 0x06,
+	NVME_AUTH_DHCHAP_FAILURE_INCORRECT_MESSAGE	= 0x07,
+};
+
+
 struct nvme_dbbuf {
 	__u8			opcode;
 	__u8			flags;

@@ -1508,6 +1707,9 @@ struct nvme_command {
 		struct nvmf_connect_command connect;
 		struct nvmf_property_set_command prop_set;
 		struct nvmf_property_get_command prop_get;
+		struct nvmf_auth_common_command auth_common;
+		struct nvmf_auth_send_command auth_send;
+		struct nvmf_auth_receive_command auth_receive;
 		struct nvme_dbbuf dbbuf;
 		struct nvme_directive_cmd directive;
 	};

2.29.2

The 'connect' command might fail with NVME_SC_AUTH_REQUIRED, so we should be decoding this error, too.

---
 drivers/nvme/host/fabrics.c | 4 ++++
 1 file changed, 4 insertions(+)

index ee79a6d639b4..953e3076aab1 100644
@@ -331,6 +331,10 @@ static void nvmf_log_connect_error(struct nvme_ctrl *ctrl,
 		dev_err(ctrl->device,
 			"Connect command failed: host path error\n");
 		break;
+	case NVME_SC_AUTH_REQUIRED:
+		dev_err(ctrl->device,
+			"Connect command failed: authentication required\n");
+		break;
 	default:
 		dev_err(ctrl->device,
 			"Connect command failed, error wo/DNR bit: %d\n",

2.29.2

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006. This patch adds two new fabric options 'dhchap_secret' to specify the pre-shared key (in ASCII respresentation according to NVMe 2.0 section 8.13.5.8 'Secret representation') and 'dhchap_ctrl_secret' to specify the pre-shared controller key for bi-directional authentication of both the host and the controller.
Re-authentication can be triggered by writing the PSK into the new controller sysfs attribute 'dhchap_secret' or 'dhchap_ctrl_secret'.

---
 drivers/nvme/Kconfig         |   1 +
 drivers/nvme/Makefile        |   1 +
 drivers/nvme/common/Kconfig  |   4 +
 drivers/nvme/common/Makefile |   7 +
 drivers/nvme/common/auth.c   | 328 ++++++++++++++
 drivers/nvme/host/Kconfig    |  12 +
 drivers/nvme/host/Makefile   |   1 +
 drivers/nvme/host/auth.c     | 813 +++++++++++++++++++++++++++++++++++
 drivers/nvme/host/core.c     | 143 +++++-
 drivers/nvme/host/fabrics.c  |  79 +++-
 drivers/nvme/host/fabrics.h  |   7 +
 drivers/nvme/host/nvme.h     |  30 ++
 drivers/nvme/host/rdma.c     |   1 +
 drivers/nvme/host/tcp.c      |   1 +
 drivers/nvme/host/trace.c    |  32 ++
 include/linux/nvme-auth.h    |  33 ++
 16 files changed, 1486 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/common/Kconfig
 create mode 100644 drivers/nvme/common/Makefile
 create mode 100644 drivers/nvme/common/auth.c
 create mode 100644 drivers/nvme/host/auth.c
 create mode 100644 include/linux/nvme-auth.h

index 87ae409a32b9..656e46d938da 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 menu "NVME Support"
 
+source "drivers/nvme/common/Kconfig"
 source "drivers/nvme/host/Kconfig"
 source "drivers/nvme/target/Kconfig"
 

index fb42c44609a8..eedca8c72098 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
+obj-$(CONFIG_NVME_COMMON)		+= common/
 obj-y		+= host/
 obj-y		+= target/

new file mode 100644
index 000000000000..4514f44362dd
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config NVME_COMMON
+       tristate

new file mode 100644
index 000000000000..720c625b8a52
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ccflags-y			+= -I$(src)
+
+obj-$(CONFIG_NVME_COMMON)	+= nvme-common.o
+
+nvme-common-y			+= auth.o

index d6d056963c06..abed8f59d3f2 100644
@@ -91,3 +91,15 @@ config NVME_TCP
 	  from https://github.com/linux-nvme/nvme-cli.
 
 	  If unsure, say N.
+
+config NVME_AUTH
+	bool "NVM Express over Fabrics In-Band Authentication"
+	depends on NVME_CORE
+	select NVME_COMMON
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This provides support for NVMe over Fabrics In-Band Authentication.
+
+	  If unsure, say N.

index 476c5c988496..7755f5e3b281 100644
@@ -15,6 +15,7 @@ nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o
 nvme-core-$(CONFIG_BLK_DEV_ZONED)	+= zns.o
 nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS)	+= fault_inject.o
 nvme-core-$(CONFIG_NVME_HWMON)		+= hwmon.o
+nvme-core-$(CONFIG_NVME_AUTH)		+= auth.o
 
 nvme-y					+= pci.o
 

index 1e3a09cad961..15c142b277ab 100644
@@ -68,6 +68,8 @@ enum {
 	NVMF_OPT_FAIL_FAST_TMO	= 1 << 20,
 	NVMF_OPT_HOST_IFACE	= 1 << 21,
 	NVMF_OPT_DISCOVERY	= 1 << 22,
+	NVMF_OPT_DHCHAP_SECRET	= 1 << 23,
+	NVMF_OPT_DHCHAP_CTRL_SECRET = 1 << 24,
 };
 
 /**

@@ -97,6 +99,9 @@ enum {
  * @max_reconnects: maximum number of allowed reconnect attempts before removing
  *              the controller, (-1) means reconnect forever, zero means remove
  *              immediately;
+ * @dhchap_secret: DH-HMAC-CHAP secret
+ * @dhchap_ctrl_secret: DH-HMAC-CHAP controller secret for bi-directional
+ *              authentication
  * @disable_sqflow: disable controller sq flow control
  * @hdr_digest: generate/verify header digest (TCP)
  * @data_digest: generate/verify data digest (TCP)

@@ -121,6 +126,8 @@ struct nvmf_ctrl_options {
 	unsigned int		kato;
 	struct nvmf_host	*host;
 	int			max_reconnects;
+	char			*dhchap_secret;
+	char			*dhchap_ctrl_secret;
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;

index a2b53ca63335..f46b752db02c 100644
@@ -327,6 +327,15 @@ struct nvme_ctrl {
 	struct work_struct ana_work;
 #endif
 
+#ifdef CONFIG_NVME_AUTH
+	struct work_struct dhchap_auth_work;
+	struct list_head dhchap_auth_list;
+	struct mutex dhchap_auth_mutex;
+	struct nvme_dhchap_key *host_key;
+	struct nvme_dhchap_key *ctrl_key;
+	u16 transaction;
+#endif
+
 	/* Power saving configuration */
 	u64 ps_max_latency_us;
 	bool apst_enabled;

@@ -957,6 +966,27 @@ static inline bool nvme_ctrl_sgl_supported(struct nvme_ctrl *ctrl)
 	return ctrl->sgls & ((1 << 0) | (1 << 1));
 }
 
+#ifdef CONFIG_NVME_AUTH
+void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl);
+void nvme_auth_stop(struct nvme_ctrl *ctrl);
+int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid);
+int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid);
+void nvme_auth_reset(struct nvme_ctrl *ctrl);
+void nvme_auth_free(struct nvme_ctrl *ctrl);
+#else
+static inline void nvme_auth_init_ctrl(struct nvme_ctrl *ctrl) {};
+static inline void nvme_auth_stop(struct nvme_ctrl *ctrl) {};
+static inline int nvme_auth_negotiate(struct nvme_ctrl *ctrl, int qid)
+{
+	return -EPROTONOSUPPORT;
+}
+static inline int nvme_auth_wait(struct nvme_ctrl *ctrl, int qid)
+{
+	return NVME_SC_AUTH_REQUIRED;
+}
+static inline void nvme_auth_free(struct nvme_ctrl *ctrl) {};
+#endif
+
 u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			 u8 opcode);
 int nvme_execute_passthru_rq(struct request *rq);

index d9f19d901313..bd63568c0068 100644
@@ -1197,6 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 	struct nvme_rdma_ctrl *ctrl = container_of(work,
 			struct nvme_rdma_ctrl, err_work);
 
+	nvme_auth_stop(&ctrl->ctrl);
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
 	nvme_rdma_teardown_io_queues(ctrl, false);

index ad3a2bf2f1e9..be52902b5f3b 100644
@@ -2174,6 +2174,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 				struct nvme_tcp_ctrl, err_work);
 	struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
 
+	nvme_auth_stop(ctrl);
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
 	nvme_tcp_teardown_io_queues(ctrl, false);

index 2a89c5aa0790..1c36fcedea20 100644
@@ -287,6 +287,34 @@ static const char *nvme_trace_fabrics_property_get(struct trace_seq *p, u8 *spc)
 	return ret;
 }
 
+static const char *nvme_trace_fabrics_auth_send(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 tl = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, tl=%u",
+			 spsp0, spsp1, secp, tl);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
+static const char *nvme_trace_fabrics_auth_receive(struct trace_seq *p, u8 *spc)
+{
+	const char *ret = trace_seq_buffer_ptr(p);
+	u8 spsp0 = spc[1];
+	u8 spsp1 = spc[2];
+	u8 secp = spc[3];
+	u32 al = get_unaligned_le32(spc + 4);
+
+	trace_seq_printf(p, "spsp0=%02x, spsp1=%02x, secp=%02x, al=%u",
+			 spsp0, spsp1, secp, al);
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 static const char *nvme_trace_fabrics_common(struct trace_seq *p, u8 *spc)
 {
 	const char *ret = trace_seq_buffer_ptr(p);

@@ -306,6 +334,10 @@ const char *nvme_trace_parse_fabrics_cmd(struct trace_seq *p,
 		return nvme_trace_fabrics_connect(p, spc);
 	case nvme_fabrics_type_property_get:
 		return nvme_trace_fabrics_property_get(p, spc);
+	case nvme_fabrics_type_auth_send:
+		return nvme_trace_fabrics_auth_send(p, spc);
+	case nvme_fabrics_type_auth_receive:
+		return nvme_trace_fabrics_auth_receive(p, spc);
 	default:
 		return nvme_trace_fabrics_common(p, spc);
 	}

new file mode 100644
index 000000000000..354456826221
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Hannes Reinecke, SUSE Software Solutions
+ */
+
+#ifndef _NVME_AUTH_H
+#define _NVME_AUTH_H
+
+#include <crypto/kpp.h>
+
+struct nvme_dhchap_key {
+	u8 *key;
+	size_t len;
+	u8 hash;
+};
+
+u32 nvme_auth_get_seqnum(void);
+const char *nvme_auth_dhgroup_name(u8 dhgroup_id);
+const char *nvme_auth_dhgroup_kpp(u8 dhgroup_id);
+u8 nvme_auth_dhgroup_id(const char *dhgroup_name);
+
+const char *nvme_auth_hmac_name(u8 hmac_id);
+const char *nvme_auth_digest_name(u8 hmac_id);
+size_t nvme_auth_hmac_hash_len(u8 hmac_id);
+u8 nvme_auth_hmac_id(const char *hmac_name);
+
+struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
+					      u8 key_hash);
+void nvme_auth_free_key(struct nvme_dhchap_key *key);
+u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
+int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+
+#endif /* _NVME_AUTH_H */

2.29.2

The Kconfig doesn't apply due to the new nvme-apple driver.

Also when I fix it up manually Kconfig and sparse are both not happy:

WARNING: unmet direct dependencies detected for CRYPTO_DH_RFC7919_GROUPS
Depends on [n]: CRYPTO [=y] && CRYPTO_DH [=n]
Selected by [y]:
- NVME_AUTH [=y] && NVME_CORE [=y]

i.e. when you select something you also need to select its dependencies

sparse also isn't exactly happy with the series:

rivers/nvme/host/core.c:3667:1: warning: symbol 'dev_attr_dhchap_secret' was not declared. Should it be static? drivers/nvme/host/core.c:3717:1: warning: symbol 'dev_attr_dhchap_ctrl_secret' was not declared. Should it be static?

drivers/nvme/target/fabrics-cmd-auth.c:33:30: warning: invalid assignment: |=
drivers/nvme/target/fabrics-cmd-auth.c:33:30: left side has type restricted __le32
drivers/nvme/target/fabrics-cmd-auth.c:33:30: right side has type int
drivers/nvme/target/fabrics-cmd-auth.c:118:22: warning: cast to restricted __le32 drivers/nvme/target/fabrics-cmd-auth.c:118:22: warning: cast from restricted __le16 drivers/nvme/target/fabrics-cmd-auth.c:389:30: warning: incorrect type in assignment (different base types)
drivers/nvme/target/fabrics-cmd-auth.c:389:30: expected restricted __le16 [usertype] dhvlen
drivers/nvme/target/fabrics-cmd-auth.c:389:30: got restricted __le32 [usertype]

Right. Will be fixing it up.

Cheers,

Hannes

Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Implement Diffie-Hellman key exchange using FFDHE groups for NVMe In-Band Authentication.

---
 drivers/nvme/common/auth.c | 153 ++++++++++++++++++++++++++++
 drivers/nvme/host/Kconfig  |   1 +
 drivers/nvme/host/auth.c   | 199 +++++++++++++++++++++++++++++++++++--
 include/linux/nvme-auth.h  |   8 ++
 4 files changed, 355 insertions(+), 6 deletions(-)

index 1cccaf6afdd2..945f6bb6eb1f 100644
@@ -301,6 +301,159 @@ u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn)
 }
 EXPORT_SYMBOL_GPL(nvme_auth_transform_key);
 
+static int nvme_auth_hash_skey(int hmac_id, u8 *skey, size_t skey_len, u8 *hkey)
+{
+	const char *digest_name;
+	struct crypto_shash *tfm;
+	int ret;
+
+	digest_name = nvme_auth_digest_name(hmac_id);
+	if (!digest_name) {
+		pr_debug("%s: failed to get digest for %d\n", __func__,
+			 hmac_id);
+		return -EINVAL;
+	}
+	tfm = crypto_alloc_shash(digest_name, 0, 0);
+	if (IS_ERR(tfm))
+		return -ENOMEM;
+
+	ret = crypto_shash_tfm_digest(tfm, skey, skey_len, hkey);
+	if (ret < 0)
+		pr_debug("%s: Failed to hash digest len %zu\n", __func__,
+			 skey_len);
+
+	crypto_free_shash(tfm);
+	return ret;
+}
+
+int nvme_auth_augmented_challenge(u8 hmac_id, u8 *skey, size_t skey_len,
+		u8 *challenge, u8 *aug, size_t hlen)
+{
+	struct crypto_shash *tfm;
+	struct shash_desc *desc;
+	u8 *hashed_key;
+	const char *hmac_name;
+	int ret;
+
+	hashed_key = kmalloc(hlen, GFP_KERNEL);
+	if (!hashed_key)
+		return -ENOMEM;
+
+	ret = nvme_auth_hash_skey(hmac_id, skey,
+				  skey_len, hashed_key);
+	if (ret < 0)
+		goto out_free_key;
+
+	hmac_name = nvme_auth_hmac_name(hmac_id);
+	if (!hmac_name) {
+		pr_warn("%s: invalid hash algoritm %d\n",
+			__func__, hmac_id);
+		ret = -EINVAL;
+		goto out_free_key;
+	}
+
+	tfm = crypto_alloc_shash(hmac_name, 0, 0);
+	if (IS_ERR(tfm)) {
+		ret = PTR_ERR(tfm);
+		goto out_free_key;
+	}
+
+	desc = kmalloc(sizeof(struct shash_desc) + crypto_shash_descsize(tfm),
+		       GFP_KERNEL);
+	if (!desc) {
+		ret = -ENOMEM;
+		goto out_free_hash;
+	}
+	desc->tfm = tfm;
+
+	ret = crypto_shash_setkey(tfm, hashed_key, hlen);
+	if (ret)
+		goto out_free_desc;
+
+	ret = crypto_shash_init(desc);
+	if (ret)
+		goto out_free_desc;
+
+	ret = crypto_shash_update(desc, challenge, hlen);
+	if (ret)
+		goto out_free_desc;
+
+	ret = crypto_shash_final(desc, aug);
+out_free_desc:
+	kfree_sensitive(desc);
+out_free_hash:
+	crypto_free_shash(tfm);
+out_free_key:
+	kfree_sensitive(hashed_key);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_augmented_challenge);
+
+int nvme_auth_gen_privkey(struct crypto_kpp *dh_tfm, u8 dh_gid)
+{
+	int ret;
+
+	ret = crypto_kpp_set_secret(dh_tfm, NULL, 0);
+	if (ret)
+		pr_debug("failed to set private key, error %d\n", ret);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_gen_privkey);
+
+int nvme_auth_gen_pubkey(struct crypto_kpp *dh_tfm,
+		u8 *host_key, size_t host_key_len)
+{
+	struct kpp_request *req;
+	struct crypto_wait wait;
+	struct scatterlist dst;
+	int ret;
+
+	req = kpp_request_alloc(dh_tfm, GFP_KERNEL);
+	if (!req)
+		return -ENOMEM;
+
+	crypto_init_wait(&wait);
+	kpp_request_set_input(req, NULL, 0);
+	sg_init_one(&dst, host_key, host_key_len);
+	kpp_request_set_output(req, &dst, host_key_len);
+	kpp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+				 crypto_req_done, &wait);
+
+	ret = crypto_wait_req(crypto_kpp_generate_public_key(req), &wait);
+	kpp_request_free(req);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_gen_pubkey);
+
+int nvme_auth_gen_shared_secret(struct crypto_kpp *dh_tfm,
+		u8 *ctrl_key, size_t ctrl_key_len,
+		u8 *sess_key, size_t sess_key_len)
+{
+	struct kpp_request *req;
+	struct crypto_wait wait;
+	struct scatterlist src, dst;
+	int ret;
+
+	req = kpp_request_alloc(dh_tfm, GFP_KERNEL);
+	if (!req)
+		return -ENOMEM;
+
+	crypto_init_wait(&wait);
+	sg_init_one(&src, ctrl_key, ctrl_key_len);
+	kpp_request_set_input(req, &src, ctrl_key_len);
+	sg_init_one(&dst, sess_key, sess_key_len);
+	kpp_request_set_output(req, &dst, sess_key_len);
+	kpp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+				 crypto_req_done, &wait);
+
+	ret = crypto_wait_req(crypto_kpp_compute_shared_secret(req), &wait);
+
+	kpp_request_free(req);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(nvme_auth_gen_shared_secret);
+
 int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key)
 {
 	struct nvme_dhchap_key *key;

index abed8f59d3f2..026b75013744 100644
@@ -99,6 +99,7 @@ config NVME_AUTH
 	select CRYPTO_HMAC
 	select CRYPTO_SHA256
 	select CRYPTO_SHA512
+	select CRYPTO_DH_RFC7919_GROUPS
 	help
 	  This provides support for NVMe over Fabrics In-Band Authentication.
 

index 354456826221..dcb8030062dd 100644
@@ -29,5 +29,13 @@ struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret,
 void nvme_auth_free_key(struct nvme_dhchap_key *key);
 u8 *nvme_auth_transform_key(struct nvme_dhchap_key *key, char *nqn);
 int nvme_auth_generate_key(u8 *secret, struct nvme_dhchap_key **ret_key);
+int nvme_auth_augmented_challenge(u8 hmac_id, u8 *skey, size_t skey_len,
+				  u8 *challenge, u8 *aug, size_t hlen);
+int nvme_auth_gen_privkey(struct crypto_kpp *dh_tfm, u8 dh_gid);
+int nvme_auth_gen_pubkey(struct crypto_kpp *dh_tfm,
+			 u8 *host_key, size_t host_key_len);
+int nvme_auth_gen_shared_secret(struct crypto_kpp *dh_tfm,
+				u8 *ctrl_key, size_t ctrl_key_len,
+				u8 *sess_key, size_t sess_key_len);
 
 #endif /* _NVME_AUTH_H */

2.29.2

Some fabrics commands can be sent via io queues, so add a new function nvmet_parse_fabrics_io_cmd() and rename the existing nvmet_parse_fabrics_cmd() to nvmet_parse_fabrics_admin_cmd().

---
 drivers/nvme/target/admin-cmd.c   |  2 +-
 drivers/nvme/target/core.c        |  4 ++++
 drivers/nvme/target/fabrics-cmd.c | 17 ++++++++++++++++-
 drivers/nvme/target/nvmet.h       |  3 ++-
 4 files changed, 23 insertions(+), 3 deletions(-)

index 397daaf51f1b..31df40ac828f 100644
@@ -1017,7 +1017,7 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
 	u16 ret;
 
 	if (nvme_is_fabrics(cmd))
-		return nvmet_parse_fabrics_cmd(req);
+		return nvmet_parse_fabrics_admin_cmd(req);
 	if (nvmet_is_disc_subsys(nvmet_req_subsys(req)))
 		return nvmet_parse_discovery_cmd(req);
 

index 90e75324dae0..792f15621173 100644
@@ -865,8 +865,12 @@ static inline u16 nvmet_io_cmd_check_access(struct nvmet_req *req)
 
 static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 {
+	struct nvme_command *cmd = req->cmd;
 	u16 ret;
 
+	if (nvme_is_fabrics(cmd))
+		return nvmet_parse_fabrics_io_cmd(req);
+
 	ret = nvmet_check_ctrl_status(req);
 	if (unlikely(ret))
 		return ret;

index 70fb587e9413..f23c28729908 100644
@@ -82,7 +82,7 @@ static void nvmet_execute_prop_get(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req)
+u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req)
 {
 	struct nvme_command *cmd = req->cmd;
 

@@ -103,6 +103,21 @@ u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req)
 	return 0;
 }
 
+u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req)
+{
+	struct nvme_command *cmd = req->cmd;
+
+	switch (cmd->fabrics.fctype) {
+	default:
+		pr_debug("received unknown capsule type 0x%x\n",
+			cmd->fabrics.fctype);
+		req->error_loc = offsetof(struct nvmf_common_command, fctype);
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+
+	return 0;
+}
+
 static u16 nvmet_install_queue(struct nvmet_ctrl *ctrl, struct nvmet_req *req)
 {
 	struct nvmf_connect_command *c = &req->cmd->connect;

index 69818752a33a..c37f41eafc2f 100644
@@ -419,7 +419,8 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req);
 u16 nvmet_parse_admin_cmd(struct nvmet_req *req);
 u16 nvmet_parse_discovery_cmd(struct nvmet_req *req);
-u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req);
+u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req);
+u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req);
 
 bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
 		struct nvmet_sq *sq, const struct nvmet_fabrics_ops *ops);

2.29.2

Implement NVMe-oF In-Band authentication according to NVMe TPAR 8006. This patch adds three additional configfs entries 'dhchap_key', 'dhchap_ctrl_key', and 'dhchap_hash' to the 'host' configfs directory. The 'dhchap_key' and 'dhchap_ctrl_key' entries need to be in the ASCII format as specified in NVMe Base Specification v2.0 section 8.13.5.8 'Secret representation'.
'dhchap_hash' defaults to 'hmac(sha256)', and can be written to to switch to a different HMAC algorithm.

---
 drivers/nvme/target/Kconfig            |  12 +
 drivers/nvme/target/Makefile           |   1 +
 drivers/nvme/target/admin-cmd.c        |   2 +
 drivers/nvme/target/auth.c             | 367 ++++++++++++++++++
 drivers/nvme/target/configfs.c         | 107 +++++-
 drivers/nvme/target/core.c             |  11 +
 drivers/nvme/target/fabrics-cmd-auth.c | 499 +++++++++++++++++++++++++
 drivers/nvme/target/fabrics-cmd.c      |  38 +-
 drivers/nvme/target/nvmet.h            |  62 +++
 9 files changed, 1096 insertions(+), 3 deletions(-)
 create mode 100644 drivers/nvme/target/auth.c
 create mode 100644 drivers/nvme/target/fabrics-cmd-auth.c

index 973561c93888..f1fbfc9b2f45 100644
@@ -83,3 +83,15 @@ config NVME_TARGET_TCP
 	  devices over TCP.
 
 	  If unsure, say N.
+
+config NVME_TARGET_AUTH
+	bool "NVMe over Fabrics In-band Authentication support"
+	depends on NVME_TARGET
+	select NVME_COMMON
+	select CRYPTO_HMAC
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
+	help
+	  This enables support for NVMe over Fabrics In-band Authentication
+
+	  If unsure, say N.

index 9837e580fa7e..c66820102493 100644
@@ -13,6 +13,7 @@ nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
+nvmet-$(CONFIG_NVME_TARGET_AUTH)	+= fabrics-cmd-auth.o auth.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o

index 31df40ac828f..fc8a957fad0a 100644
@@ -1018,6 +1018,8 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
 
 	if (nvme_is_fabrics(cmd))
 		return nvmet_parse_fabrics_admin_cmd(req);
+	if (unlikely(!nvmet_check_auth_status(req)))
+		return NVME_SC_AUTH_REQUIRED | NVME_SC_DNR;
 	if (nvmet_is_disc_subsys(nvmet_req_subsys(req)))
 		return nvmet_parse_discovery_cmd(req);
 

index 792f15621173..eec0351e1022 100644
@@ -795,6 +795,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq)
 	wait_for_completion(&sq->confirm_done);
 	wait_for_completion(&sq->free_done);
 	percpu_ref_exit(&sq->ref);
+	nvmet_auth_sq_free(sq);
 
 	if (ctrl) {
 		/*

@@ -871,6 +872,9 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 	if (nvme_is_fabrics(cmd))
 		return nvmet_parse_fabrics_io_cmd(req);
 
+	if (unlikely(!nvmet_check_auth_status(req)))
+		return NVME_SC_AUTH_REQUIRED | NVME_SC_DNR;
+
 	ret = nvmet_check_ctrl_status(req);
 	if (unlikely(ret))
 		return ret;

@@ -1275,6 +1279,11 @@ u16 nvmet_check_ctrl_status(struct nvmet_req *req)
 		       req->cmd->common.opcode, req->sq->qid);
 		return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
 	}
+
+	if (unlikely(!nvmet_check_auth_status(req))) {
+		pr_warn("qid %d not authenticated\n", req->sq->qid);
+		return NVME_SC_AUTH_REQUIRED | NVME_SC_DNR;
+	}
 	return 0;
 }
 

@@ -1465,6 +1474,8 @@ static void nvmet_ctrl_free(struct kref *ref)
 	flush_work(&ctrl->async_event_work);
 	cancel_work_sync(&ctrl->fatal_err_work);
 
+	nvmet_destroy_auth(ctrl);
+
 	ida_free(&cntlid_ida, ctrl->cntlid);
 
 	nvmet_async_events_free(ctrl);

index f23c28729908..f91a56180d3d 100644
@@ -93,6 +93,14 @@ u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req)
 	case nvme_fabrics_type_property_get:
 		req->execute = nvmet_execute_prop_get;
 		break;
+#ifdef CONFIG_NVME_TARGET_AUTH
+	case nvme_fabrics_type_auth_send:
+		req->execute = nvmet_execute_auth_send;
+		break;
+	case nvme_fabrics_type_auth_receive:
+		req->execute = nvmet_execute_auth_receive;
+		break;
+#endif
 	default:
 		pr_debug("received unknown capsule type 0x%x\n",
 			cmd->fabrics.fctype);

@@ -108,6 +116,14 @@ u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req)
 	struct nvme_command *cmd = req->cmd;
 
 	switch (cmd->fabrics.fctype) {
+#ifdef CONFIG_NVME_TARGET_AUTH
+	case nvme_fabrics_type_auth_send:
+		req->execute = nvmet_execute_auth_send;
+		break;
+	case nvme_fabrics_type_auth_receive:
+		req->execute = nvmet_execute_auth_receive;
+		break;
+#endif
 	default:
 		pr_debug("received unknown capsule type 0x%x\n",
 			cmd->fabrics.fctype);

@@ -188,6 +204,7 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 	struct nvmf_connect_data *d;
 	struct nvmet_ctrl *ctrl = NULL;
 	u16 status = 0;
+	int ret;
 
 	if (!nvmet_check_transfer_len(req, sizeof(struct nvmf_connect_data)))
 		return;

@@ -230,18 +247,32 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 
 	uuid_copy(&ctrl->hostid, &d->hostid);
 
+	ret = nvmet_setup_auth(ctrl);
+	if (ret < 0) {
+		pr_err("Failed to setup authentication, error %d\n", ret);
+		nvmet_ctrl_put(ctrl);
+		if (ret == -EPERM)
+			status = (NVME_SC_CONNECT_INVALID_HOST | NVME_SC_DNR);
+		else
+			status = NVME_SC_INTERNAL;
+		goto out;
+	}
+
 	status = nvmet_install_queue(ctrl, req);
 	if (status) {
 		nvmet_ctrl_put(ctrl);
 		goto out;
 	}
 
-	pr_info("creating %s controller %d for subsystem %s for NQN %s%s.\n",
+	pr_info("creating %s controller %d for subsystem %s for NQN %s%s%s.\n",
 		nvmet_is_disc_subsys(ctrl->subsys) ? "discovery" : "nvm",
 		ctrl->cntlid, ctrl->subsys->subsysnqn, ctrl->hostnqn,
-		ctrl->pi_support ? " T10-PI is enabled" : "");
+		ctrl->pi_support ? " T10-PI is enabled" : "",
+		nvmet_has_auth(ctrl) ? " with DH-HMAC-CHAP" : "");
 	req->cqe->result.u16 = cpu_to_le16(ctrl->cntlid);
 
+	if (nvmet_has_auth(ctrl))
+		nvmet_init_auth(ctrl, req);
 out:
 	kfree(d);
 complete:

@@ -301,6 +332,9 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
 	req->cqe->result.u16 = cpu_to_le16(ctrl->cntlid);
 
 	pr_debug("adding queue %d to ctrl %d.\n", qid, ctrl->cntlid);
+	req->cqe->result.u16 = cpu_to_le16(ctrl->cntlid);
+	if (nvmet_has_auth(ctrl))
+		nvmet_init_auth(ctrl, req);
 
 out:
 	kfree(d);

2.29.2

Implement Diffie-Hellman key exchange using FFDHE groups for NVMe In-Band Authentication.
This patch adds a new host configfs attribute 'dhchap_dhgroup' to select the FFDHE group to use.

---
 drivers/nvme/target/Kconfig            |   1 +
 drivers/nvme/target/auth.c             | 157 +++++++++++++++++++++++++
 drivers/nvme/target/configfs.c         |  31 +++++
 drivers/nvme/target/fabrics-cmd-auth.c |  41 +++++--
 drivers/nvme/target/nvmet.h            |   9 ++
 5 files changed, 231 insertions(+), 8 deletions(-)

index f1fbfc9b2f45..a1d9ddf57671 100644
@@ -91,6 +91,7 @@ config NVME_TARGET_AUTH
 	select CRYPTO_HMAC
 	select CRYPTO_SHA256
 	select CRYPTO_SHA512
+	select CRYPTO_DH_GROUPS_RFC7919
 	help
 	  This enables support for NVMe over Fabrics In-band Authentication
 

index 9a7d91c64fcd..1b11f6a83bb6 100644
@@ -1746,10 +1746,41 @@ static ssize_t nvmet_host_dhchap_hash_store(struct config_item *item,
 
 CONFIGFS_ATTR(nvmet_host_, dhchap_hash);
 
+static ssize_t nvmet_host_dhchap_dhgroup_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_host *host = to_host(item);
+	const char *dhgroup = nvme_auth_dhgroup_name(host->dhchap_dhgroup_id);
+
+	return sprintf(page, "%s\n", dhgroup ? dhgroup : "none");
+}
+
+static ssize_t nvmet_host_dhchap_dhgroup_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_host *host = to_host(item);
+	int dhgroup_id;
+
+	dhgroup_id = nvme_auth_dhgroup_id(page);
+	if (dhgroup_id == NVME_AUTH_DHGROUP_INVALID)
+		return -EINVAL;
+	if (dhgroup_id != NVME_AUTH_DHGROUP_NULL) {
+		const char *kpp = nvme_auth_dhgroup_kpp(dhgroup_id);
+
+		if (!crypto_has_kpp(kpp, 0, 0))
+			return -EINVAL;
+	}
+	host->dhchap_dhgroup_id = dhgroup_id;
+	return count;
+}
+
+CONFIGFS_ATTR(nvmet_host_, dhchap_dhgroup);
+
 static struct configfs_attribute *nvmet_host_attrs[] = {
 	&nvmet_host_attr_dhchap_key,
 	&nvmet_host_attr_dhchap_ctrl_key,
 	&nvmet_host_attr_dhchap_hash,
+	&nvmet_host_attr_dhchap_dhgroup,
 	NULL,
 };
 #endif /* CONFIG_NVME_TARGET_AUTH */

index 765db7541a87..8b239aec3ca2 100644
@@ -225,6 +225,10 @@ struct nvmet_ctrl {
 	struct nvme_dhchap_key	*host_key;
 	struct nvme_dhchap_key	*ctrl_key;
 	u8			shash_id;
+	struct crypto_kpp	*dh_tfm;
+	u8			dh_gid;
+	u8			*dh_key;
+	size_t			dh_keysize;
 #endif
 };
 

@@ -701,6 +705,7 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl);
 void nvmet_init_auth(struct nvmet_ctrl *ctrl, struct nvmet_req *req);
 void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);
 void nvmet_auth_sq_free(struct nvmet_sq *sq);
+int nvmet_setup_dhgroup(struct nvmet_ctrl *ctrl, u8 dhgroup_id);
 bool nvmet_check_auth_status(struct nvmet_req *req);
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			 unsigned int hash_len);

@@ -710,6 +715,10 @@ static inline bool nvmet_has_auth(struct nvmet_ctrl *ctrl)
 {
 	return ctrl->host_key != NULL;
 }
+int nvmet_auth_ctrl_exponential(struct nvmet_req *req,
+				u8 *buf, int buf_size);
+int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
+			    u8 *buf, int buf_size);
 #else
 static inline int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
 {

2.29.2

Each authentication step is required to be completed within the KATO interval (or two minutes if not set). So add a workqueue function to reset the transaction ID and the expected next protocol step; this will automatically the next authentication command referring to the terminated authentication.

---
 drivers/nvme/target/auth.c             |  1 +
 drivers/nvme/target/fabrics-cmd-auth.c | 20 +++++++++++++++++++-
 drivers/nvme/target/nvmet.h            |  1 +
 3 files changed, 21 insertions(+), 1 deletion(-)

index d5624bdf834b..bf92435c783c 100644
@@ -218,6 +218,7 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
 
 void nvmet_auth_sq_free(struct nvmet_sq *sq)
 {
+	cancel_delayed_work(&sq->auth_expired_work);
 	kfree(sq->dhchap_c1);
 	sq->dhchap_c1 = NULL;
 	kfree(sq->dhchap_c2);

index f32e2a4aeaa0..f2a355b8833f 100644
@@ -12,9 +12,22 @@
 #include <crypto/kpp.h>
 #include "nvmet.h"
 
+static void nvmet_auth_expired_work(struct work_struct *work)
+{
+	struct nvmet_sq *sq = container_of(to_delayed_work(work),
+			struct nvmet_sq, auth_expired_work);
+
+	pr_debug("%s: ctrl %d qid %d transaction %u expired, resetting\n",
+		 __func__, sq->ctrl->cntlid, sq->qid, sq->dhchap_tid);
+	sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+	sq->dhchap_tid = -1;
+}
+
 void nvmet_init_auth(struct nvmet_ctrl *ctrl, struct nvmet_req *req)
 {
 	/* Initialize in-band authentication */
+	INIT_DELAYED_WORK(&req->sq->auth_expired_work,
+			  nvmet_auth_expired_work);
 	req->sq->authenticated = false;
 	req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
 	req->cqe->result.u32 |= 0x2 << 16;

@@ -330,8 +343,13 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	req->cqe->result.u64 = 0;
 	nvmet_req_complete(req, status);
 	if (req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2 &&
-	    req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
+	    req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_FAILURE2) {
+		unsigned long auth_expire_secs = ctrl->kato ? ctrl->kato : 120;
+
+		mod_delayed_work(system_wq, &req->sq->auth_expired_work,
+				 auth_expire_secs * HZ);
 		return;
+	}
 	/* Final states, clear up variables */
 	nvmet_auth_sq_free(req->sq);
 	if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)

index 8b239aec3ca2..829fb1d78ee1 100644
@@ -109,6 +109,7 @@ struct nvmet_sq {
 	u32			sqhd;
 	bool			sqhd_disabled;
 #ifdef CONFIG_NVME_TARGET_AUTH
+	struct delayed_work	auth_expired_work;
 	bool			authenticated;
 	u16			dhchap_tid;
 	u16			dhchap_status;

2.29.2

blktests (master) # ./check nvme/039
nvme/039 (Create authenticated connections) [failed]

     runtime  1.400s  ...  1.707s
     --- tests/nvme/039.out	2022-06-08 18:09:06.239931529 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/039.out.bad	2022-06-08 

18:09:40.596663692 -0700
@@ -1,6 +1,7 @@
Running nvme/039
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Test unauthenticated connection
-no controller found
+no controller found: failed to write to nvme-fabrics device
NQN:blktests-subsystem-1 disconnected 0 controller(s) Test authenticated connection
...
(Run 'diff -u tests/nvme/039.out
/mnt/data/blktests/results/nodev/nvme/039.out.bad' to see the entire diff) blktests (master) # ./check nvme/040
nvme/040 (Test dhchap key types for authenticated connections) [failed]

     runtime  0.119s  ...  11.141s
     --- tests/nvme/040.out	2022-06-08 18:09:06.473936524 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/040.out.bad	2022-06-08 

18:09:58.453043923 -0700
@@ -1,4 +1,5 @@
Running nvme/040
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Testing hmac 0
NQN:blktests-subsystem-1 disconnected 1 controller(s) Testing hmac 1
blktests (master) # ./check nvme/041
nvme/041 (Test hash and DH group variations for authenticated connections) [failed]

     runtime  10.814s  ...  14.777s
     --- tests/nvme/041.out	2022-06-08 18:09:06.708941544 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/041.out.bad	2022-06-08 

18:10:39.374915307 -0700
@@ -1,4 +1,5 @@
Running nvme/041
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Testing hash hmac(sha256)
NQN:blktests-subsystem-1 disconnected 1 controller(s) Testing hash hmac(sha384)
blktests (master) # ./check nvme/042
nvme/042 (Test bi-directional authentication) [failed]

     runtime  0.066s  ...  3.889s
     --- tests/nvme/042.out	2022-06-08 18:09:06.939946473 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/042.out.bad	2022-06-08 

18:10:46.273062196 -0700
@@ -1,8 +1,9 @@
Running nvme/042
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Test host authentication
NQN:blktests-subsystem-1 disconnected 1 controller(s) Test host authentication and invalid ctrl authentication
-no controller found
+no controller found: failed to write to nvme-fabrics device
...
(Run 'diff -u tests/nvme/042.out
/mnt/data/blktests/results/nodev/nvme/042.out.bad' to see the entire diff) blktests (master) # nvme_trtype=tcp ./check nvme/042
nvme/042 (Test bi-directional authentication) [failed]

     runtime  3.889s  ...  3.293s
     --- tests/nvme/042.out	2022-06-08 18:09:06.939946473 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/042.out.bad	2022-06-08 

18:11:44.248296709 -0700
@@ -1,8 +1,9 @@
Running nvme/042
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Test host authentication
NQN:blktests-subsystem-1 disconnected 1 controller(s) Test host authentication and invalid ctrl authentication
-no controller found
+no controller found: failed to write to nvme-fabrics device
...
(Run 'diff -u tests/nvme/042.out
/mnt/data/blktests/results/nodev/nvme/042.out.bad' to see the entire diff) blktests (master) # nvme_trtype=tcp ./check nvme/039
nvme/039 (Create authenticated connections) [failed]

     runtime  1.707s  ...  1.391s
     --- tests/nvme/039.out	2022-06-08 18:09:06.239931529 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/039.out.bad	2022-06-08 

18:11:51.896459567 -0700
@@ -1,6 +1,7 @@
Running nvme/039
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Test unauthenticated connection
-no controller found
+no controller found: failed to write to nvme-fabrics device
NQN:blktests-subsystem-1 disconnected 0 controller(s) Test authenticated connection
...
(Run 'diff -u tests/nvme/039.out
/mnt/data/blktests/results/nodev/nvme/039.out.bad' to see the entire diff) blktests (master) # nvme_trtype=tcp ./check nvme/040
nvme/040 (Test dhchap key types for authenticated connections) [failed]

     runtime  11.141s  ...  9.133s
     --- tests/nvme/040.out	2022-06-08 18:09:06.473936524 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/040.out.bad	2022-06-08 

18:12:05.467748553 -0700
@@ -1,4 +1,5 @@
Running nvme/040
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Testing hmac 0
NQN:blktests-subsystem-1 disconnected 1 controller(s) Testing hmac 1
blktests (master) # nvme_trtype=tcp ./check nvme/041 nvme/041 (Test hash and DH group variations for authenticated connections) [failed]

     runtime  14.777s  ...  8.555s
     --- tests/nvme/041.out	2022-06-08 18:09:06.708941544 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/041.out.bad	2022-06-08 

18:12:17.358001741 -0700
@@ -1,4 +1,5 @@
Running nvme/041
+tests/nvme/rc: line 269: printf: write error: Invalid argument
Testing hash hmac(sha256)
NQN:blktests-subsystem-1 disconnected 1 controller(s) Testing hash hmac(sha384)
blktests (master) #

diff)

Hmm. Not sure what has happened here, but all blktests worked on my testbed. I'll be rechecking with the latest nvme-cli build.

Which nvme-cli version did you use?

Cheers,

Hannes

Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Retested with latest nvme-cli:

# bash ./check tests/nvme/039
nvme/039 (Create authenticated connections)                  [passed]
    runtime  1.625s  ...  1.667s
# bash ./check tests/nvme/040
nvme/040 (Test dhchap key types for authenticated connections) [passed]
    runtime  10.497s  ...  10.657s

So not sure what is happening at your end; the 'invalid argument' seems to indicate that the 'connect' arguments are not understood. Maybe a missing config option during kernel build?

Cheers,

Hannes

Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

This is the nvme config I've :-

nvme (nvme-5.18) # grep NVME .config
# NVME Support
CONFIG_NVME_COMMON=m
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_VERBOSE_ERRORS=y
CONFIG_NVME_HWMON=y
CONFIG_NVME_FABRICS=m
CONFIG_NVME_RDMA=m
CONFIG_NVME_FC=m
CONFIG_NVME_TCP=m
CONFIG_NVME_AUTH=y
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_PASSTHRU=y
CONFIG_NVME_TARGET_LOOP=m
CONFIG_NVME_TARGET_RDMA=m
CONFIG_NVME_TARGET_FC=m
CONFIG_NVME_TARGET_FCLOOP=m
CONFIG_NVME_TARGET_TCP=m
CONFIG_NVME_TARGET_AUTH=y

# end of NVME Support
CONFIG_RTC_NVMEM=y
CONFIG_NVMEM=y
CONFIG_NVMEM_SYSFS=y
# CONFIG_NVMEM_RMEM is not set
nvme (nvme-5.18) # grep NVME .config | grep AUTH
CONFIG_NVME_AUTH=y
CONFIG_NVME_TARGET_AUTH=y
nvme (nvme-5.18) #

after some debugging I found this :-

# ./check nvme/039
nvme/039 (Create authenticated connections)                  [failed]
     runtime  1.636s  ...  1.656s
     --- tests/nvme/039.out	2022-06-08 18:09:06.239931529 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/039.out.bad	2022-06-10 
02:02:03.734310155 -0700
     @@ -1,6 +1,7 @@
      Running nvme/039
     +tests/nvme/rc: line 269: printf: write error: Invalid argument
      Test unauthenticated connection
     -no controller found
     +no controller found: failed to write to nvme-fabrics device
      NQN:blktests-subsystem-1 disconnected 0 controller(s)
      Test authenticated connection
     ...
     (Run 'diff -u tests/nvme/039.out 
/mnt/data/blktests/results/nodev/nvme/039.out.bad' to see the entire diff)

blktests (master) # dmesg -c
810.765135] run blktests nvme/039 at 2022-06-10 02:03:42

[  810.794825] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[  810.795094] nvmet: nvmet_ns_enable_store 535
[  810.795098] nvmet: nvmet_ns_enable 555
[  810.795100] nvmet: nvmet_ns_enable 559
[  810.795101] nvmet: nvmet_ns_enable 565
[  810.795102] nvmet: nvmet_ns_enable 573
*[  810.795108] nvmet: failed to open file /tmp/blktest-ns1.img: (-22)*0
[  810.797146] nvmet: nvmet_ns_enable 610
[  810.805542] nvmet: creating nvm controller 1 for subsystem 

blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:dbdf14ac-d4e5-4a3c-b547-3c39899650e7 with DH-HMAC-CHAP.
*[ 810.805556] nvme nvme1: qid 0: no key*
*[ 810.805559] nvme nvme1: qid 0: authentication setup failed*

[  810.824145] nvmet: creating nvm controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:dbdf14ac-d4e5-4a3c-b547-3c39899650e7 
with DH-HMAC-CHAP.
[  810.824174] __nvme_auth_work 701

*[ 810.829094] nvme nvme1: qid 0: authenticated with hash hmac(sha256) dhgroup null*
*[ 810.829115] nvme nvme1: qid 0: authenticated*

Even after using a loop device insted of passing file to the target ns it still fails :-

blktests (master) # ./check nvme/039
nvme/039 (Create authenticated connections) [failed]

     runtime  1.665s  ...  1.713s
     --- tests/nvme/039.out	2022-06-08 18:09:06.239931529 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/039.out.bad	2022-06-10 

02:15:36.123178797 -0700
@@ -1,7 +1,10 @@
Running nvme/039
+losetup: /tmp/blktest-ns1.img: failed to set up loop device:
Device or resource busy
Test unauthenticated connection
-no controller found
+no controller found: failed to write to nvme-fabrics device
NQN:blktests-subsystem-1 disconnected 0 controller(s) Test authenticated connection
...
(Run 'diff -u tests/nvme/039.out
/mnt/data/blktests/results/nodev/nvme/039.out.bad' to see the entire diff)

blktests (master) # dmesg -c

[ 1522.531012] run blktests nvme/039 at 2022-06-10 02:15:34
[ 1522.574487] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[ 1522.574773] nvmet: nvmet_ns_enable_store 535
[ 1522.574777] nvmet: nvmet_ns_enable 555
[ 1522.574779] nvmet: nvmet_ns_enable 559
[ 1522.574780] nvmet: nvmet_ns_enable 565
[ 1522.574781] nvmet: nvmet_ns_enable 573
[ 1522.584408] nvmet: nvmet_ns_enable 580
[ 1522.584418] nvmet: nvmet_ns_enable 596
[ 1522.584420] nvmet: nvmet_ns_enable 601
[ 1522.584421] nvmet: nvmet_ns_enable 605
[ 1522.584421] nvmet: nvmet_ns_enable 607
[ 1522.584422] nvmet: nvmet_ns_enable 610
[ 1522.592585] nvmet: creating nvm controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:d3c7574b-dcf2-4882-8141-93baf2c9291c 
with DH-HMAC-CHAP.
[ 1522.592601] nvme nvme1: qid 0: no key
[ 1522.592604] nvme nvme1: qid 0: authentication setup failed
[ 1522.614144] nvmet: creating nvm controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:d3c7574b-dcf2-4882-8141-93baf2c9291c 
with DH-HMAC-CHAP.
[ 1522.614171] __nvme_auth_work 701
[ 1522.619125] nvme nvme1: qid 0: authenticated with hash hmac(sha256) 
dhgroup null
[ 1522.619154] nvme nvme1: qid 0: authenticated
[ 1522.619275] nvme nvme1: creating 48 I/O queues.
[ 1522.620508] __nvme_auth_work 701
[ 1522.625672] __nvme_auth_work 701
[ 1522.630451] __nvme_auth_work 701
[ 1522.635257] __nvme_auth_work 701
[ 1522.640227] __nvme_auth_work 701
[ 1522.644709] __nvme_auth_work 701
[ 1522.649660] __nvme_auth_work 701
[ 1522.654324] __nvme_auth_work 701
[ 1522.659129] __nvme_auth_work 701
[ 1522.663996] __nvme_auth_work 701
[ 1522.668686] __nvme_auth_work 701
[ 1522.673422] __nvme_auth_work 701
[ 1522.678153] __nvme_auth_work 701
[ 1522.682897] __nvme_auth_work 701
[ 1522.687692] __nvme_auth_work 701
[ 1522.692478] __nvme_auth_work 701
[ 1522.697329] __nvme_auth_work 701
[ 1522.701962] __nvme_auth_work 701
[ 1522.706674] __nvme_auth_work 701
[ 1522.711382] __nvme_auth_work 701
[ 1522.715998] __nvme_auth_work 701
[ 1522.720803] __nvme_auth_work 701
[ 1522.725550] __nvme_auth_work 701
[ 1522.730237] __nvme_auth_work 701
[ 1522.734947] __nvme_auth_work 701
[ 1522.739615] __nvme_auth_work 701
[ 1522.744340] __nvme_auth_work 701
[ 1522.749115] __nvme_auth_work 701
[ 1522.753804] __nvme_auth_work 701
[ 1522.758516] __nvme_auth_work 701
[ 1522.763498] __nvme_auth_work 701
[ 1522.768125] __nvme_auth_work 701
[ 1522.772960] __nvme_auth_work 701
[ 1522.777941] __nvme_auth_work 701
[ 1522.782815] __nvme_auth_work 701
[ 1522.787598] __nvme_auth_work 701
[ 1522.792304] __nvme_auth_work 701
[ 1522.796911] __nvme_auth_work 701
[ 1522.801722] __nvme_auth_work 701
[ 1522.806501] __nvme_auth_work 701
[ 1522.811436] __nvme_auth_work 701
[ 1522.816282] __nvme_auth_work 701
[ 1522.821029] __nvme_auth_work 701
[ 1522.825746] __nvme_auth_work 701
[ 1522.830410] __nvme_auth_work 701
[ 1522.835514] __nvme_auth_work 701
[ 1522.840346] __nvme_auth_work 701
[ 1522.845145] __nvme_auth_work 701
[ 1522.849755] nvme nvme1: new ctrl: "blktests-subsystem-1"
[ 1523.870772] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1"
[ 1524.212770] nvmet: nvmet_ns_enable_store 535

On 6/10/22 11:06, Chaitanya Kulkarni wrote:
[ .. ]

Weird. But I'll update the testcases for the latest master branch; maybe it's due to some infrastructure changes.

Cheers,

Hannes

Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

~ # nvme --version
nvme version 2.0 (git 2.0-96-g0b286b1)
libnvme version 1.0 (git 1.0-62-gaacaeea)
~ #

I'm I using wrong version ?

Can we please add check in the blktests to skip the test if we have wrong wrong nvme-cli version ? it will help others to not misdiagnosed the problem...

-ck

Hmm. No, that version looks okay.

What's the kernel message log? Does it say something about 'dev_loss_tmo' ?

If so, can you try with 'export nvme_trtype=tcp' and then running the tests? Previous versions of nvme-cli had issues with nvme-loop, and were trying to set the 'dev_loss_tmo' parameter, which isn't valid for loop.

Cheers,

Hannes

Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

blktests (master) # nvme_trtype=tcp ./check nvme/039
nvme/039 (Create authenticated connections) [failed]

     runtime  1.398s  ...  1.397s
     --- tests/nvme/039.out	2022-06-08 18:09:06.239931529 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/039.out.bad	2022-06-10 

02:28:28.077262601 -0700
@@ -1,7 +1,10 @@
Running nvme/039
+losetup: /tmp/blktest-ns1.img: failed to set up loop device:
Device or resource busy
Test unauthenticated connection
-no controller found
+no controller found: failed to write to nvme-fabrics device
NQN:blktests-subsystem-1 disconnected 0 controller(s) Test authenticated connection
...
(Run 'diff -u tests/nvme/039.out
/mnt/data/blktests/results/nodev/nvme/039.out.bad' to see the entire diff) blktests (master) # dmesg -c

[ 2294.783157] run blktests nvme/039 at 2022-06-10 02:28:26
[ 2294.829279] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[ 2294.829568] nvmet: nvmet_ns_enable_store 535
[ 2294.829573] nvmet: nvmet_ns_enable 555
[ 2294.829574] nvmet: nvmet_ns_enable 559
[ 2294.829575] nvmet: nvmet_ns_enable 565
[ 2294.829576] nvmet: nvmet_ns_enable 573
[ 2294.832318] nvmet: nvmet_ns_enable 580
[ 2294.832325] nvmet: nvmet_ns_enable 596
[ 2294.832327] nvmet: nvmet_ns_enable 601
[ 2294.832328] nvmet: nvmet_ns_enable 605
[ 2294.832329] nvmet: nvmet_ns_enable 607
[ 2294.832330] nvmet: nvmet_ns_enable 610
[ 2294.833335] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
[ 2294.840597] nvmet: creating nvm controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:e4f6b1f3-b215-498e-88c0-cfd20c7dee98 
with DH-HMAC-CHAP.
[ 2294.840641] nvme nvme1: qid 0: no key
[ 2294.840645] nvme nvme1: qid 0: authentication setup failed
[ 2294.840668] nvme nvme1: failed to connect queue: 0 ret=401
[ 2294.857350] nvmet: creating nvm controller 2 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:e4f6b1f3-b215-498e-88c0-cfd20c7dee98 
with DH-HMAC-CHAP.
[ 2294.857405] __nvme_auth_work 701
[ 2294.862741] nvme nvme1: qid 0: authenticated with hash hmac(sha256) 
dhgroup null
[ 2294.862771] nvme nvme1: qid 0: authenticated
[ 2294.864236] nvme nvme1: creating 48 I/O queues.
[ 2294.866802] nvme nvme1: mapped 48/0/0 default/read/poll queues.
[ 2294.867942] __nvme_auth_work 701
[ 2294.873099] __nvme_auth_work 701
[ 2294.878303] __nvme_auth_work 701
[ 2294.883186] __nvme_auth_work 701
[ 2294.888048] __nvme_auth_work 701
[ 2294.893031] __nvme_auth_work 701
[ 2294.897962] __nvme_auth_work 701
[ 2294.902783] __nvme_auth_work 701
[ 2294.907737] __nvme_auth_work 701
[ 2294.912721] __nvme_auth_work 701
[ 2294.917574] __nvme_auth_work 701
[ 2294.922468] __nvme_auth_work 701
[ 2294.927642] __nvme_auth_work 701
[ 2294.932643] __nvme_auth_work 701
[ 2294.937493] __nvme_auth_work 701
[ 2294.942385] __nvme_auth_work 701
[ 2294.947261] __nvme_auth_work 701
[ 2294.952312] __nvme_auth_work 701
[ 2294.957299] __nvme_auth_work 701
[ 2294.962158] __nvme_auth_work 701
[ 2294.967057] __nvme_auth_work 701
[ 2294.972046] __nvme_auth_work 701
[ 2294.977111] __nvme_auth_work 701
[ 2294.982095] __nvme_auth_work 701
[ 2294.987101] __nvme_auth_work 701
[ 2294.992158] __nvme_auth_work 701
[ 2294.997097] __nvme_auth_work 701
[ 2295.001999] __nvme_auth_work 701
[ 2295.006776] __nvme_auth_work 701
[ 2295.011645] __nvme_auth_work 701
[ 2295.016524] __nvme_auth_work 701
[ 2295.021228] __nvme_auth_work 701
[ 2295.026451] __nvme_auth_work 701
[ 2295.031347] __nvme_auth_work 701
[ 2295.036301] __nvme_auth_work 701
[ 2295.041172] __nvme_auth_work 701
[ 2295.046149] __nvme_auth_work 701
[ 2295.051108] __nvme_auth_work 701
[ 2295.056102] __nvme_auth_work 701
[ 2295.061153] __nvme_auth_work 701
[ 2295.066004] __nvme_auth_work 701
[ 2295.070828] __nvme_auth_work 701
[ 2295.075662] __nvme_auth_work 701
[ 2295.080518] __nvme_auth_work 701
[ 2295.085397] __nvme_auth_work 701
[ 2295.090413] __nvme_auth_work 701
[ 2295.095427] __nvme_auth_work 701
[ 2295.100301] __nvme_auth_work 701
[ 2295.105378] nvme nvme1: new ctrl: NQN "blktests-subsystem-1", addr 
127.0.0.1:4420
[ 2296.126821] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1"
[ 2296.159534] nvmet: nvmet_ns_enable_store 535

blktests (master) #

Please ignore my earlier log and consider this :-

blktests (master) # nvme_trtype=tcp ./check nvme/039
nvme/039 (Create authenticated connections) [failed]

     runtime  1.397s  ...  1.416s
     --- tests/nvme/039.out	2022-06-08 18:09:06.239931529 -0700
     +++ /mnt/data/blktests/results/nodev/nvme/039.out.bad	2022-06-10 

02:34:02.813590992 -0700
@@ -1,7 +1,9 @@
Running nvme/039
Test unauthenticated connection
-no controller found
+no controller found: failed to write to nvme-fabrics device
NQN:blktests-subsystem-1 disconnected 0 controller(s) Test authenticated connection
NQN:blktests-subsystem-1 disconnected 1 controller(s)
...
(Run 'diff -u tests/nvme/039.out
/mnt/data/blktests/results/nodev/nvme/039.out.bad' to see the entire diff) blktests (master) # dmesg -c

[ 2629.493178] run blktests nvme/039 at 2022-06-10 02:34:01
[ 2629.526813] loop0: detected capacity change from 0 to 1048576
[ 2629.540273] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[ 2629.540512] nvmet: nvmet_ns_enable_store 535
[ 2629.540516] nvmet: nvmet_ns_enable 555
[ 2629.540518] nvmet: nvmet_ns_enable 559
[ 2629.540519] nvmet: nvmet_ns_enable 565
[ 2629.540520] nvmet: nvmet_ns_enable 573
[ 2629.550996] nvmet: nvmet_ns_enable 580
[ 2629.551005] nvmet: nvmet_ns_enable 596
[ 2629.551007] nvmet: nvmet_ns_enable 601
[ 2629.551008] nvmet: nvmet_ns_enable 605
[ 2629.551009] nvmet: nvmet_ns_enable 607
[ 2629.551010] nvmet: nvmet_ns_enable 610
[ 2629.552027] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
[ 2629.560307] nvmet: creating nvm controller 1 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:dfcc37bd-2779-4b29-9f36-7c0386851263 
with DH-HMAC-CHAP.
[ 2629.560343] nvme nvme1: qid 0: no key
[ 2629.560347] nvme nvme1: qid 0: authentication setup failed
[ 2629.560365] nvme nvme1: failed to connect queue: 0 ret=401
[ 2629.577013] nvmet: creating nvm controller 2 for subsystem 
blktests-subsystem-1 for NQN 
nqn.2014-08.org.nvmexpress:uuid:dfcc37bd-2779-4b29-9f36-7c0386851263 
with DH-HMAC-CHAP.
[ 2629.577063] __nvme_auth_work 701
[ 2629.582375] nvme nvme1: qid 0: authenticated with hash hmac(sha256) 
dhgroup null
[ 2629.582395] nvme nvme1: qid 0: authenticated
[ 2629.582863] nvme nvme1: creating 48 I/O queues.
[ 2629.585466] nvme nvme1: mapped 48/0/0 default/read/poll queues.
[ 2629.586634] __nvme_auth_work 701
[ 2629.592092] __nvme_auth_work 701
[ 2629.597131] __nvme_auth_work 701
[ 2629.602217] __nvme_auth_work 701
[ 2629.607362] __nvme_auth_work 701
[ 2629.612357] __nvme_auth_work 701
[ 2629.617396] __nvme_auth_work 701
[ 2629.622605] __nvme_auth_work 701
[ 2629.627642] __nvme_auth_work 701
[ 2629.632645] __nvme_auth_work 701
[ 2629.637676] __nvme_auth_work 701
[ 2629.642738] __nvme_auth_work 701
[ 2629.647647] __nvme_auth_work 701
[ 2629.652817] __nvme_auth_work 701
[ 2629.657795] __nvme_auth_work 701
[ 2629.662618] __nvme_auth_work 701
[ 2629.667358] __nvme_auth_work 701
[ 2629.672319] __nvme_auth_work 701
[ 2629.677234] __nvme_auth_work 701
[ 2629.682177] __nvme_auth_work 701
[ 2629.687177] __nvme_auth_work 701
[ 2629.692211] __nvme_auth_work 701
[ 2629.697234] __nvme_auth_work 701
[ 2629.702091] __nvme_auth_work 701
[ 2629.707177] __nvme_auth_work 701
[ 2629.712087] __nvme_auth_work 701
[ 2629.717093] __nvme_auth_work 701
[ 2629.721979] __nvme_auth_work 701
[ 2629.726928] __nvme_auth_work 701
[ 2629.731843] __nvme_auth_work 701
[ 2629.736852] __nvme_auth_work 701
[ 2629.741845] __nvme_auth_work 701
[ 2629.746871] __nvme_auth_work 701
[ 2629.752102] __nvme_auth_work 701
[ 2629.757202] __nvme_auth_work 701
[ 2629.762209] __nvme_auth_work 701
[ 2629.767301] __nvme_auth_work 701
[ 2629.772238] __nvme_auth_work 701
[ 2629.777158] __nvme_auth_work 701
[ 2629.782157] __nvme_auth_work 701
[ 2629.787153] __nvme_auth_work 701
[ 2629.792119] __nvme_auth_work 701
[ 2629.797402] __nvme_auth_work 701
[ 2629.802400] __nvme_auth_work 701
[ 2629.807405] __nvme_auth_work 701
[ 2629.812316] __nvme_auth_work 701
[ 2629.817405] __nvme_auth_work 701
[ 2629.822526] __nvme_auth_work 701
[ 2629.827648] nvme nvme1: new ctrl: NQN "blktests-subsystem-1", addr 
127.0.0.1:4420
[ 2630.848759] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1"
[ 2630.881149] nvmet: nvmet_ns_enable_store 535

Let me know if you need any more information, I'll also try and debug the problem tomorrow.

-ck

On 6/10/22 11:35, Chaitanya Kulkarni wrote:
[ .. ]

I've now created a new pull request for blktests (#94), that one should be good for the recent changes to nvme-cli.

Can you test with that or do I need to send a patchset to the linux-nvme mailing list?

Cheers,

Hannes

Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Can you please send it over the linux-nvme if already not so I can test it along with your new version of in-band with Christoph's comments ?

Still catching-up with my emails...

[ .. ]

Ah. Found it.
It's just that the output of the nvme-cli command changed. I'll be sending a new version of the blktest scripts.

Cheers,

Hannes

Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer