Skip to contentSkip to conversation

Implement the NVMe reservation feature

PATCH Superseded Guixin Liu 4 participants 13 messages Oct 11, 2024 – Oct 15, 2024

Subsystems (2)

v14 of 19 Superseded Latest is v19 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v14 cover Implement the NVMe reservation feature

Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.

Changes from v13 to v14:
- Fix some grammatical mistake in commit message.
- Add two characters to bulleted lists in commit message to make it more readble.
- Adjust some tab indent.

Changes from v12 to v13:
- Set regctl to the number of all registrants instead of reported when reporting.

Changes from v11 to v12:
- Change patch to fit newest 6.12-rc2.
- Fix possible out-of-bounds access in nvmet_execute_pr_report.
- Use sysfs_emit instead of printf in nvmet_ns_resv_enable_show.
- Add struct rcu_head to nvmet_pr_registrant to free.
- Introduce a helper nvmet_pr_parse_ignore_key.
- Use NVME_CNTLID_DYNAMIC when report cntlid.
- Fix some over long line.
- Add more details to commit body.
- Separate out the include/linux/nvme.h changes to a single patch.
- Add a nvmetcli patch to support resv_enable configuration.

Changes from v10 to v11:
- Remove the cntlid from struct nvmet_pr_registrant. And add hostid to nvmet_pr_per_ctrl_ref to set the correct ctrl to do abort.
- Report all registrants, and report cntlid 0xffff.
- Change nvmet_req's flags to pc_ref.

Changes from v9 to v10:
- Fix the misjudgement in nvmet_pr_unregister_one.
- Fix the non-atomicity problem in nvmet_pr_preempt, now we set the new holder first before unregistering, this can not only make sure that during unregistering other host can not access, but also ensure that nvmet_pr_unregister_one will not unregiter the new holder(In nvmet_pr_unreg_all_others_by_prkey, I exclude current host).
- Remove the pr_abort in nvmet_ctrl, instead kill the per-controller percpu ref first and then wait to zero at the end of preempt_and_abort.
- Fix some spelling mistakes.

Changes from v8 to v9:
- Remove the maintianer request.

- Support "preemt and abort" by adding a per-controller percpu ref to ns, doing wait per-controller percpu ref to zero when a controller's reservation or registration is preempt and the racqa == "preempt and abort". Currently add per-controller percpu ref only when nvmet_pr_check_cmd_access success, others are not cared.

- Report ns support reservation when resv is enabled.

- Report ctrl support reservation.

- Change the log level to info when log lost.

- Fix the UAF issue in nvmet_pr_unreg_by_prkey, and change nvmet_pr_unreg_by_prkey to nvmet_pr_unreg_all_host_by_prkey.

- Dont unregister the host when ctrl is destroyed for keep the reservation info when reconnect.

- Remove the rcu lock and mutex lock when free ns's pr info.

- Fix the situation of log.count is zero.

- Move the rtype check to the start of preemtion, Dmitry suggests that:
1.4.1.6 reserved
Receipt of reserved coded values in defined fields in commands shall be reported as an error.
Look forward the suggestions.
And also, this avoid the non-atomic change of unregister and set new holder.

- Fix the compile error when close CONFIG_LOCKDEP.

- Change nvmet_pr_send_event_by_hostid to nvmet_pr_send_event_to_host.

- Change nvmet_pr_unreg_by_prkey_except_hostid to nvmet_pr_unreg_all_others_by_prkey.

- Change nvmet_pr_unreg_by_prkey to nvmet_pr_unreg_all_host_by_prkey.

Changes from v7 to v8:
- Add me as the new file pr.c's maintainer.

Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.

- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.

- Fix the resv log page is random when there is no resv log page.

- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().

- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().

- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().

- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().

- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().

- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.

Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().

- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().

- Add the print fmt to the head.

- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.

- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.

Changes from v4 to v5:

- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.

- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.

- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.

- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.

- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.

- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.

- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.

- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.

Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.

- Change the function calling alignment style to:

    nvmet_pr_send_event_by_hostid(pr, hostid,
            NVME_PR_LOG_RESERVATOPM_PREEMPTED); 

- Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().

- Remove the goto in __nvmet_pr_unregister_one().

- Change generation to atomic_t, and remove nvmet_pr_inc_generation().

- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.

Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.

- Limit the resv_log_list to 128.

- Change generation to atomic64.

- Put register rkey check to a warpper.

- Change nr_avl_pages to nr_pages.

- Use NVME_SC_SUCCESS instead of 0.

- Change kmalloc param to let it not sleep in mutex lock.

Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.

- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is
0xffffffff at each admin command path, if it is needed.

- Add reservation commands support in nvmet_get_cmd_effects_nvm().

- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.

- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.

- Remove kmem_cache instead to use kmalloc and kfree.

- Change others advice from Sagi.

- Add a blktest test case, this patch will be sent before these series of patches.

Guixin Liu (2):
nvme: add reservation command's defines
nvmet: support reservation feature

 drivers/nvme/target/Makefile      |    2 +-
 drivers/nvme/target/admin-cmd.c   |   24 +-
 drivers/nvme/target/configfs.c    |   27 +
 drivers/nvme/target/core.c        |   56 +-
 drivers/nvme/target/fabrics-cmd.c |    4 +-
 drivers/nvme/target/nvmet.h       |   55 +-
 drivers/nvme/target/pr.c          | 1162 +++++++++++++++++++++++++++++
 include/linux/nvme.h              |   68 ++
 8 files changed, 1386 insertions(+), 12 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v14 1/2 nvme: add reservation command's defines

This is a prepath of reservation implementation patch. Add the defines of reservation command, such as reservation log and sub operations.

---
 include/linux/nvme.h | 68 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
include/linux/nvme.h +68 -0
index b58d9405d65e..44d048d68503 100644
@@ -2037,4 +2037,72 @@ struct nvme_completion {
 #define NVME_MINOR(ver)		(((ver) >> 8) & 0xff)
 #define NVME_TERTIARY(ver)	((ver) & 0xff)
 
+enum {
+	NVME_AEN_RESV_LOG_PAGE_AVALIABLE	= 0x00,
+};
+
+enum {
+	NVME_PR_LOG_EMPTY_LOG_PAGE			= 0x00,
+	NVME_PR_LOG_REGISTRATION_PREEMPTED		= 0x01,
+	NVME_PR_LOG_RESERVATION_RELEASED		= 0x02,
+	NVME_PR_LOG_RESERVATOPM_PREEMPTED		= 0x03,
+};
+
+enum {
+	NVME_PR_NOTIFY_BIT_REG_PREEMPTED		= 1,
+	NVME_PR_NOTIFY_BIT_RESV_RELEASED		= 2,
+	NVME_PR_NOTIFY_BIT_RESV_PREEMPTED		= 3,
+};
+
+struct nvme_pr_log {
+	__le64			count;
+	__u8			type;
+	__u8			nr_pages;
+	__u8			rsvd1[2];
+	__le32			nsid;
+	__u8			rsvd2[48];
+};
+
+struct nvmet_pr_register_data {
+	__le64	crkey;
+	__le64	nrkey;
+};
+
+struct nvmet_pr_acquire_data {
+	__le64	crkey;
+	__le64	prkey;
+};
+
+struct nvmet_pr_release_data {
+	__le64	crkey;
+};
+
+enum nvme_pr_capabilities {
+	NVME_PR_SUPPORT_PTPL				= 1,
+	NVME_PR_SUPPORT_WRITE_EXCLUSIVE			= 1 << 1,
+	NVME_PR_SUPPORT_EXCLUSIVE_ACCESS		= 1 << 2,
+	NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY	= 1 << 3,
+	NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY	= 1 << 4,
+	NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS	= 1 << 5,
+	NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS	= 1 << 6,
+	NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF		= 1 << 7,
+};
+
+enum nvme_pr_register_action {
+	NVME_PR_REGISTER_ACT_REG		= 0,
+	NVME_PR_REGISTER_ACT_UNREG		= 1,
+	NVME_PR_REGISTER_ACT_REPLACE		= 1 << 1,
+};
+
+enum nvme_pr_acquire_action {
+	NVME_PR_ACQUIRE_ACT_ACQUIRE		= 0,
+	NVME_PR_ACQUIRE_ACT_PREEMPT		= 1,
+	NVME_PR_ACQUIRE_ACT_PREEMPT_AND_ABORT	= 1 << 1,
+};
+
+enum nvme_pr_release_action {
+	NVME_PR_RELEASE_ACT_RELEASE		= 0,
+	NVME_PR_RELEASE_ACT_CLEAR		= 1,
+};
+
 #endif /* _LINUX_NVME_H */
Signature · 1 line
2.43.0
Tested-by Reviewed-by on v14 1/2
#2 lore raw

Re: [PATCH v14 1/2] nvme: add reservation command's defines

Quoted 1 line · Guixin Liu

This is a prepath of reservation implementation patch.

if it makes sense please consider something like :-

This is a preparation patch for NVMeOF target reservation commands implantation.

Quoted 6 lines

Add the defines of reservation command, such as reservation log and sub operations.

---

Looks good.

-ck

author
#3 lore raw

Re: [PATCH v14 1/2] nvme: add reservation command's defines

Quoted 6 lines · Chaitanya Kulkarni
Quoted 1 line · Guixin Liu

This is a prepath of reservation implementation patch.

if it makes sense please consider something like :-

This is a preparation patch for NVMeOF target reservation commands implantation.

OK, I will send the v15 to change it.

Quoted 14 lines
Quoted 6 lines

Add the defines of reservation command, such as reservation log and sub operations.

---

Looks good.

-ck

author
#4 lore raw

PATCH v14 2/2 nvmet: support reservation feature

In reply to Guixin Liu

This patch implements the reservation feature, including:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

Not supported:
1. persistent reservation through power loss.

Test cases:
Use nvme-cli and fio to test all implemented sub features:
1. use nvme resv-register to register host a registrant or unregister or replace a new key.
2. use nvme resv-acquire to set host to the holder, and use fio to send read and write io in all reservation type. And also test preempt and "preempt and abort".
3. use nvme resv-report to show all registrants and reservation status.
4. use nvme resv-release to release all registrants.
5. use nvme get-log to get events generated by the preceding operations.

In addition, make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---
 drivers/nvme/target/Makefile      |    2 +-
 drivers/nvme/target/admin-cmd.c   |   24 +-
 drivers/nvme/target/configfs.c    |   27 +
 drivers/nvme/target/core.c        |   56 +-
 drivers/nvme/target/fabrics-cmd.c |    4 +-
 drivers/nvme/target/nvmet.h       |   55 +-
 drivers/nvme/target/pr.c          | 1162 +++++++++++++++++++++++++++++
 7 files changed, 1318 insertions(+), 12 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
drivers/nvme/target/Makefile +1 -1
index c402c44350b2..f2b025bbe10c 100644
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_TARGET_FCLOOP)	+= nvme-fcloop.o
 obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
-			discovery.o io-cmd-file.o io-cmd-bdev.o
+			discovery.o io-cmd-file.o io-cmd-bdev.o pr.o
 nvmet-$(CONFIG_NVME_TARGET_DEBUGFS)	+= debugfs.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
drivers/nvme/target/admin-cmd.c +23 -1
index 081f0473cd9e..19428745c795 100644
@@ -176,6 +176,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 	log->iocs[nvme_cmd_read] =
 	log->iocs[nvme_cmd_flush] =
 	log->iocs[nvme_cmd_dsm]	=
+	log->iocs[nvme_cmd_resv_acquire] =
+	log->iocs[nvme_cmd_resv_register] =
+	log->iocs[nvme_cmd_resv_release] =
+	log->iocs[nvme_cmd_resv_report] =
 		cpu_to_le32(NVME_CMD_EFFECTS_CSUPP);
 	log->iocs[nvme_cmd_write] =
 	log->iocs[nvme_cmd_write_zeroes] =
@@ -340,6 +344,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		return nvmet_execute_get_log_cmd_effects_ns(req);
 	case NVME_LOG_ANA:
 		return nvmet_execute_get_log_page_ana(req);
+	case NVME_LOG_RESERVATION:
+		return nvmet_execute_get_log_page_resv(req);
 	}
 	pr_debug("unhandled lid %d on qid %d\n",
 	       req->cmd->get_log_page.lid, req->sq->qid);
@@ -433,7 +439,8 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
-			NVME_CTRL_ONCS_WRITE_ZEROES);
+			NVME_CTRL_ONCS_WRITE_ZEROES |
+			NVME_CTRL_ONCS_RESERVATIONS);
 
 	/* XXX: don't report vwc if the underlying device is write through */
 	id->vwc = NVME_CTRL_VWC_PRESENT;
@@ -551,6 +558,15 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
 	id->nmic = NVME_NS_NMIC_SHARED;
 	id->anagrpid = cpu_to_le32(req->ns->anagrpid);
 
+	if (req->ns->pr.enable)
+		id->rescap = NVME_PR_SUPPORT_WRITE_EXCLUSIVE |
+			NVME_PR_SUPPORT_EXCLUSIVE_ACCESS |
+			NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY |
+			NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY |
+			NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS |
+			NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS |
+			NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF;
+
 	memcpy(&id->nguid, &req->ns->nguid, sizeof(id->nguid));
 
 	id->lbaf[0].ds = req->ns->blksize_shift;
@@ -861,6 +877,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_set_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_set_feat_resv_notif_mask(req, cdw11);
+		break;
 	default:
 		req->error_loc = offsetof(struct nvme_common_command, cdw10);
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -959,6 +978,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_get_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_get_feat_resv_notif_mask(req);
+		break;
 	default:
 		req->error_loc =
 			offsetof(struct nvme_common_command, cdw10);
drivers/nvme/target/configfs.c +27 -0
index 685e89b35d33..eeee9e9b854c 100644
@@ -769,6 +769,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,
 
 CONFIGFS_ATTR_WO(nvmet_ns_, revalidate_size);
 
+static ssize_t nvmet_ns_resv_enable_show(struct config_item *item, char *page)
+{
+	return sysfs_emit(page, "%d\n", to_nvmet_ns(item)->pr.enable);
+}
+
+static ssize_t nvmet_ns_resv_enable_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_ns *ns = to_nvmet_ns(item);
+	bool val;
+
+	if (kstrtobool(page, &val))
+		return -EINVAL;
+
+	mutex_lock(&ns->subsys->lock);
+	if (ns->enabled) {
+		pr_err("the ns:%d is already enabled.\n", ns->nsid);
+		mutex_unlock(&ns->subsys->lock);
+		return -EINVAL;
+	}
+	ns->pr.enable = val;
+	mutex_unlock(&ns->subsys->lock);
+	return count;
+}
+CONFIGFS_ATTR(nvmet_ns_, resv_enable);
+
 static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_device_path,
 	&nvmet_ns_attr_device_nguid,
@@ -777,6 +803,7 @@ static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_enable,
 	&nvmet_ns_attr_buffered_io,
 	&nvmet_ns_attr_revalidate_size,
+	&nvmet_ns_attr_resv_enable,
 #ifdef CONFIG_PCI_P2PDMA
 	&nvmet_ns_attr_p2pmem,
 #endif
drivers/nvme/target/core.c +50 -6

129 lines not on this page · raw message

drivers/nvme/target/fabrics-cmd.c +1 -3
index c4b2eddd5666..28a84af1b4c0 100644
@@ -245,12 +245,10 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 	d->subsysnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	d->hostnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	status = nvmet_alloc_ctrl(d->subsysnqn, d->hostnqn, req,
-				  le32_to_cpu(c->kato), &ctrl);
+				  le32_to_cpu(c->kato), &ctrl, &d->hostid);
 	if (status)
 		goto out;
 
-	uuid_copy(&ctrl->hostid, &d->hostid);
-
 	dhchap_status = nvmet_setup_auth(ctrl);
 	if (dhchap_status) {
 		pr_err("Failed to setup authentication, dhchap status %u\n",
drivers/nvme/target/nvmet.h +54 -1

107 lines not on this page · raw message

drivers/nvme/target/pr.c new file +1162 -0

1,162 lines not on this page · raw message

Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH v14 2/2] nvmet: support reservation feature

Quoted 25 lines · Guixin Liu

This patch implements the reservation feature, including:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

Not supported:
1. persistent reservation through power loss.

Test cases:
Use nvme-cli and fio to test all implemented sub features:
1. use nvme resv-register to register host a registrant or unregister or replace a new key.
2. use nvme resv-acquire to set host to the holder, and use fio to send read and write io in all reservation type. And also test preempt and "preempt and abort".
3. use nvme resv-report to show all registrants and reservation status.
4. use nvme resv-release to release all registrants.
5. use nvme get-log to get events generated by the preceding operations.

Quoted 5 lines

In addition, make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

Once got, you may(should for reviewers) add Revived-by tag for the patches. So, please do not forget:

#7 lore raw

Re: [PATCH v14 2/2] nvmet: support reservation feature

In reply to Guixin Liu
Quoted 6 lines · Guixin Liu
drivers/nvme/target/pr.c new file +0 -0
new file mode 100644
index 000000000000..5f3b9861533e

drivers/nvme/target/pr.c line 10

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
@@ -0,0 +1,1162 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe over Fabrics Persist Reservation.
+ * Copyright (c) 2024 Guixin Liu, Alibaba Group.
+ * All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/unaligned.h>
+#include <linux/lockdep.h>
+#include "nvmet.h"

I'm running into error when trying to test this code [1], with following seems to fix it on my platformcan you check ?:-

drivers/nvme/target/pr.c +1 -1
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
   * All rights reserved.
   */
  #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
  #include <linux/lockdep.h>
  #include "nvmet.h"
 

-ck

[1]

nvme (nvme-6.12) # make M=drivers/nvme/target/
  CC [M]  drivers/nvme/target/pr.o
drivers/nvme/target/pr.c:8:10: fatal error: linux/unaligned.h: No such file or directory
    8 | #include <linux/unaligned.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [scripts/Makefile.build:229: drivers/nvme/target/pr.o] Error 1 make[1]: *** [/mnt/data/nvme/Makefile:1936: drivers/nvme/target] Error 2
make: *** [Makefile:224: __sub-make] Error 2

nvme (nvme-6.12) # git diff

drivers/nvme/target/pr.c +1 -1
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
   * All rights reserved.
   */
  #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
  #include <linux/lockdep.h>
  #include "nvmet.h"
 

nvme (nvme-6.12) # git apply pr-fix.diff

nvme (nvme-6.12) # make M=drivers/nvme/target/
  CC [M]  drivers/nvme/target/pr.o
  LD [M]  drivers/nvme/target/nvmet.o
  MODPOST drivers/nvme/target/Module.symvers
  CC [M]  drivers/nvme/target/.module-common.o
  LD [M]  drivers/nvme/target/nvmet.ko
  LD [M]  drivers/nvme/target/nvme-loop.ko
  LD [M]  drivers/nvme/target/nvmet-rdma.ko
  LD [M]  drivers/nvme/target/nvmet-fc.ko
  LD [M]  drivers/nvme/target/nvme-fcloop.ko
  LD [M]  drivers/nvme/target/nvmet-tcp.ko
nvme (nvme-6.12) #

author
#8 lore raw

Re: [PATCH v14 2/2] nvmet: support reservation feature

New 6.12-rc2 add a commit 5f60d5f6bbc12e782fac78110b0ee62698f3b576("move asm/unaligned.h to linux/unaligned.h"), I change my code to fit this.

Best Regards,

Guixin Liu

Quoted 80 lines · Chaitanya Kulkarni
Quoted 6 lines · Guixin Liu
drivers/nvme/target/pr.c new file +0 -0
new file mode 100644
index 000000000000..5f3b9861533e

drivers/nvme/target/pr.c line 10

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
@@ -0,0 +1,1162 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe over Fabrics Persist Reservation.
+ * Copyright (c) 2024 Guixin Liu, Alibaba Group.
+ * All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/unaligned.h>
+#include <linux/lockdep.h>
+#include "nvmet.h"

I'm running into error when trying to test this code [1], with following seems to fix it on my platformcan you check ?:-

drivers/nvme/target/pr.c +1 -2
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
    * All rights reserved.
    */
   #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
   #include <linux/lockdep.h>
   #include "nvmet.h"
 
-ck

[1]

nvme (nvme-6.12) # make M=drivers/nvme/target/
  CC [M]  drivers/nvme/target/pr.o
drivers/nvme/target/pr.c:8:10: fatal error: linux/unaligned.h: No such file or directory
    8 | #include <linux/unaligned.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [scripts/Makefile.build:229: drivers/nvme/target/pr.o] Error 1 make[1]: *** [/mnt/data/nvme/Makefile:1936: drivers/nvme/target] Error 2
make: *** [Makefile:224: __sub-make] Error 2

nvme (nvme-6.12) # git diff

drivers/nvme/target/pr.c +1 -1
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
    * All rights reserved.
    */
   #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
   #include <linux/lockdep.h>
   #include "nvmet.h"
 

nvme (nvme-6.12) # git apply pr-fix.diff

nvme (nvme-6.12) # make M=drivers/nvme/target/
  CC [M]  drivers/nvme/target/pr.o
  LD [M]  drivers/nvme/target/nvmet.o
  MODPOST drivers/nvme/target/Module.symvers
  CC [M]  drivers/nvme/target/.module-common.o
  LD [M]  drivers/nvme/target/nvmet.ko
  LD [M]  drivers/nvme/target/nvme-loop.ko
  LD [M]  drivers/nvme/target/nvmet-rdma.ko
  LD [M]  drivers/nvme/target/nvmet-fc.ko
  LD [M]  drivers/nvme/target/nvme-fcloop.ko
  LD [M]  drivers/nvme/target/nvmet-tcp.ko
nvme (nvme-6.12) #

#9 lore raw

Re: [PATCH v14 2/2] nvmet: support reservation feature

Quoted 8 lines · Guixin Liu

New 6.12-rc2 add a commit 5f60d5f6bbc12e782fac78110b0ee62698f3b576("move asm/unaligned.h to linux/unaligned.h"), I change my code to fit this.

Best Regards,

Guixin Liu

Let's avoid top posting if we can for everyone else on the list.

So if we merge this patch then we have to make sure before we sent the pull request above commit (f3b576) is in the tree, otherwise kernel test robot mentioned in [1] will start complaining ?

Quoted 68 lines · Chaitanya Kulkarni
Quoted 67 lines · Guixin Liu
Quoted 6 lines
drivers/nvme/target/pr.c new file +0 -0
new file mode 100644
index 000000000000..5f3b9861533e

drivers/nvme/target/pr.c line 10

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
@@ -0,0 +1,1162 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe over Fabrics Persist Reservation.
+ * Copyright (c) 2024 Guixin Liu, Alibaba Group.
+ * All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/unaligned.h>
+#include <linux/lockdep.h>
+#include "nvmet.h"

I'm running into error when trying to test this code [1], with following seems to fix it on my platformcan you check ?:-

drivers/nvme/target/pr.c +1 -2
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
     * All rights reserved.
     */
    #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
    #include <linux/lockdep.h>
    #include "nvmet.h"
 
-ck

[1]

nvme (nvme-6.12) # make M=drivers/nvme/target/
    CC [M]  drivers/nvme/target/pr.o drivers/nvme/target/pr.c:8:10: fatal error: linux/unaligned.h: No such file or directory
      8 | #include <linux/unaligned.h>
        |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [scripts/Makefile.build:229: drivers/nvme/target/pr.o] Error 1
make[1]: *** [/mnt/data/nvme/Makefile:1936: drivers/nvme/target] Error 2
make: *** [Makefile:224: __sub-make] Error 2

nvme (nvme-6.12) # git diff

drivers/nvme/target/pr.c +1 -1
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
     * All rights reserved.
     */
    #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
    #include <linux/lockdep.h>
    #include "nvmet.h"
 

nvme (nvme-6.12) # git apply pr-fix.diff

Quoted 14 lines
Quoted 4 lines

nvme (nvme-6.12) # make M=drivers/nvme/target/
    CC [M]  drivers/nvme/target/pr.o
    LD [M]  drivers/nvme/target/nvmet.o
    MODPOST drivers/nvme/target/Module.symvers

Quoted 10 lines

    CC [M]  drivers/nvme/target/.module-common.o
    LD [M]  drivers/nvme/target/nvmet.ko
    LD [M]  drivers/nvme/target/nvme-loop.ko
    LD [M]  drivers/nvme/target/nvmet-rdma.ko
    LD [M]  drivers/nvme/target/nvmet-fc.ko
    LD [M]  drivers/nvme/target/nvme-fcloop.ko
    LD [M]  drivers/nvme/target/nvmet-tcp.ko
nvme (nvme-6.12) #

-ck

[1]

E.g.
https://lists.infradead.org/pipermail/linux-nvme/2024-September/049831.html

author
#10 lore raw

Re: [PATCH v14 2/2] nvmet: support reservation feature

Quoted 10 lines · Chaitanya Kulkarni
Quoted 8 lines · Guixin Liu

New 6.12-rc2 add a commit 5f60d5f6bbc12e782fac78110b0ee62698f3b576("move asm/unaligned.h to linux/unaligned.h"), I change my code to fit this.

Best Regards,

Guixin Liu

Let's avoid top posting if we can for everyone else on the list.

Oh, sorry.

Quoted 3 lines

So if we merge this patch then we have to make sure before we sent the pull request above commit (f3b576) is in the tree, otherwise kernel test robot mentioned in [1] will start complaining ?

Yes, commit (f3b576) changed all "#include <asm/unaligned.h>" to "#include <linux/unaligned.h>",

I think we should do a rebase before sending the pull request.

Best Regards,

Guixin Liu

Quoted 90 lines
Quoted 68 lines · Chaitanya Kulkarni
Quoted 67 lines · Guixin Liu
Quoted 6 lines
drivers/nvme/target/pr.c new file +0 -0
new file mode 100644
index 000000000000..5f3b9861533e

drivers/nvme/target/pr.c line 10

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
@@ -0,0 +1,1162 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe over Fabrics Persist Reservation.
+ * Copyright (c) 2024 Guixin Liu, Alibaba Group.
+ * All rights reserved.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/unaligned.h>
+#include <linux/lockdep.h>
+#include "nvmet.h"

I'm running into error when trying to test this code [1], with following seems to fix it on my platformcan you check ?:-

drivers/nvme/target/pr.c +1 -2
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
     * All rights reserved.
     */
    #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
    #include <linux/lockdep.h>
    #include "nvmet.h"
 
-ck

[1]

nvme (nvme-6.12) # make M=drivers/nvme/target/
    CC [M]  drivers/nvme/target/pr.o
drivers/nvme/target/pr.c:8:10: fatal error: linux/unaligned.h: No such file or directory
      8 | #include <linux/unaligned.h>
        |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [scripts/Makefile.build:229: drivers/nvme/target/pr.o] Error 1
make[1]: *** [/mnt/data/nvme/Makefile:1936: drivers/nvme/target] Error 2
make: *** [Makefile:224: __sub-make] Error 2

nvme (nvme-6.12) # git diff

drivers/nvme/target/pr.c +1 -1
index 5f3b9861533e..000c83fbda52 100644
@@ -5,7 +5,7 @@
     * All rights reserved.
     */
    #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/unaligned.h>
+#include <asm-generic/unaligned.h>
    #include <linux/lockdep.h>
    #include "nvmet.h"
 

nvme (nvme-6.12) # git apply pr-fix.diff

Quoted 14 lines
Quoted 4 lines

nvme (nvme-6.12) # make M=drivers/nvme/target/
    CC [M]  drivers/nvme/target/pr.o
    LD [M]  drivers/nvme/target/nvmet.o
    MODPOST drivers/nvme/target/Module.symvers

Quoted 10 lines

    CC [M]  drivers/nvme/target/.module-common.o
    LD [M]  drivers/nvme/target/nvmet.ko
    LD [M]  drivers/nvme/target/nvme-loop.ko
    LD [M]  drivers/nvme/target/nvmet-rdma.ko
    LD [M]  drivers/nvme/target/nvmet-fc.ko
    LD [M]  drivers/nvme/target/nvme-fcloop.ko
    LD [M]  drivers/nvme/target/nvmet-tcp.ko
nvme (nvme-6.12) #

-ck

[1]

E.g.
https://lists.infradead.org/pipermail/linux-nvme/2024-September/049831.html

#11 lore raw

Re: [PATCH v14 2/2] nvmet: support reservation feature

In reply to Guixin Liu
Quoted 2 lines · Guixin Liu

New 6.12-rc2 add a commit 5f60d5f6bbc12e782fac78110b0ee62698f3b576("move asm/unaligned.h to linux/unaligned.h"), I change my code to fit this.

Requiring a -rc2 base for the next merge window is perfectly normal.

Tested-by Reviewed-by on v14 2/2
#12 lore raw

Re: [PATCH v14 2/2] nvmet: support reservation feature

In reply to Guixin Liu
Quoted 24 lines · Guixin Liu

This patch implements the reservation feature, including:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

Not supported:
1. persistent reservation through power loss.

Test cases:
Use nvme-cli and fio to test all implemented sub features:
1. use nvme resv-register to register host a registrant or unregister or replace a new key.
2. use nvme resv-acquire to set host to the holder, and use fio to send read and write io in all reservation type. And also test preempt and "preempt and abort".
3. use nvme resv-report to show all registrants and reservation status.
4. use nvme resv-release to release all registrants.
5. use nvme get-log to get events generated by the preceding operations.

Quoted 5 lines

In addition, make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

Looks good.

-ck

blktests (master) # ./check nvme
nvme/002 (tr=loop) (create many subsystems and test discovery) [passed]
    runtime  29.316s  ...  28.922s
nvme/003 (tr=loop) (test if we're sending keep-alives to a discovery controller) [passed]
    runtime  11.222s  ...  11.247s
nvme/004 (tr=loop) (test nvme and nvmet UUID NS descriptors) [passed]
    runtime  0.717s  ...  0.712s
nvme/005 (tr=loop) (reset local loopback target) [passed]
    runtime  1.287s  ...  1.258s
nvme/006 (tr=loop bd=device) (create an NVMeOF target) [passed]
    runtime  0.077s  ...  0.073s
nvme/006 (tr=loop bd=file) (create an NVMeOF target) [passed]
    runtime  0.049s  ...  0.048s
nvme/008 (tr=loop bd=device) (create an NVMeOF host) [passed]
    runtime  0.746s  ...  0.709s
nvme/008 (tr=loop bd=file) (create an NVMeOF host) [passed]
    runtime  0.682s  ...  0.701s
nvme/010 (tr=loop bd=device) (run data verification fio job) [passed]
    runtime  17.291s  ...  48.777s
nvme/010 (tr=loop bd=file) (run data verification fio job) [passed]
    runtime  32.528s  ...  109.749s
nvme/012 (tr=loop bd=device) (run mkfs and data verification fio) [passed]
    runtime  18.424s  ...  41.073s
nvme/012 (tr=loop bd=file) (run mkfs and data verification fio) [passed]
    runtime  38.336s  ...  90.011s
nvme/014 (tr=loop bd=device) (flush a command from host) [passed]
    runtime  6.030s  ...  7.619s
nvme/014 (tr=loop bd=file) (flush a command from host) [passed]
    runtime  4.471s  ...  6.199s
nvme/016 (tr=loop) (create/delete many NVMeOF block device-backed ns and test discovery) [passed]
    runtime  19.666s  ...  19.725s
nvme/017 (tr=loop) (create/delete many file-ns and test discovery) [passed]
    runtime  20.157s  ...  19.392s
nvme/018 (tr=loop) (unit test NVMe-oF out of range access on a file backend) [passed]
    runtime  0.703s  ...  0.684s
nvme/019 (tr=loop bd=device) (test NVMe DSM Discard command) [passed]
    runtime  0.715s  ...  0.685s
nvme/019 (tr=loop bd=file) (test NVMe DSM Discard command) [passed]
    runtime  0.688s  ...  0.679s
nvme/021 (tr=loop bd=device) (test NVMe list command) [passed]
    runtime  0.728s  ...  0.699s
nvme/021 (tr=loop bd=file) (test NVMe list command) [passed]
    runtime  0.692s  ...  0.687s
nvme/022 (tr=loop bd=device) (test NVMe reset command) [passed]
    runtime  1.272s  ...  1.260s
nvme/022 (tr=loop bd=file) (test NVMe reset command) [passed]
    runtime  1.267s  ...  1.250s
nvme/023 (tr=loop bd=device) (test NVMe smart-log command) [passed]
    runtime  0.722s  ...  0.686s
nvme/023 (tr=loop bd=file) (test NVMe smart-log command) [passed]
    runtime  0.702s  ...  0.671s
nvme/025 (tr=loop bd=device) (test NVMe effects-log) [passed]
    runtime  0.548s  ...  0.692s
nvme/025 (tr=loop bd=file) (test NVMe effects-log) [passed]
    runtime  0.694s  ...  0.681s
nvme/026 (tr=loop bd=device) (test NVMe ns-descs) [passed]
    runtime  0.717s  ...  0.675s
nvme/026 (tr=loop bd=file) (test NVMe ns-descs) [passed]
    runtime  0.703s  ...  0.666s
nvme/027 (tr=loop bd=device) (test NVMe ns-rescan command) [passed]
    runtime  0.748s  ...  0.728s
nvme/027 (tr=loop bd=file) (test NVMe ns-rescan command) [passed]
    runtime  0.724s  ...  0.705s
nvme/028 (tr=loop bd=device) (test NVMe list-subsys) [passed]
    runtime  0.733s  ...  0.683s
nvme/028 (tr=loop bd=file) (test NVMe list-subsys) [passed]
    runtime  0.699s  ...  0.675s
nvme/029 (tr=loop) (test userspace IO via nvme-cli read/write interface) [passed]
    runtime  0.808s  ...  0.807s
nvme/030 (tr=loop) (ensure the discovery generation counter is updated appropriately) [passed]
    runtime  0.299s  ...  0.317s
nvme/031 (tr=loop) (test deletion of NVMeOF controllers immediately after setup) [passed]
    runtime  6.820s  ...  6.698s
nvme/038 (tr=loop) (test deletion of NVMeOF subsystem without enabling) [passed]
    runtime  0.012s  ...  0.012s
nvme/040 (tr=loop) (test nvme fabrics controller reset/disconnect operation during I/O) [passed]
    runtime  7.129s  ...  7.542s
nvme/041 (tr=loop) (Create authenticated connections) [passed]
    runtime  2.070s  ...  2.019s
nvme/042 (tr=loop) (Test dhchap key types for authenticated connections) [passed]
    runtime  6.337s  ...  6.198s
nvme/043 (tr=loop) (Test hash and DH group variations for authenticated connections) [passed]
    runtime  25.100s  ...  24.496s
nvme/044 (tr=loop) (Test bi-directional authentication) [passed]
    runtime  4.452s  ...  4.387s
nvme/045 (tr=loop) (Test re-authentication) [passed]
    runtime  1.678s  ...  1.817s
common/rc: line 213: 0vme: value too great for base (error token is "0vme") nvme/048 (tr=loop) (Test queue count changes on reconnect)   [not run]
    nvme_trtype=loop is not supported in this test
nvme/051 (tr=loop) (test nvmet concurrent ns enable/disable) [passed]
    runtime  2.238s  ...  2.306s
nvme/052 (tr=loop) (Test file-ns creation/deletion under one subsystem) [passed]
    runtime  20.621s  ...  5.493s
nvme/054 (Test the NVMe reservation feature) [passed]
    runtime  0.739s  ...  0.724s
blktests (master) # nvme_trtype=tcp ./check nvme
nvme/002 (tr=tcp) (create many subsystems and test discovery) [not run]
    nvme_trtype=tcp is not supported in this test
nvme/003 (tr=tcp) (test if we're sending keep-alives to a discovery controller) [passed]
    runtime  11.262s  ...  11.241s
nvme/004 (tr=tcp) (test nvme and nvmet UUID NS descriptors) [passed]
    runtime  0.261s  ...  0.179s
nvme/005 (tr=tcp) (reset local loopback target) [passed]
    runtime  0.311s  ...  0.244s
nvme/006 (tr=tcp bd=device) (create an NVMeOF target) [passed]
    runtime  0.090s  ...  0.069s
nvme/006 (tr=tcp bd=file) (create an NVMeOF target) [passed]
    runtime  0.072s  ...  0.051s
nvme/008 (tr=tcp bd=device) (create an NVMeOF host) [passed]
    runtime  0.259s  ...  0.194s
nvme/008 (tr=tcp bd=file) (create an NVMeOF host) [passed]
    runtime  0.192s  ...  0.181s
nvme/010 (tr=tcp bd=device) (run data verification fio job) [passed]
    runtime  62.326s  ...  53.529s
nvme/010 (tr=tcp bd=file) (run data verification fio job) [passed]
    runtime  95.985s  ...  106.414s
nvme/012 (tr=tcp bd=device) (run mkfs and data verification fio) [passed]
    runtime  63.920s  ...  53.094s
nvme/012 (tr=tcp bd=file) (run mkfs and data verification fio) [passed]
    runtime  105.927s  ...  83.156s
nvme/014 (tr=tcp bd=device) (flush a command from host) [passed]
    runtime  8.228s  ...  6.341s
nvme/014 (tr=tcp bd=file) (flush a command from host) [passed]
    runtime  6.434s  ...  5.502s
nvme/016 (tr=tcp) (create/delete many NVMeOF block device-backed ns and test discovery) [not run]
    nvme_trtype=tcp is not supported in this test
nvme/017 (tr=tcp) (create/delete many file-ns and test discovery) [not run]
    nvme_trtype=tcp is not supported in this test
nvme/018 (tr=tcp) (unit test NVMe-oF out of range access on a file backend) [passed]
    runtime  0.213s  ...  0.202s
nvme/019 (tr=tcp bd=device) (test NVMe DSM Discard command) [passed]
    runtime  0.250s  ...  0.170s
nvme/019 (tr=tcp bd=file) (test NVMe DSM Discard command) [passed]
    runtime  0.210s  ...  0.168s
nvme/021 (tr=tcp bd=device) (test NVMe list command) [passed]
    runtime  0.246s  ...  0.215s
nvme/021 (tr=tcp bd=file) (test NVMe list command) [passed]
    runtime  0.204s  ...  0.160s
nvme/022 (tr=tcp bd=device) (test NVMe reset command) [passed]
    runtime  0.320s  ...  0.213s
nvme/022 (tr=tcp bd=file) (test NVMe reset command) [passed]
    runtime  0.293s  ...  0.208s
nvme/023 (tr=tcp bd=device) (test NVMe smart-log command) [passed]
    runtime  0.236s  ...  0.180s
nvme/023 (tr=tcp bd=file) (test NVMe smart-log command) [passed]
    runtime  0.224s  ...  0.175s
nvme/025 (tr=tcp bd=device) (test NVMe effects-log) [passed]
    runtime  0.254s  ...  0.184s
nvme/025 (tr=tcp bd=file) (test NVMe effects-log) [passed]
    runtime  0.225s  ...  0.168s
nvme/026 (tr=tcp bd=device) (test NVMe ns-descs) [passed]
    runtime  0.239s  ...  0.159s
nvme/026 (tr=tcp bd=file) (test NVMe ns-descs) [passed]
    runtime  0.230s  ...  0.157s
nvme/027 (tr=tcp bd=device) (test NVMe ns-rescan command) [passed]
    runtime  0.277s  ...  0.195s
nvme/027 (tr=tcp bd=file) (test NVMe ns-rescan command) [passed]
    runtime  0.260s  ...  0.195s
nvme/028 (tr=tcp bd=device) (test NVMe list-subsys) [passed]
    runtime  0.246s  ...  0.176s
nvme/028 (tr=tcp bd=file) (test NVMe list-subsys) [passed]
    runtime  0.232s  ...  0.160s
nvme/029 (tr=tcp) (test userspace IO via nvme-cli read/write interface) [passed]
    runtime  0.406s  ...  0.293s
nvme/030 (tr=tcp) (ensure the discovery generation counter is updated appropriately) [passed]
    runtime  0.169s  ...  0.135s
nvme/031 (tr=tcp) (test deletion of NVMeOF controllers immediately after setup) [passed]
    runtime  2.121s  ...  1.639s
nvme/038 (tr=tcp) (test deletion of NVMeOF subsystem without enabling) [passed]
    runtime  0.022s  ...  0.015s
nvme/040 (tr=tcp) (test nvme fabrics controller reset/disconnect operation during I/O) [passed]
    runtime  6.289s  ...  6.239s
nvme/041 (tr=tcp) (Create authenticated connections) [passed]
    runtime  1.710s  ...  1.542s
nvme/042 (tr=tcp) (Test dhchap key types for authenticated connections) [passed]
    runtime  3.992s  ...  2.755s
nvme/043 (tr=tcp) (Test hash and DH group variations for authenticated connections) [passed]
    runtime  19.801s  ...  17.088s
nvme/044 (tr=tcp) (Test bi-directional authentication) [passed]
    runtime  3.609s  ...  3.385s
nvme/045 (tr=tcp) (Test re-authentication) [passed]
    runtime  1.440s  ...  1.230s
common/rc: line 213: 0vme: value too great for base (error token is "0vme") nvme/048 (tr=tcp) (Test queue count changes on reconnect) [passed]
    runtime  6.308s  ...  5.221s
nvme/051 (tr=tcp) (test nvmet concurrent ns enable/disable) [passed]
    runtime    ...  2.589s
nvme/052 (tr=tcp) (Test file-ns creation/deletion under one subsystem) [not run]
    runtime  2.234s  ...
    nvme_trtype=tcp is not supported in this test
nvme/054 (Test the NVMe reservation feature) [passed]
    runtime  0.724s  ...  0.511s
blktests (master) #