Skip to contentSkip to conversation

Implement the NVMe reservation feature

PATCH Superseded Guixin Liu 5 participants 12 messages Feb 29, 2024 – Sep 19, 2024

Subsystems (2)

v8 of 19 Superseded Latest is v19 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v8 cover Implement the NVMe reservation feature

Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.

Changes from v7 to v8:
- Add me as the new file pr.c's maintainer.

Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.

- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.

- Fix the resv log page is random when there is no resv log page.

- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().

- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().

- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().

- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().

- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().

- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.

Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().

- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().

- Add the print fmt to the head.

- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.

- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.

Changes from v4 to v5:

- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.

- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.

- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.

- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.

- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.

- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.

- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.

- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.

Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.

- Change the function calling alignment style to:

    nvmet_pr_send_event_by_hostid(pr, hostid,
            NVME_PR_LOG_RESERVATOPM_PREEMPTED); 

- Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().

- Remove the goto in __nvmet_pr_unregister_one().

- Change generation to atomic_t, and remove nvmet_pr_inc_generation().

- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.

Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.

- Limit the resv_log_list to 128.

- Change generation to atomic64.

- Put register rkey check to a warpper.

- Change nr_avl_pages to nr_pages.

- Use NVME_SC_SUCCESS instead of 0.

- Change kmalloc param to let it not sleep in mutex lock.

Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.

- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is
0xffffffff at each admin command path, if it is needed.

- Add reservation commands support in nvmet_get_cmd_effects_nvm().

- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.

- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.

- Remove kmem_cache instead to use kmalloc and kfree.

- Change others advice from Sagi.

- Add a blktest test case, this patch will be sent before these series of patches.

Guixin Liu (1):
nvmet: support reservation feature

 MAINTAINERS                     |    6 +
 drivers/nvme/target/Makefile    |    2 +-
 drivers/nvme/target/admin-cmd.c |   20 +-
 drivers/nvme/target/configfs.c  |   27 +
 drivers/nvme/target/core.c      |   51 +-
 drivers/nvme/target/nvmet.h     |   36 ++
 drivers/nvme/target/pr.c        | 1041 +++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   54 ++
 8 files changed, 1230 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v8 1/1 nvmet: support reservation feature

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---
 MAINTAINERS                     |    6 +
 drivers/nvme/target/Makefile    |    2 +-
 drivers/nvme/target/admin-cmd.c |   20 +-
 drivers/nvme/target/configfs.c  |   27 +
 drivers/nvme/target/core.c      |   51 +-
 drivers/nvme/target/nvmet.h     |   36 ++
 drivers/nvme/target/pr.c        | 1041 +++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   54 ++
 8 files changed, 1230 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
MAINTAINERS +6 -0
index 8999497011a2..8e8162736233 100644
@@ -15660,6 +15660,12 @@ W:	http://git.infradead.org/nvme.git
 T:	git git://git.infradead.org/nvme.git
 F:	drivers/nvme/target/
 
+NVM EXPRESS TARGET PERSISTENT RESERVATION
+M:	Guixin Liu <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/nvme/target/pr.c
+
 NVMEM FRAMEWORK
 M:	Srinivas Kandagatla <[email protected]>
 S:	Maintained
drivers/nvme/target/Makefile +1 -1
index c66820102493..f9bfc904a5b3 100644
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_TARGET_FCLOOP)	+= nvme-fcloop.o
 obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
-			discovery.o io-cmd-file.o io-cmd-bdev.o
+			discovery.o io-cmd-file.o io-cmd-bdev.o pr.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
 nvmet-$(CONFIG_NVME_TARGET_AUTH)	+= fabrics-cmd-auth.o auth.o
drivers/nvme/target/admin-cmd.c +19 -1
index 39cb570f833d..eb1d85701441 100644
@@ -176,6 +176,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 	log->iocs[nvme_cmd_read] =
 	log->iocs[nvme_cmd_flush] =
 	log->iocs[nvme_cmd_dsm]	=
+	log->iocs[nvme_cmd_resv_acquire] =
+	log->iocs[nvme_cmd_resv_register] =
+	log->iocs[nvme_cmd_resv_release] =
+	log->iocs[nvme_cmd_resv_report] =
 		cpu_to_le32(NVME_CMD_EFFECTS_CSUPP);
 	log->iocs[nvme_cmd_write] =
 	log->iocs[nvme_cmd_write_zeroes] =
@@ -340,6 +344,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		return nvmet_execute_get_log_cmd_effects_ns(req);
 	case NVME_LOG_ANA:
 		return nvmet_execute_get_log_page_ana(req);
+	case NVME_LOG_RESERVATION:
+		return nvmet_execute_get_log_page_resv(req);
 	}
 	pr_debug("unhandled lid %d on qid %d\n",
 	       req->cmd->get_log_page.lid, req->sq->qid);
@@ -550,7 +556,13 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
 	 */
 	id->nmic = NVME_NS_NMIC_SHARED;
 	id->anagrpid = cpu_to_le32(req->ns->anagrpid);
-
+	id->rescap = NVME_PR_SUPPORT_WRITE_EXCLUSIVE |
+		     NVME_PR_SUPPORT_EXCLUSIVE_ACCESS |
+		     NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY |
+		     NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY |
+		     NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS |
+		     NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS |
+		     NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF;
 	memcpy(&id->nguid, &req->ns->nguid, sizeof(id->nguid));
 
 	id->lbaf[0].ds = req->ns->blksize_shift;
@@ -851,6 +863,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_set_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_set_feat_resv_notif_mask(req, cdw11);
+		break;
 	default:
 		req->error_loc = offsetof(struct nvme_common_command, cdw10);
 		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
@@ -949,6 +964,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_get_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_get_feat_resv_notif_mask(req);
+		break;
 	default:
 		req->error_loc =
 			offsetof(struct nvme_common_command, cdw10);
drivers/nvme/target/configfs.c +27 -0
index 2482a0db2504..56aead89d21a 100644
@@ -714,6 +714,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,
 
 CONFIGFS_ATTR_WO(nvmet_ns_, revalidate_size);
 
+static ssize_t nvmet_ns_resv_enable_show(struct config_item *item, char *page)
+{
+	return sprintf(page, "%d\n", to_nvmet_ns(item)->pr.enable);
+}
+
+static ssize_t nvmet_ns_resv_enable_store(struct config_item *item,
+					const char *page, size_t count)
+{
+	struct nvmet_ns *ns = to_nvmet_ns(item);
+	bool val;
+
+	if (kstrtobool(page, &val))
+		return -EINVAL;
+
+	mutex_lock(&ns->subsys->lock);
+	if (ns->enabled) {
+		pr_err("the ns:%d is already enabled.\n", ns->nsid);
+		mutex_unlock(&ns->subsys->lock);
+		return -EINVAL;
+	}
+	ns->pr.enable = val;
+	mutex_unlock(&ns->subsys->lock);
+	return count;
+}
+CONFIGFS_ATTR(nvmet_ns_, resv_enable);
+
 static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_device_path,
 	&nvmet_ns_attr_device_nguid,
@@ -722,6 +748,7 @@ static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_enable,
 	&nvmet_ns_attr_buffered_io,
 	&nvmet_ns_attr_revalidate_size,
+	&nvmet_ns_attr_resv_enable,
 #ifdef CONFIG_PCI_P2PDMA
 	&nvmet_ns_attr_p2pmem,
 #endif
drivers/nvme/target/core.c +46 -5

94 lines not on this page · raw message

drivers/nvme/target/nvmet.h +36 -0
index 6c8acebe1a1a..4f8416e7353d 100644
@@ -20,6 +20,7 @@
 #include <linux/blkdev.h>
 #include <linux/radix-tree.h>
 #include <linux/t10-pi.h>
+#include <linux/kfifo.h>
 
 #define NVMET_DEFAULT_VS		NVME_VS(1, 3, 0)
 
@@ -30,6 +31,7 @@
 #define NVMET_MN_MAX_SIZE		40
 #define NVMET_SN_MAX_SIZE		20
 #define NVMET_FR_MAX_SIZE		8
+#define NVMET_PR_LOG_QUEUE_SIZE		64
 
 /*
  * Supported optional AENs:
@@ -56,6 +58,22 @@
 #define IPO_IATTR_CONNECT_SQE(x)	\
 	(cpu_to_le32(offsetof(struct nvmf_connect_command, x)))
 
+struct nvmet_pr_registrant {
+	u64			rkey;
+	uuid_t			hostid;
+	enum nvme_pr_type	rtype;
+	struct list_head	entry;
+};
+
+struct nvmet_pr {
+	bool			enable;
+	unsigned long		notify_mask;
+	atomic_t		generation;
+	struct nvmet_pr_registrant __rcu *holder;
+	struct mutex		pr_lock;
+	struct list_head	registrant_list;
+};
+
 struct nvmet_ns {
 	struct percpu_ref	ref;
 	struct bdev_handle	*bdev_handle;
@@ -85,6 +103,7 @@ struct nvmet_ns {
 	int			pi_type;
 	int			metadata_size;
 	u8			csi;
+	struct nvmet_pr		pr;
 };
 
 static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item)
@@ -190,6 +209,13 @@ static inline bool nvmet_port_secure_channel_required(struct nvmet_port *port)
     return nvmet_port_disc_addr_treq_secure_channel(port) == NVMF_TREQ_REQUIRED;
 }
 
+struct nvmet_pr_log_mgr {
+	struct mutex		lock;
+	u64			lost_count;
+	u64			counter;
+	DECLARE_KFIFO(log_queue, struct nvme_pr_log, NVMET_PR_LOG_QUEUE_SIZE);
+};
+
 struct nvmet_ctrl {
 	struct nvmet_subsys	*subsys;
 	struct nvmet_sq		**sqs;
@@ -243,6 +269,7 @@ struct nvmet_ctrl {
 	u8			*dh_key;
 	size_t			dh_keysize;
 #endif
+	struct nvmet_pr_log_mgr pr_log_mgr;
 };
 
 struct nvmet_subsys {
@@ -750,4 +777,13 @@ static inline bool nvmet_has_auth(struct nvmet_ctrl *ctrl)
 static inline const char *nvmet_dhchap_dhgroup_name(u8 dhgid) { return NULL; }
 #endif
 
+void nvmet_pr_init_ns(struct nvmet_ns *ns);
+u16 nvmet_parse_pr_cmd(struct nvmet_req *req);
+u16 nvmet_pr_check_cmd_access(struct nvmet_req *req);
+void nvmet_ctrl_destroy_pr(struct nvmet_ctrl *ctrl);
+void nvmet_pr_exit_ns(struct nvmet_ns *ns);
+bool nvmet_is_host_connected(uuid_t *hostid, struct nvmet_subsys *subsys);
+void nvmet_execute_get_log_page_resv(struct nvmet_req *req);
+u16 nvmet_set_feat_resv_notif_mask(struct nvmet_req *req, u32 mask);
+u16 nvmet_get_feat_resv_notif_mask(struct nvmet_req *req);
 #endif /* _NVMET_H */
drivers/nvme/target/pr.c new file +1041 -0

1,041 lines not on this page · raw message

include/linux/nvme.h +54 -0

72 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v8 1/1] nvmet: support reservation feature

Hi,

Waiting for the final solution half an year we taken this patch as is. Here is my comments on it. Hope, this will bring the life to the patch.

Quoted 14 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---

drivers/nvme/target/admin-cmd.c +0 -0
@@ -550,7 +556,13 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
         */

drivers/nvme/target/admin-cmd.c static void nvmet_execute_identify_ns(struct nvmet_req *req) · line 565

Quoted 10 lines
drivers/nvme/target/admin-cmd.c +7 -1
        id->nmic = NVME_NS_NMIC_SHARED;
        id->anagrpid = cpu_to_le32(req->ns->anagrpid);
-
+       id->rescap = NVME_PR_SUPPORT_WRITE_EXCLUSIVE |
+                    NVME_PR_SUPPORT_EXCLUSIVE_ACCESS |
+                    NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY |
+                    NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY |
+                    NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS |
+                    NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS |
+                    NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF;

You have a flag enabling PR on a namespace, so report rescap only if it is enabled.

Also I didnot find ONCS, is it not necessarily?

Quoted 23 lines
drivers/nvme/target/pr.c new file +23 -0
+void nvmet_execute_get_log_page_resv(struct nvmet_req *req)
+{
+       struct nvmet_pr_log_mgr *log_mgr = &req->sq->ctrl->pr_log_mgr;
+       struct nvme_pr_log next_log = {0};
+       struct nvme_pr_log log = {0};
+       u16 status = NVME_SC_SUCCESS;
+       u64 lost_count;
+       u64 cur_count;
+       u64 next_count;
+
+       mutex_lock(&log_mgr->lock);
+       if (!kfifo_get(&log_mgr->log_queue, &log))
+               goto out;
+
+       /*
+        * We can't get the last in kfifo.
+        * Utilize the current count and the count from the next log to
+        * calculate the number of lost logs, while also addressing cases
+        * of overflow. If there is no subsequent log, the number of lost
+        * logs is equal to the lost_count within the nvmet_pr_log_mgr.
+        */
+       cur_count = le64_to_cpu(log.count);
+       if (kfifo_peek(&log_mgr->log_queue, &next_log)) {

drivers/nvme/target/pr.c line 144

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+               next_count = le64_to_cpu(next_log.count);
+               if (next_count > cur_count)
+                       lost_count = next_count - cur_count - 1;
+               else
+                       lost_count = U64_MAX - cur_count + next_count - 1;
+       } else {
+               lost_count = log_mgr->lost_count;
+       }
+
+       log.count = cpu_to_le64(cur_count + lost_count);

This value should be wrapped by U64_MAX too but in reverse direction (count is next_count-1). If == 0 then = -1.

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+static void nvmet_pr_add_resv_log(struct nvmet_ctrl *ctrl, u8 log_type, u32 nsid)
+{
+       struct nvmet_pr_log_mgr *log_mgr = &ctrl->pr_log_mgr;
+       struct nvme_pr_log log = {0};
+
+       mutex_lock(&log_mgr->lock);
+       log_mgr->counter++;

drivers/nvme/target/pr.c line 171

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+       if (log_mgr->counter == 0)
+               log_mgr->counter = 1;
+
+       log.count = cpu_to_le64(log_mgr->counter);
+       log.type = log_type;
+       log.nsid = cpu_to_le32(nsid);
+
+       if (!kfifo_put(&log_mgr->log_queue, log)) {
+               pr_warn("a reservation log lost, cntlid:%d, log_type:%d, nsid:%d\n",
+                       ctrl->cntlid, log_type, nsid);

I think this log should be Info level, because it is not an error situation. Size of this queue is fixed and it is obvious that it can be overflowed.

drivers/nvme/target/pr.c line 244

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+static u16 nvmet_pr_create_new_reservation(struct nvmet_pr *pr, u8 new_rtype,
+                                          struct nvmet_pr_registrant *reg)
+{
+       u16 status;
+
+       status = nvmet_pr_validate_rtype(new_rtype);

You shall check the validity of the command in the very beginning of command handling, not at the very end.

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+       if (!status) {
+               reg->rtype = new_rtype;
+               rcu_assign_pointer(pr->holder, reg);
+       }
+       return status;
+}
+

...

Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+static u16 nvmet_pr_unreg_by_prkey(struct nvmet_pr *pr, u64 prkey,
+                                  uuid_t *send_hostid)
+{
+       u16 status = NVME_SC_RESERVATION_CONFLICT | NVME_SC_DNR;
+       struct nvmet_pr_registrant *reg, *tmp;

drivers/nvme/target/pr.c line 489

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+       uuid_t hostid;
+
+       lockdep_assert_held(&pr->pr_lock);
+
+       list_for_each_entry_safe(reg, tmp, &pr->registrant_list, entry) {
+               if (reg->rkey == prkey) {
+                       status = NVME_SC_SUCCESS;
+                       uuid_copy(&hostid, &reg->hostid);
+                       nvmet_pr_unregister_one(pr, reg);
+                       if (!uuid_equal(&reg->hostid, send_hostid))

reg is free already, use just hostid as line below:

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+                               nvmet_pr_registration_preempted(pr, &hostid);
+               }
+       }
+       return status;
+}
+

...

Quoted 15 lines
drivers/nvme/target/pr.c new file +15 -0
+static u16 nvmet_pr_preempt(struct nvmet_req *req,
+                           struct nvmet_pr_registrant *reg,
+                           u8 rtype,
+                           struct nvmet_pr_acquire_data *d)
+{
+       struct nvmet_ctrl *ctrl = req->sq->ctrl;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       enum nvme_pr_type original_rtype;
+       u16 status;
+
+       lockdep_assert_held(&pr->pr_lock);
+       holder = rcu_dereference_protected(pr->holder,
+                       lockdep_is_held(&pr->pr_lock));
+       if (!holder)

drivers/nvme/target/pr.c line 572

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+               return nvmet_pr_unreg_by_prkey(pr, le64_to_cpu(d->prkey),
+                               &ctrl->hostid);
+
+       original_rtype = holder->rtype;
+       if (original_rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+           original_rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS) {
+               if (!le64_to_cpu(d->prkey)) {
+                       nvmet_pr_unreg_except_hostid(pr, &ctrl->hostid);
+                       return nvmet_pr_create_new_reservation(pr,
+                                       rtype, reg);

There is not an atomic change of the reservation here. Probably if you swap these lines with each other, it will fix that.

Quoted 9 lines
drivers/nvme/target/pr.c new file +9 -0
+               }
+               return nvmet_pr_unreg_by_prkey(pr, le64_to_cpu(d->prkey),
+                               &ctrl->hostid);
+       }
+
+       if (holder == reg) {
+               status = nvmet_pr_update_reg_attr(pr, holder,
+                               nvmet_pr_update_holder_rtype, &rtype);
+               if (!status && original_rtype != rtype)

drivers/nvme/target/pr.c line 591

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+                       nvmet_pr_resv_released(pr, &reg->hostid);
+               return status;
+       }
+
+       if (le64_to_cpu(d->prkey) == holder->rkey) {
+               status = nvmet_pr_unreg_by_prkey_except_hostid(pr,
+                               le64_to_cpu(d->prkey), &ctrl->hostid);
+               if (status)
+                       return status;
+               status = nvmet_pr_create_new_reservation(pr, rtype, reg);

Here that time gap too.

Quoted 11 lines
drivers/nvme/target/pr.c new file +11 -0
+               if (!status && original_rtype != rtype)
+                       nvmet_pr_resv_released(pr, &reg->hostid);
+               return status;
+       }
+
+       if (le64_to_cpu(d->prkey))
+               return nvmet_pr_unreg_by_prkey(pr, le64_to_cpu(d->prkey),
+                               &ctrl->hostid);
+       return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+}
+

...

drivers/nvme/target/pr.c line 996

Quoted 9 lines
drivers/nvme/target/pr.c new file +9 -0
+void nvmet_ctrl_destroy_pr(struct nvmet_ctrl *ctrl)
+{
+       struct nvmet_pr_registrant *reg;
+       struct nvmet_ns *ns;
+       unsigned long idx;
+
+       kfifo_free(&ctrl->pr_log_mgr.log_queue);
+       mutex_destroy(&ctrl->pr_log_mgr.lock);
+

From here and until the end of the function it is against the standard.

Quoted 4 lines
drivers/nvme/target/pr.c new file +4 -0
+       if (nvmet_is_host_connected(&ctrl->hostid, ctrl->subsys))
+               return;
+
+       xa_for_each(&ctrl->subsys->namespaces, idx, ns) {

drivers/nvme/target/pr.c line 1010

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+               mutex_lock(&ns->pr.pr_lock);
+               list_for_each_entry_rcu(reg, &ns->pr.registrant_list, entry,
+                                       lockdep_is_held(&ns->pr.pr_lock)) {
+                       if (uuid_equal(&reg->hostid, &ctrl->hostid)) {
+                               nvmet_pr_unregister_one(&ns->pr, reg);
+                               break;
+                       }
+               }
+               mutex_unlock(&ns->pr.pr_lock);
+       }

You shall not modify Reservation due to host connectivity.

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+}
+
+void nvmet_pr_exit_ns(struct nvmet_ns *ns)
+{
+       struct nvmet_pr_registrant *reg, *tmp;
+       struct nvmet_pr *pr = &ns->pr;

drivers/nvme/target/pr.c line 1026

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+       LIST_HEAD(free_list);
+
+       mutex_lock(&pr->pr_lock);
+       rcu_assign_pointer(pr->holder, NULL);
+       list_for_each_entry_safe(reg, tmp, &pr->registrant_list, entry) {
+               list_del_rcu(&reg->entry);
+               list_add(&reg->entry, &free_list);
+       }
+       mutex_unlock(&pr->pr_lock);
+       synchronize_rcu();

IMHO, there is no need in two lists and synchronize_rcu here. At this moment there shall be no other users of ns.

drivers/nvme/target/pr.c line 1032

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+       list_for_each_entry_safe(reg, tmp, &free_list, entry) {
+               kfree(reg);
+       }
+
+       mutex_destroy(&pr->pr_lock);
+}

BR,
Dmitry

author
#3 lore raw

Re: [PATCH v8 1/1] nvmet: support reservation feature

Hi Dmitry,

    My thanks for your advice, I will work on this back recently.

Quoted 15 lines · Dmitry Bogdanov

Hi,

Waiting for the final solution half an year we taken this patch as is. Here is my comments on it. Hope, this will bring the life to the patch.

Quoted 9 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

Quoted 16 lines
Quoted 4 lines

---

drivers/nvme/target/admin-cmd.c +0 -0
@@ -550,7 +556,13 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
          */

drivers/nvme/target/admin-cmd.c static void nvmet_execute_identify_ns(struct nvmet_req *req) · line 565

Quoted 10 lines
drivers/nvme/target/admin-cmd.c +7 -1
         id->nmic = NVME_NS_NMIC_SHARED;
         id->anagrpid = cpu_to_le32(req->ns->anagrpid);
-
+       id->rescap = NVME_PR_SUPPORT_WRITE_EXCLUSIVE |
+                    NVME_PR_SUPPORT_EXCLUSIVE_ACCESS |
+                    NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY |
+                    NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY |
+                    NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS |
+                    NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS |
+                    NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF;

You have a flag enabling PR on a namespace, so report rescap only if it is enabled.

Sure, I will add a "if" statement.

Quoted 2 lines

Also I didnot find ONCS, is it not necessarily?

Oh, I missed this, thanks, I will change oncs.

Quoted 27 lines
Quoted 27 lines
drivers/nvme/target/pr.c new file +27 -0
+void nvmet_execute_get_log_page_resv(struct nvmet_req *req)
+{
+       struct nvmet_pr_log_mgr *log_mgr = &req->sq->ctrl->pr_log_mgr;
+       struct nvme_pr_log next_log = {0};
+       struct nvme_pr_log log = {0};
+       u16 status = NVME_SC_SUCCESS;
+       u64 lost_count;
+       u64 cur_count;
+       u64 next_count;
+
+       mutex_lock(&log_mgr->lock);
+       if (!kfifo_get(&log_mgr->log_queue, &log))
+               goto out;
+
+       /*
+        * We can't get the last in kfifo.
+        * Utilize the current count and the count from the next log to
+        * calculate the number of lost logs, while also addressing cases
+        * of overflow. If there is no subsequent log, the number of lost
+        * logs is equal to the lost_count within the nvmet_pr_log_mgr.
+        */
+       cur_count = le64_to_cpu(log.count);
+       if (kfifo_peek(&log_mgr->log_queue, &next_log)) {
+               next_count = le64_to_cpu(next_log.count);
+               if (next_count > cur_count)
+                       lost_count = next_count - cur_count - 1;
+               else
Quoted 9 lines

drivers/nvme/target/pr.c line 144

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+                       lost_count = U64_MAX - cur_count + next_count - 1;
+       } else {
+               lost_count = log_mgr->lost_count;
+       }
+
+       log.count = cpu_to_le64(cur_count + lost_count);

This value should be wrapped by U64_MAX too but in reverse direction (count is next_count-1). If == 0 then = -1.

The next_count will never be 0, you can see in nvmet_pr_add_resv_log, when

the log_mgr->counter == 0, I set the log_mgr->counter to 1.

Quoted 12 lines
Quoted 12 lines
drivers/nvme/target/pr.c new file +12 -0
+static void nvmet_pr_add_resv_log(struct nvmet_ctrl *ctrl, u8 log_type, u32 nsid)
+{
+       struct nvmet_pr_log_mgr *log_mgr = &ctrl->pr_log_mgr;
+       struct nvme_pr_log log = {0};
+
+       mutex_lock(&log_mgr->lock);
+       log_mgr->counter++;
+       if (log_mgr->counter == 0)
+               log_mgr->counter = 1;
+
+       log.count = cpu_to_le64(log_mgr->counter);
+       log.type = log_type;
Quoted 8 lines

drivers/nvme/target/pr.c line 171

Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+       log.nsid = cpu_to_le32(nsid);
+
+       if (!kfifo_put(&log_mgr->log_queue, log)) {
+               pr_warn("a reservation log lost, cntlid:%d, log_type:%d, nsid:%d\n",
+                       ctrl->cntlid, log_type, nsid);

I think this log should be Info level, because it is not an error situation. Size of this queue is fixed and it is obvious that it can be overflowed.

Agree.

Quoted 8 lines

drivers/nvme/target/pr.c line 244

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+static u16 nvmet_pr_create_new_reservation(struct nvmet_pr *pr, u8 new_rtype,
+                                          struct nvmet_pr_registrant *reg)
+{
+       u16 status;
+
+       status = nvmet_pr_validate_rtype(new_rtype);

You shall check the validity of the command in the very beginning of command handling, not at the very end.

In some situations, we dont care the value of rtype, for example

using preempt to unregister other host.

So I only check it if needed.

Quoted 9 lines
Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+       if (!status) {
+               reg->rtype = new_rtype;
+               rcu_assign_pointer(pr->holder, reg);
+       }
+       return status;
+}
+

...

Quoted 16 lines
Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+static u16 nvmet_pr_unreg_by_prkey(struct nvmet_pr *pr, u64 prkey,
+                                  uuid_t *send_hostid)
+{
+       u16 status = NVME_SC_RESERVATION_CONFLICT | NVME_SC_DNR;
+       struct nvmet_pr_registrant *reg, *tmp;

drivers/nvme/target/pr.c line 489

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+       uuid_t hostid;
+
+       lockdep_assert_held(&pr->pr_lock);
+
+       list_for_each_entry_safe(reg, tmp, &pr->registrant_list, entry) {
+               if (reg->rkey == prkey) {
+                       status = NVME_SC_SUCCESS;
+                       uuid_copy(&hostid, &reg->hostid);
+                       nvmet_pr_unregister_one(pr, reg);
+                       if (!uuid_equal(&reg->hostid, send_hostid))

reg is free already, use just hostid as line below:

Yeah sure, thanks a lot.

Quoted 27 lines
Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+                               nvmet_pr_registration_preempted(pr, &hostid);
+               }
+       }
+       return status;
+}
+

...

Quoted 19 lines
drivers/nvme/target/pr.c new file +19 -0
+static u16 nvmet_pr_preempt(struct nvmet_req *req,
+                           struct nvmet_pr_registrant *reg,
+                           u8 rtype,
+                           struct nvmet_pr_acquire_data *d)
+{
+       struct nvmet_ctrl *ctrl = req->sq->ctrl;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       enum nvme_pr_type original_rtype;
+       u16 status;
+
+       lockdep_assert_held(&pr->pr_lock);
+       holder = rcu_dereference_protected(pr->holder,
+                       lockdep_is_held(&pr->pr_lock));
+       if (!holder)
+               return nvmet_pr_unreg_by_prkey(pr, le64_to_cpu(d->prkey),
+                               &ctrl->hostid);
+
+       original_rtype = holder->rtype;
Quoted 9 lines

drivers/nvme/target/pr.c line 572

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+       if (original_rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+           original_rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS) {
+               if (!le64_to_cpu(d->prkey)) {
+                       nvmet_pr_unreg_except_hostid(pr, &ctrl->hostid);
+                       return nvmet_pr_create_new_reservation(pr,
+                                       rtype, reg);

There is not an atomic change of the reservation here. Probably if you swap these lines with each other, it will fix that.

You are right, we need create new reservation first.

Quoted 12 lines
Quoted 12 lines
drivers/nvme/target/pr.c new file +12 -0
+               }
+               return nvmet_pr_unreg_by_prkey(pr, le64_to_cpu(d->prkey),
+                               &ctrl->hostid);
+       }
+
+       if (holder == reg) {
+               status = nvmet_pr_update_reg_attr(pr, holder,
+                               nvmet_pr_update_holder_rtype, &rtype);
+               if (!status && original_rtype != rtype)
+                       nvmet_pr_resv_released(pr, &reg->hostid);
+               return status;
+       }
Quoted 8 lines

drivers/nvme/target/pr.c line 591

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+
+       if (le64_to_cpu(d->prkey) == holder->rkey) {
+               status = nvmet_pr_unreg_by_prkey_except_hostid(pr,
+                               le64_to_cpu(d->prkey), &ctrl->hostid);
+               if (status)
+                       return status;
+               status = nvmet_pr_create_new_reservation(pr, rtype, reg);

Here that time gap too.

Changed too.

Quoted 39 lines
Quoted 12 lines
drivers/nvme/target/pr.c new file +11 -0
+               if (!status && original_rtype != rtype)
+                       nvmet_pr_resv_released(pr, &reg->hostid);
+               return status;
+       }
+
+       if (le64_to_cpu(d->prkey))
+               return nvmet_pr_unreg_by_prkey(pr, le64_to_cpu(d->prkey),
+                               &ctrl->hostid);
+       return NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+}
+

...

drivers/nvme/target/pr.c line 996

Quoted 9 lines
drivers/nvme/target/pr.c new file +9 -0
+void nvmet_ctrl_destroy_pr(struct nvmet_ctrl *ctrl)
+{
+       struct nvmet_pr_registrant *reg;
+       struct nvmet_ns *ns;
+       unsigned long idx;
+
+       kfifo_free(&ctrl->pr_log_mgr.log_queue);
+       mutex_destroy(&ctrl->pr_log_mgr.lock);
+

From here and until the end of the function it is against the standard.

Quoted 4 lines
drivers/nvme/target/pr.c new file +4 -0
+       if (nvmet_is_host_connected(&ctrl->hostid, ctrl->subsys))
+               return;
+
+       xa_for_each(&ctrl->subsys->namespaces, idx, ns) {

drivers/nvme/target/pr.c line 1010

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+               mutex_lock(&ns->pr.pr_lock);
+               list_for_each_entry_rcu(reg, &ns->pr.registrant_list, entry,
+                                       lockdep_is_held(&ns->pr.pr_lock)) {
+                       if (uuid_equal(&reg->hostid, &ctrl->hostid)) {
+                               nvmet_pr_unregister_one(&ns->pr, reg);
+                               break;
+                       }
+               }
+               mutex_unlock(&ns->pr.pr_lock);
+       }
Quoted 1 line

You shall not modify Reservation due to host connectivity.

You are right, we should not unregister the host when disconnect, we

should keep it.

Quoted 11 lines
Quoted 11 lines
drivers/nvme/target/pr.c new file +10 -0
+}
+
+void nvmet_pr_exit_ns(struct nvmet_ns *ns)
+{
+       struct nvmet_pr_registrant *reg, *tmp;
+       struct nvmet_pr *pr = &ns->pr;
+       LIST_HEAD(free_list);
+
+       mutex_lock(&pr->pr_lock);
+       rcu_assign_pointer(pr->holder, NULL);
Quoted 8 lines

drivers/nvme/target/pr.c line 1026

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+       list_for_each_entry_safe(reg, tmp, &pr->registrant_list, entry) {
+               list_del_rcu(&reg->entry);
+               list_add(&reg->entry, &free_list);
+       }
+       mutex_unlock(&pr->pr_lock);
+       synchronize_rcu();

IMHO, there is no need in two lists and synchronize_rcu here. At this moment there shall be no other users of ns.

Yeah, we can just free all regs simply here.

Quoted 11 lines

drivers/nvme/target/pr.c line 1032

Quoted 7 lines
drivers/nvme/target/pr.c new file +6 -0
+       list_for_each_entry_safe(reg, tmp, &free_list, entry) {
+               kfree(reg);
+       }
+
+       mutex_destroy(&pr->pr_lock);
+}

BR,
Dmitry

Best Regards,

Guixin Liu

#4 lore raw

Re: [PATCH v8 1/1] nvmet: support reservation feature

Quoted 48 lines · Guixin Liu

Hi Dmitry,

My thanks for your advice, I will work on this back recently.

Quoted 32 lines · Dmitry Bogdanov

Hi,

Waiting for the final solution half an year we taken this patch as is. Here is my comments on it. Hope, this will bring the life to the patch.

Quoted 27 lines
drivers/nvme/target/pr.c new file +27 -0
+void nvmet_execute_get_log_page_resv(struct nvmet_req *req)
+{
+       struct nvmet_pr_log_mgr *log_mgr = &req->sq->ctrl->pr_log_mgr;
+       struct nvme_pr_log next_log = {0};
+       struct nvme_pr_log log = {0};
+       u16 status = NVME_SC_SUCCESS;
+       u64 lost_count;
+       u64 cur_count;
+       u64 next_count;
+
+       mutex_lock(&log_mgr->lock);
+       if (!kfifo_get(&log_mgr->log_queue, &log))
+               goto out;
+
+       /*
+        * We can't get the last in kfifo.
+        * Utilize the current count and the count from the next log to
+        * calculate the number of lost logs, while also addressing cases
+        * of overflow. If there is no subsequent log, the number of lost
+        * logs is equal to the lost_count within the nvmet_pr_log_mgr.
+        */
+       cur_count = le64_to_cpu(log.count);
+       if (kfifo_peek(&log_mgr->log_queue, &next_log)) {
+               next_count = le64_to_cpu(next_log.count);
+               if (next_count > cur_count)
+                       lost_count = next_count - cur_count - 1;
+               else
Quoted 9 lines

drivers/nvme/target/pr.c line 144

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+                       lost_count = U64_MAX - cur_count + next_count - 1;
+       } else {
+               lost_count = log_mgr->lost_count;
+       }
+
+       log.count = cpu_to_le64(cur_count + lost_count);

This value should be wrapped by U64_MAX too but in reverse direction (count is next_count-1). If == 0 then = -1.

Quoted 3 lines

The next_count will never be 0, you can see in nvmet_pr_add_resv_log, when

the log_mgr->counter == 0, I set the log_mgr->counter to 1.

I mean that herer tou should add the same logic as in nvmet_pr_add_resv_log. I am talking about this case:
next_count = 2
cur_count = 0xffffffff
lost_count = U64_MAX - cur_count + next_count - 1 = 0xffffffff - 0xffffffff + 2 - 1 = 1 log.count = cpu_to_le64(cur_count + lost_count) = 0xffffffff + 1 = 0

0 in this case shall became 1 (not -1 as in my original comment)

if (log.count == 0)

	log.count = 1;

would fix that.

Quoted 9 lines
Quoted 8 lines

drivers/nvme/target/pr.c line 244

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+static u16 nvmet_pr_create_new_reservation(struct nvmet_pr *pr, u8 new_rtype,
+                                          struct nvmet_pr_registrant *reg)
+{
+       u16 status;
+
+       status = nvmet_pr_validate_rtype(new_rtype);

You shall check the validity of the command in the very beginning of command handling, not at the very end.

Quoted 5 lines

In some situations, we dont care the value of rtype, for example

using preempt to unregister other host.

So I only check it if needed.

I have two arguemnts for:
1. It is more convenient to check a validity of a command in the beginning - there will be less changes to rollback. Now you do some unregistrations and then may fail the command due to invalid rtype, but Reservation data you does not change back.

2. Yes, the spec does not state that RTYPE shall be valid in Reservation Acquire Action (RACQA) field to 001b (Preempt) when reservation is not going to be changed. But it doesnot state that it
might be ignored too :). In some places I see that such a statement there is.
Also, there is such general statement for reserved values
1.4.1.6 reserved
Receipt of reserved coded values in defined fields in
commands shall be reported as an error.

Quoted 7 lines
Quoted 7 lines

drivers/nvme/target/pr.c line 251

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+       if (!status) {
+               reg->rtype = new_rtype;
+               rcu_assign_pointer(pr->holder, reg);
+       }
+       return status;
+}
+

BR,
Dmitry

author
#5 lore raw

Re: [PATCH v8 1/1] nvmet: support reservation feature

Quoted 58 lines · Dmitry Bogdanov
Quoted 47 lines · Guixin Liu

Hi Dmitry,

My thanks for your advice, I will work on this back recently.

Quoted 32 lines · Dmitry Bogdanov

Hi,

Waiting for the final solution half an year we taken this patch as is. Here is my comments on it. Hope, this will bring the life to the patch.

Quoted 27 lines
drivers/nvme/target/pr.c new file +27 -0
+void nvmet_execute_get_log_page_resv(struct nvmet_req *req)
+{
+       struct nvmet_pr_log_mgr *log_mgr = &req->sq->ctrl->pr_log_mgr;
+       struct nvme_pr_log next_log = {0};
+       struct nvme_pr_log log = {0};
+       u16 status = NVME_SC_SUCCESS;
+       u64 lost_count;
+       u64 cur_count;
+       u64 next_count;
+
+       mutex_lock(&log_mgr->lock);
+       if (!kfifo_get(&log_mgr->log_queue, &log))
+               goto out;
+
+       /*
+        * We can't get the last in kfifo.
+        * Utilize the current count and the count from the next log to
+        * calculate the number of lost logs, while also addressing cases
+        * of overflow. If there is no subsequent log, the number of lost
+        * logs is equal to the lost_count within the nvmet_pr_log_mgr.
+        */
+       cur_count = le64_to_cpu(log.count);
+       if (kfifo_peek(&log_mgr->log_queue, &next_log)) {
+               next_count = le64_to_cpu(next_log.count);
+               if (next_count > cur_count)
+                       lost_count = next_count - cur_count - 1;
+               else
Quoted 9 lines

drivers/nvme/target/pr.c line 144

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+                       lost_count = U64_MAX - cur_count + next_count - 1;
+       } else {
+               lost_count = log_mgr->lost_count;
+       }
+
+       log.count = cpu_to_le64(cur_count + lost_count);

This value should be wrapped by U64_MAX too but in reverse direction (count is next_count-1). If == 0 then = -1.

Quoted 3 lines

The next_count will never be 0, you can see in nvmet_pr_add_resv_log, when

the log_mgr->counter == 0, I set the log_mgr->counter to 1.

I mean that herer tou should add the same logic as in nvmet_pr_add_resv_log. I am talking about this case:
next_count = 2
cur_count = 0xffffffff
lost_count = U64_MAX - cur_count + next_count - 1 = 0xffffffff - 0xffffffff + 2 - 1 = 1 log.count = cpu_to_le64(cur_count + lost_count) = 0xffffffff + 1 = 0

Quoted 6 lines

0 in this case shall became 1 (not -1 as in my original comment)

if (log.count == 0)

	log.count = 1;

would fix that.

Your right, it will be changed in v9.

Quoted 19 lines
Quoted 8 lines
Quoted 8 lines

drivers/nvme/target/pr.c line 244

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+static u16 nvmet_pr_create_new_reservation(struct nvmet_pr *pr, u8 new_rtype,
+                                          struct nvmet_pr_registrant *reg)
+{
+       u16 status;
+
+       status = nvmet_pr_validate_rtype(new_rtype);

You shall check the validity of the command in the very beginning of command handling, not at the very end.

Quoted 5 lines

In some situations, we dont care the value of rtype, for example

using preempt to unregister other host.

So I only check it if needed.

I have two arguemnts for:
1. It is more convenient to check a validity of a command in the beginning - there will be less changes to rollback. Now you do some unregistrations and then may fail the command due to invalid rtype, but Reservation data you does not change back.

Quoted 7 lines

2. Yes, the spec does not state that RTYPE shall be valid in Reservation Acquire Action (RACQA) field to 001b (Preempt) when reservation is not going to be changed. But it doesnot state that it
might be ignored too :). In some places I see that such a statement there is.
Also, there is such general statement for reserved values
1.4.1.6 reserved
Receipt of reserved coded values in defined fields in
commands shall be reported as an error.

I changed rtype check to the beginning of acquire to see maintainer`s opinion.

Quoted 9 lines
Quoted 7 lines
Quoted 7 lines

drivers/nvme/target/pr.c line 251

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+       if (!status) {
+               reg->rtype = new_rtype;
+               rcu_assign_pointer(pr->holder, reg);
+       }
+       return status;
+}
+

BR,
Dmitry

Best Regards,

Guixin Liu

author
#6 lore raw
In reply to Guixin Liu

Hi Sagi,

    Friendly ping. If you have no further concerns with this patch, I would be very grateful

if you could provide a "Reviewed-by" tag. Such an endorsement would be immensely encouraging

for my work in this area.

    Thank you very much for reviewing this patch and for the valuable suggestions you have provided.

Best regards,

Guixin Liu

Quoted 6 lines · Guixin Liu

Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.

maintainer
#7 lore raw
Quoted 9 lines · Guixin Liu

Hi Sagi,

    Friendly ping. If you have no further concerns with this patch, I would be very grateful

if you could provide a "Reviewed-by" tag. Such an endorsement would be immensely encouraging

for my work in this area.

    Thank you very much for reviewing this patch and for the valuable suggestions you have provided.

Sure, I do think we need at least one more reviewer.

Plus, I don't think a target-side reservations maintainers entry is justified here, it is a part of nvmet. So please drop it from your next submission.

#8 lore raw
Quoted 13 lines · Sagi Grimberg
Quoted 10 lines · Guixin Liu

Hi Sagi,

    Friendly ping. If you have no further concerns with this patch, I would be very grateful

if you could provide a "Reviewed-by" tag. Such an endorsement would be immensely encouraging

for my work in this area.

Quoted 8 lines
Quoted 2 lines

    Thank you very much for reviewing this patch and for the valuable suggestions you have provided.

Sure, I do think we need at least one more reviewer.

Plus, I don't think a target-side reservations maintainers entry is justified here, it is a part of nvmet.
So please drop it from your next submission.

Can you please send a next version with Sagi's comments ? also can you please send blktests if you have any so I can run some tests once I finish the review on next version ?

-ck

author
#9 lore raw
Quoted 11 lines
Quoted 6 lines

Sure, I do think we need at least one more reviewer.

Plus, I don't think a target-side reservations maintainers entry is justified here, it is a part of nvmet.
So please drop it from your next submission.

Can you please send a next version with Sagi's comments ? also can you please send blktests if you have any so I can run some tests once I finish the review on next version ?

-ck

Sure,but there is still a discussion going on in v7, I may need to change

the code, I will send a v9 after the discussion and with the blktests patch.

Best regards,

Guixin Liu

Quoted 2 lines
#10 lore raw

Hi Liu,

did you manage to get back to this? I just played around with the feature to test my RFC 9561 implementatiomn and it would be great to get it merged.

author
#11 lore raw

Hi,

    My apologize for not being able to continue my modifications recently, I have been assigned to other work over past half year, I will resume my modifications soon to get this patch merged.

Best Regards,

Guixin Liu

Quoted 5 lines · 在 2024/7/6 00:45, [email protected] 写道:

Hi Liu,

did you manage to get back to this? I just played around with the feature to test my RFC 9561 implementatiomn and it would be great to get it merged.