Skip to contentSkip to conversation

Implement the NVMe reservation feature

PATCH Superseded Guixin Liu 3 participants 11 messages Sep 29, 2024 – Oct 8, 2024

Subsystems (2)

v11 of 19 Superseded Latest is v19 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v11 cover Implement the NVMe reservation feature

Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.

Changes from v10 to v11:
- Remove the cntlid from struct nvmet_pr_registrant. And add hostid to nvmet_pr_per_ctrl_ref to set the correct ctrl to do abort.
- Report all registrants, and report cntlid 0xffff.
- Change nvmet_req's flags to pc_ref.

Changes from v9 to v10:
- Fix the misjudgement in nvmet_pr_unregister_one.
- Fix the non-atomicity problem in nvmet_pr_preempt, now we set the new holder first before unregistering, this can not only make sure that during unregistering other host can not access, but also ensure that nvmet_pr_unregister_one will not unregiter the new holder(In nvmet_pr_unreg_all_others_by_prkey, I exclude current host).
- Remove the pr_abort in nvmet_ctrl, instead kill the per-controller percpu ref first and then wait to zero at the end of preempt_and_abort.
- Fix some spelling mistakes.

Changes from v8 to v9:
- Remove the maintianer request.

- Support "preemt and abort" by adding a per-controller percpu ref to ns, doing wait per-controller percpu ref to zero when a controller's reservation or registration is preempt and the racqa == "preempt and abort". Currently add per-controller percpu ref only when nvmet_pr_check_cmd_access success, others are not cared.

- Report ns support reservation when resv is enabled.

- Report ctrl support reservation.

- Change the log level to info when log lost.

- Fix the UAF issue in nvmet_pr_unreg_by_prkey, and change nvmet_pr_unreg_by_prkey to nvmet_pr_unreg_all_host_by_prkey.

- Dont unregister the host when ctrl is destroyed for keep the reservation info when reconnect.

- Remove the rcu lock and mutex lock when free ns's pr info.

- Fix the situation of log.count is zero.

- Move the rtype check to the start of preemtion, Dmitry suggests that:
1.4.1.6 reserved
Receipt of reserved coded values in defined fields in commands shall be reported as an error.
Look forward the suggestions.
And also, this avoid the non-atomic change of unregister and set new holder.

- Fix the compile error when close CONFIG_LOCKDEP.

- Change nvmet_pr_send_event_by_hostid to nvmet_pr_send_event_to_host.

- Change nvmet_pr_unreg_by_prkey_except_hostid to nvmet_pr_unreg_all_others_by_prkey.

- Change nvmet_pr_unreg_by_prkey to nvmet_pr_unreg_all_host_by_prkey.

Changes from v7 to v8:
- Add me as the new file pr.c's maintainer.

Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.

- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.

- Fix the resv log page is random when there is no resv log page.

- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().

- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().

- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().

- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().

- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().

- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.

Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().

- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().

- Add the print fmt to the head.

- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.

- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.

Changes from v4 to v5:

- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.

- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.

- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.

- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.

- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.

- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.

- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.

- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.

Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.

- Change the function calling alignment style to:

    nvmet_pr_send_event_by_hostid(pr, hostid,
            NVME_PR_LOG_RESERVATOPM_PREEMPTED); 

- Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().

- Remove the goto in __nvmet_pr_unregister_one().

- Change generation to atomic_t, and remove nvmet_pr_inc_generation().

- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.

Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.

- Limit the resv_log_list to 128.

- Change generation to atomic64.

- Put register rkey check to a warpper.

- Change nr_avl_pages to nr_pages.

- Use NVME_SC_SUCCESS instead of 0.

- Change kmalloc param to let it not sleep in mutex lock.

Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.

- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is
0xffffffff at each admin command path, if it is needed.

- Add reservation commands support in nvmet_get_cmd_effects_nvm().

- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.

- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.

- Remove kmem_cache instead to use kmalloc and kfree.

- Change others advice from Sagi.

- Add a blktest test case, this patch will be sent before these series of patches.

Guixin Liu (1):
nvmet: support reservation feature

 drivers/nvme/target/Makefile      |    2 +-
 drivers/nvme/target/admin-cmd.c   |   24 +-
 drivers/nvme/target/configfs.c    |   27 +
 drivers/nvme/target/core.c        |   56 +-
 drivers/nvme/target/fabrics-cmd.c |    4 +-
 drivers/nvme/target/nvmet.h       |   50 +-
 drivers/nvme/target/pr.c          | 1177 +++++++++++++++++++++++++++++
 include/linux/nvme.h              |   54 ++
 8 files changed, 1382 insertions(+), 12 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v11 1/1 nvmet: support reservation feature

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---
 drivers/nvme/target/Makefile      |    2 +-
 drivers/nvme/target/admin-cmd.c   |   24 +-
 drivers/nvme/target/configfs.c    |   27 +
 drivers/nvme/target/core.c        |   56 +-
 drivers/nvme/target/fabrics-cmd.c |    4 +-
 drivers/nvme/target/nvmet.h       |   50 +-
 drivers/nvme/target/pr.c          | 1177 +++++++++++++++++++++++++++++
 include/linux/nvme.h              |   54 ++
 8 files changed, 1382 insertions(+), 12 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
drivers/nvme/target/Makefile +1 -1
index c66820102493..f9bfc904a5b3 100644
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_TARGET_FCLOOP)	+= nvme-fcloop.o
 obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
-			discovery.o io-cmd-file.o io-cmd-bdev.o
+			discovery.o io-cmd-file.o io-cmd-bdev.o pr.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
 nvmet-$(CONFIG_NVME_TARGET_AUTH)	+= fabrics-cmd-auth.o auth.o
drivers/nvme/target/admin-cmd.c +23 -1
index f5b7054a4a05..2cd3560f5c6f 100644
@@ -176,6 +176,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 	log->iocs[nvme_cmd_read] =
 	log->iocs[nvme_cmd_flush] =
 	log->iocs[nvme_cmd_dsm]	=
+	log->iocs[nvme_cmd_resv_acquire] =
+	log->iocs[nvme_cmd_resv_register] =
+	log->iocs[nvme_cmd_resv_release] =
+	log->iocs[nvme_cmd_resv_report] =
 		cpu_to_le32(NVME_CMD_EFFECTS_CSUPP);
 	log->iocs[nvme_cmd_write] =
 	log->iocs[nvme_cmd_write_zeroes] =
@@ -340,6 +344,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		return nvmet_execute_get_log_cmd_effects_ns(req);
 	case NVME_LOG_ANA:
 		return nvmet_execute_get_log_page_ana(req);
+	case NVME_LOG_RESERVATION:
+		return nvmet_execute_get_log_page_resv(req);
 	}
 	pr_debug("unhandled lid %d on qid %d\n",
 	       req->cmd->get_log_page.lid, req->sq->qid);
@@ -433,7 +439,8 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES);
 	id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM |
-			NVME_CTRL_ONCS_WRITE_ZEROES);
+			NVME_CTRL_ONCS_WRITE_ZEROES |
+			NVME_CTRL_ONCS_RESERVATIONS);
 
 	/* XXX: don't report vwc if the underlying device is write through */
 	id->vwc = NVME_CTRL_VWC_PRESENT;
@@ -551,6 +558,15 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
 	id->nmic = NVME_NS_NMIC_SHARED;
 	id->anagrpid = cpu_to_le32(req->ns->anagrpid);
 
+	if (req->ns->pr.enable)
+		id->rescap = NVME_PR_SUPPORT_WRITE_EXCLUSIVE |
+			NVME_PR_SUPPORT_EXCLUSIVE_ACCESS |
+			NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY |
+			NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY |
+			NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS |
+			NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS |
+			NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF;
+
 	memcpy(&id->nguid, &req->ns->nguid, sizeof(id->nguid));
 
 	id->lbaf[0].ds = req->ns->blksize_shift;
@@ -851,6 +867,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_set_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_set_feat_resv_notif_mask(req, cdw11);
+		break;
 	default:
 		req->error_loc = offsetof(struct nvme_common_command, cdw10);
 		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
@@ -949,6 +968,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_get_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_get_feat_resv_notif_mask(req);
+		break;
 	default:
 		req->error_loc =
 			offsetof(struct nvme_common_command, cdw10);
drivers/nvme/target/configfs.c +27 -0
index bd87dfd173a4..9a56269cefd0 100644
@@ -748,6 +748,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,
 
 CONFIGFS_ATTR_WO(nvmet_ns_, revalidate_size);
 
+static ssize_t nvmet_ns_resv_enable_show(struct config_item *item, char *page)
+{
+	return sprintf(page, "%d\n", to_nvmet_ns(item)->pr.enable);
+}
+
+static ssize_t nvmet_ns_resv_enable_store(struct config_item *item,
+					const char *page, size_t count)
+{
+	struct nvmet_ns *ns = to_nvmet_ns(item);
+	bool val;
+
+	if (kstrtobool(page, &val))
+		return -EINVAL;
+
+	mutex_lock(&ns->subsys->lock);
+	if (ns->enabled) {
+		pr_err("the ns:%d is already enabled.\n", ns->nsid);
+		mutex_unlock(&ns->subsys->lock);
+		return -EINVAL;
+	}
+	ns->pr.enable = val;
+	mutex_unlock(&ns->subsys->lock);
+	return count;
+}
+CONFIGFS_ATTR(nvmet_ns_, resv_enable);
+
 static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_device_path,
 	&nvmet_ns_attr_device_nguid,
@@ -756,6 +782,7 @@ static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_enable,
 	&nvmet_ns_attr_buffered_io,
 	&nvmet_ns_attr_revalidate_size,
+	&nvmet_ns_attr_resv_enable,
 #ifdef CONFIG_PCI_P2PDMA
 	&nvmet_ns_attr_p2pmem,
 #endif
drivers/nvme/target/core.c +50 -6

129 lines not on this page · raw message

drivers/nvme/target/fabrics-cmd.c +1 -3
index 69d77d34bec1..edba81796f99 100644
@@ -245,12 +245,10 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 	d->subsysnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	d->hostnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	status = nvmet_alloc_ctrl(d->subsysnqn, d->hostnqn, req,
-				  le32_to_cpu(c->kato), &ctrl);
+				  le32_to_cpu(c->kato), &ctrl, &d->hostid);
 	if (status)
 		goto out;
 
-	uuid_copy(&ctrl->hostid, &d->hostid);
-
 	dhchap_status = nvmet_setup_auth(ctrl);
 	if (dhchap_status) {
 		pr_err("Failed to setup authentication, dhchap status %u\n",
drivers/nvme/target/nvmet.h +49 -1

102 lines not on this page · raw message

drivers/nvme/target/pr.c new file +1177 -0

1,177 lines not on this page · raw message

include/linux/nvme.h +54 -0

72 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 29 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---
 drivers/nvme/target/Makefile      |    2 +-
 drivers/nvme/target/admin-cmd.c   |   24 +-
 drivers/nvme/target/configfs.c    |   27 +
 drivers/nvme/target/core.c        |   56 +-
 drivers/nvme/target/fabrics-cmd.c |    4 +-
 drivers/nvme/target/nvmet.h       |   50 +-
 drivers/nvme/target/pr.c          | 1177 +++++++++++++++++++++++++++++
 include/linux/nvme.h              |   54 ++
 8 files changed, 1382 insertions(+), 12 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
drivers/nvme/target/configfs.c +0 -0
index bd87dfd173a4..9a56269cefd0 100644

drivers/nvme/target/configfs.c static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item, · line 753

Quoted 6 lines
drivers/nvme/target/configfs.c +3 -0
@@ -748,6 +748,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,
 
 CONFIGFS_ATTR_WO(nvmet_ns_, revalidate_size);
 
+static ssize_t nvmet_ns_resv_enable_show(struct config_item *item, char *page)
+{
+       return sprintf(page, "%d\n", to_nvmet_ns(item)->pr.enable);

sysfs_emit is to be used in show()

Quoted 49 lines
drivers/nvme/target/configfs.c +47 -0
+}
+
 
 
+
+static void nvmet_execute_pr_report(struct nvmet_req *req)
+{
+       u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
+       u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
+       u32 num_bytes = 4 * (cdw10 + 1); /* cdw10 is number of dwords */
+       u8 eds = cdw11 & 1; /* Extended data structure, bit 00 */
+       struct nvme_registered_ctrl_ext *ctrl_eds;
+       struct nvme_reservation_status_ext *data;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       struct nvmet_pr_registrant *reg;
+       u16 num_ctrls = 0;
+       u16 status;
+       u8 rtype;
+
+       /* nvmet hostid(uuid_t) is 128 bit. */
+       if (!eds) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw11);
+               status = NVME_SC_HOST_ID_INCONSIST | NVME_SC_DNR;
+               goto out;
+       }
+
+       if (num_bytes < sizeof(struct nvme_reservation_status_ext)) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw10);
+               status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+               goto out;
+       }
+
+       data = kmalloc(num_bytes, GFP_KERNEL);
+       if (!data) {
+               status = NVME_SC_INTERNAL;
+               goto out;
+       }
+       memset(data, 0, num_bytes);
+       data->gen = cpu_to_le32(atomic_read(&pr->generation));
+       data->ptpls = 0;
+       ctrl_eds = data->regctl_eds;
+
+       rcu_read_lock();
+       holder = rcu_dereference(pr->holder);
+       rtype = holder ? holder->rtype : 0;
+       data->rtype = rtype;
+
+       list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {

drivers/nvme/target/configfs.c static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,

Quoted 10 lines
drivers/nvme/target/configfs.c +10 -0
+               if ((void *)ctrl_eds >= (void *)(data + num_bytes))
+                       break;
+               /*
+                * Dynamic controller, set cntlid to 0xffff.
+                */
+               ctrl_eds->cntlid = 0xffff;
+               if (rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+                   rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS)
+                       ctrl_eds->rcsts = 1;
+               if (holder && uuid_equal(&reg->hostid, &holder->hostid))

is just (reg == holder) not enough here?

drivers/nvme/target/configfs.c

Quoted 9 lines
drivers/nvme/target/configfs.c +9 -0
+                       ctrl_eds->rcsts = 1;
+               uuid_copy((uuid_t *)&ctrl_eds->hostid, &reg->hostid);
+               ctrl_eds->rkey = cpu_to_le64(reg->rkey);
+               ctrl_eds++;
+               num_ctrls++;
+       }
+       rcu_read_unlock();
+
+       put_unaligned_le16(num_ctrls, data->regctl);

Here you should report the number of all registered registrants in ns, not the only reported. It is used by the host to understand that it got not a full response.
NUMD field (cdw11) is about transfering the response. But the structure itsef in the reposnse has to be complete.

drivers/nvme/target/configfs.c

Quoted 6 lines
drivers/nvme/target/configfs.c +6 -0
+       status = nvmet_copy_to_sgl(req, 0, data, num_bytes);
+       kfree(data);
+out:
+       nvmet_req_complete(req, status);
+}
+

BR,
Dmitry

author
#3 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 31 lines · Dmitry Bogdanov
Quoted 30 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---
  drivers/nvme/target/Makefile      |    2 +-
  drivers/nvme/target/admin-cmd.c   |   24 +-
  drivers/nvme/target/configfs.c    |   27 +
  drivers/nvme/target/core.c        |   56 +-
  drivers/nvme/target/fabrics-cmd.c |    4 +-
  drivers/nvme/target/nvmet.h       |   50 +-
  drivers/nvme/target/pr.c          | 1177 +++++++++++++++++++++++++++++
  include/linux/nvme.h              |   54 ++
  8 files changed, 1382 insertions(+), 12 deletions(-)

create mode 100644 drivers/nvme/target/pr.c

drivers/nvme/target/configfs.c +0 -0
index bd87dfd173a4..9a56269cefd0 100644
@@ -748,6 +748,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,
 
Quoted 6 lines

drivers/nvme/target/configfs.c static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item, · line 753

Quoted 5 lines
drivers/nvme/target/configfs.c +3 -0
  CONFIGFS_ATTR_WO(nvmet_ns_, revalidate_size);
 
+static ssize_t nvmet_ns_resv_enable_show(struct config_item *item, char *page)
+{
+       return sprintf(page, "%d\n", to_nvmet_ns(item)->pr.enable);

sysfs_emit is to be used in show()

OK

Quoted 51 lines
Quoted 51 lines
drivers/nvme/target/configfs.c +50 -0
+}
+
 
+
+static void nvmet_execute_pr_report(struct nvmet_req *req)
+{
+       u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
+       u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
+       u32 num_bytes = 4 * (cdw10 + 1); /* cdw10 is number of dwords */
+       u8 eds = cdw11 & 1; /* Extended data structure, bit 00 */
+       struct nvme_registered_ctrl_ext *ctrl_eds;
+       struct nvme_reservation_status_ext *data;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       struct nvmet_pr_registrant *reg;
+       u16 num_ctrls = 0;
+       u16 status;
+       u8 rtype;
+
+       /* nvmet hostid(uuid_t) is 128 bit. */
+       if (!eds) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw11);
+               status = NVME_SC_HOST_ID_INCONSIST | NVME_SC_DNR;
+               goto out;
+       }
+
+       if (num_bytes < sizeof(struct nvme_reservation_status_ext)) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw10);
+               status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+               goto out;
+       }
+
+       data = kmalloc(num_bytes, GFP_KERNEL);
+       if (!data) {
+               status = NVME_SC_INTERNAL;
+               goto out;
+       }
+       memset(data, 0, num_bytes);
+       data->gen = cpu_to_le32(atomic_read(&pr->generation));
+       data->ptpls = 0;
+       ctrl_eds = data->regctl_eds;
+
+       rcu_read_lock();
+       holder = rcu_dereference(pr->holder);
+       rtype = holder ? holder->rtype : 0;
+       data->rtype = rtype;
+
+       list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
+               if ((void *)ctrl_eds >= (void *)(data + num_bytes))
+                       break;
+               /*
Quoted 8 lines

drivers/nvme/target/configfs.c

Quoted 7 lines
drivers/nvme/target/configfs.c +7 -0
+                * Dynamic controller, set cntlid to 0xffff.
+                */
+               ctrl_eds->cntlid = 0xffff;
+               if (rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+                   rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS)
+                       ctrl_eds->rcsts = 1;
+               if (holder && uuid_equal(&reg->hostid, &holder->hostid))

is just (reg == holder) not enough here?

Yes.

Quoted 9 lines

drivers/nvme/target/configfs.c

Quoted 9 lines
drivers/nvme/target/configfs.c +9 -0
+                       ctrl_eds->rcsts = 1;
+               uuid_copy((uuid_t *)&ctrl_eds->hostid, &reg->hostid);
+               ctrl_eds->rkey = cpu_to_le64(reg->rkey);
+               ctrl_eds++;
+               num_ctrls++;
+       }
+       rcu_read_unlock();
+
+       put_unaligned_le16(num_ctrls, data->regctl);
Quoted 5 lines

Here you should report the number of all registered registrants in ns, not the only reported. It is used by the host to understand that it got not a full response.
NUMD field (cdw11) is about transfering the response. But the structure itsef in the reposnse has to be complete.

The NVMe spec says that:

    "This indicates the number of Registered Controller data structures and/or Registered Controller extended data structures contained in this data structure."

It means that host use this filed to obtain the reported registrants number.

Best Regards,

Guixin Liu

Quoted 8 lines

drivers/nvme/target/configfs.c

Quoted 6 lines
drivers/nvme/target/configfs.c +6 -0
+       status = nvmet_copy_to_sgl(req, 0, data, num_bytes);
+       kfree(data);
+out:
+       nvmet_req_complete(req, status);
+}
+

BR,
Dmitry

#4 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 87 lines · Guixin Liu
Quoted 62 lines · Dmitry Bogdanov
Quoted 61 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---

drivers/nvme/target/pr.c new file +48 -0
+
+static void nvmet_execute_pr_report(struct nvmet_req *req)
+{
+       u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
+       u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
+       u32 num_bytes = 4 * (cdw10 + 1); /* cdw10 is number of dwords */
+       u8 eds = cdw11 & 1; /* Extended data structure, bit 00 */
+       struct nvme_registered_ctrl_ext *ctrl_eds;
+       struct nvme_reservation_status_ext *data;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       struct nvmet_pr_registrant *reg;
+       u16 num_ctrls = 0;
+       u16 status;
+       u8 rtype;
+
+       /* nvmet hostid(uuid_t) is 128 bit. */
+       if (!eds) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw11);
+               status = NVME_SC_HOST_ID_INCONSIST | NVME_SC_DNR;
+               goto out;
+       }
+
+       if (num_bytes < sizeof(struct nvme_reservation_status_ext)) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw10);
+               status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+               goto out;
+       }
+
+       data = kmalloc(num_bytes, GFP_KERNEL);
+       if (!data) {
+               status = NVME_SC_INTERNAL;
+               goto out;
+       }
+       memset(data, 0, num_bytes);
+       data->gen = cpu_to_le32(atomic_read(&pr->generation));
+       data->ptpls = 0;
+       ctrl_eds = data->regctl_eds;
+
+       rcu_read_lock();
+       holder = rcu_dereference(pr->holder);
+       rtype = holder ? holder->rtype : 0;
+       data->rtype = rtype;
+
+       list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
+               if ((void *)ctrl_eds >= (void *)(data + num_bytes))
+                       break;
+               /*
Quoted 8 lines

drivers/nvme/target/pr.c line 885

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+                * Dynamic controller, set cntlid to 0xffff.
+                */
+               ctrl_eds->cntlid = 0xffff;
+               if (rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+                   rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS)
+                       ctrl_eds->rcsts = 1;
+               if (holder && uuid_equal(&reg->hostid, &holder->hostid))

is just (reg == holder) not enough here?

Yes.

Quoted 9 lines

drivers/nvme/target/pr.c line 894

Quoted 9 lines
drivers/nvme/target/pr.c new file +9 -0
+                       ctrl_eds->rcsts = 1;
+               uuid_copy((uuid_t *)&ctrl_eds->hostid, &reg->hostid);
+               ctrl_eds->rkey = cpu_to_le64(reg->rkey);
+               ctrl_eds++;
+               num_ctrls++;
+       }
+       rcu_read_unlock();
+
+       put_unaligned_le16(num_ctrls, data->regctl);
Quoted 5 lines

Here you should report the number of all registered registrants in ns, not the only reported. It is used by the host to understand that it got not a full response.
NUMD field (cdw11) is about transfering the response. But the structure itsef in the reposnse has to be complete.

Quoted 5 lines

The NVMe spec says that:

"This indicates the number of Registered Controller data structures
and/or Registered Controller extended data structures contained in this data structure."

You read not the latest Spec again. Spec v2.1 has another statement:
Number of Registrants (REGSTRNT): This field indicates the number of registrants of the namespace. This indicates the number of Registrant data structures or Registrant Extended data structures contained in this data structure.

Quoted 3 lines

It means that host use this filed to obtain the reported registrants number.

Think from the host point of view - how are you going to understand how much registrants are there in the namespace?

BR,
Dmitry

author
#5 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 89 lines · Dmitry Bogdanov
Quoted 88 lines · Guixin Liu
Quoted 62 lines · Dmitry Bogdanov
Quoted 61 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---

drivers/nvme/target/pr.c new file +48 -0
+
+static void nvmet_execute_pr_report(struct nvmet_req *req)
+{
+       u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
+       u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
+       u32 num_bytes = 4 * (cdw10 + 1); /* cdw10 is number of dwords */
+       u8 eds = cdw11 & 1; /* Extended data structure, bit 00 */
+       struct nvme_registered_ctrl_ext *ctrl_eds;
+       struct nvme_reservation_status_ext *data;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       struct nvmet_pr_registrant *reg;
+       u16 num_ctrls = 0;
+       u16 status;
+       u8 rtype;
+
+       /* nvmet hostid(uuid_t) is 128 bit. */
+       if (!eds) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw11);
+               status = NVME_SC_HOST_ID_INCONSIST | NVME_SC_DNR;
+               goto out;
+       }
+
+       if (num_bytes < sizeof(struct nvme_reservation_status_ext)) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw10);
+               status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+               goto out;
+       }
+
+       data = kmalloc(num_bytes, GFP_KERNEL);
+       if (!data) {
+               status = NVME_SC_INTERNAL;
+               goto out;
+       }
+       memset(data, 0, num_bytes);
+       data->gen = cpu_to_le32(atomic_read(&pr->generation));
+       data->ptpls = 0;
+       ctrl_eds = data->regctl_eds;
+
+       rcu_read_lock();
+       holder = rcu_dereference(pr->holder);
+       rtype = holder ? holder->rtype : 0;
+       data->rtype = rtype;
+
+       list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
+               if ((void *)ctrl_eds >= (void *)(data + num_bytes))
+                       break;
+               /*
Quoted 8 lines

drivers/nvme/target/pr.c line 885

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+                * Dynamic controller, set cntlid to 0xffff.
+                */
+               ctrl_eds->cntlid = 0xffff;
+               if (rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+                   rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS)
+                       ctrl_eds->rcsts = 1;
+               if (holder && uuid_equal(&reg->hostid, &holder->hostid))

is just (reg == holder) not enough here?

Yes.

Quoted 9 lines

drivers/nvme/target/pr.c line 894

Quoted 9 lines
drivers/nvme/target/pr.c new file +9 -0
+                       ctrl_eds->rcsts = 1;
+               uuid_copy((uuid_t *)&ctrl_eds->hostid, &reg->hostid);
+               ctrl_eds->rkey = cpu_to_le64(reg->rkey);
+               ctrl_eds++;
+               num_ctrls++;
+       }
+       rcu_read_unlock();
+
+       put_unaligned_le16(num_ctrls, data->regctl);
Quoted 5 lines

Here you should report the number of all registered registrants in ns, not the only reported. It is used by the host to understand that it got not a full response.
NUMD field (cdw11) is about transfering the response. But the structure itsef in the reposnse has to be complete.

The NVMe spec says that:

Quoted 7 lines
Quoted 3 lines

"This indicates the number of Registered Controller data structures
and/or Registered Controller extended data structures contained in this data structure."

You read not the latest Spec again. Spec v2.1 has another statement:
Number of Registrants (REGSTRNT): This field indicates the number of registrants of the namespace. This indicates the number of Registrant data structures or Registrant Extended data structures contained in this data structure.

Well, still, "contained in this data structure" means that we should report how many

registrants we reported.

I do a research on nvme-cli and SPDK:

1. nvme-cli: In json_nvme_resv_report, use reported regctls to traverse reported registrants.

2. SPDK: In nvmf_ns_reservation_report, set regctl to the number of reported registrants.

Best Regards,

Guixin Liu

Quoted 7 lines
Quoted 2 lines

It means that host use this filed to obtain the reported registrants number.

Think from the host point of view - how are you going to understand how much registrants are there in the namespace?

BR,
Dmitry

#6 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 93 lines · Guixin Liu
Quoted 92 lines · Dmitry Bogdanov
Quoted 91 lines · Guixin Liu
Quoted 62 lines · Dmitry Bogdanov
Quoted 61 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---

drivers/nvme/target/pr.c new file +48 -0
+
+static void nvmet_execute_pr_report(struct nvmet_req *req)
+{
+       u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
+       u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
+       u32 num_bytes = 4 * (cdw10 + 1); /* cdw10 is number of dwords */
+       u8 eds = cdw11 & 1; /* Extended data structure, bit 00 */
+       struct nvme_registered_ctrl_ext *ctrl_eds;
+       struct nvme_reservation_status_ext *data;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       struct nvmet_pr_registrant *reg;
+       u16 num_ctrls = 0;
+       u16 status;
+       u8 rtype;
+
+       /* nvmet hostid(uuid_t) is 128 bit. */
+       if (!eds) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw11);
+               status = NVME_SC_HOST_ID_INCONSIST | NVME_SC_DNR;
+               goto out;
+       }
+
+       if (num_bytes < sizeof(struct nvme_reservation_status_ext)) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw10);
+               status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+               goto out;
+       }
+
+       data = kmalloc(num_bytes, GFP_KERNEL);
+       if (!data) {
+               status = NVME_SC_INTERNAL;
+               goto out;
+       }
+       memset(data, 0, num_bytes);
+       data->gen = cpu_to_le32(atomic_read(&pr->generation));
+       data->ptpls = 0;
+       ctrl_eds = data->regctl_eds;
+
+       rcu_read_lock();
+       holder = rcu_dereference(pr->holder);
+       rtype = holder ? holder->rtype : 0;
+       data->rtype = rtype;
+
+       list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
+               if ((void *)ctrl_eds >= (void *)(data + num_bytes))
+                       break;
+               /*
Quoted 8 lines

drivers/nvme/target/pr.c line 885

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+                * Dynamic controller, set cntlid to 0xffff.
+                */
+               ctrl_eds->cntlid = 0xffff;
+               if (rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+                   rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS)
+                       ctrl_eds->rcsts = 1;
+               if (holder && uuid_equal(&reg->hostid, &holder->hostid))

is just (reg == holder) not enough here?

Yes.

Quoted 9 lines

drivers/nvme/target/pr.c line 894

Quoted 9 lines
drivers/nvme/target/pr.c new file +9 -0
+                       ctrl_eds->rcsts = 1;
+               uuid_copy((uuid_t *)&ctrl_eds->hostid, &reg->hostid);
+               ctrl_eds->rkey = cpu_to_le64(reg->rkey);
+               ctrl_eds++;
+               num_ctrls++;
+       }
+       rcu_read_unlock();
+
+       put_unaligned_le16(num_ctrls, data->regctl);
Quoted 5 lines

Here you should report the number of all registered registrants in ns, not the only reported. It is used by the host to understand that it got not a full response.
NUMD field (cdw11) is about transfering the response. But the structure itsef in the reposnse has to be complete.

The NVMe spec says that:

"This indicates the number of Registered Controller data structures
and/or Registered Controller extended data structures contained in this data structure."

Quoted 7 lines
Quoted 4 lines

You read not the latest Spec again. Spec v2.1 has another statement:
Number of Registrants (REGSTRNT): This field indicates the number of registrants of the namespace. This indicates the number of Registrant data structures or Registrant Extended data structures contained in this data structure.

Well, still, "contained in this data structure" means that we should report how many registrants we reported.

No, vice-versa :), that means that "this data structure" shall contain "the number of registrants of the namespace".

Quoted 5 lines

I do a research on nvme-cli and SPDK:

1. nvme-cli: In json_nvme_resv_report, use reported regctls to traverse reported registrants.

I saw, and that is a security violation - to trust a data from a network. Try to set a small NUMD and ask ReservationRepost from SPDK nvme target - you will get a coredump of nvme-cli due to aouut-of-bound access of its buffer (NUMD sized).

Instead, it shall send ReservationReport twice - the first with small buffer to get just REGSTRNT, and the second with a full buffer to get the whole list. That is a common handling of a variable length response in SCSI, and NVME too.

Quoted 3 lines

2. SPDK: In nvmf_ns_reservation_report, set regctl to the number of reported registrants.

Looks like you see again not the last version of the reference. The latest SPDK has a correct implementation - It reports a whole registrants list since Feb 13, 2023.
Please, follow it.

Quoted 8 lines

Best Regards,

Guixin Liu

Quoted 4 lines
Quoted 2 lines

It means that host use this filed to obtain the reported registrants number.

Think from the host point of view - how are you going to understand how much registrants are there in the namespace?

BR,
Dmitry

author
#7 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 112 lines · Dmitry Bogdanov
Quoted 93 lines · Guixin Liu
Quoted 92 lines · Dmitry Bogdanov
Quoted 91 lines · Guixin Liu
Quoted 62 lines · Dmitry Bogdanov
Quoted 61 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---

drivers/nvme/target/pr.c new file +48 -0
+
+static void nvmet_execute_pr_report(struct nvmet_req *req)
+{
+       u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
+       u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
+       u32 num_bytes = 4 * (cdw10 + 1); /* cdw10 is number of dwords */
+       u8 eds = cdw11 & 1; /* Extended data structure, bit 00 */
+       struct nvme_registered_ctrl_ext *ctrl_eds;
+       struct nvme_reservation_status_ext *data;
+       struct nvmet_pr *pr = &req->ns->pr;
+       struct nvmet_pr_registrant *holder;
+       struct nvmet_pr_registrant *reg;
+       u16 num_ctrls = 0;
+       u16 status;
+       u8 rtype;
+
+       /* nvmet hostid(uuid_t) is 128 bit. */
+       if (!eds) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw11);
+               status = NVME_SC_HOST_ID_INCONSIST | NVME_SC_DNR;
+               goto out;
+       }
+
+       if (num_bytes < sizeof(struct nvme_reservation_status_ext)) {
+               req->error_loc = offsetof(struct nvme_common_command, cdw10);
+               status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
+               goto out;
+       }
+
+       data = kmalloc(num_bytes, GFP_KERNEL);
+       if (!data) {
+               status = NVME_SC_INTERNAL;
+               goto out;
+       }
+       memset(data, 0, num_bytes);
+       data->gen = cpu_to_le32(atomic_read(&pr->generation));
+       data->ptpls = 0;
+       ctrl_eds = data->regctl_eds;
+
+       rcu_read_lock();
+       holder = rcu_dereference(pr->holder);
+       rtype = holder ? holder->rtype : 0;
+       data->rtype = rtype;
+
+       list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
+               if ((void *)ctrl_eds >= (void *)(data + num_bytes))
+                       break;
+               /*
Quoted 8 lines

drivers/nvme/target/pr.c line 885

Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+                * Dynamic controller, set cntlid to 0xffff.
+                */
+               ctrl_eds->cntlid = 0xffff;
+               if (rtype == NVME_PR_WRITE_EXCLUSIVE_ALL_REGS ||
+                   rtype == NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS)
+                       ctrl_eds->rcsts = 1;
+               if (holder && uuid_equal(&reg->hostid, &holder->hostid))

is just (reg == holder) not enough here?

Yes.

Quoted 9 lines

drivers/nvme/target/pr.c line 894

Quoted 9 lines
drivers/nvme/target/pr.c new file +9 -0
+                       ctrl_eds->rcsts = 1;
+               uuid_copy((uuid_t *)&ctrl_eds->hostid, &reg->hostid);
+               ctrl_eds->rkey = cpu_to_le64(reg->rkey);
+               ctrl_eds++;
+               num_ctrls++;
+       }
+       rcu_read_unlock();
+
+       put_unaligned_le16(num_ctrls, data->regctl);
Quoted 5 lines

Here you should report the number of all registered registrants in ns, not the only reported. It is used by the host to understand that it got not a full response.
NUMD field (cdw11) is about transfering the response. But the structure itsef in the reposnse has to be complete.

The NVMe spec says that:

"This indicates the number of Registered Controller data structures
and/or Registered Controller extended data structures contained in this data structure."

Quoted 6 lines
Quoted 4 lines

You read not the latest Spec again. Spec v2.1 has another statement:
Number of Registrants (REGSTRNT): This field indicates the number of registrants of the namespace. This indicates the number of Registrant data structures or Registrant Extended data structures contained in this data structure.

Well, still, "contained in this data structure" means that we should report how many registrants we reported.

No, vice-versa :), that means that "this data structure" shall contain "the number of registrants of the namespace".

Quoted 4 lines

I do a research on nvme-cli and SPDK:

1. nvme-cli: In json_nvme_resv_report, use reported regctls to traverse reported registrants.

I saw, and that is a security violation - to trust a data from a network. Try to set a small NUMD and ask ReservationRepost from SPDK nvme target - you will get a coredump of nvme-cli due to aouut-of-bound access of its buffer (NUMD sized).

Quoted 4 lines

Instead, it shall send ReservationReport twice - the first with small buffer to get just REGSTRNT, and the second with a full buffer to get the whole list. That is a common handling of a variable length response in SCSI, and NVME too.

My mistake, I miss that before traverse, nvme-cli cut down the regctl by

request bytes(numd).

Quoted 7 lines
Quoted 2 lines

2. SPDK: In nvmf_ns_reservation_report, set regctl to the number of reported registrants.

Looks like you see again not the last version of the reference. The latest SPDK has a correct implementation - It reports a whole registrants list since Feb 13, 2023.
Please, follow it.

Yeah, I see, SPDK changed to all registrants number.

I will change this in v13, thanks.

Best Regards,

Guixin Liu

Quoted 10 lines
Quoted 8 lines

Best Regards,

Guixin Liu

Quoted 4 lines
Quoted 2 lines

It means that host use this filed to obtain the reported registrants number.

Think from the host point of view - how are you going to understand how much registrants are there in the namespace?

BR,
Dmitry

#8 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

In reply to Guixin Liu
Quoted 10 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

The explanation feels a bit sparse. It could also mentioned that no support for persistent reservation exists, and how this code was tested.

Also, do you have a corresponding nvmetcli patch?

drivers/nvme/target/pr.c line 30

Quoted 13 lines
drivers/nvme/target/pr.c new file +13 -0
+struct nvmet_pr_register_data {
+	__le64	crkey;
+	__le64	nrkey;
+};
+
+struct nvmet_pr_acquire_data {
+	__le64	crkey;
+	__le64	prkey;
+};
+
+struct nvmet_pr_release_data {
+	__le64	crkey;
+};

Using little endian fields for purely in-memory data feels strange. Is there a good reason for this?

drivers/nvme/target/pr.c line 339

Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+static u16 nvmet_pr_update_reg_attr(struct nvmet_pr *pr,
+				    struct nvmet_pr_registrant *reg,
+				    void (*change_attr)(struct nvmet_pr_registrant *reg,
+					void *attr),
+				    void *attr)

Please avoid the overly long line here. That's easiest done by following the style used elsewhere in the nvme code using two tab continuations:

static u16 nvmet_pr_update_reg_attr(struct nvmet_pr *pr,
struct nvmet_pr_registrant *reg,
void (*change_attr)(struct nvmet_pr_registrant *reg,
void *attr),
void *attr)

drivers/nvme/target/pr.c line 365

Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+	change_attr(new, attr);
+	list_replace_rcu(&holder->entry, &new->entry);
+	rcu_assign_pointer(pr->holder, new);
+	synchronize_rcu();
+	kfree(holder);

Does this really need a full blown expensive synchronize_rcu vs just a cheaper kfree_rcu_mightsleep or kfree_rcu?

Review

Quoted 1 line
(quoted diff) +1 -0
+	bool ignore_key = (bool)((cdw10 >> 3) & 1); /* Ignore existing key, bit 03 */

Overly long line. This might also benefit from adding symbolic constants and/or extraction helpers.

The explicit cast to bool should also not be needed.

Quoted 8 lines
drivers/nvme/target/pr.c new file +8 -0
+	struct nvmet_pr_registrant *reg, *tmp;
+	struct nvmet_pr *pr = &req->ns->pr;
+	LIST_HEAD(free_list);
+
+	lockdep_assert_held(&pr->pr_lock);
+
+	rcu_assign_pointer(pr->holder, NULL);
+

drivers/nvme/target/pr.c line 766

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+	list_for_each_entry_safe(reg, tmp, &pr->registrant_list, entry) {
+		list_del_rcu(&reg->entry);
+		if (!uuid_equal(&req->sq->ctrl->hostid, &reg->hostid))
+			nvmet_pr_resv_preempted(pr, &reg->hostid);
+		list_add(&reg->entry, &free_list);
+	}
+	synchronize_rcu();
+	list_for_each_entry_safe(reg, tmp, &free_list, entry) {
+		kfree(reg);
+	}

No nee for the outer braces here. But why do we we need the expensive synchronize_rcu and two-step operation here anyway vs just using kfree_rcu?

drivers/nvme/target/pr.c line 881

Quoted 4 lines
drivers/nvme/target/pr.c new file +4 -0
+		/*
+		 * Dynamic controller, set cntlid to 0xffff.
+		 */
+		ctrl_eds->cntlid = 0xffff;

NVME_CNTLID_DYNAMIC

drivers/nvme/target/pr.c line 1019

Quoted 1 line
drivers/nvme/target/pr.c new file +1 -0
+	req->pc_ref = xa_load(&req->ns->pr_per_ctrl_refs, req->sq->ctrl->cntlid);

Overly long line.

drivers/nvme/target/pr.c line 1029

Quoted 10 lines
drivers/nvme/target/pr.c new file +10 -0
+	if (unlikely(!percpu_ref_tryget_live(&req->pc_ref->ref)))
+		return NVME_SC_INTERNAL;
+	return NVME_SC_SUCCESS;
+}
+
+void nvmet_pr_put_ns_pc_ref(struct nvmet_req *req)
+{
+	if (req->pc_ref)
+		percpu_ref_put(&req->pc_ref->ref);
+}

It would be niceto have the NULL check inline to avoid the call for for namespaces without reservation support.

include/linux/nvme.h

Quoted 4 lines
include/linux/nvme.h +0 -0
index 425573202295..b1be3d313bee 100644

Please split out adding the new code points to nvme.h to a separate prep patch.

author
#9 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.

Quoted 5 lines
Quoted 2 lines

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

The explanation feels a bit sparse. It could also mentioned that no support for persistent reservation exists, and how this code was tested.

I will add more detail in commit in v12.

Quoted 2 lines

Also, do you have a corresponding nvmetcli patch?

Sure, I think so.

Quoted 7 lines
Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+struct nvmet_pr_register_data {
+	__le64	crkey;
+	__le64	nrkey;
+};
+
+struct nvmet_pr_acquire_data {
+	__le64	crkey;
Quoted 9 lines

drivers/nvme/target/pr.c line 30

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+	__le64	prkey;
+};
+
+struct nvmet_pr_release_data {
+	__le64	crkey;
+};

Using little endian fields for purely in-memory data feels strange. Is there a good reason for this?

I use nvmet_copy_from_sgl to obtain the command's detail, just like

nvme_dsm_range, maybe I can put them in to include/linux/nvme.h.

Quoted 9 lines

drivers/nvme/target/pr.c line 339

Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+static u16 nvmet_pr_update_reg_attr(struct nvmet_pr *pr,
+				    struct nvmet_pr_registrant *reg,
+				    void (*change_attr)(struct nvmet_pr_registrant *reg,
+					void *attr),
+				    void *attr)

Please avoid the overly long line here. That's easiest done by following the style used elsewhere in the nvme code using two tab continuations:

Quoted 5 lines

static u16 nvmet_pr_update_reg_attr(struct nvmet_pr *pr,
struct nvmet_pr_registrant *reg,
void (*change_attr)(struct nvmet_pr_registrant *reg,
void *attr),
void *attr)

OK.

Quoted 7 lines

drivers/nvme/target/pr.c line 365

Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+	change_attr(new, attr);
+	list_replace_rcu(&holder->entry, &new->entry);
+	rcu_assign_pointer(pr->holder, new);
+	synchronize_rcu();
+	kfree(holder);

Does this really need a full blown expensive synchronize_rcu vs just a cheaper kfree_rcu_mightsleep or kfree_rcu?

Yes, I will use kfree_rcu instead.

Quoted 3 lines

Review

Quoted 1 line
(quoted diff) +1 -0
+	bool ignore_key = (bool)((cdw10 >> 3) & 1); /* Ignore existing key, bit 03 */

Overly long line. This might also benefit from adding symbolic constants and/or extraction helpers.

OK.

Quoted 3 lines

The explicit cast to bool should also not be needed.

OK.

Quoted 13 lines
Quoted 13 lines
drivers/nvme/target/pr.c new file +13 -0
+	struct nvmet_pr_registrant *reg, *tmp;
+	struct nvmet_pr *pr = &req->ns->pr;
+	LIST_HEAD(free_list);
+
+	lockdep_assert_held(&pr->pr_lock);
+
+	rcu_assign_pointer(pr->holder, NULL);
+
+	list_for_each_entry_safe(reg, tmp, &pr->registrant_list, entry) {
+		list_del_rcu(&reg->entry);
+		if (!uuid_equal(&req->sq->ctrl->hostid, &reg->hostid))
+			nvmet_pr_resv_preempted(pr, &reg->hostid);
+		list_add(&reg->entry, &free_list);
Quoted 8 lines

drivers/nvme/target/pr.c line 766

Quoted 5 lines
drivers/nvme/target/pr.c new file +5 -0
+	}
+	synchronize_rcu();
+	list_for_each_entry_safe(reg, tmp, &free_list, entry) {
+		kfree(reg);
+	}

No nee for the outer braces here. But why do we we need the expensive synchronize_rcu and two-step operation here anyway vs just using kfree_rcu?

Sure.

Quoted 6 lines

drivers/nvme/target/pr.c line 881

Quoted 4 lines
drivers/nvme/target/pr.c new file +4 -0
+		/*
+		 * Dynamic controller, set cntlid to 0xffff.
+		 */
+		ctrl_eds->cntlid = 0xffff;

NVME_CNTLID_DYNAMIC

OK, will be changed in v12.

Quoted 2 lines

drivers/nvme/target/pr.c line 1019

Quoted 1 line
drivers/nvme/target/pr.c new file +1 -0
+	req->pc_ref = xa_load(&req->ns->pr_per_ctrl_refs, req->sq->ctrl->cntlid);

Overly long line.

Changed.

Quoted 13 lines

drivers/nvme/target/pr.c line 1029

Quoted 11 lines
drivers/nvme/target/pr.c new file +10 -0
+	if (unlikely(!percpu_ref_tryget_live(&req->pc_ref->ref)))
+		return NVME_SC_INTERNAL;
+	return NVME_SC_SUCCESS;
+}
+
+void nvmet_pr_put_ns_pc_ref(struct nvmet_req *req)
+{
+	if (req->pc_ref)
+		percpu_ref_put(&req->pc_ref->ref);
+}

It would be niceto have the NULL check inline to avoid the call for for namespaces without reservation support.

Understood, I will change nvmet_pr_put_ns_pc_ref to inline.

Quoted 6 lines

include/linux/nvme.h

Quoted 4 lines
include/linux/nvme.h +0 -0
index 425573202295..b1be3d313bee 100644

Please split out adding the new code points to nvme.h to a separate prep patch.

OK, I will split it in v12.

Best regards,

Guixin Liu

#10 lore raw

Re: [PATCH v11 1/1] nvmet: support reservation feature

Quoted 16 lines · Guixin Liu
Quoted 7 lines
Quoted 7 lines
drivers/nvme/target/pr.c new file +7 -0
+struct nvmet_pr_register_data {
+	__le64	crkey;
+	__le64	nrkey;
+};
+
+struct nvmet_pr_acquire_data {
+	__le64	crkey;
Quoted 9 lines

drivers/nvme/target/pr.c line 30

Quoted 6 lines
drivers/nvme/target/pr.c new file +6 -0
+	__le64	prkey;
+};
+
+struct nvmet_pr_release_data {
+	__le64	crkey;
+};

Using little endian fields for purely in-memory data feels strange. Is there a good reason for this?

Quoted 3 lines

I use nvmet_copy_from_sgl to obtain the command's detail, just like

nvme_dsm_range, maybe I can put them in to include/linux/nvme.h.

Ah right, you just store the on the wire format locally. Yes, pleaße ad them to nvme.h.