Implement the NVMe reservation feature
Subsystems (2)
v10 cover Implement the NVMe reservation feature
Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.
Changes from v9 to v10:
- Fix the misjudgement in nvmet_pr_unregister_one.
- Fix the non-atomicity problem in nvmet_pr_preempt, now we set the new holder first before unregistering, this can not only make sure that during unregistering other host can not access, but also ensure that nvmet_pr_unregister_one will not unregiter the new holder(In nvmet_pr_unreg_all_others_by_prkey, I exclude current host).
- Remove the pr_abort in nvmet_ctrl, instead kill the per-controller percpu ref first and then wait to zero at the end of preempt_and_abort.
- Fix some spelling mistakes.
Changes from v8 to v9:
- Remove the maintianer request.
- Support "preemt and abort" by adding a per-controller percpu ref to ns, doing wait per-controller percpu ref to zero when a controller's reservation or registration is preempt and the racqa == "preempt and abort". Currently add per-controller percpu ref only when nvmet_pr_check_cmd_access success, others are not cared.
- Report ns support reservation when resv is enabled.
- Report ctrl support reservation.
- Change the log level to info when log lost.
- Fix the UAF issue in nvmet_pr_unreg_by_prkey, and change nvmet_pr_unreg_by_prkey to nvmet_pr_unreg_all_host_by_prkey.
- Dont unregister the host when ctrl is destroyed for keep the reservation info when reconnect.
- Remove the rcu lock and mutex lock when free ns's pr info.
- Fix the situation of log.count is zero.
- Move the rtype check to the start of preemtion, Dmitry suggests that:
1.4.1.6 reserved
Receipt of reserved coded values in defined fields in commands shall be reported as an error.
Look forward the suggestions.
And also, this avoid the non-atomic change of unregister and set new holder.
- Fix the compile error when close CONFIG_LOCKDEP.
- Change nvmet_pr_send_event_by_hostid to nvmet_pr_send_event_to_host.
- Change nvmet_pr_unreg_by_prkey_except_hostid to nvmet_pr_unreg_all_others_by_prkey.
- Change nvmet_pr_unreg_by_prkey to nvmet_pr_unreg_all_host_by_prkey.
Changes from v7 to v8:
- Add me as the new file pr.c's maintainer.
Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.
- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.
- Fix the resv log page is random when there is no resv log page.
- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().
- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().
- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().
- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().
- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().
- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.
Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().
- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().
- Add the print fmt to the head.
- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.
- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.
Changes from v4 to v5:
- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.
- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.
- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.
- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.
- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.
- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.
- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.
- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.
Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.
- Change the function calling alignment style to:
nvmet_pr_send_event_by_hostid(pr, hostid,
NVME_PR_LOG_RESERVATOPM_PREEMPTED); - Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().
- Remove the goto in __nvmet_pr_unregister_one().
- Change generation to atomic_t, and remove nvmet_pr_inc_generation().
- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.
Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.
- Limit the resv_log_list to 128.
- Change generation to atomic64.
- Put register rkey check to a warpper.
- Change nr_avl_pages to nr_pages.
- Use NVME_SC_SUCCESS instead of 0.
- Change kmalloc param to let it not sleep in mutex lock.
Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.
- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is0xffffffff at each admin command path, if it is needed.
- Add reservation commands support in nvmet_get_cmd_effects_nvm().
- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.
- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.
- Remove kmem_cache instead to use kmalloc and kfree.
- Change others advice from Sagi.
- Add a blktest test case, this patch will be sent before these series of patches.
Guixin Liu (1):
nvmet: support reservation feature
drivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 24 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 58 +- drivers/nvme/target/nvmet.h | 48 ++ drivers/nvme/target/pr.c | 1197 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1401 insertions(+), 9 deletions(-) create mode 100644 drivers/nvme/target/pr.c
Signature · 1 line
2.43.0
PATCH v10 1/1 nvmet: support reservation feature
This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.
And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.
--- drivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 24 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 58 +- drivers/nvme/target/nvmet.h | 48 ++ drivers/nvme/target/pr.c | 1197 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1401 insertions(+), 9 deletions(-) create mode 100644 drivers/nvme/target/pr.c
drivers/nvme/target/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_TARGET_FCLOOP) += nvme-fcloop.o obj-$(CONFIG_NVME_TARGET_TCP) += nvmet-tcp.o nvmet-y += core.o configfs.o admin-cmd.o fabrics-cmd.o \ - discovery.o io-cmd-file.o io-cmd-bdev.o + discovery.o io-cmd-file.o io-cmd-bdev.o pr.o nvmet-$(CONFIG_NVME_TARGET_PASSTHRU) += passthru.o nvmet-$(CONFIG_BLK_DEV_ZONED) += zns.o nvmet-$(CONFIG_NVME_TARGET_AUTH) += fabrics-cmd-auth.o auth.o
drivers/nvme/target/admin-cmd.c +23 -1
@@ -176,6 +176,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log) log->iocs[nvme_cmd_read] = log->iocs[nvme_cmd_flush] = log->iocs[nvme_cmd_dsm] = + log->iocs[nvme_cmd_resv_acquire] = + log->iocs[nvme_cmd_resv_register] = + log->iocs[nvme_cmd_resv_release] = + log->iocs[nvme_cmd_resv_report] =cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); log->iocs[nvme_cmd_write] = log->iocs[nvme_cmd_write_zeroes] =
@@ -340,6 +344,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)return nvmet_execute_get_log_cmd_effects_ns (req);case NVME_LOG_ANA :return nvmet_execute_get_log_page_ana (req); +case NVME_LOG_RESERVATION : +return nvmet_execute_get_log_page_resv (req); }pr_debug ("unhandled lid %d on qid %d\n" , req->cmd->get_log_page.lid, req->sq->qid);
@@ -433,7 +439,8 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn =cpu_to_le32 (NVMET_MAX_NAMESPACES ); id->mnan =cpu_to_le32 (NVMET_MAX_NAMESPACES ); id->oncs =cpu_to_le16 (NVME_CTRL_ONCS_DSM | -NVME_CTRL_ONCS_WRITE_ZEROES ); +NVME_CTRL_ONCS_WRITE_ZEROES | +NVME_CTRL_ONCS_RESERVATIONS );/* XXX: don't report vwc if the underlying device is write through */ id->vwc =NVME_CTRL_VWC_PRESENT ;
@@ -551,6 +558,15 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) id->nmic =NVME_NS_NMIC_SHARED ; id->anagrpid =cpu_to_le32 (req->ns->anagrpid); +if (req->ns->pr.enable) + id->rescap =NVME_PR_SUPPORT_WRITE_EXCLUSIVE | +NVME_PR_SUPPORT_EXCLUSIVE_ACCESS | +NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY | +NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY | +NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS | +NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS | +NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF ; +memcpy (&id->nguid, &req->ns->nguid,sizeof (id->nguid)); id->lbaf[0 ].ds = req->ns->blksize_shift;
@@ -851,6 +867,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)case NVME_FEAT_WRITE_PROTECT : status =nvmet_set_feat_write_protect (req);break ; +case NVME_FEAT_RESV_MASK : + status =nvmet_set_feat_resv_notif_mask (req, cdw11); +break ;default : req->error_loc = offsetof(struct nvme_common_command , cdw10); status =NVME_SC_INVALID_FIELD |NVME_SC_DNR ;
@@ -949,6 +968,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)case NVME_FEAT_WRITE_PROTECT : status =nvmet_get_feat_write_protect (req);break ; +case NVME_FEAT_RESV_MASK : + status =nvmet_get_feat_resv_notif_mask (req); +break ;default : req->error_loc = offsetof(struct nvme_common_command , cdw10);
drivers/nvme/target/configfs.c +27 -0
@@ -748,6 +748,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,CONFIGFS_ATTR_WO (nvmet_ns_, revalidate_size); +static ssize_t nvmet_ns_resv_enable_show (struct config_item *item,char *page) +{ +return sprintf (page,"%d\n" ,to_nvmet_ns (item)->pr.enable); +} + +static ssize_t nvmet_ns_resv_enable_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_ns *ns =to_nvmet_ns (item); +bool val; + +if (kstrtobool (page, &val)) +return -EINVAL ; + +mutex_lock (&ns->subsys->lock); +if (ns->enabled) { +pr_err ("the ns:%d is already enabled.\n" , ns->nsid); +mutex_unlock (&ns->subsys->lock); +return -EINVAL ; + } + ns->pr.enable = val; +mutex_unlock (&ns->subsys->lock); +return count; +} +CONFIGFS_ATTR (nvmet_ns_, resv_enable); +static struct configfs_attribute *nvmet_ns_attrs[] = { &nvmet_ns_attr_device_path, &nvmet_ns_attr_device_nguid,
@@ -756,6 +782,7 @@ static struct configfs_attribute *nvmet_ns_attrs[] = { &nvmet_ns_attr_enable, &nvmet_ns_attr_buffered_io, &nvmet_ns_attr_revalidate_size, + &nvmet_ns_attr_resv_enable, #ifdef CONFIG_PCI_P2PDMA &nvmet_ns_attr_p2pmem, #endif
drivers/nvme/target/core.c +51 -7
127 lines not on this page · raw message
drivers/nvme/target/nvmet.h +48 -0
94 lines not on this page · raw message
drivers/nvme/target/pr.c new file +1197 -0
1,197 lines not on this page · raw message
include/linux/nvme.h +54 -0
72 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v10 1/1] nvmet: support reservation feature
Quoted 87 lines · Guixin Liu
This patch implements the reservation feature, includes:
1. reservationregister(register, unregister and replace).
2. reservationacquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.And also make reservation configurable, one can set ns to support reservation before enable ns. The default of
resv_enableis false.Signed-off-by Guixin Liu <[email protected]>--- drivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 24 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 58 +- drivers/nvme/target/nvmet.h | 48 ++ drivers/nvme/target/pr.c | 1197 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1401 insertions(+), 9 deletions(-) create mode 100644 drivers/nvme/target/pr.cdrivers/nvme/target/pr.c new file +63
-0+static void nvmet_execute_pr_report (struct nvmet_req *req) +{ +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +u32 cdw10 =le32_to_cpu (req->cmd->common.cdw10); +u32 num_bytes =4 * (cdw10 +1 );/* cdw10 is number of dwords */ +u8 eds = cdw11 &1 ;/* Extended data structure, bit 00 */ +struct nvmet_subsys *subsys = req->sq->ctrl->subsys; +struct nvme_registered_ctrl_ext *ctrl_eds; +struct nvme_reservation_status_ext *data; +struct nvmet_pr *pr = &req->ns->pr; +struct nvmet_pr_registrant *holder; +struct nvmet_pr_registrant *reg; +struct nvmet_ctrl *ctrl; +u16 num_ctrls =0 ; +u16 status; +u8 rtype; + +/* nvmet hostid(uuid_t) is 128 bit. */ +if (!eds) { + req->error_loc = offsetof(struct nvme_common_command , cdw11); + status =NVME_SC_HOST_ID_INCONSIST |NVME_SC_DNR ; + goto out; + } + +if (num_bytes <sizeof (struct nvme_reservation_status_ext )) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); + status =NVME_SC_INVALID_FIELD |NVME_SC_DNR ; + goto out; + } + +mutex_lock (&subsys->lock); + num_ctrls =list_count_nodes (&subsys->ctrls); +mutex_unlock (&subsys->lock); + + num_bytes =min (num_bytes, +sizeof (struct nvme_reservation_status_ext ) + +sizeof (struct nvme_registered_ctrl_ext ) * num_ctrls); + + data =kmalloc (num_bytes,GFP_KERNEL ); +if (!data) { + status =NVME_SC_INTERNAL ; + goto out; + } +memset (data,0 , num_bytes); + data->gen =cpu_to_le32 (atomic_read (&pr->generation)); + data->ptpls =0 ; + ctrl_eds = data->regctl_eds; + +mutex_lock (&subsys->lock); +rcu_read_lock (); + holder =rcu_dereference (pr->holder); + rtype = holder ? holder->rtype :0 ; + data->rtype = rtype; + num_ctrls =0 ; +list_for_each_entry (ctrl, &subsys->ctrls, subsys_entry) { +if ((void *)ctrl_eds >= (void *)(data + num_bytes)) +break ; + reg =nvmet_pr_find_registrant (pr, &ctrl->hostid); +if (!reg) +continue ; + ctrl_eds->cntlid =cpu_to_le16 (ctrl->cntlid); +if (rtype ==NVME_PR_WRITE_EXCLUSIVE_ALL_REGS || + rtype ==NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS )
drivers/nvme/target/pr.c line 904
Quoted 10 lines
drivers/nvme/target/pr.c new file +10
-0+ ctrl_eds->rcsts =1 ; +if (holder &&uuid_equal (&ctrl->hostid, &holder->hostid)) + ctrl_eds->rcsts =1 ; +uuid_copy ((uuid_t *)&ctrl_eds->hostid, &ctrl->hostid); + ctrl_eds->rkey =cpu_to_le64 (reg->rkey); + ctrl_eds++; + num_ctrls++; + } +rcu_read_unlock (); +mutex_unlock (&subsys->lock);
That code fully confroms NVMe Base Spec 2.0c but it is completely wrong from the logic point of view.
ControllerId is a dynamic thing - it's allocated for the each fabric session(Connect command). It has no meaning for an initiator. The association between Registrant (uniq key HostId) and Controller (uniq key CtrlId) is not 1..1, it is 1..N.
NVMe Base Spec 2.1 addressed this issue and it states, that here target shall report just list if the registrants:
Number of Registrants (REGSTRNT): This field indicates the number of registrants of the namespace. This indicates the number of Registrant data structures or Registrant Extended data structures contained in this data structure.
Note: This field was formerly named Number of Registered Controllers (REGCTL).
Though, v2.1 still does not clearly states what exact CNTLID is to be reported from 1..N association.
I suggest you to update your patch to conform the Spec v2.1 - remove cntlid from nvmet_pr_registrant and report here just ns->pr.registrant_list. And not to track the connectivity of registrants - report always CNTLID=0xFFFF as a some compromise value that is allowed by the Spec.
drivers/nvme/target/pr.c line 908
Quoted 4 lines
drivers/nvme/target/pr.c new file +4
-0+ +put_unaligned_le16 (num_ctrls, data->regctl); + num_bytes =sizeof (struct nvme_reservation_status_ext ) + +sizeof (struct nvme_registered_ctrl_ext ) * num_ctrls;
num_bytes was a size of requested buffer (cdw10), you even allocate that size. You shall not extend it here. Instead you should allocate maximum possible size and copy here only cdw10 dwords from your buffer to sgl.
drivers/nvme/target/pr.c line 914
Quoted 6 lines
drivers/nvme/target/pr.c new file +6
-0+ status =nvmet_copy_to_sgl (req,0 , data, num_bytes); +kfree (data); +out: +nvmet_req_complete (req, status); +} +
BR,
Dmitry
Re: [PATCH v10 1/1] nvmet: support reservation feature
Quoted 113 lines · Dmitry Bogdanov
Quoted 86 lines · Guixin Liu
This patch implements the reservation feature, includes:
1. reservationregister(register, unregister and replace).
2. reservationacquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
5. set feature and get feature of reservation notify mask.
6. get log page of reservation event.And also make reservation configurable, one can set ns to support reservation before enable ns. The default of
resv_enableis false.Signed-off-by Guixin Liu <[email protected]>--- drivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 24 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 58 +- drivers/nvme/target/nvmet.h | 48 ++ drivers/nvme/target/pr.c | 1197 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1401 insertions(+), 9 deletions(-)create mode 100644
drivers/nvme/target/pr.cdrivers/nvme/target/pr.c new file +63
-0+static void nvmet_execute_pr_report (struct nvmet_req *req) +{ +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +u32 cdw10 =le32_to_cpu (req->cmd->common.cdw10); +u32 num_bytes =4 * (cdw10 +1 );/* cdw10 is number of dwords */ +u8 eds = cdw11 &1 ;/* Extended data structure, bit 00 */ +struct nvmet_subsys *subsys = req->sq->ctrl->subsys; +struct nvme_registered_ctrl_ext *ctrl_eds; +struct nvme_reservation_status_ext *data; +struct nvmet_pr *pr = &req->ns->pr; +struct nvmet_pr_registrant *holder; +struct nvmet_pr_registrant *reg; +struct nvmet_ctrl *ctrl; +u16 num_ctrls =0 ; +u16 status; +u8 rtype; + +/* nvmet hostid(uuid_t) is 128 bit. */ +if (!eds) { + req->error_loc = offsetof(struct nvme_common_command , cdw11); + status =NVME_SC_HOST_ID_INCONSIST |NVME_SC_DNR ; + goto out; + } + +if (num_bytes <sizeof (struct nvme_reservation_status_ext )) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); + status =NVME_SC_INVALID_FIELD |NVME_SC_DNR ; + goto out; + } + +mutex_lock (&subsys->lock); + num_ctrls =list_count_nodes (&subsys->ctrls); +mutex_unlock (&subsys->lock); + + num_bytes =min (num_bytes, +sizeof (struct nvme_reservation_status_ext ) + +sizeof (struct nvme_registered_ctrl_ext ) * num_ctrls); + + data =kmalloc (num_bytes,GFP_KERNEL ); +if (!data) { + status =NVME_SC_INTERNAL ; + goto out; + } +memset (data,0 , num_bytes); + data->gen =cpu_to_le32 (atomic_read (&pr->generation)); + data->ptpls =0 ; + ctrl_eds = data->regctl_eds; + +mutex_lock (&subsys->lock); +rcu_read_lock (); + holder =rcu_dereference (pr->holder); + rtype = holder ? holder->rtype :0 ; + data->rtype = rtype; + num_ctrls =0 ; +list_for_each_entry (ctrl, &subsys->ctrls, subsys_entry) { +if ((void *)ctrl_eds >= (void *)(data + num_bytes)) +break ; + reg =nvmet_pr_find_registrant (pr, &ctrl->hostid); +if (!reg) +continue ; + ctrl_eds->cntlid =cpu_to_le16 (ctrl->cntlid); +if (rtype ==NVME_PR_WRITE_EXCLUSIVE_ALL_REGS || + rtype ==NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS )drivers/nvme/target/pr.c line 904
Quoted 10 lines
drivers/nvme/target/pr.c new file +10
-0+ ctrl_eds->rcsts =1 ; +if (holder &&uuid_equal (&ctrl->hostid, &holder->hostid)) + ctrl_eds->rcsts =1 ; +uuid_copy ((uuid_t *)&ctrl_eds->hostid, &ctrl->hostid); + ctrl_eds->rkey =cpu_to_le64 (reg->rkey); + ctrl_eds++; + num_ctrls++; + } +rcu_read_unlock (); +mutex_unlock (&subsys->lock);That code fully confroms NVMe Base Spec 2.0c but it is completely wrong from the logic point of view.
ControllerId is a dynamic thing - it's allocated for the each fabric session(Connect command). It has no meaning for an initiator. The association between Registrant (uniq key HostId) and Controller (uniq key CtrlId) is not 1..1, it is 1..N.NVMe Base Spec 2.1 addressed this issue and it states, that here target shall report just list if the registrants:
Number of Registrants (REGSTRNT): This field indicates the number of registrants of the namespace. This indicates the number of Registrant data structures or Registrant Extended data structures contained in this data structure.
Note: This field was formerly named Number of Registered Controllers (REGCTL).
Quoted 8 lines
Though, v2.1 still does not clearly states what exact CNTLID is to be reported from 1..N association.
I suggest you to update your patch to conform the Spec v2.1 - remove cntlid from
nvmet_pr_registrantand report here justns->pr.registrant_list. And not to track the connectivity of registrants - report always CNTLID=0xFFFF as a some compromise value that is allowed by the Spec.
I dont notice that another host can connect with already exist same hostid,
I remove the cntlid in nvmet_pr_registrant and add hostid to nvmet_pr_per_ctrl_ref to
find unregisted controller to kill the per-controlelr percpu ref.
And also report cntlid to 0xffff.
Quoted 7 lines
drivers/nvme/target/pr.c line 908
Quoted 4 lines
drivers/nvme/target/pr.c new file +4
-0+ +put_unaligned_le16 (num_ctrls, data->regctl); + num_bytes =sizeof (struct nvme_reservation_status_ext ) + +sizeof (struct nvme_registered_ctrl_ext ) * num_ctrls;
num_byteswas a size of requested buffer (cdw10), you even allocate that size. You shall not extend it here. Instead you should allocate maximum possible size and copy here only cdw10 dwords from your buffer to sgl.
OK, I will adjust code here.
Best Regards,
Guixin Liu
Quoted 8 lines
drivers/nvme/target/pr.c line 914
Quoted 6 lines
drivers/nvme/target/pr.c new file +6
-0+ status =nvmet_copy_to_sgl (req,0 , data, num_bytes); +kfree (data); +out: +nvmet_req_complete (req, status); +} +BR,
Dmitry