Implement the NVMe reservation feature
Subsystems (2)
v7 cover Implement the NVMe reservation feature
Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.
Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.
- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.
- Fix the resv log page is random when there is no resv log page.
- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().
- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().
- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().
- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().
- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().
- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.
Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().
- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().
- Add the print fmt to the head.
- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.
- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.
Changes from v4 to v5:
- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.
- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.
- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.
- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.
- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.
- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.
- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.
- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.
Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.
- Change the function calling alignment style to:
nvmet_pr_send_event_by_hostid(pr, hostid,
NVME_PR_LOG_RESERVATOPM_PREEMPTED); - Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().
- Remove the goto in __nvmet_pr_unregister_one().
- Change generation to atomic_t, and remove nvmet_pr_inc_generation().
- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.
Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.
- Limit the resv_log_list to 128.
- Change generation to atomic64.
- Put register rkey check to a warpper.
- Change nr_avl_pages to nr_pages.
- Use NVME_SC_SUCCESS instead of 0.
- Change kmalloc param to let it not sleep in mutex lock.
Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.
- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is0xffffffff at each admin command path, if it is needed.
- Add reservation commands support in nvmet_get_cmd_effects_nvm().
- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.
- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.
- Remove kmem_cache instead to use kmalloc and kfree.
- Change others advice from Sagi.
- Add a blktest test case, this patch will be sent before these series of patches.
Guixin Liu (1):
nvmet: support reservation feature
drivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 20 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 51 +- drivers/nvme/target/nvmet.h | 36 ++ drivers/nvme/target/pr.c | 1041 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1224 insertions(+), 7 deletions(-) create mode 100644 drivers/nvme/target/pr.c
Signature · 1 line
2.43.0
PATCH v7 1/1 nvmet: support reservation feature
This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.
And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.
--- drivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 20 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 51 +- drivers/nvme/target/nvmet.h | 36 ++ drivers/nvme/target/pr.c | 1041 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1224 insertions(+), 7 deletions(-) create mode 100644 drivers/nvme/target/pr.c
drivers/nvme/target/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_TARGET_FCLOOP) += nvme-fcloop.o obj-$(CONFIG_NVME_TARGET_TCP) += nvmet-tcp.o nvmet-y += core.o configfs.o admin-cmd.o fabrics-cmd.o \ - discovery.o io-cmd-file.o io-cmd-bdev.o + discovery.o io-cmd-file.o io-cmd-bdev.o pr.o nvmet-$(CONFIG_NVME_TARGET_PASSTHRU) += passthru.o nvmet-$(CONFIG_BLK_DEV_ZONED) += zns.o nvmet-$(CONFIG_NVME_TARGET_AUTH) += fabrics-cmd-auth.o auth.o
drivers/nvme/target/admin-cmd.c +19 -1
@@ -176,6 +176,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log) log->iocs[nvme_cmd_read] = log->iocs[nvme_cmd_flush] = log->iocs[nvme_cmd_dsm] = + log->iocs[nvme_cmd_resv_acquire] = + log->iocs[nvme_cmd_resv_register] = + log->iocs[nvme_cmd_resv_release] = + log->iocs[nvme_cmd_resv_report] =cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); log->iocs[nvme_cmd_write] = log->iocs[nvme_cmd_write_zeroes] =
@@ -340,6 +344,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)return nvmet_execute_get_log_cmd_effects_ns (req);case NVME_LOG_ANA :return nvmet_execute_get_log_page_ana (req); +case NVME_LOG_RESERVATION : +return nvmet_execute_get_log_page_resv (req); }pr_debug ("unhandled lid %d on qid %d\n" , req->cmd->get_log_page.lid, req->sq->qid);
@@ -550,7 +556,13 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)*/ id->nmic =NVME_NS_NMIC_SHARED ; id->anagrpid =cpu_to_le32 (req->ns->anagrpid); - + id->rescap =NVME_PR_SUPPORT_WRITE_EXCLUSIVE | +NVME_PR_SUPPORT_EXCLUSIVE_ACCESS | +NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY | +NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY | +NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS | +NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS | +NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF ;memcpy (&id->nguid, &req->ns->nguid,sizeof (id->nguid)); id->lbaf[0 ].ds = req->ns->blksize_shift;
@@ -851,6 +863,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)case NVME_FEAT_WRITE_PROTECT : status =nvmet_set_feat_write_protect (req);break ; +case NVME_FEAT_RESV_MASK : + status =nvmet_set_feat_resv_notif_mask (req, cdw11); +break ;default : req->error_loc = offsetof(struct nvme_common_command , cdw10); status =NVME_SC_INVALID_FIELD |NVME_SC_DNR ;
@@ -949,6 +964,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)case NVME_FEAT_WRITE_PROTECT : status =nvmet_get_feat_write_protect (req);break ; +case NVME_FEAT_RESV_MASK : + status =nvmet_get_feat_resv_notif_mask (req); +break ;default : req->error_loc = offsetof(struct nvme_common_command , cdw10);
drivers/nvme/target/configfs.c +27 -0
@@ -714,6 +714,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,CONFIGFS_ATTR_WO (nvmet_ns_, revalidate_size); +static ssize_t nvmet_ns_resv_enable_show (struct config_item *item,char *page) +{ +return sprintf (page,"%d\n" ,to_nvmet_ns (item)->pr.enable); +} + +static ssize_t nvmet_ns_resv_enable_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_ns *ns =to_nvmet_ns (item); +bool val; + +if (kstrtobool (page, &val)) +return -EINVAL ; + +mutex_lock (&ns->subsys->lock); +if (ns->enabled) { +pr_err ("the ns:%d is already enabled.\n" , ns->nsid); +mutex_unlock (&ns->subsys->lock); +return -EINVAL ; + } + ns->pr.enable = val; +mutex_unlock (&ns->subsys->lock); +return count; +} +CONFIGFS_ATTR (nvmet_ns_, resv_enable); +static struct configfs_attribute *nvmet_ns_attrs[] = { &nvmet_ns_attr_device_path, &nvmet_ns_attr_device_nguid,
@@ -722,6 +748,7 @@ static struct configfs_attribute *nvmet_ns_attrs[] = { &nvmet_ns_attr_enable, &nvmet_ns_attr_buffered_io, &nvmet_ns_attr_revalidate_size, + &nvmet_ns_attr_resv_enable, #ifdef CONFIG_PCI_P2PDMA &nvmet_ns_attr_p2pmem, #endif
drivers/nvme/target/core.c +46 -5
94 lines not on this page · raw message
drivers/nvme/target/nvmet.h +36 -0
@@ -20,6 +20,7 @@#include <linux/blkdev.h> #include <linux/radix-tree.h> #include <linux/t10-pi.h> +#include <linux/kfifo.h> #define NVMET_DEFAULT_VS NVME_VS(1, 3, 0)
@@ -30,6 +31,7 @@#define NVMET_MN_MAX_SIZE 40#define NVMET_SN_MAX_SIZE 20#define NVMET_FR_MAX_SIZE 8 +#define NVMET_PR_LOG_QUEUE_SIZE 64/* * Supported optional AENs:
@@ -56,6 +58,22 @@#define IPO_IATTR_CONNECT_SQE (x) \ (cpu_to_le32(offsetof(struct nvmf_connect_command, x))) +struct nvmet_pr_registrant { +u64 rkey; +uuid_t hostid; +enum nvme_pr_type rtype; +struct list_head entry; +}; + +struct nvmet_pr { +bool enable; +unsigned long notify_mask; +atomic_t generation; +struct nvmet_pr_registrant __rcu *holder; +struct mutex pr_lock; +struct list_head registrant_list; +}; +struct nvmet_ns {struct percpu_ref ref;struct bdev_handle *bdev_handle;
@@ -85,6 +103,7 @@ struct nvmet_ns {int pi_type;int metadata_size;u8 csi; +struct nvmet_pr pr; };static inline struct nvmet_ns *to_nvmet_ns (struct config_item *item)
@@ -190,6 +209,13 @@ static inline bool nvmet_port_secure_channel_required(struct nvmet_port *port)return nvmet_port_disc_addr_treq_secure_channel (port) ==NVMF_TREQ_REQUIRED ; } +struct nvmet_pr_log_mgr { +struct mutex lock; +u64 lost_count; +u64 counter; + DECLARE_KFIFO(log_queue, struct nvme_pr_log, NVMET_PR_LOG_QUEUE_SIZE); +}; +struct nvmet_ctrl {struct nvmet_subsys *subsys;struct nvmet_sq **sqs;
@@ -243,6 +269,7 @@ struct nvmet_ctrl {u8 *dh_key;size_t dh_keysize;#endif +struct nvmet_pr_log_mgr pr_log_mgr; };struct nvmet_subsys {
@@ -750,4 +777,13 @@ static inline bool nvmet_has_auth(struct nvmet_ctrl *ctrl)static inline const char *nvmet_dhchap_dhgroup_name (u8 dhgid) {return NULL ; }#endif +void nvmet_pr_init_ns (struct nvmet_ns *ns); +u16 nvmet_parse_pr_cmd (struct nvmet_req *req); +u16 nvmet_pr_check_cmd_access (struct nvmet_req *req); +void nvmet_ctrl_destroy_pr (struct nvmet_ctrl *ctrl); +void nvmet_pr_exit_ns (struct nvmet_ns *ns); +bool nvmet_is_host_connected (uuid_t *hostid,struct nvmet_subsys *subsys); +void nvmet_execute_get_log_page_resv (struct nvmet_req *req); +u16 nvmet_set_feat_resv_notif_mask (struct nvmet_req *req,u32 mask); +u16 nvmet_get_feat_resv_notif_mask (struct nvmet_req *req);#endif /* _NVMET_H */
drivers/nvme/target/pr.c new file +1041 -0
1,041 lines not on this page · raw message
include/linux/nvme.h +54 -0
72 lines not on this page · raw message
Signature · 1 line
2.43.0
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Hey Guixin,
Quoted 3 lines · Guixin Liu
This patch implements the reservation feature, includes:
1. reservationregister(register, unregister and replace).
2. reservationacquire(acquire, preempt, preempt and abort).
Overall patchset looks in decent shape to me now.
I do have one more question though.
You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 11 lines · Sagi Grimberg
Hey Guixin,
Quoted 3 lines · Guixin Liu
This patch implements the reservation feature, includes:
1. reservationregister(register, unregister and replace).
2. reservationacquire(acquire, preempt, preempt and abort).Overall patchset looks in decent shape to me now.
I do have one more question though.
My thanks for your patient review.
Quoted 2 lines
You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.
Yes, the nvme target does not do any useful workto abort command, the comment of nvmet_execute_abort() says that
"we are not required to do any useful work, and we couldn't really do a useful abort",
so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.
My statement here may be a little inaccurate.
Best Regards,
Guixin Liu
Keith Busch
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 22 lines · Guixin Liu
Quoted 11 lines · Sagi Grimberg
Hey Guixin,
Quoted 3 lines · Guixin Liu
This patch implements the reservation feature, includes:
1. reservationregister(register, unregister and replace).
2. reservationacquire(acquire, preempt, preempt and abort).Overall patchset looks in decent shape to me now.
I do have one more question though.
My thanks for your patient review.
Quoted 4 lines
You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.
Yes, the nvme target does not do any useful workto abort command, the comment of
nvmet_execute_abort()says that
Quoted 7 lines
"we are not required to do any useful work, and we couldn't really do a useful abort",
so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.
My statement here may be a little inaccurate.
I think you may be mixing up the NVMe "Abort" command with a different command, "Reservation Acquire", where RACQA is set to value 010b, "Preempt and Abort".
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 22 lines · Keith Busch
Quoted 21 lines · Guixin Liu
Quoted 10 lines · Sagi Grimberg
Hey Guixin,
Quoted 3 lines · Guixin Liu
This patch implements the reservation feature, includes:
1. reservationregister(register, unregister and replace).
2. reservationacquire(acquire, preempt, preempt and abort).Overall patchset looks in decent shape to me now.
I do have one more question though.
My thanks for your patient review.
Quoted 3 lines
You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.
Yes, the nvme target does not do any useful workto abort command, the comment of
nvmet_execute_abort()says that"we are not required to do any useful work, and we couldn't really do a useful abort",
Quoted 7 lines
Quoted 4 lines
so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.
My statement here may be a little inaccurate.
I think you may be mixing up the NVMe "Abort" command with a different command, "Reservation Acquire", where RACQA is set to value 010b, "Preempt and Abort".
Sorry, I just want to explain that I dont know how to implement "preempt and abort"
either.
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 27 lines · Guixin Liu
Quoted 25 lines · Keith Busch
Quoted 24 lines · Guixin Liu
Quoted 10 lines · Sagi Grimberg
Hey Guixin,
Quoted 3 lines · Guixin Liu
This patch implements the reservation feature, includes:
1. reservationregister(register, unregister and replace).
2. reservationacquire(acquire, preempt, preempt and abort).Overall patchset looks in decent shape to me now.
I do have one more question though.
My thanks for your patient review.
Quoted 3 lines
You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.
Yes, the nvme target does not do any useful workto abort command, the comment of
nvmet_execute_abort()says that"we are not required to do any useful work, and we couldn't really do a useful abort",
so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
My statement here may be a little inaccurate.
I think you may be mixing up the NVMe "Abort" command with a different command, "Reservation Acquire", where RACQA is set to value 010b, "Preempt and Abort".
Sorry, I just want to explain that I dont know how to implement "preempt and abort"
unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.
A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using percpu_ref_kill and percpu_ref_resurrect on ns->ref). This won't work very efficiently withALL_REGS reservations though.
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 7 lines · Sagi Grimberg
unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.
A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using
percpu_ref_killandpercpu_ref_resurrectonns->ref). This won't work very efficientlywithALL_REGSreservations though.
I will think deeply,and send a v9 patch.
Best regards,
Guixin Liu
Re: [PATCH v7 1/1] nvmet: support reservation feature
In reply to Sagi GrimbergQuoted 8 lines
unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.
A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using
percpu_ref_killandpercpu_ref_resurrectonns->ref). This won't work very efficientlywithALL_REGSreservations though.
Hi Sagi,
I found that if we return an error when the call to percpu_ref_tryget_live(&ns->ref) fails,
it might cause hosts that still have permissions to interrupt their IO. Additionally,
preempt_and_abort itself holds an ns->ref, we cannot wait the ref to become to zero.
The solution I can think of is to add a "per-namespace" percpu_ref to the controller for
counting IO issued to a particular namespace by that controller. Then, during the execution
of preempt_and_abort, we wait for the count of those preempted and unregistered controllers
to drop to zero.
The nsid is user-specified, so we can not use array to store the per-namespace percpu_ref,
this will increase lookup overhead if we use xarray.
What do you think Sagi? Or may be we can declare that preempt_and_abort is not supported, just
like SPDK does.
Best Regards,
Guixin Liu
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 17 lines · Guixin Liu
Quoted 8 lines
unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.
A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using
percpu_ref_killandpercpu_ref_resurrectonns->ref). This won't work very efficientlywithALL_REGSreservations though.Hi Sagi,
I found that if we return an error when the call to
percpu_ref_tryget_live(&ns->ref)fails,it might cause hosts that still have permissions to interrupt their IO. Additionally,
preempt_and_abortitself holds anns->ref, we cannot wait the ref to become to zero.
Quoted 7 lines
The solution I can think of is to add a "per-namespace"
percpu_refto the controller forcounting IO issued to a particular namespace by that controller. Then, during the execution
of
preempt_and_abort, we wait for the count of those preempted and unregistered controllersto drop to zero.
Yes, that is what I had in mind as well. Obviously the ns->ref cannot be used for this purpose.
Quoted 4 lines
The nsid is user-specified, so we can not use array to store the per-namespace
percpu_ref,this will increase lookup overhead if we use xarray.
Yes, that is tricky to get right.
Quoted 4 lines
What do you think Sagi? Or may be we can declare that
preempt_and_abortis not supported, justlike SPDK does.
It can definitely come incrementally, but at the very least it should be incorrectly supported.
Out of curiosity, doesn't your use-case need a fencing protection against inflight I/Os reordering during preemption?
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 9 lines
Quoted 4 lines
What do you think Sagi? Or may be we can declare that
preempt_and_abortis not supported, justlike SPDK does.
It can definitely come incrementally, but at the very least it should be incorrectly supported.
Out of curiosity, doesn't your use-case need a fencing protection against inflight I/Os reordering during preemption?
The cluster use stonith(shoot the other node in the head) to protect sources,
and the backend storage(our storage system goes like this, so does disk firmware I think.)
can process write I/O operations on a first-come,first-served basis if the range of
data accessed by two I/O operation overlaps, for example, hostB preempt and abort
hostA, then host B send I/Os, the backend storage will handle hostA's I/O first(by the timestamp).
Best regards,
Guixin Liu
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 12 lines · Guixin Liu
Quoted 9 lines
Quoted 4 lines
What do you think Sagi? Or may be we can declare that
preempt_and_abortis not supported, justlike SPDK does.
It can definitely come incrementally, but at the very least it should be incorrectly supported.
Out of curiosity, doesn't your use-case need a fencing protection against inflight I/Os reordering during preemption?
The cluster use stonith(shoot the other node in the head) to protect sources,
Quoted 7 lines
and the backend
storage(our storage system goes like this, so does disk firmware I think.)can process write I/O operations on a first-come,first-served basis if the range of
data accessed by two I/O operation overlaps, for example, hostB preempt and abort
hostA, then host B send I/Os, the backend storage will handle hostA's I/O first(by the timestamp).
So I understand from you that there is out-of-band mechanism for fence guarantee?
In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.
I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.
btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 5 lines · Sagi Grimberg
So I understand from you that there is out-of-band mechanism for fence guarantee?
In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.
Agree.
Quoted 1 line
I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.
I will implement it in v9, and I also think that we should a time-limited wait.
Quoted 4 lines
btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).
You mean one host can reserve all of namespaces by sending one reservation command?
Well, Could you please tell me which two use-cases?
I think this will make the target code too complicated, because the reservation situation on each
namespace may different, we should handle it one-by-one, and if failed in middle, we should revert
the handled namespaces.
Best regards,
Guixin Liu
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 11 lines · Guixin Liu
Quoted 5 lines · Sagi Grimberg
So I understand from you that there is out-of-band mechanism for fence guarantee?
In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.
Agree.
Quoted 1 line
I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.
I will implement it in v9, and I also think that we should a time-limited wait.
If it is time-limited, and you didn't manage to wait, so you need to fail the preempt and abort action (without DNR set so the reservation holder can retry if it chooses to).
Quoted 7 lines
Quoted 5 lines
btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).
You mean one host can reserve all of namespaces by sending one reservation command?
Yes. The use-case is to have the reservation in the subsystem level. The spec seems to imply that this is a valid reservation request.
Quoted 2 lines
Well, Could you please tell me which two use-cases?
Don't remember for sure, but IIRC I've seen people run oracle rac with all namespaces in the nvmet subsystem is exposed to DB hosts, and there were many namespaces.
Quoted 6 lines
I think this will make the target code too complicated, because the reservation situation on each
namespace may different, we should handle it one-by-one, and if failed in middle, we should revert
the handled namespaces.
Not sure why you'd assume that, this would be a subsystem-wide reservation, which should simply cross-check individual namespace reservations for conflicts. Sure it adds complication, but I think implementing it on every individual namespace is a naive implementation.
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 10 lines · Sagi Grimberg
Quoted 7 lines · Guixin Liu
Quoted 5 lines · Sagi Grimberg
So I understand from you that there is out-of-band mechanism for fence guarantee?
In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.
Quoted 8 lines
Quoted 4 lines
Agree.
Quoted 1 line
I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.
I will implement it in v9, and I also think that we should a time-limited wait.
If it is time-limited, and you didn't manage to wait, so you need to fail the preempt and abort action (without DNR set so the reservation holder can retry if it chooses to).
It can wait.
Quoted 20 lines
Quoted 7 lines
Quoted 5 lines
btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).
You mean one host can reserve all of namespaces by sending one reservation command?
Yes. The use-case is to have the reservation in the subsystem level. The spec seems to imply that this is a valid reservation request.
Quoted 2 lines
Well, Could you please tell me which two use-cases?
Don't remember for sure, but IIRC I've seen people run oracle rac with all namespaces in the nvmet subsystem is exposed to DB hosts, and there were many namespaces.
Quoted 3 lines
I think this will make the target code too complicated, because the reservation situation on each
Quoted 7 lines
Quoted 3 lines
namespace may different, we should handle it one-by-one, and if failed in middle, we should revert
the handled namespaces.
Not sure why you'd assume that, this would be a subsystem-wide reservation, which should simply cross-check individual namespace reservations for conflicts. Sure it adds complication, but I think implementing it on every individual namespace is a naive implementation.
Well, this is strange, a few years ago, when I tested SCSI reservations with Oracle RAC,
I did not observe any "all LUN" reservation commands being issued.
I also researched LIO and SPDK, and found that neither supports "all LUN/NS" reservation commands.
When a host issued a reservation command without carrying a specific LUN or NS ID, both returned an error.
Best regards,
Guixin Liu
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 49 lines · Guixin Liu
Quoted 10 lines · Sagi Grimberg
Quoted 7 lines · Guixin Liu
Quoted 5 lines · Sagi Grimberg
So I understand from you that there is out-of-band mechanism for fence guarantee?
In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.
Quoted 8 lines
Quoted 4 lines
Agree.
Quoted 1 line
I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.
I will implement it in v9, and I also think that we should a time-limited wait.
If it is time-limited, and you didn't manage to wait, so you need to fail the preempt and abort action (without DNR set so the reservation holder can retry if it chooses to).
It can wait.
Quoted 20 lines
Quoted 7 lines
Quoted 5 lines
btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).
You mean one host can reserve all of namespaces by sending one reservation command?
Yes. The use-case is to have the reservation in the subsystem level. The spec seems to imply that this is a valid reservation request.
Quoted 2 lines
Well, Could you please tell me which two use-cases?
Don't remember for sure, but IIRC I've seen people run oracle rac with all namespaces in the nvmet subsystem is exposed to DB hosts, and there were many namespaces.
Quoted 3 lines
I think this will make the target code too complicated, because the reservation situation on each
Quoted 7 lines
Quoted 3 lines
namespace may different, we should handle it one-by-one, and if failed in middle, we should revert
the handled namespaces.
Not sure why you'd assume that, this would be a subsystem-wide reservation, which should simply cross-check individual namespace reservations for conflicts. Sure it adds complication, but I think implementing it on every individual namespace is a naive implementation.
Quoted 7 lines
Well, this is strange, a few years ago, when I tested SCSI reservations with Oracle RAC,
I did not observe any "all LUN" reservation commands being issued.
I also researched LIO and SPDK, and found that neither supports "all LUN/NS" reservation commands.
When a host issued a reservation command without carrying a specific LUN or NS ID, both returned an error.
Didn't say this is actually in use, but a use-case that would benefit.
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 3 lines
Didn't say this is actually in use, but a use-case that would benefit.
I will add a TODO-list to research main cluster softwares
to see if this is actually a significant use-case, after my patch merged.
I'll change my code then.
Best regards,
Guixin Liu
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 6 lines · Guixin Liu
Quoted 3 lines
Didn't say this is actually in use, but a use-case that would benefit.
I will add a TODO-list to research main cluster softwares
Quoted 8 lines
to see if this is actually a significant use-case, after my patch merged.
I'll change my code then.
Best regards,
Guixin Liu
so which one is the latest version I use for testing ? I want to run some tests so I can provide reviewed-by tag ...
-ck
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 9 lines · Chaitanya Kulkarni
Quoted 8 lines · Guixin Liu
Quoted 2 lines
Didn't say this is actually in use, but a use-case that would benefit.
I will add a TODO-list to research main cluster softwares
to see if this is actually a significant use-case, after my patch merged.
I'll change my code then.
Quoted 8 lines
Quoted 4 lines
Best regards,
Guixin Liu
so which one is the latest version I use for testing ? I want to run some tests so I can provide reviewed-by tag ...
-ck
I am adding a per-controller percpu-ref to ns to implement preempt_and_abort, the v9 will be soon.
Thanks,
Guixin Liu
Quoted 1 line
[email protected]
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 2 lines · Guixin Liu
I am adding a per-controller percpu-ref to ns to implement
preempt_and_abort, the v9 will be soon.
Do we need that? For blk-mq based backends just doing a queue freeze would be enough. That doesn't work for bio based drivers yet, but I'd rather improve the block layer freezing for them rather than reimplenting the whole thing.
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 7 lines · On 20/03/2024 3:59, [email protected] wrote:
Quoted 2 lines · Guixin Liu
I am adding a per-controller percpu-ref to ns to implement
preempt_and_abort, the v9 will be soon.Do we need that? For blk-mq based backends just doing a queue freeze would be enough. That doesn't work for bio based drivers yet, but I'd rather improve the block layer freezing for them rather than reimplenting the whole thing.
That's a neat idea, we also have file to think about though.
Re: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 4 lines · Sagi Grimberg
Quoted 1 line · On 20/03/2024 3:59, [email protected] wrote:
On Tue, Mar 19, 2024 at 11:19:50AM +0800, Guixin Liu wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I am adding a per-controller percpu-ref to ns to implement
preempt_and_abort, the v9 will be soon.Do we need that? For blk-mq based backends just doing a queue freeze would be enough. That doesn't work for bio based drivers yet, but I'd rather improve the block layer freezing for them rather than reimplenting the whole thing.
That's a neat idea, we also have file to think about though.
I failed to understand how this will work for non block device (passthru or bio mode) backend(s). Unless there is a plan to create similar API for non block device backhend(s) and if that turns out to be overkill then why not go with current per-controller percpu-ref ?
-ck
[email protected]
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
In reply to Sagi GrimbergQuoted 1 line · Sagi Grimberg
That's a neat idea, we also have file to think about though.
Indeed. It still would be good to avoid the extra overhead when not needed, though.
Re: [PATCH v7 1/1] nvmet: support reservation feature
Christoph,
Quoted 5 lines · On 3/21/2024 2:02 PM, [email protected] wrote:
Quoted 1 line · Sagi Grimberg
That's a neat idea, we also have file to think about though.
Indeed. It still would be good to avoid the extra overhead when not needed, though.
I still don't understand how this solves the case for file back-end ? can you please explain in detail ?
as far as I can there is no equivalent provision that we can use for file back-end as you have suggested for block (bio/blk-mq)back-end, if there is can you please provide some details ?
In case there is no such provision, how do you see file backends to handle this case ? perhaps you are referring to use above suggested mechanism for block backend and implement percpu-ref to for file only backend so we can keep the code clean for bio and passthru back-end vs file back-end ?
-ck
Sagi Grimberg
maintainerRe: [PATCH v7 1/1] nvmet: support reservation feature
Quoted 14 lines · Chaitanya Kulkarni
Christoph,
Quoted 4 lines · On 3/21/2024 2:02 PM, [email protected] wrote:
Quoted 1 line · Sagi Grimberg
That's a neat idea, we also have file to think about though.
Indeed. It still would be good to avoid the extra overhead when not needed, though.
I still don't understand how this solves the case for file back-end ? can you please explain in detail ?
as far as I can there is no equivalent provision that we can use for file back-end as you have suggested for block (bio/blk-mq)back-end, if there is can you please provide some details ?
Quoted 5 lines
In case there is no such provision, how do you see file backends to handle this case ? perhaps you are referring to use above suggested mechanism for block backend and implement percpu-ref to for file only backend so we can keep the code clean for bio and passthru back-end vs file back-end ?
Yes, the latter.
Hi,
Gentle ping...
Best regards.
Guixin Liu
Quoted 143 lines · Guixin Liu
Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.- Add all the registrants that need to be freed to a temporary list fist, and then after calling
synchronize_rcu(), release all the registrants on the temporary list.- Fix the resv log page is random when there is no resv log page.
- Change
nvmet_is_host_still_connected()tonvmet_is_host_connected().- Remove
nvmet_pr_set_rtype_and_holder()and changenvmet_pr_create_new_resv()tonvmet_pr_create_new_reservation().- Change
nvmet_pr_find_registrant_by_hostid()tonvmet_pr_find_registrant().- Change
nvmet_pr_send_resv_released()tonvmet_pr_resv_released().- Change
__nvmet_pr_unregister_one()tonvmet_pr_unregister_one().- In
nvmet_pr_unreg_by_prkey(),nvmet_pr_unreg_by_prkey_except_hostid()andnvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.Changes from v5 to v6:
- Usesynchronize_rcu()andkfree()to free registrant instead ofkfree_rcu().- Remove
nvmet_pr_register_check_rkey(), put the check intopr_lockwarp. And refactor thenvmet_pr_register().- Add the print fmt to the head.
- Add
lockdep_is_held(&pr->pr_lock)condition tolist_for_each_entry_rcu.- Fix the bug in
nvmet_pr_update_reg_attr(), when thechange_attrhook return fail, we should not replace the holder.Changes from v4 to v5:
- Use rculist macros to handle
registration_listinstead of list macros regardless of in mutex lock or not.- Use goto statement instead of return in
nvmet_is_host_still_connectedand__nvmet_pr_unregister_one.- Add
lockdep_assert_heldandrcu_read_lock_heldassert to many functions, if it's necessary.- Add a comment to
nvmet_execute_get_log_page_resvto explain howlost_countworks.- In
nvmet_pr_clear, we should set holder to NULL first, I fixed this.- Unify
nvmet_pr_update_holder_rtypeand__nvmet_pr_do_replacetonvmet_pr_update_reg_attr.- Fix wrong
nr_pagesinnvmet_execute_get_log_page_resv.- Fix the deadlock issue of
nvmet_pr_exit_ns, put it out of the subsys lock.Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.- Change the function calling alignment style to:
nvmet_pr_send_event_by_hostid(pr, hostid, NVME_PR_LOG_RESERVATOPM_PREEMPTED);- Put kmalloc out of
rcu_read_lockinnvmet_execute_pr_report().- Remove the goto in
__nvmet_pr_unregister_one().- Change generation to
atomic_t, and removenvmet_pr_inc_generation().- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.
Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the structnvmet_pr_registrant.- Limit the
resv_log_listto 128.- Change generation to atomic64.
- Put register rkey check to a warpper.
- Change
nr_avl_pagestonr_pages.- Use
NVME_SC_SUCCESSinstead of 0.- Change kmalloc param to let it not sleep in mutex lock.
Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is0xffffffffat each admin command path, if it is needed.- Add reservation commands support in
nvmet_get_cmd_effects_nvm().- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.- Put the
nvmet_pr_check_cmd_accessandnvmet_parse_pr_cmdinto reservation enable check warp.- Remove
kmem_cacheinstead to use kmalloc and kfree.- Change others advice from Sagi.
- Add a blktest test case, this patch will be sent before these series of patches.
Guixin Liu (1):
nvmet: support reservation featuredrivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 20 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 51 +- drivers/nvme/target/nvmet.h | 36 ++ drivers/nvme/target/pr.c | 1041 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1224 insertions(+), 7 deletions(-)create mode 100644
drivers/nvme/target/pr.c
Hi Christoph and Chaitanya,
I noticed that Sagi hasn't replied to any emails for a month, could
you please review this patch?
Best Regards,
Guixin Liu
Quoted 167 lines · Guixin Liu
Hi,
Gentle ping...
Best regards.
Guixin Liu
Quoted 158 lines · Guixin Liu
Hi guys,
I've implemented the NVMe reservation feature. Please review it, all comments are welcome as usual.Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation
log.- Add all the registrants that need to be freed to a temporary list fist,
and then after callingsynchronize_rcu(), release all the registrants on the temporary list.- Fix the resv log page is random when there is no resv log page.
- Change
nvmet_is_host_still_connected()tonvmet_is_host_connected().- Remove
nvmet_pr_set_rtype_and_holder()and changenvmet_pr_create_new_resv()
tonvmet_pr_create_new_reservation().- Change
nvmet_pr_find_registrant_by_hostid()tonvmet_pr_find_registrant().- Change
nvmet_pr_send_resv_released()tonvmet_pr_resv_released().- Change
__nvmet_pr_unregister_one()tonvmet_pr_unregister_one().- In
nvmet_pr_unreg_by_prkey(),nvmet_pr_unreg_by_prkey_except_hostid()andnvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.Changes from v5 to v6:
- Usesynchronize_rcu()andkfree()to free registrant instead ofkfree_rcu().- Remove
nvmet_pr_register_check_rkey(), put the check intopr_lockwarp.
And refactor thenvmet_pr_register().- Add the print fmt to the head.
- Add
lockdep_is_held(&pr->pr_lock)condition tolist_for_each_entry_rcu.- Fix the bug in
nvmet_pr_update_reg_attr(), when thechange_attrhook return fail, we should not replace the holder.Changes from v4 to v5:
- Use rculist macros to handle
registration_listinstead of list macros regardless of in mutex lock or not.- Use goto statement instead of return in
nvmet_is_host_still_connectedand__nvmet_pr_unregister_one.- Add
lockdep_assert_heldandrcu_read_lock_heldassert to many functions,
if it's necessary.- Add a comment to
nvmet_execute_get_log_page_resvto explain howlost_count
works.- In
nvmet_pr_clear, we should set holder to NULL first, I fixed this.- Unify
nvmet_pr_update_holder_rtypeand__nvmet_pr_do_replacetonvmet_pr_update_reg_attr.- Fix wrong
nr_pagesinnvmet_execute_get_log_page_resv.- Fix the deadlock issue of
nvmet_pr_exit_ns, put it out of the subsys lock.Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.- Change the function calling alignment style to:
nvmet_pr_send_event_by_hostid(pr, hostid,
NVME_PR_LOG_RESERVATOPM_PREEMPTED);- Put kmalloc out of
rcu_read_lockinnvmet_execute_pr_report().- Remove the goto in
__nvmet_pr_unregister_one().- Change generation to
atomic_t, and removenvmet_pr_inc_generation().- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.
Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype
into the structnvmet_pr_registrant.- Limit the
resv_log_listto 128.- Change generation to atomic64.
- Put register rkey check to a warpper.
- Change
nr_avl_pagestonr_pages.- Use
NVME_SC_SUCCESSinstead of 0.- Change kmalloc param to let it not sleep in mutex lock.
Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get reservation log page command to clear log page at host.- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check, so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is0xffffffffat each admin command path, if it is needed.- Add reservation commands support in
nvmet_get_cmd_effects_nvm().- From Chaitanya, change the local variable tree style to make it cleaner,
and add some comments about NVMe spec.
And also change others advice from chaitanya.- Put the
nvmet_pr_check_cmd_accessandnvmet_parse_pr_cmdinto reservation
enable check warp.- Remove
kmem_cacheinstead to use kmalloc and kfree.- Change others advice from Sagi.
- Add a blktest test case, this patch will be sent before these series of
patches.Guixin Liu (1):
nvmet: support reservation featuredrivers/nvme/target/Makefile | 2 +-
drivers/nvme/target/admin-cmd.c| 20 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 51 +- drivers/nvme/target/nvmet.h | 36 ++ drivers/nvme/target/pr.c | 1041 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++7 files changed, 1224 insertions(+), 7 deletions(-)create mode 100644
drivers/nvme/target/pr.c
Quoted 10 lines · Guixin Liu
Hi Christoph and Chaitanya,
I noticed that Sagi hasn't replied to any emails for a month, could
you please review this patch?
Best Regards,
Guixin Liu
Hi,
You will have my review by end of this week worst case on weekend.
-ck
Hi guys,
I will send a v8 patch to add me as the new file's maintainer.
Best Regards,
Guixin Liu
Quoted 143 lines · Guixin Liu
Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.- Add all the registrants that need to be freed to a temporary list fist, and then after calling
synchronize_rcu(), release all the registrants on the temporary list.- Fix the resv log page is random when there is no resv log page.
- Change
nvmet_is_host_still_connected()tonvmet_is_host_connected().- Remove
nvmet_pr_set_rtype_and_holder()and changenvmet_pr_create_new_resv()tonvmet_pr_create_new_reservation().- Change
nvmet_pr_find_registrant_by_hostid()tonvmet_pr_find_registrant().- Change
nvmet_pr_send_resv_released()tonvmet_pr_resv_released().- Change
__nvmet_pr_unregister_one()tonvmet_pr_unregister_one().- In
nvmet_pr_unreg_by_prkey(),nvmet_pr_unreg_by_prkey_except_hostid()andnvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.Changes from v5 to v6:
- Usesynchronize_rcu()andkfree()to free registrant instead ofkfree_rcu().- Remove
nvmet_pr_register_check_rkey(), put the check intopr_lockwarp. And refactor thenvmet_pr_register().- Add the print fmt to the head.
- Add
lockdep_is_held(&pr->pr_lock)condition tolist_for_each_entry_rcu.- Fix the bug in
nvmet_pr_update_reg_attr(), when thechange_attrhook return fail, we should not replace the holder.Changes from v4 to v5:
- Use rculist macros to handle
registration_listinstead of list macros regardless of in mutex lock or not.- Use goto statement instead of return in
nvmet_is_host_still_connectedand__nvmet_pr_unregister_one.- Add
lockdep_assert_heldandrcu_read_lock_heldassert to many functions, if it's necessary.- Add a comment to
nvmet_execute_get_log_page_resvto explain howlost_countworks.- In
nvmet_pr_clear, we should set holder to NULL first, I fixed this.- Unify
nvmet_pr_update_holder_rtypeand__nvmet_pr_do_replacetonvmet_pr_update_reg_attr.- Fix wrong
nr_pagesinnvmet_execute_get_log_page_resv.- Fix the deadlock issue of
nvmet_pr_exit_ns, put it out of the subsys lock.Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.- Change the function calling alignment style to:
nvmet_pr_send_event_by_hostid(pr, hostid, NVME_PR_LOG_RESERVATOPM_PREEMPTED);- Put kmalloc out of
rcu_read_lockinnvmet_execute_pr_report().- Remove the goto in
__nvmet_pr_unregister_one().- Change generation to
atomic_t, and removenvmet_pr_inc_generation().- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.
Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the structnvmet_pr_registrant.- Limit the
resv_log_listto 128.- Change generation to atomic64.
- Put register rkey check to a warpper.
- Change
nr_avl_pagestonr_pages.- Use
NVME_SC_SUCCESSinstead of 0.- Change kmalloc param to let it not sleep in mutex lock.
Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is0xffffffffat each admin command path, if it is needed.- Add reservation commands support in
nvmet_get_cmd_effects_nvm().- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.- Put the
nvmet_pr_check_cmd_accessandnvmet_parse_pr_cmdinto reservation enable check warp.- Remove
kmem_cacheinstead to use kmalloc and kfree.- Change others advice from Sagi.
- Add a blktest test case, this patch will be sent before these series of patches.
Guixin Liu (1):
nvmet: support reservation featuredrivers/nvme/target/Makefile | 2 +- drivers/nvme/target/admin-cmd.c | 20 +- drivers/nvme/target/configfs.c | 27 + drivers/nvme/target/core.c | 51 +- drivers/nvme/target/nvmet.h | 36 ++ drivers/nvme/target/pr.c | 1041 +++++++++++++++++++++++++++++++ include/linux/nvme.h | 54 ++ 7 files changed, 1224 insertions(+), 7 deletions(-)create mode 100644
drivers/nvme/target/pr.c