Skip to contentSkip to conversation

Implement the NVMe reservation feature

PATCH Superseded Guixin Liu 5 participants 29 messages Feb 1, 2024 – Mar 23, 2024

Subsystems (2)

v7 of 19 Superseded Latest is v19 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v7 cover Implement the NVMe reservation feature

Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.

Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.

- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.

- Fix the resv log page is random when there is no resv log page.

- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().

- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().

- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().

- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().

- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().

- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.

Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().

- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().

- Add the print fmt to the head.

- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.

- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.

Changes from v4 to v5:

- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.

- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.

- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.

- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.

- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.

- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.

- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.

- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.

Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.

- Change the function calling alignment style to:

    nvmet_pr_send_event_by_hostid(pr, hostid,
            NVME_PR_LOG_RESERVATOPM_PREEMPTED); 

- Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().

- Remove the goto in __nvmet_pr_unregister_one().

- Change generation to atomic_t, and remove nvmet_pr_inc_generation().

- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.

Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.

- Limit the resv_log_list to 128.

- Change generation to atomic64.

- Put register rkey check to a warpper.

- Change nr_avl_pages to nr_pages.

- Use NVME_SC_SUCCESS instead of 0.

- Change kmalloc param to let it not sleep in mutex lock.

Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.

- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is
0xffffffff at each admin command path, if it is needed.

- Add reservation commands support in nvmet_get_cmd_effects_nvm().

- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.

- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.

- Remove kmem_cache instead to use kmalloc and kfree.

- Change others advice from Sagi.

- Add a blktest test case, this patch will be sent before these series of patches.

Guixin Liu (1):
nvmet: support reservation feature

 drivers/nvme/target/Makefile    |    2 +-
 drivers/nvme/target/admin-cmd.c |   20 +-
 drivers/nvme/target/configfs.c  |   27 +
 drivers/nvme/target/core.c      |   51 +-
 drivers/nvme/target/nvmet.h     |   36 ++
 drivers/nvme/target/pr.c        | 1041 +++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   54 ++
 7 files changed, 1224 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v7 1/1 nvmet: support reservation feature

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).
3. reservation release(release and clear).
4. reservation report.

And also make reservation configurable, one can set ns to support reservation before enable ns. The default of resv_enable is false.

---
 drivers/nvme/target/Makefile    |    2 +-
 drivers/nvme/target/admin-cmd.c |   20 +-
 drivers/nvme/target/configfs.c  |   27 +
 drivers/nvme/target/core.c      |   51 +-
 drivers/nvme/target/nvmet.h     |   36 ++
 drivers/nvme/target/pr.c        | 1041 +++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   54 ++
 7 files changed, 1224 insertions(+), 7 deletions(-)
 create mode 100644 drivers/nvme/target/pr.c
drivers/nvme/target/Makefile +1 -1
index c66820102493..f9bfc904a5b3 100644
@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_TARGET_FCLOOP)	+= nvme-fcloop.o
 obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
-			discovery.o io-cmd-file.o io-cmd-bdev.o
+			discovery.o io-cmd-file.o io-cmd-bdev.o pr.o
 nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvmet-$(CONFIG_BLK_DEV_ZONED)		+= zns.o
 nvmet-$(CONFIG_NVME_TARGET_AUTH)	+= fabrics-cmd-auth.o auth.o
drivers/nvme/target/admin-cmd.c +19 -1
index 39cb570f833d..eb1d85701441 100644
@@ -176,6 +176,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log)
 	log->iocs[nvme_cmd_read] =
 	log->iocs[nvme_cmd_flush] =
 	log->iocs[nvme_cmd_dsm]	=
+	log->iocs[nvme_cmd_resv_acquire] =
+	log->iocs[nvme_cmd_resv_register] =
+	log->iocs[nvme_cmd_resv_release] =
+	log->iocs[nvme_cmd_resv_report] =
 		cpu_to_le32(NVME_CMD_EFFECTS_CSUPP);
 	log->iocs[nvme_cmd_write] =
 	log->iocs[nvme_cmd_write_zeroes] =
@@ -340,6 +344,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
 		return nvmet_execute_get_log_cmd_effects_ns(req);
 	case NVME_LOG_ANA:
 		return nvmet_execute_get_log_page_ana(req);
+	case NVME_LOG_RESERVATION:
+		return nvmet_execute_get_log_page_resv(req);
 	}
 	pr_debug("unhandled lid %d on qid %d\n",
 	       req->cmd->get_log_page.lid, req->sq->qid);
@@ -550,7 +556,13 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req)
 	 */
 	id->nmic = NVME_NS_NMIC_SHARED;
 	id->anagrpid = cpu_to_le32(req->ns->anagrpid);
-
+	id->rescap = NVME_PR_SUPPORT_WRITE_EXCLUSIVE |
+		     NVME_PR_SUPPORT_EXCLUSIVE_ACCESS |
+		     NVME_PR_SUPPORT_WRITE_EXCLUSIVE_REG_ONLY |
+		     NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_REG_ONLY |
+		     NVME_PR_SUPPORT_WRITE_EXCLUSIVE_ALL_REGS |
+		     NVME_PR_SUPPORT_EXCLUSIVE_ACCESS_ALL_REGS |
+		     NVME_PR_SUPPORT_IEKEY_VER_1_3_DEF;
 	memcpy(&id->nguid, &req->ns->nguid, sizeof(id->nguid));
 
 	id->lbaf[0].ds = req->ns->blksize_shift;
@@ -851,6 +863,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_set_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_set_feat_resv_notif_mask(req, cdw11);
+		break;
 	default:
 		req->error_loc = offsetof(struct nvme_common_command, cdw10);
 		status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
@@ -949,6 +964,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)
 	case NVME_FEAT_WRITE_PROTECT:
 		status = nvmet_get_feat_write_protect(req);
 		break;
+	case NVME_FEAT_RESV_MASK:
+		status = nvmet_get_feat_resv_notif_mask(req);
+		break;
 	default:
 		req->error_loc =
 			offsetof(struct nvme_common_command, cdw10);
drivers/nvme/target/configfs.c +27 -0
index 2482a0db2504..56aead89d21a 100644
@@ -714,6 +714,32 @@ static ssize_t nvmet_ns_revalidate_size_store(struct config_item *item,
 
 CONFIGFS_ATTR_WO(nvmet_ns_, revalidate_size);
 
+static ssize_t nvmet_ns_resv_enable_show(struct config_item *item, char *page)
+{
+	return sprintf(page, "%d\n", to_nvmet_ns(item)->pr.enable);
+}
+
+static ssize_t nvmet_ns_resv_enable_store(struct config_item *item,
+					const char *page, size_t count)
+{
+	struct nvmet_ns *ns = to_nvmet_ns(item);
+	bool val;
+
+	if (kstrtobool(page, &val))
+		return -EINVAL;
+
+	mutex_lock(&ns->subsys->lock);
+	if (ns->enabled) {
+		pr_err("the ns:%d is already enabled.\n", ns->nsid);
+		mutex_unlock(&ns->subsys->lock);
+		return -EINVAL;
+	}
+	ns->pr.enable = val;
+	mutex_unlock(&ns->subsys->lock);
+	return count;
+}
+CONFIGFS_ATTR(nvmet_ns_, resv_enable);
+
 static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_device_path,
 	&nvmet_ns_attr_device_nguid,
@@ -722,6 +748,7 @@ static struct configfs_attribute *nvmet_ns_attrs[] = {
 	&nvmet_ns_attr_enable,
 	&nvmet_ns_attr_buffered_io,
 	&nvmet_ns_attr_revalidate_size,
+	&nvmet_ns_attr_resv_enable,
 #ifdef CONFIG_PCI_P2PDMA
 	&nvmet_ns_attr_p2pmem,
 #endif
drivers/nvme/target/core.c +46 -5

94 lines not on this page · raw message

drivers/nvme/target/nvmet.h +36 -0
index 6c8acebe1a1a..4f8416e7353d 100644
@@ -20,6 +20,7 @@
 #include <linux/blkdev.h>
 #include <linux/radix-tree.h>
 #include <linux/t10-pi.h>
+#include <linux/kfifo.h>
 
 #define NVMET_DEFAULT_VS		NVME_VS(1, 3, 0)
 
@@ -30,6 +31,7 @@
 #define NVMET_MN_MAX_SIZE		40
 #define NVMET_SN_MAX_SIZE		20
 #define NVMET_FR_MAX_SIZE		8
+#define NVMET_PR_LOG_QUEUE_SIZE		64
 
 /*
  * Supported optional AENs:
@@ -56,6 +58,22 @@
 #define IPO_IATTR_CONNECT_SQE(x)	\
 	(cpu_to_le32(offsetof(struct nvmf_connect_command, x)))
 
+struct nvmet_pr_registrant {
+	u64			rkey;
+	uuid_t			hostid;
+	enum nvme_pr_type	rtype;
+	struct list_head	entry;
+};
+
+struct nvmet_pr {
+	bool			enable;
+	unsigned long		notify_mask;
+	atomic_t		generation;
+	struct nvmet_pr_registrant __rcu *holder;
+	struct mutex		pr_lock;
+	struct list_head	registrant_list;
+};
+
 struct nvmet_ns {
 	struct percpu_ref	ref;
 	struct bdev_handle	*bdev_handle;
@@ -85,6 +103,7 @@ struct nvmet_ns {
 	int			pi_type;
 	int			metadata_size;
 	u8			csi;
+	struct nvmet_pr		pr;
 };
 
 static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item)
@@ -190,6 +209,13 @@ static inline bool nvmet_port_secure_channel_required(struct nvmet_port *port)
     return nvmet_port_disc_addr_treq_secure_channel(port) == NVMF_TREQ_REQUIRED;
 }
 
+struct nvmet_pr_log_mgr {
+	struct mutex		lock;
+	u64			lost_count;
+	u64			counter;
+	DECLARE_KFIFO(log_queue, struct nvme_pr_log, NVMET_PR_LOG_QUEUE_SIZE);
+};
+
 struct nvmet_ctrl {
 	struct nvmet_subsys	*subsys;
 	struct nvmet_sq		**sqs;
@@ -243,6 +269,7 @@ struct nvmet_ctrl {
 	u8			*dh_key;
 	size_t			dh_keysize;
 #endif
+	struct nvmet_pr_log_mgr pr_log_mgr;
 };
 
 struct nvmet_subsys {
@@ -750,4 +777,13 @@ static inline bool nvmet_has_auth(struct nvmet_ctrl *ctrl)
 static inline const char *nvmet_dhchap_dhgroup_name(u8 dhgid) { return NULL; }
 #endif
 
+void nvmet_pr_init_ns(struct nvmet_ns *ns);
+u16 nvmet_parse_pr_cmd(struct nvmet_req *req);
+u16 nvmet_pr_check_cmd_access(struct nvmet_req *req);
+void nvmet_ctrl_destroy_pr(struct nvmet_ctrl *ctrl);
+void nvmet_pr_exit_ns(struct nvmet_ns *ns);
+bool nvmet_is_host_connected(uuid_t *hostid, struct nvmet_subsys *subsys);
+void nvmet_execute_get_log_page_resv(struct nvmet_req *req);
+u16 nvmet_set_feat_resv_notif_mask(struct nvmet_req *req, u32 mask);
+u16 nvmet_get_feat_resv_notif_mask(struct nvmet_req *req);
 #endif /* _NVMET_H */
drivers/nvme/target/pr.c new file +1041 -0

1,041 lines not on this page · raw message

include/linux/nvme.h +54 -0

72 lines not on this page · raw message

Signature · 1 line
2.43.0
maintainer
#2 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Hey Guixin,

Quoted 3 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).

Overall patchset looks in decent shape to me now.

I do have one more question though.

You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.

author
#3 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 11 lines · Sagi Grimberg

Hey Guixin,

Quoted 3 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).

Overall patchset looks in decent shape to me now.

I do have one more question though.

My thanks for your patient review.

Quoted 2 lines

You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.

Yes, the nvme target does not do any useful workto abort command, the comment of nvmet_execute_abort() says that

"we are not required to do any useful work, and we couldn't really do a useful abort",

so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.

My statement here may be a little inaccurate.

Best Regards,

Guixin Liu

maintainer
#4 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 22 lines · Guixin Liu
Quoted 11 lines · Sagi Grimberg

Hey Guixin,

Quoted 3 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).

Overall patchset looks in decent shape to me now.

I do have one more question though.

My thanks for your patient review.

Quoted 4 lines

You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.

Yes, the nvme target does not do any useful workto abort command, the comment of nvmet_execute_abort() says that

Quoted 7 lines

"we are not required to do any useful work, and we couldn't really do a useful abort",

so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.

My statement here may be a little inaccurate.

I think you may be mixing up the NVMe "Abort" command with a different command, "Reservation Acquire", where RACQA is set to value 010b, "Preempt and Abort".

author
#5 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 22 lines · Keith Busch
Quoted 21 lines · Guixin Liu
Quoted 10 lines · Sagi Grimberg

Hey Guixin,

Quoted 3 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).

Overall patchset looks in decent shape to me now.

I do have one more question though.

My thanks for your patient review.

Quoted 3 lines

You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.

Yes, the nvme target does not do any useful workto abort command, the comment of nvmet_execute_abort() says that

"we are not required to do any useful work, and we couldn't really do a useful abort",

Quoted 7 lines
Quoted 4 lines

so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.

My statement here may be a little inaccurate.

I think you may be mixing up the NVMe "Abort" command with a different command, "Reservation Acquire", where RACQA is set to value 010b, "Preempt and Abort".

Sorry, I just want to explain that I dont know how to implement "preempt and abort"

either.

maintainer
#6 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 27 lines · Guixin Liu
Quoted 25 lines · Keith Busch
Quoted 24 lines · Guixin Liu
Quoted 10 lines · Sagi Grimberg

Hey Guixin,

Quoted 3 lines · Guixin Liu

This patch implements the reservation feature, includes:
1. reservation register(register, unregister and replace).
2. reservation acquire(acquire, preempt, preempt and abort).

Overall patchset looks in decent shape to me now.

I do have one more question though.

My thanks for your patient review.

Quoted 3 lines

You say that this patchset implements preempt and abort, however I don't see how this is actually implemented or even how can it be implemented at all at this point.

Yes, the nvme target does not do any useful workto abort command, the comment of nvmet_execute_abort() says that

"we are not required to do any useful work, and we couldn't really do a useful abort",

so I don't know what to do about preempt and abort, and the NVMe spec says that the abort is best effor.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

My statement here may be a little inaccurate.

I think you may be mixing up the NVMe "Abort" command with a different command, "Reservation Acquire", where RACQA is set to value 010b, "Preempt and Abort".

Sorry, I just want to explain that I dont know how to implement "preempt and abort"

unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.

A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using percpu_ref_kill and percpu_ref_resurrect on ns->ref). This won't work very efficiently withALL_REGS reservations though.

author
#7 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 7 lines · Sagi Grimberg

unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.

A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using percpu_ref_kill and percpu_ref_resurrect on ns->ref). This won't work very efficiently withALL_REGS reservations though.

I will think deeply,and send a v9 patch.

Best regards,

Guixin Liu

author
#8 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

In reply to Sagi Grimberg
Quoted 8 lines

unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.

A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using percpu_ref_kill and percpu_ref_resurrect on ns->ref). This won't work very efficiently withALL_REGS reservations though.

Hi Sagi,

I found that if we return an error when the call to percpu_ref_tryget_live(&ns->ref) fails,

it might cause hosts that still have permissions to interrupt their IO. Additionally,

preempt_and_abort itself holds an ns->ref, we cannot wait the ref to become to zero.

The solution I can think of is to add a "per-namespace" percpu_ref to the controller for

counting IO issued to a particular namespace by that controller. Then, during the execution

of preempt_and_abort, we wait for the count of those preempted and unregistered controllers

to drop to zero.

The nsid is user-specified, so we can not use array to store the per-namespace percpu_ref,

this will increase lookup overhead if we use xarray.

What do you think Sagi? Or may be we can declare that preempt_and_abort is not supported, just

like SPDK does.

Best Regards,

Guixin Liu

maintainer
#9 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 17 lines · Guixin Liu
Quoted 8 lines

unlike abort, preempt-and-abort needs a semantic guarantee because the consumer may rely on this for fencing purposes. So it cannot be supported in "best effort" I think.

A possible implementation would be not to abort as there is no such interface, but nvmet may wait for all pending ns IO to complete and disallowing new IO to come in (using percpu_ref_kill and percpu_ref_resurrect on ns->ref). This won't work very efficiently withALL_REGS reservations though.

Hi Sagi,

I found that if we return an error when the call to percpu_ref_tryget_live(&ns->ref) fails,

it might cause hosts that still have permissions to interrupt their IO. Additionally,

preempt_and_abort itself holds an ns->ref, we cannot wait the ref to become to zero.

Quoted 7 lines

The solution I can think of is to add a "per-namespace" percpu_ref to the controller for

counting IO issued to a particular namespace by that controller. Then, during the execution

of preempt_and_abort, we wait for the count of those preempted and unregistered controllers

to drop to zero.

Yes, that is what I had in mind as well. Obviously the ns->ref cannot be used for this purpose.

Quoted 4 lines

The nsid is user-specified, so we can not use array to store the per-namespace percpu_ref,

this will increase lookup overhead if we use xarray.

Yes, that is tricky to get right.

Quoted 4 lines

What do you think Sagi? Or may be we can declare that preempt_and_abort is not supported, just

like SPDK does.

It can definitely come incrementally, but at the very least it should be incorrectly supported.

Out of curiosity, doesn't your use-case need a fencing protection against inflight I/Os reordering during preemption?

author
#10 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 9 lines
Quoted 4 lines

What do you think Sagi? Or may be we can declare that preempt_and_abort is not supported, just

like SPDK does.

It can definitely come incrementally, but at the very least it should be incorrectly supported.

Out of curiosity, doesn't your use-case need a fencing protection against inflight I/Os reordering during preemption?

The cluster use stonith(shoot the other node in the head) to protect sources,

and the backend storage(our storage system goes like this, so does disk firmware I think.)

can process write I/O operations on a first-come,first-served basis if the range of

data accessed by two I/O operation overlaps, for example, hostB preempt and abort

hostA, then host B send I/Os, the backend storage will handle hostA's I/O first(by the timestamp).

Best regards,

Guixin Liu

maintainer
#11 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 12 lines · Guixin Liu
Quoted 9 lines
Quoted 4 lines

What do you think Sagi? Or may be we can declare that preempt_and_abort is not supported, just

like SPDK does.

It can definitely come incrementally, but at the very least it should be incorrectly supported.

Out of curiosity, doesn't your use-case need a fencing protection against inflight I/Os reordering during preemption?

The cluster use stonith(shoot the other node in the head) to protect sources,

Quoted 7 lines

and the backend storage(our storage system goes like this, so does disk firmware I think.)

can process write I/O operations on a first-come,first-served basis if the range of

data accessed by two I/O operation overlaps, for example, hostB preempt and abort

hostA, then host B send I/Os, the backend storage will handle hostA's I/O first(by the timestamp).

So I understand from you that there is out-of-band mechanism for fence guarantee?

In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.

I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.

btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).

author
#12 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 5 lines · Sagi Grimberg

So I understand from you that there is out-of-band mechanism for fence guarantee?

In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.

Agree.

Quoted 1 line

I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.

I will implement it in v9, and I also think that we should a time-limited wait.

Quoted 4 lines

btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).

You mean one host can reserve all of namespaces by sending one reservation command?

Well, Could you please tell me which two use-cases?

I think this will make the target code too complicated, because the reservation situation on each

namespace may different, we should handle it one-by-one, and if failed in middle, we should revert

the handled namespaces.

Best regards,

Guixin Liu

maintainer
#13 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 11 lines · Guixin Liu
Quoted 5 lines · Sagi Grimberg

So I understand from you that there is out-of-band mechanism for fence guarantee?

In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.

Agree.

Quoted 1 line

I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.

I will implement it in v9, and I also think that we should a time-limited wait.

If it is time-limited, and you didn't manage to wait, so you need to fail the preempt and abort action (without DNR set so the reservation holder can retry if it chooses to).

Quoted 7 lines
Quoted 5 lines

btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).

You mean one host can reserve all of namespaces by sending one reservation command?

Yes. The use-case is to have the reservation in the subsystem level. The spec seems to imply that this is a valid reservation request.

Quoted 2 lines

Well, Could you please tell me which two use-cases?

Don't remember for sure, but IIRC I've seen people run oracle rac with all namespaces in the nvmet subsystem is exposed to DB hosts, and there were many namespaces.

Quoted 6 lines

I think this will make the target code too complicated, because the reservation situation on each

namespace may different, we should handle it one-by-one, and if failed in middle, we should revert

the handled namespaces.

Not sure why you'd assume that, this would be a subsystem-wide reservation, which should simply cross-check individual namespace reservations for conflicts. Sure it adds complication, but I think implementing it on every individual namespace is a naive implementation.

author
#14 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 10 lines · Sagi Grimberg
Quoted 7 lines · Guixin Liu
Quoted 5 lines · Sagi Grimberg

So I understand from you that there is out-of-band mechanism for fence guarantee?

In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.

Quoted 8 lines
Quoted 4 lines

Agree.

Quoted 1 line

I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.

I will implement it in v9, and I also think that we should a time-limited wait.

If it is time-limited, and you didn't manage to wait, so you need to fail the preempt and abort action (without DNR set so the reservation holder can retry if it chooses to).

It can wait.

Quoted 20 lines
Quoted 7 lines
Quoted 5 lines

btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).

You mean one host can reserve all of namespaces by sending one reservation command?

Yes. The use-case is to have the reservation in the subsystem level. The spec seems to imply that this is a valid reservation request.

Quoted 2 lines

Well, Could you please tell me which two use-cases?

Don't remember for sure, but IIRC I've seen people run oracle rac with all namespaces in the nvmet subsystem is exposed to DB hosts, and there were many namespaces.

Quoted 3 lines

I think this will make the target code too complicated, because the reservation situation on each

Quoted 7 lines
Quoted 3 lines

namespace may different, we should handle it one-by-one, and if failed in middle, we should revert

the handled namespaces.

Not sure why you'd assume that, this would be a subsystem-wide reservation, which should simply cross-check individual namespace reservations for conflicts. Sure it adds complication, but I think implementing it on every individual namespace is a naive implementation.

Well, this is strange, a few years ago, when I tested SCSI reservations with Oracle RAC,

I did not observe any "all LUN" reservation commands being issued.

I also researched LIO and SPDK, and found that neither supports "all LUN/NS" reservation commands.

When a host issued a reservation command without carrying a specific LUN or NS ID, both returned an error.

Best regards,

Guixin Liu

maintainer
#15 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 49 lines · Guixin Liu
Quoted 10 lines · Sagi Grimberg
Quoted 7 lines · Guixin Liu
Quoted 5 lines · Sagi Grimberg

So I understand from you that there is out-of-band mechanism for fence guarantee?

In any event, Christoph reminded me that preempt and abort is mandatory, so we cannot get away without doing it.

Quoted 8 lines
Quoted 4 lines

Agree.

Quoted 1 line

I suggest to introduce a per-ns-per-controller percpuref, with an xarray that is rcu protected.

I will implement it in v9, and I also think that we should a time-limited wait.

If it is time-limited, and you didn't manage to wait, so you need to fail the preempt and abort action (without DNR set so the reservation holder can retry if it chooses to).

It can wait.

Quoted 20 lines
Quoted 7 lines
Quoted 5 lines

btw, I also think that reservations on nsid=0xffffffff should also work. IIRC there were at least two use-cases that would have benefited from reservations on all subsystem namespaces. It can also be a lot more lightweight to implement that (outside of conflicting reservations checks).

You mean one host can reserve all of namespaces by sending one reservation command?

Yes. The use-case is to have the reservation in the subsystem level. The spec seems to imply that this is a valid reservation request.

Quoted 2 lines

Well, Could you please tell me which two use-cases?

Don't remember for sure, but IIRC I've seen people run oracle rac with all namespaces in the nvmet subsystem is exposed to DB hosts, and there were many namespaces.

Quoted 3 lines

I think this will make the target code too complicated, because the reservation situation on each

Quoted 7 lines
Quoted 3 lines

namespace may different, we should handle it one-by-one, and if failed in middle, we should revert

the handled namespaces.

Not sure why you'd assume that, this would be a subsystem-wide reservation, which should simply cross-check individual namespace reservations for conflicts. Sure it adds complication, but I think implementing it on every individual namespace is a naive implementation.

Quoted 7 lines

Well, this is strange, a few years ago, when I tested SCSI reservations with Oracle RAC,

I did not observe any "all LUN" reservation commands being issued.

I also researched LIO and SPDK, and found that neither supports "all LUN/NS" reservation commands.

When a host issued a reservation command without carrying a specific LUN or NS ID, both returned an error.

Didn't say this is actually in use, but a use-case that would benefit.

author
#16 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 3 lines

Didn't say this is actually in use, but a use-case that would benefit.

I will add a TODO-list to research main cluster softwares

to see if this is actually a significant use-case, after my patch merged.

I'll change my code then.

Best regards,

Guixin Liu

#17 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 6 lines · Guixin Liu
Quoted 3 lines

Didn't say this is actually in use, but a use-case that would benefit.

I will add a TODO-list to research main cluster softwares

Quoted 8 lines

to see if this is actually a significant use-case, after my patch merged.

I'll change my code then.

Best regards,

Guixin Liu

so which one is the latest version I use for testing ? I want to run some tests so I can provide reviewed-by tag ...

-ck

author
#18 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 9 lines · Chaitanya Kulkarni
Quoted 8 lines · Guixin Liu
Quoted 2 lines

Didn't say this is actually in use, but a use-case that would benefit.

I will add a TODO-list to research main cluster softwares

to see if this is actually a significant use-case, after my patch merged.

I'll change my code then.

Quoted 8 lines
Quoted 4 lines

Best regards,

Guixin Liu

so which one is the latest version I use for testing ? I want to run some tests so I can provide reviewed-by tag ...

-ck

I am adding a per-controller percpu-ref to ns to implement preempt_and_abort, the v9 will be soon.

Thanks,

Guixin Liu

Quoted 1 line
#19 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 2 lines · Guixin Liu

I am adding a per-controller percpu-ref to ns to implement preempt_and_abort, the v9 will be soon.

Do we need that? For blk-mq based backends just doing a queue freeze would be enough. That doesn't work for bio based drivers yet, but I'd rather improve the block layer freezing for them rather than reimplenting the whole thing.

maintainer
#20 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 7 lines · On 20/03/2024 3:59, [email protected] wrote:
Quoted 2 lines · Guixin Liu

I am adding a per-controller percpu-ref to ns to implement preempt_and_abort, the v9 will be soon.

Do we need that? For blk-mq based backends just doing a queue freeze would be enough. That doesn't work for bio based drivers yet, but I'd rather improve the block layer freezing for them rather than reimplenting the whole thing.

That's a neat idea, we also have file to think about though.

#21 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 4 lines · Sagi Grimberg
Quoted 1 line · On 20/03/2024 3:59, [email protected] wrote:

On Tue, Mar 19, 2024 at 11:19:50AM +0800, Guixin Liu wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I am adding a per-controller percpu-ref to ns to implement preempt_and_abort, the v9 will be soon.

Do we need that?  For blk-mq based backends just doing a queue freeze would be enough.  That doesn't work for bio based drivers yet, but I'd rather improve the block layer freezing for them rather than reimplenting the whole thing.

That's a neat idea, we also have file to think about though.

I failed to understand how this will work for non block device (passthru or bio mode) backend(s). Unless there is a plan to create similar API for non block device backhend(s) and if that turns out to be overkill then why not go with current per-controller percpu-ref ?

-ck

#22 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

In reply to Sagi Grimberg
Quoted 1 line · Sagi Grimberg

That's a neat idea, we also have file to think about though.

Indeed. It still would be good to avoid the extra overhead when not needed, though.

#23 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Christoph,

Quoted 5 lines · On 3/21/2024 2:02 PM, [email protected] wrote:
Quoted 1 line · Sagi Grimberg

That's a neat idea, we also have file to think about though.

Indeed. It still would be good to avoid the extra overhead when not needed, though.

I still don't understand how this solves the case for file back-end ? can you please explain in detail ?

as far as I can there is no equivalent provision that we can use for file back-end as you have suggested for block (bio/blk-mq)back-end, if there is can you please provide some details ?

In case there is no such provision, how do you see file backends to handle this case ? perhaps you are referring to use above suggested mechanism for block backend and implement percpu-ref to for file only backend so we can keep the code clean for bio and passthru back-end vs file back-end ?

-ck

maintainer
#24 lore raw

Re: [PATCH v7 1/1] nvmet: support reservation feature

Quoted 14 lines · Chaitanya Kulkarni

Christoph,

Quoted 4 lines · On 3/21/2024 2:02 PM, [email protected] wrote:
Quoted 1 line · Sagi Grimberg

That's a neat idea, we also have file to think about though.

Indeed. It still would be good to avoid the extra overhead when not needed, though.

I still don't understand how this solves the case for file back-end ? can you please explain in detail ?

as far as I can there is no equivalent provision that we can use for file back-end as you have suggested for block (bio/blk-mq)back-end, if there is can you please provide some details ?

Quoted 5 lines

In case there is no such provision, how do you see file backends to handle this case ? perhaps you are referring to use above suggested mechanism for block backend and implement percpu-ref to for file only backend so we can keep the code clean for bio and passthru back-end vs file back-end ?

Yes, the latter.

author
#25 lore raw
In reply to Guixin Liu

Hi,

    Gentle ping...

Best regards.

Guixin Liu

Quoted 143 lines · Guixin Liu

Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.

Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.

- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.

- Fix the resv log page is random when there is no resv log page.

- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().

- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().

- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().

- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().

- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().

- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.

Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().

- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().

- Add the print fmt to the head.

- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.

- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.

Changes from v4 to v5:

- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.

- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.

- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.

- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.

- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.

- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.

- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.

- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.

Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.

- Change the function calling alignment style to:

     nvmet_pr_send_event_by_hostid(pr, hostid,
             NVME_PR_LOG_RESERVATOPM_PREEMPTED);

- Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().

- Remove the goto in __nvmet_pr_unregister_one().

- Change generation to atomic_t, and remove nvmet_pr_inc_generation().

- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.

Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.

- Limit the resv_log_list to 128.

- Change generation to atomic64.

- Put register rkey check to a warpper.

- Change nr_avl_pages to nr_pages.

- Use NVME_SC_SUCCESS instead of 0.

- Change kmalloc param to let it not sleep in mutex lock.

Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.

- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is
0xffffffff at each admin command path, if it is needed.

- Add reservation commands support in nvmet_get_cmd_effects_nvm().

- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.

- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.

- Remove kmem_cache instead to use kmalloc and kfree.

- Change others advice from Sagi.

- Add a blktest test case, this patch will be sent before these series of patches.

Guixin Liu (1):
nvmet: support reservation feature

  drivers/nvme/target/Makefile    |    2 +-
  drivers/nvme/target/admin-cmd.c |   20 +-
  drivers/nvme/target/configfs.c  |   27 +
  drivers/nvme/target/core.c      |   51 +-
  drivers/nvme/target/nvmet.h     |   36 ++
  drivers/nvme/target/pr.c        | 1041 +++++++++++++++++++++++++++++++
  include/linux/nvme.h            |   54 ++
  7 files changed, 1224 insertions(+), 7 deletions(-)

create mode 100644 drivers/nvme/target/pr.c

author
#26 lore raw

Hi Christoph and Chaitanya,

    I noticed that Sagi hasn't replied to any emails for a month, could

you please review this patch?

Best Regards,

Guixin Liu

Quoted 167 lines · Guixin Liu

Hi,

    Gentle ping...

Best regards.

Guixin Liu

Quoted 158 lines · Guixin Liu

Hi guys,
     I've implemented the NVMe reservation feature. Please review it, all comments are welcome as usual.

Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation
log.

- Add all the registrants that need to be freed to a temporary list fist,
and then after calling synchronize_rcu(), release all the registrants on the temporary list.

- Fix the resv log page is random when there is no resv log page.

- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().

- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv()
to nvmet_pr_create_new_reservation().

- Change nvmet_pr_find_registrant_by_hostid() to
nvmet_pr_find_registrant().

- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().

- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().

- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.

Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().

- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp.
And refactor the nvmet_pr_register().

- Add the print fmt to the head.

- Add lockdep_is_held(&pr->pr_lock) condition to
list_for_each_entry_rcu.

- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.

Changes from v4 to v5:

- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.

- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.

- Add lockdep_assert_held and rcu_read_lock_held assert to many functions,
if it's necessary.

- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count
works.

- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.

- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.

- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.

- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.

Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.

- Change the function calling alignment style to:
     nvmet_pr_send_event_by_hostid(pr, hostid,
             NVME_PR_LOG_RESERVATOPM_PREEMPTED);

- Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().

- Remove the goto in __nvmet_pr_unregister_one().

- Change generation to atomic_t, and remove nvmet_pr_inc_generation().

- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.

Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype
into the struct nvmet_pr_registrant.

- Limit the resv_log_list to 128.

- Change generation to atomic64.

- Put register rkey check to a warpper.

- Change nr_avl_pages to nr_pages.

- Use NVME_SC_SUCCESS instead of 0.

- Change kmalloc param to let it not sleep in mutex lock.

Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get reservation log page command to clear log page at host.

- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
   Now there is no admin opcode nvmet implemented needs reservation check, so I dont add reservation check to admin command path.
   Next we need to do reservation check includes the situation of nsid is 0xffffffff at each admin command path, if it is needed.

- Add reservation commands support in nvmet_get_cmd_effects_nvm().

- From Chaitanya, change the local variable tree style to make it cleaner,
and add some comments about NVMe spec.
   And also change others advice from chaitanya.

- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation
enable check warp.

- Remove kmem_cache instead to use kmalloc and kfree.

- Change others advice from Sagi.

- Add a blktest test case, this patch will be sent before these series of
patches.

Guixin Liu (1):
   nvmet: support reservation feature

  drivers/nvme/target/Makefile    |    2 +- drivers/nvme/target/admin-cmd.c |   20 +- drivers/nvme/target/configfs.c  |   27 + drivers/nvme/target/core.c      |   51 +- drivers/nvme/target/nvmet.h     |   36 ++ drivers/nvme/target/pr.c        | 1041 +++++++++++++++++++++++++++++++ include/linux/nvme.h            |   54 ++

  7 files changed, 1224 insertions(+), 7 deletions(-)

  create mode 100644 drivers/nvme/target/pr.c

#27 lore raw
Quoted 10 lines · Guixin Liu

Hi Christoph and Chaitanya,

    I noticed that Sagi hasn't replied to any emails for a month, could

you please review this patch?

Best Regards,

Guixin Liu

Hi,

You will have my review by end of this week worst case on weekend.

-ck

author
#28 lore raw
In reply to Guixin Liu

Hi guys,

    I will send a v8 patch to add me as the new file's maintainer.

Best Regards,

Guixin Liu

Quoted 143 lines · Guixin Liu

Hi guys,
I've implemented the NVMe reservation feature. Please review it, all
comments are welcome as usual.

Changes from v6 to v7:
- Handle "reservation notification mask" feature command to mask reservation log.

- Add all the registrants that need to be freed to a temporary list fist, and then after calling synchronize_rcu(), release all the registrants on the temporary list.

- Fix the resv log page is random when there is no resv log page.

- Change nvmet_is_host_still_connected() to nvmet_is_host_connected().

- Remove nvmet_pr_set_rtype_and_holder() and change nvmet_pr_create_new_resv() to nvmet_pr_create_new_reservation().

- Change nvmet_pr_find_registrant_by_hostid() to nvmet_pr_find_registrant().

- Change nvmet_pr_send_resv_released() to nvmet_pr_resv_released().

- Change __nvmet_pr_unregister_one() to nvmet_pr_unregister_one().

- In nvmet_pr_unreg_by_prkey(), nvmet_pr_unreg_by_prkey_except_hostid() and nvmet_pr_unreg_except_hostid(), first do unregistering and then do event sending.

Changes from v5 to v6:
- Use synchronize_rcu() and kfree() to free registrant instead of kfree_rcu().

- Remove nvmet_pr_register_check_rkey(), put the check into pr_lock warp. And refactor the nvmet_pr_register().

- Add the print fmt to the head.

- Add lockdep_is_held(&pr->pr_lock) condition to list_for_each_entry_rcu.

- Fix the bug in nvmet_pr_update_reg_attr(), when the change_attr hook return fail, we should not replace the holder.

Changes from v4 to v5:

- Use rculist macros to handle registration_list instead of list macros regardless of in mutex lock or not.

- Use goto statement instead of return in nvmet_is_host_still_connected and __nvmet_pr_unregister_one.

- Add lockdep_assert_held and rcu_read_lock_held assert to many functions, if it's necessary.

- Add a comment to nvmet_execute_get_log_page_resv to explain how lost_count works.

- In nvmet_pr_clear, we should set holder to NULL first, I fixed this.

- Unify nvmet_pr_update_holder_rtype and __nvmet_pr_do_replace to nvmet_pr_update_reg_attr.

- Fix wrong nr_pages in nvmet_execute_get_log_page_resv.

- Fix the deadlock issue of nvmet_pr_exit_ns, put it out of the subsys lock.

Changes from v3 to v4:
- Use kfifo to handle resv log page instead of list, and also limit the resv log queue to 64.

- Change the function calling alignment style to:

     nvmet_pr_send_event_by_hostid(pr, hostid,
             NVME_PR_LOG_RESERVATOPM_PREEMPTED);

- Put kmalloc out of rcu_read_lock in nvmet_execute_pr_report().

- Remove the goto in __nvmet_pr_unregister_one().

- Change generation to atomic_t, and remove nvmet_pr_inc_generation().

- In addtion, the number2 patch "nvmet: unify aer type enum" is not relate with this patch, so I will send it separately.

Changes from v2 to v3:
- Use rcu instead of rwlock to make IO path run faster, and put the rtype into the struct nvmet_pr_registrant.

- Limit the resv_log_list to 128.

- Change generation to atomic64.

- Put register rkey check to a warpper.

- Change nr_avl_pages to nr_pages.

- Use NVME_SC_SUCCESS instead of 0.

- Change kmalloc param to let it not sleep in mutex lock.

Changes from v1 to v2:
- Implement the reservation notification report, includes registration preempted, reservation released and reservation preempted. And also handle the reservation log page available event and send get
reservation log page command to clear log page at host.

- Put the reservation check access after validate opcode. And remove opcodes which nvmet not implement yet check.
Now there is no admin opcode nvmet implemented needs reservation check,
so I dont add reservation check to admin command path.
Next we need to do reservation check includes the situation of nsid is
0xffffffff at each admin command path, if it is needed.

- Add reservation commands support in nvmet_get_cmd_effects_nvm().

- From Chaitanya, change the local variable tree style to make it cleaner, and add some comments about NVMe spec.
And also change others advice from chaitanya.

- Put the nvmet_pr_check_cmd_access and nvmet_parse_pr_cmd into reservation enable check warp.

- Remove kmem_cache instead to use kmalloc and kfree.

- Change others advice from Sagi.

- Add a blktest test case, this patch will be sent before these series of patches.

Guixin Liu (1):
nvmet: support reservation feature

  drivers/nvme/target/Makefile    |    2 +-
  drivers/nvme/target/admin-cmd.c |   20 +-
  drivers/nvme/target/configfs.c  |   27 +
  drivers/nvme/target/core.c      |   51 +-
  drivers/nvme/target/nvmet.h     |   36 ++
  drivers/nvme/target/pr.c        | 1041 +++++++++++++++++++++++++++++++
  include/linux/nvme.h            |   54 ++
  7 files changed, 1224 insertions(+), 7 deletions(-)

create mode 100644 drivers/nvme/target/pr.c