Skip to contentSkip to conversation

nvme: add support for command quiesce timeout

RFC PATCH In review Daniel Wagner 8 participants 39 messages Mar 24, 2025 – Apr 17, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover nvme: add support for command quiesce timeout

This is thought just as discussion input for the KATO session during LSFMM.

The last patch is the interesting part. When commands is failed over to the next path, it delays the requeuing on ctrl level. The implemention is on purpose not implementing all the details. Just the very basics so it's easy to understand what the main idea is.

---
Daniel Wagner (3):
nvmet: add command quiesce time
nvme: store cqt value into nvme ctrl object
nvme: delay failover by command quiesce timeout

 drivers/nvme/host/core.c        | 20 ++++++++++++++++
 drivers/nvme/host/fc.c          |  4 ++++
 drivers/nvme/host/multipath.c   | 52 ++++++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h        | 16 +++++++++++++
 drivers/nvme/host/rdma.c        |  2 ++
 drivers/nvme/host/tcp.c         |  1 +
 drivers/nvme/target/admin-cmd.c |  6 +++++
 drivers/nvme/target/nvmet.h     |  1 +
 include/linux/nvme.h            |  4 +++-
 9 files changed, 102 insertions(+), 4 deletions(-)

---
base-commit: 25462bccac81486c2fff0fa4b4c2b1e56751b8de
change-id: 20241128-tp4129-a45b998b20cb

Best regards,

Signature · 1 line
Daniel Wagner <[email protected]>
#1 lore raw

RFC PATCH 1/3 nvmet: add command quiesce time

TP4129 introduces Command Quiesce Time (CQT) for coordinating the shutdown sequence when for example KATO expires.

Add support to nvmet but only report CQT is available but the controller doesn't need any additional time when shutting down. In this case the spec says nvmet should report a value of 1.

---
 drivers/nvme/target/admin-cmd.c | 6 ++++++
 drivers/nvme/target/nvmet.h     | 1 +
 include/linux/nvme.h            | 4 +++-
 3 files changed, 10 insertions(+), 1 deletion(-)
drivers/nvme/target/admin-cmd.c +6 -0
index e670dc185a967dc69c9b7d23930bb52bdcc3271a..09ac5a43f70dbe3889c1b404d6b59c0053337192 100644
@@ -733,6 +733,12 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	/* We support keep-alive timeout in granularity of seconds */
 	id->kas = cpu_to_le16(NVMET_KAS);
 
+	/*
+	 * Command Quiesce Time in milliseconds. If the controller is not
+	 * need any quiencse time, the controller should set it to 1.
+	 */
+	id->cqt = cpu_to_le16(NVMET_CQT);
+
 	id->sqes = (0x6 << 4) | 0x6;
 	id->cqes = (0x4 << 4) | 0x4;
 
drivers/nvme/target/nvmet.h +1 -0
index b540216c0c9a9138f0913f8df28fa5ae13c6397f..47ae8be6200054eaaad2dbacf23db080bf0c56c2 100644
@@ -671,6 +671,7 @@ bool nvmet_subsys_nsid_exists(struct nvmet_subsys *subsys, u32 nsid);
 
 #define NVMET_KAS		10
 #define NVMET_DISC_KATO_MS		120000
+#define NVMET_CQT		1
 
 int __init nvmet_init_configfs(void);
 void __exit nvmet_exit_configfs(void);
include/linux/nvme.h +3 -1
index fe3b60818fdcfbb4baabce59f7499bc1fa07e855..983b047e7158dcb33da66a25c67684b8f1ef5a7e 100644
@@ -335,7 +335,9 @@ struct nvme_id_ctrl {
 	__u8			anacap;
 	__le32			anagrpmax;
 	__le32			nanagrpid;
-	__u8			rsvd352[160];
+	__u8			rsvd352[34];
+	__u16			cqt;
+	__u8			rsvd388[124];
 	__u8			sqes;
 	__u8			cqes;
 	__le16			maxcmd;
Signature · 1 line
2.48.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH RFC 1/3] nvmet: add command quiesce time

Quoted 7 lines · Daniel Wagner

TP4129 introduces Command Quiesce Time (CQT) for coordinating the shutdown sequence when for example KATO expires.

Add support to nvmet but only report CQT is available but the controller doesn't need any additional time when shutting down. In this case the spec says nvmet should report a value of 1.

Quoted 7 lines
---
  drivers/nvme/target/admin-cmd.c | 6 ++++++
  drivers/nvme/target/nvmet.h     | 1 +
  include/linux/nvme.h            | 4 +++-
  3 files changed, 10 insertions(+), 1 deletion(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#3 lore raw

Re: [PATCH RFC 1/3] nvmet: add command quiesce time

In reply to Daniel Wagner
Quoted 36 lines · Daniel Wagner

TP4129 introduces Command Quiesce Time (CQT) for coordinating the shutdown sequence when for example KATO expires.

Add support to nvmet but only report CQT is available but the controller doesn't need any additional time when shutting down. In this case the spec says nvmet should report a value of 1.

---
 drivers/nvme/target/admin-cmd.c | 6 ++++++
 drivers/nvme/target/nvmet.h     | 1 +
 include/linux/nvme.h            | 4 +++-
 3 files changed, 10 insertions(+), 1 deletion(-)
drivers/nvme/target/admin-cmd.c +6 -0
index e670dc185a967dc69c9b7d23930bb52bdcc3271a..09ac5a43f70dbe3889c1b404d6b59c0053337192 100644
@@ -733,6 +733,12 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
 	/* We support keep-alive timeout in granularity of seconds */
 	id->kas = cpu_to_le16(NVMET_KAS);
 
+	/*
+	 * Command Quiesce Time in milliseconds. If the controller is not
+	 * need any quiencse time, the controller should set it to 1.
+	 */
+	id->cqt = cpu_to_le16(NVMET_CQT);
+
 	id->sqes = (0x6 << 4) | 0x6;
 	id->cqes = (0x4 << 4) | 0x4;
 
drivers/nvme/target/nvmet.h +0 -0
index b540216c0c9a9138f0913f8df28fa5ae13c6397f..47ae8be6200054eaaad2dbacf23db080bf0c56c2 100644

drivers/nvme/target/nvmet.h bool nvmet_subsys_nsid_exists(struct nvmet_subsys *subsys, u32 nsid); · line 674

Quoted 4 lines
drivers/nvme/target/nvmet.h +1 -0
@@ -671,6 +671,7 @@ bool nvmet_subsys_nsid_exists(struct nvmet_subsys *subsys, u32 nsid);
 
 #define NVMET_KAS		10
 #define NVMET_DISC_KATO_MS		120000
+#define NVMET_CQT		1

Setting CQT to 1 is a promise to initiator that target will quiesce pending requests in 1ms after it detects loss of kato traffic. For initiator, it means these requests can be retried safely in 1m after kato delay. Is this guaranteed by target?

I thought leaving the value 0 means CQT is not implemented, no?

Quoted 22 lines
drivers/nvme/target/nvmet.h +0 -0
 int __init nvmet_init_configfs(void);
 void __exit nvmet_exit_configfs(void);
include/linux/nvme.h +3 -1
index fe3b60818fdcfbb4baabce59f7499bc1fa07e855..983b047e7158dcb33da66a25c67684b8f1ef5a7e 100644
@@ -335,7 +335,9 @@ struct nvme_id_ctrl {
 	__u8			anacap;
 	__le32			anagrpmax;
 	__le32			nanagrpid;
-	__u8			rsvd352[160];
+	__u8			rsvd352[34];
+	__u16			cqt;
+	__u8			rsvd388[124];
 	__u8			sqes;
 	__u8			cqes;
 	__le16			maxcmd;
Signature · 2 lines
-- 
2.48.1
#4 lore raw

Re: [PATCH RFC 1/3] nvmet: add command quiesce time

Quoted 13 lines · Mohamed Khalfella

drivers/nvme/target/nvmet.h bool nvmet_subsys_nsid_exists(struct nvmet_subsys *subsys, u32 nsid); · line 674

Quoted 7 lines
drivers/nvme/target/nvmet.h +1 -0
@@ -671,6 +671,7 @@ bool nvmet_subsys_nsid_exists(struct nvmet_subsys *subsys, u32 nsid);
 
 #define NVMET_KAS		10
 #define NVMET_DISC_KATO_MS		120000
+#define NVMET_CQT		1

Setting CQT to 1 is a promise to initiator that target will quiesce pending requests in 1ms after it detects loss of kato traffic. For initiator, it means these requests can be retried safely in 1m after kato delay. Is this guaranteed by target?

Quoted 1 line

I thought leaving the value 0 means CQT is not implemented, no?

I decided to go with the first clause (9.6.2 End of Controller Processing of Outstanding Commands)

- If the CQT field (refer to Figure 312) is non-zero, wait for the amount of time indicated in the CQT field to elapse; or

- If the CQT field is cleared to 0h, wait for an implementation specific amount of time (e.g. 10 seconds). The host should allow this value to be administratively configured.

The nvmet doesn't need the CQT thus set it to the smallest amount of time. But yes this should be configurable for testing purposes with blktests.

#5 lore raw

RFC PATCH 2/3 nvme: store cqt value into nvme ctrl object

In reply to Daniel Wagner
---
 drivers/nvme/host/core.c | 1 +
 drivers/nvme/host/nvme.h | 1 +
 2 files changed, 2 insertions(+)
drivers/nvme/host/core.c +1 -0
index 40046770f1bf0b98261d8b80e21aa0cc04ebb7a0..135045528ea1c79eac0d6d47d5f7f05a7c98acc4 100644
@@ -3380,6 +3380,7 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl)
 	ctrl->kas = le16_to_cpu(id->kas);
 	ctrl->max_namespaces = le32_to_cpu(id->mnan);
 	ctrl->ctratt = le32_to_cpu(id->ctratt);
+	ctrl->cqt = le16_to_cpu(id->cqt);
 
 	ctrl->cntrltype = id->cntrltype;
 	ctrl->dctype = id->dctype;
drivers/nvme/host/nvme.h +1 -0
index 7be92d07430e950c3faa764514daf3808009e223..7563332b5b7b76fc6165ec8c6f2d144737d4fe85 100644
@@ -344,6 +344,7 @@ struct nvme_ctrl {
 	u32 oaes;
 	u32 aen_result;
 	u32 ctratt;
+	u16 cqt;
 	unsigned int shutdown_timeout;
 	unsigned int kato;
 	bool subsystem;
Signature · 1 line
2.48.1
Reviewed-by on v1 2/3
#6 lore raw

Re: [PATCH RFC 2/3] nvme: store cqt value into nvme ctrl object

Quoted 24 lines · Daniel Wagner
---
  drivers/nvme/host/core.c | 1 +
  drivers/nvme/host/nvme.h | 1 +
  2 files changed, 2 insertions(+)
drivers/nvme/host/core.c +1 -0
index 40046770f1bf0b98261d8b80e21aa0cc04ebb7a0..135045528ea1c79eac0d6d47d5f7f05a7c98acc4 100644
@@ -3380,6 +3380,7 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl)
  	ctrl->kas = le16_to_cpu(id->kas);
  	ctrl->max_namespaces = le32_to_cpu(id->mnan);
  	ctrl->ctratt = le32_to_cpu(id->ctratt);
+	ctrl->cqt = le16_to_cpu(id->cqt);
 
  	ctrl->cntrltype = id->cntrltype;
  	ctrl->dctype = id->dctype;
drivers/nvme/host/nvme.h +0 -0
index 7be92d07430e950c3faa764514daf3808009e223..7563332b5b7b76fc6165ec8c6f2d144737d4fe85 100644

drivers/nvme/host/nvme.h struct nvme_ctrl { · line 350

Quoted 7 lines
drivers/nvme/host/nvme.h +1 -0
@@ -344,6 +344,7 @@ struct nvme_ctrl {
  	u32 oaes;
  	u32 aen_result;
  	u32 ctratt;
+	u16 cqt;
  	unsigned int shutdown_timeout;
  	unsigned int kato;
  	bool subsystem;

Hmm. I would squash it with the next patch, but I'm sure others have other opinions.

So:

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#7 lore raw

RFC PATCH 3/3 nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner

The TP4129 mendates that the failover should be delayed by CQT. Thus when nvme_decide_disposition returns FAILOVER do not immediately re-queue it on the namespace level instead queue it on the ctrl's request_list and moved later to the namespace's requeue_list.

---
 drivers/nvme/host/core.c      | 19 ++++++++++++++++
 drivers/nvme/host/fc.c        |  4 ++++
 drivers/nvme/host/multipath.c | 52 ++++++++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h      | 15 +++++++++++++
 drivers/nvme/host/rdma.c      |  2 ++
 drivers/nvme/host/tcp.c       |  1 +
 6 files changed, 90 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +19 -0
index 135045528ea1c79eac0d6d47d5f7f05a7c98acc4..f3155c7735e75e06c4359c26db8931142c067e1d 100644
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 
 	flush_work(&ctrl->reset_work);
 	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);
 	nvme_remove_namespaces(ctrl);
 	ctrl->ops->delete_ctrl(ctrl);
 	nvme_uninit_ctrl(ctrl);
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
 	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
 }
 
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;
+
+	queue_delayed_work(nvme_wq, &ctrl->failover_work, delay);
+}
+EXPORT_SYMBOL_GPL(nvme_schedule_failover);
+
 static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 						 blk_status_t status)
 {
@@ -1336,6 +1350,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 		dev_err(ctrl->device,
 			"failed nvme_keep_alive_end_io error=%d\n",
 				status);
+
+		nvme_schedule_failover(ctrl);
 		return RQ_END_IO_NONE;
 	}
 
@@ -4716,6 +4732,7 @@ EXPORT_SYMBOL_GPL(nvme_remove_io_tag_set);
 
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
+	nvme_schedule_failover(ctrl);
 	nvme_mpath_stop(ctrl);
 	nvme_auth_stop(ctrl);
 	nvme_stop_failfast_work(ctrl);
@@ -4842,6 +4859,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	INIT_DELAYED_WORK(&ctrl->ka_work, nvme_keep_alive_work);
 	INIT_DELAYED_WORK(&ctrl->failfast_work, nvme_failfast_work);
+	INIT_DELAYED_WORK(&ctrl->failover_work, nvme_failover_work);
+	INIT_LIST_HEAD(&ctrl->failover_list);
 	memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd));
 	ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive;
 	ctrl->ka_last_check_time = jiffies;
drivers/nvme/host/fc.c +4 -0
index cdc1ba277a5c23ef1afd26e6911b082f3d12b215..bd897b29cd286008b781bbcb4230e08019da6b6b 100644
@@ -2553,6 +2553,8 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
 {
 	enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	/*
 	 * if an error (io timeout, etc) while (re)connecting, the remote
 	 * port requested terminating of the association (disconnect_ls)
@@ -3378,6 +3380,8 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 	/* will block will waiting for io to terminate */
 	nvme_fc_delete_association(ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
 		dev_err(ctrl->ctrl.device,
 			"NVME-FC{%d}: error_recovery: Couldn't change state "
drivers/nvme/host/multipath.c +49 -3

67 lines not on this page · raw message

drivers/nvme/host/nvme.h +15 -0
index 7563332b5b7b76fc6165ec8c6f2d144737d4fe85..10eb323bdaf139526959180c1e66ab4579bb145d 100644
@@ -411,6 +411,9 @@ struct nvme_ctrl {
 
 	enum nvme_ctrl_type cntrltype;
 	enum nvme_dctype dctype;
+
+	struct delayed_work failover_work;
+	struct list_head failover_list;
 };
 
 static inline enum nvme_ctrl_state nvme_ctrl_state(struct nvme_ctrl *ctrl)
@@ -954,6 +957,9 @@ void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
 void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
 void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys);
 void nvme_failover_req(struct request *req);
+void nvme_failover_work(struct work_struct *work);
+void nvme_schedule_failover(struct nvme_ctrl *ctrl);
+void nvme_flush_failover(struct nvme_ctrl *ctrl);
 void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
 int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
 void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid);
@@ -996,6 +1002,15 @@ static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
 static inline void nvme_failover_req(struct request *req)
 {
 }
+static inline void nvme_failover_work(struct work_struct *work)
+{
+}
+static inline void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+}
+static inline void nvme_flush_failover(struct nvme_ctrl *ctrl)
+{
+}
 static inline void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
 {
 }
drivers/nvme/host/rdma.c +2 -0
index 86a2891d9bcc7a990cd214a7fe93fa5c55b292c7..9bee376f881b4c3ebe5502abf23a8e76829780ff 100644
@@ -1127,6 +1127,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
+	nvme_schedule_failover(&ctrl->ctrl);
 	nvme_rdma_teardown_io_queues(ctrl, false);
 	nvme_unquiesce_io_queues(&ctrl->ctrl);
 	nvme_rdma_teardown_admin_queue(ctrl, false);
@@ -2153,6 +2154,7 @@ static const struct blk_mq_ops nvme_rdma_admin_mq_ops = {
 
 static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
 {
+	nvme_schedule_failover(&ctrl->ctrl);
 	nvme_rdma_teardown_io_queues(ctrl, shutdown);
 	nvme_quiesce_admin_queue(&ctrl->ctrl);
 	nvme_disable_ctrl(&ctrl->ctrl, shutdown);
drivers/nvme/host/tcp.c +1 -0
index d0023bcfd8a79a193adf2807a24481c8c164a174..3a6c1d3febaf233996e4dcf684793327b5d1412f 100644
@@ -2345,6 +2345,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
+	nvme_schedule_failover(ctrl);
 	nvme_tcp_teardown_io_queues(ctrl, false);
 	/* unquiesce to fail fast pending requests */
 	nvme_unquiesce_io_queues(ctrl);
Signature · 1 line
2.48.1
#8 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 234 lines · Daniel Wagner

The TP4129 mendates that the failover should be delayed by CQT. Thus when nvme_decide_disposition returns FAILOVER do not immediately re-queue it on the namespace level instead queue it on the ctrl's request_list and moved later to the namespace's requeue_list.

---
  drivers/nvme/host/core.c      | 19 ++++++++++++++++
  drivers/nvme/host/fc.c        |  4 ++++
  drivers/nvme/host/multipath.c | 52 ++++++++++++++++++++++++++++++++++++++++---
  drivers/nvme/host/nvme.h      | 15 +++++++++++++
  drivers/nvme/host/rdma.c      |  2 ++
  drivers/nvme/host/tcp.c       |  1 +
  6 files changed, 90 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +19 -0
index 135045528ea1c79eac0d6d47d5f7f05a7c98acc4..f3155c7735e75e06c4359c26db8931142c067e1d 100644
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 
  	flush_work(&ctrl->reset_work);
  	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);
  	nvme_remove_namespaces(ctrl);
  	ctrl->ops->delete_ctrl(ctrl);
  	nvme_uninit_ctrl(ctrl);
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
  	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
  }
 
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;
+
+	queue_delayed_work(nvme_wq, &ctrl->failover_work, delay);
+}
+EXPORT_SYMBOL_GPL(nvme_schedule_failover);
+
  static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
  						 blk_status_t status)
  {
@@ -1336,6 +1350,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
  		dev_err(ctrl->device,
  			"failed nvme_keep_alive_end_io error=%d\n",
  				status);
+
+		nvme_schedule_failover(ctrl);
  		return RQ_END_IO_NONE;
  	}
 
@@ -4716,6 +4732,7 @@ EXPORT_SYMBOL_GPL(nvme_remove_io_tag_set);
 
  void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
  {
+	nvme_schedule_failover(ctrl);
  	nvme_mpath_stop(ctrl);
  	nvme_auth_stop(ctrl);
  	nvme_stop_failfast_work(ctrl);
@@ -4842,6 +4859,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
  	INIT_DELAYED_WORK(&ctrl->ka_work, nvme_keep_alive_work);
  	INIT_DELAYED_WORK(&ctrl->failfast_work, nvme_failfast_work);
+	INIT_DELAYED_WORK(&ctrl->failover_work, nvme_failover_work);
+	INIT_LIST_HEAD(&ctrl->failover_list);
  	memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd));
  	ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive;
  	ctrl->ka_last_check_time = jiffies;
drivers/nvme/host/fc.c +4 -0
index cdc1ba277a5c23ef1afd26e6911b082f3d12b215..bd897b29cd286008b781bbcb4230e08019da6b6b 100644
@@ -2553,6 +2553,8 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
  {
  	enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
  	/*
  	 * if an error (io timeout, etc) while (re)connecting, the remote
  	 * port requested terminating of the association (disconnect_ls)
@@ -3378,6 +3380,8 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
  	/* will block will waiting for io to terminate */
  	nvme_fc_delete_association(ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
  	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
  		dev_err(ctrl->ctrl.device,
  			"NVME-FC{%d}: error_recovery: Couldn't change state "
drivers/nvme/host/multipath.c +49 -3
index 2a7635565083046c575efe1793362ae10581defd..a14b055796b982df96609f53174a5d1334c1c0c4 100644
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
  void nvme_failover_req(struct request *req)
  {
  	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
  	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
  	unsigned long flags;
  	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
  	nvme_mpath_clear_current_path(ns);
 
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
  	blk_steal_bios(&ns->head->requeue_list, req);
  	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);
+
+	if (state == NVME_CTRL_DELETING) {
+		/*
+		 * request which fail over in the DELETING state were
+		 * canceled and blk_mq_tagset_wait_completed_request will
+		 * block until they have been proceed. Though
+		 * nvme_failover_work is already stopped. Thus schedule
+		 * a failover; it's still necessary to delay these commands
+		 * by CQT.
+		 */
+		nvme_schedule_failover(ctrl);
+	}
+}
+
+void nvme_flush_failover(struct nvme_ctrl *ctrl)
+{
+	LIST_HEAD(failover_list);
+	struct request *rq;
+	bool kick = false;
+
+	spin_lock_irq(&ctrl->lock);
+	list_splice_init(&ctrl->failover_list, &failover_list);
+	spin_unlock_irq(&ctrl->lock);
+
+	while (!list_empty(&failover_list)) {
+		rq = list_entry(failover_list.next,
+				struct request, queuelist);
+		list_del_init(&rq->queuelist);
+
+		nvme_req(rq)->status = 0;
+		nvme_end_req(rq);
+		kick = true;
+	}
+
+	if (kick)
+		nvme_kick_requeue_lists(ctrl);
+}
+
+void nvme_failover_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl = container_of(to_delayed_work(work),
+					struct nvme_ctrl, failover_work);
+
+	nvme_flush_failover(ctrl);
  }
 
  void nvme_mpath_start_request(struct request *rq)
drivers/nvme/host/nvme.h +15 -0
index 7563332b5b7b76fc6165ec8c6f2d144737d4fe85..10eb323bdaf139526959180c1e66ab4579bb145d 100644
@@ -411,6 +411,9 @@ struct nvme_ctrl {
 
  	enum nvme_ctrl_type cntrltype;
  	enum nvme_dctype dctype;
+
+	struct delayed_work failover_work;
+	struct list_head failover_list;
  };
 
  static inline enum nvme_ctrl_state nvme_ctrl_state(struct nvme_ctrl *ctrl)
@@ -954,6 +957,9 @@ void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
  void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
  void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys);
  void nvme_failover_req(struct request *req);
+void nvme_failover_work(struct work_struct *work);
+void nvme_schedule_failover(struct nvme_ctrl *ctrl);
+void nvme_flush_failover(struct nvme_ctrl *ctrl);
  void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
  int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
  void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid);
@@ -996,6 +1002,15 @@ static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
  static inline void nvme_failover_req(struct request *req)
  {
  }
+static inline void nvme_failover_work(struct work_struct *work)
+{
+}
+static inline void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+}
+static inline void nvme_flush_failover(struct nvme_ctrl *ctrl)
+{
+}
  static inline void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
  {
  }
drivers/nvme/host/rdma.c +2 -0
index 86a2891d9bcc7a990cd214a7fe93fa5c55b292c7..9bee376f881b4c3ebe5502abf23a8e76829780ff 100644
@@ -1127,6 +1127,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 
  	nvme_stop_keep_alive(&ctrl->ctrl);
  	flush_work(&ctrl->ctrl.async_event_work);
+	nvme_schedule_failover(&ctrl->ctrl);
  	nvme_rdma_teardown_io_queues(ctrl, false);
  	nvme_unquiesce_io_queues(&ctrl->ctrl);
  	nvme_rdma_teardown_admin_queue(ctrl, false);
@@ -2153,6 +2154,7 @@ static const struct blk_mq_ops nvme_rdma_admin_mq_ops = {
 
  static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
  {
+	nvme_schedule_failover(&ctrl->ctrl);
  	nvme_rdma_teardown_io_queues(ctrl, shutdown);
  	nvme_quiesce_admin_queue(&ctrl->ctrl);
  	nvme_disable_ctrl(&ctrl->ctrl, shutdown);
drivers/nvme/host/tcp.c +0 -0
index d0023bcfd8a79a193adf2807a24481c8c164a174..3a6c1d3febaf233996e4dcf684793327b5d1412f 100644

drivers/nvme/host/tcp.c static void nvme_tcp_error_recovery_work(struct work_struct *work) · line 2351

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -0
@@ -2345,6 +2345,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 
  	nvme_stop_keep_alive(ctrl);
  	flush_work(&ctrl->async_event_work);
+	nvme_schedule_failover(ctrl);
  	nvme_tcp_teardown_io_queues(ctrl, false);
  	/* unquiesce to fail fast pending requests */
  	nvme_unquiesce_io_queues(ctrl);

Hmm. Rather not.

Why do we have to have a separate failover queue? Can't we simply delay the error recovery by the cqt value?

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#9 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 12 lines · Hannes Reinecke

drivers/nvme/host/tcp.c static void nvme_tcp_error_recovery_work(struct work_struct *work) · line 2351

Quoted 10 lines
drivers/nvme/host/tcp.c +1 -0
@@ -2345,6 +2345,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
  	nvme_stop_keep_alive(ctrl);
  	flush_work(&ctrl->async_event_work);
+	nvme_schedule_failover(ctrl);
  	nvme_tcp_teardown_io_queues(ctrl, false);
  	/* unquiesce to fail fast pending requests */
  	nvme_unquiesce_io_queues(ctrl);
 

Hmm. Rather not.

Quoted 1 line

Why do we have to have a separate failover queue?

This RFC plays with the idea to handle the request which timeout on ctrl level. The main point is to avoid touching every single transport.

An additional failover queue is likely to introduce new problems and additional complexity. There is no free lunch. So I don't think it's a good concept afterall.

And as discussed during LSFMM I think it would best to focus on factor out the common code (the project I wanted to work on for a while...) from the transports first and then figure out how to get the CQT, CCF working.

Quoted 1 line

Can't we simply delay the error recovery by the cqt value?

Yes and no. As we know from our in house testing, it fixes the problem for customers but it is not spec compliant.

#10 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner
Quoted 112 lines · Daniel Wagner

The TP4129 mendates that the failover should be delayed by CQT. Thus when nvme_decide_disposition returns FAILOVER do not immediately re-queue it on the namespace level instead queue it on the ctrl's request_list and moved later to the namespace's requeue_list.

---
 drivers/nvme/host/core.c      | 19 ++++++++++++++++
 drivers/nvme/host/fc.c        |  4 ++++
 drivers/nvme/host/multipath.c | 52 ++++++++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h      | 15 +++++++++++++
 drivers/nvme/host/rdma.c      |  2 ++
 drivers/nvme/host/tcp.c       |  1 +
 6 files changed, 90 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +19 -0
index 135045528ea1c79eac0d6d47d5f7f05a7c98acc4..f3155c7735e75e06c4359c26db8931142c067e1d 100644
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 
 	flush_work(&ctrl->reset_work);
 	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);
 	nvme_remove_namespaces(ctrl);
 	ctrl->ops->delete_ctrl(ctrl);
 	nvme_uninit_ctrl(ctrl);
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
 	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
 }
 
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;
+
+	queue_delayed_work(nvme_wq, &ctrl->failover_work, delay);
+}
+EXPORT_SYMBOL_GPL(nvme_schedule_failover);
+
 static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 						 blk_status_t status)
 {
@@ -1336,6 +1350,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 		dev_err(ctrl->device,
 			"failed nvme_keep_alive_end_io error=%d\n",
 				status);
+
+		nvme_schedule_failover(ctrl);
 		return RQ_END_IO_NONE;
 	}
 
@@ -4716,6 +4732,7 @@ EXPORT_SYMBOL_GPL(nvme_remove_io_tag_set);
 
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
+	nvme_schedule_failover(ctrl);
 	nvme_mpath_stop(ctrl);
 	nvme_auth_stop(ctrl);
 	nvme_stop_failfast_work(ctrl);
@@ -4842,6 +4859,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	INIT_DELAYED_WORK(&ctrl->ka_work, nvme_keep_alive_work);
 	INIT_DELAYED_WORK(&ctrl->failfast_work, nvme_failfast_work);
+	INIT_DELAYED_WORK(&ctrl->failover_work, nvme_failover_work);
+	INIT_LIST_HEAD(&ctrl->failover_list);
 	memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd));
 	ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive;
 	ctrl->ka_last_check_time = jiffies;
drivers/nvme/host/fc.c +4 -0
index cdc1ba277a5c23ef1afd26e6911b082f3d12b215..bd897b29cd286008b781bbcb4230e08019da6b6b 100644
@@ -2553,6 +2553,8 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
 {
 	enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	/*
 	 * if an error (io timeout, etc) while (re)connecting, the remote
 	 * port requested terminating of the association (disconnect_ls)
@@ -3378,6 +3380,8 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 	/* will block will waiting for io to terminate */
 	nvme_fc_delete_association(ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
 		dev_err(ctrl->ctrl.device,
 			"NVME-FC{%d}: error_recovery: Couldn't change state "
drivers/nvme/host/multipath.c +2 -0
index 2a7635565083046c575efe1793362ae10581defd..a14b055796b982df96609f53174a5d1334c1c0c4 100644
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
 void nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
 	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
 	unsigned long flags;
 	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
 	nvme_mpath_clear_current_path(ns);
 

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 129

Quoted 10 lines
drivers/nvme/host/multipath.c +4 -3
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
 	blk_steal_bios(&ns->head->requeue_list, req);
 	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);
+

Why do we need to wait until error_recovery for scheduling failover? Can't we schedule failover as soon as we get path error? Also can't we avoid failover_list?

Thanks,
--Nilay

#11 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 7 lines · Nilay Shroff
drivers/nvme/host/multipath.c +0 -1
-	kblockd_schedule_work(&ns->head->requeue_work);

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 129

Quoted 4 lines
drivers/nvme/host/multipath.c +4 -0
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);
+

Why do we need to wait until error_recovery for scheduling failover?

This is where the delay is added to the processing. The failed requests (timeout) are held back by the delay here and after the wait the are immediately fall over

Quoted 2 lines

Can't we schedule failover as soon as we get path error? Also can't we avoid failover_list?

Then we have exactly what we have now. An failed request is rescheduled to the next path immediately.

#12 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner
Quoted 29 lines · Daniel Wagner

The TP4129 mendates that the failover should be delayed by CQT. Thus when nvme_decide_disposition returns FAILOVER do not immediately re-queue it on the namespace level instead queue it on the ctrl's request_list and moved later to the namespace's requeue_list.

---
 drivers/nvme/host/core.c      | 19 ++++++++++++++++
 drivers/nvme/host/fc.c        |  4 ++++
 drivers/nvme/host/multipath.c | 52 ++++++++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h      | 15 +++++++++++++
 drivers/nvme/host/rdma.c      |  2 ++
 drivers/nvme/host/tcp.c       |  1 +
 6 files changed, 90 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +1 -0
index 135045528ea1c79eac0d6d47d5f7f05a7c98acc4..f3155c7735e75e06c4359c26db8931142c067e1d 100644
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 
 	flush_work(&ctrl->reset_work);
 	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);
 	nvme_remove_namespaces(ctrl);
 	ctrl->ops->delete_ctrl(ctrl);
 	nvme_uninit_ctrl(ctrl);
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
 	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 10 lines
drivers/nvme/host/core.c +8 -0
 }
 
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

Quoted 73 lines
drivers/nvme/host/core.c +10 -0
+
+	queue_delayed_work(nvme_wq, &ctrl->failover_work, delay);
+}
+EXPORT_SYMBOL_GPL(nvme_schedule_failover);
+
 static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 						 blk_status_t status)
 {
@@ -1336,6 +1350,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 		dev_err(ctrl->device,
 			"failed nvme_keep_alive_end_io error=%d\n",
 				status);
+
+		nvme_schedule_failover(ctrl);
 		return RQ_END_IO_NONE;
 	}
 
@@ -4716,6 +4732,7 @@ EXPORT_SYMBOL_GPL(nvme_remove_io_tag_set);
 
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
+	nvme_schedule_failover(ctrl);
 	nvme_mpath_stop(ctrl);
 	nvme_auth_stop(ctrl);
 	nvme_stop_failfast_work(ctrl);
@@ -4842,6 +4859,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	INIT_DELAYED_WORK(&ctrl->ka_work, nvme_keep_alive_work);
 	INIT_DELAYED_WORK(&ctrl->failfast_work, nvme_failfast_work);
+	INIT_DELAYED_WORK(&ctrl->failover_work, nvme_failover_work);
+	INIT_LIST_HEAD(&ctrl->failover_list);
 	memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd));
 	ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive;
 	ctrl->ka_last_check_time = jiffies;
drivers/nvme/host/fc.c +4 -0
index cdc1ba277a5c23ef1afd26e6911b082f3d12b215..bd897b29cd286008b781bbcb4230e08019da6b6b 100644
@@ -2553,6 +2553,8 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
 {
 	enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	/*
 	 * if an error (io timeout, etc) while (re)connecting, the remote
 	 * port requested terminating of the association (disconnect_ls)
@@ -3378,6 +3380,8 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 	/* will block will waiting for io to terminate */
 	nvme_fc_delete_association(ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
 		dev_err(ctrl->ctrl.device,
 			"NVME-FC{%d}: error_recovery: Couldn't change state "
drivers/nvme/host/multipath.c +2 -0
index 2a7635565083046c575efe1793362ae10581defd..a14b055796b982df96609f53174a5d1334c1c0c4 100644
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
 void nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
 	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
 	unsigned long flags;
 	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
 	nvme_mpath_clear_current_path(ns);
 

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 128

Quoted 9 lines
drivers/nvme/host/multipath.c +3 -3
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
 	blk_steal_bios(&ns->head->requeue_list, req);
 	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);

I see this is the only place where held requests are added to failover_list.

- Will this hold admin requests in failover_list?

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

- In case of controller reset or delete if nvme_disable_ctrl() successfully disables the controller, then we do not want to add canceled requests to failover_list, right? Does this implementation consider this case?

Quoted 123 lines
drivers/nvme/host/multipath.c +44 -0
+
+	if (state == NVME_CTRL_DELETING) {
+		/*
+		 * request which fail over in the DELETING state were
+		 * canceled and blk_mq_tagset_wait_completed_request will
+		 * block until they have been proceed. Though
+		 * nvme_failover_work is already stopped. Thus schedule
+		 * a failover; it's still necessary to delay these commands
+		 * by CQT.
+		 */
+		nvme_schedule_failover(ctrl);
+	}
+}
+
+void nvme_flush_failover(struct nvme_ctrl *ctrl)
+{
+	LIST_HEAD(failover_list);
+	struct request *rq;
+	bool kick = false;
+
+	spin_lock_irq(&ctrl->lock);
+	list_splice_init(&ctrl->failover_list, &failover_list);
+	spin_unlock_irq(&ctrl->lock);
+
+	while (!list_empty(&failover_list)) {
+		rq = list_entry(failover_list.next,
+				struct request, queuelist);
+		list_del_init(&rq->queuelist);
+
+		nvme_req(rq)->status = 0;
+		nvme_end_req(rq);
+		kick = true;
+	}
+
+	if (kick)
+		nvme_kick_requeue_lists(ctrl);
+}
+
+void nvme_failover_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl = container_of(to_delayed_work(work),
+					struct nvme_ctrl, failover_work);
+
+	nvme_flush_failover(ctrl);
 }
 
 void nvme_mpath_start_request(struct request *rq)
drivers/nvme/host/nvme.h +15 -0
index 7563332b5b7b76fc6165ec8c6f2d144737d4fe85..10eb323bdaf139526959180c1e66ab4579bb145d 100644
@@ -411,6 +411,9 @@ struct nvme_ctrl {
 
 	enum nvme_ctrl_type cntrltype;
 	enum nvme_dctype dctype;
+
+	struct delayed_work failover_work;
+	struct list_head failover_list;
 };
 
 static inline enum nvme_ctrl_state nvme_ctrl_state(struct nvme_ctrl *ctrl)
@@ -954,6 +957,9 @@ void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
 void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
 void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys);
 void nvme_failover_req(struct request *req);
+void nvme_failover_work(struct work_struct *work);
+void nvme_schedule_failover(struct nvme_ctrl *ctrl);
+void nvme_flush_failover(struct nvme_ctrl *ctrl);
 void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
 int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
 void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid);
@@ -996,6 +1002,15 @@ static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
 static inline void nvme_failover_req(struct request *req)
 {
 }
+static inline void nvme_failover_work(struct work_struct *work)
+{
+}
+static inline void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+}
+static inline void nvme_flush_failover(struct nvme_ctrl *ctrl)
+{
+}
 static inline void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
 {
 }
drivers/nvme/host/rdma.c +2 -0
index 86a2891d9bcc7a990cd214a7fe93fa5c55b292c7..9bee376f881b4c3ebe5502abf23a8e76829780ff 100644
@@ -1127,6 +1127,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 
 	nvme_stop_keep_alive(&ctrl->ctrl);
 	flush_work(&ctrl->ctrl.async_event_work);
+	nvme_schedule_failover(&ctrl->ctrl);
 	nvme_rdma_teardown_io_queues(ctrl, false);
 	nvme_unquiesce_io_queues(&ctrl->ctrl);
 	nvme_rdma_teardown_admin_queue(ctrl, false);
@@ -2153,6 +2154,7 @@ static const struct blk_mq_ops nvme_rdma_admin_mq_ops = {
 
 static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
 {
+	nvme_schedule_failover(&ctrl->ctrl);
 	nvme_rdma_teardown_io_queues(ctrl, shutdown);
 	nvme_quiesce_admin_queue(&ctrl->ctrl);
 	nvme_disable_ctrl(&ctrl->ctrl, shutdown);
drivers/nvme/host/tcp.c +1 -0
index d0023bcfd8a79a193adf2807a24481c8c164a174..3a6c1d3febaf233996e4dcf684793327b5d1412f 100644
@@ -2345,6 +2345,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 
 	nvme_stop_keep_alive(ctrl);
 	flush_work(&ctrl->async_event_work);
+	nvme_schedule_failover(ctrl);
 	nvme_tcp_teardown_io_queues(ctrl, false);
 	/* unquiesce to fail fast pending requests */
 	nvme_unquiesce_io_queues(ctrl);
Signature · 2 lines
-- 
2.48.1
maintainer
#13 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 32 lines · Mohamed Khalfella
Quoted 31 lines · Daniel Wagner

The TP4129 mendates that the failover should be delayed by CQT. Thus when nvme_decide_disposition returns FAILOVER do not immediately re-queue it on the namespace level instead queue it on the ctrl's request_list and moved later to the namespace's requeue_list.

---
  drivers/nvme/host/core.c      | 19 ++++++++++++++++
  drivers/nvme/host/fc.c        |  4 ++++
  drivers/nvme/host/multipath.c | 52 ++++++++++++++++++++++++++++++++++++++++---
  drivers/nvme/host/nvme.h      | 15 +++++++++++++
  drivers/nvme/host/rdma.c      |  2 ++
  drivers/nvme/host/tcp.c       |  1 +
  6 files changed, 90 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +1 -0
index 135045528ea1c79eac0d6d47d5f7f05a7c98acc4..f3155c7735e75e06c4359c26db8931142c067e1d 100644
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 
  	flush_work(&ctrl->reset_work);
  	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);
  	nvme_remove_namespaces(ctrl);
  	ctrl->ops->delete_ctrl(ctrl);
  	nvme_uninit_ctrl(ctrl);
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
  	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
  }
 
Quoted 11 lines

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 8 lines
drivers/nvme/host/core.c +8 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

This was said before, but if we are going to always start waiting for kato for failover purposes, we first need a patch that prevent kato from being arbitrarily long.

Lets cap kato to something like 10 seconds (which is 2x the default which apparently no one is touching).

#14 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 12 lines · Sagi Grimberg
Quoted 11 lines

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 8 lines
drivers/nvme/host/core.c +8 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

Quoted 3 lines

This was said before, but if we are going to always start waiting for kato for failover purposes,
we first need a patch that prevent kato from being arbitrarily long.

That should be addressed with the cross controller reset (CCR). The KATO*n
+ CQT is the upper limit for the target recovery. As soon we have CCR, the recovery delay is reduced to the time the CCR exchange takes.

Quoted 2 lines

Lets cap kato to something like 10 seconds (which is 2x the default which apparently no one is touching).

If I understood the TP4129 the upper limit is now defined, so we don't have to define our own upper limit.

maintainer
#15 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 12 lines · Daniel Wagner
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 8 lines
drivers/nvme/host/core.c +8 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

Quoted 4 lines
Quoted 3 lines

This was said before, but if we are going to always start waiting for kato for failover purposes,
we first need a patch that prevent kato from being arbitrarily long.

That should be addressed with the cross controller reset (CCR).

CCR is a better solution as it is explicit, and faster.

Quoted 3 lines

The KATO*n
+ CQT is the upper limit for the target recovery. As soon we have CCR, the recovery delay is reduced to the time the CCR exchange takes.

What I meant was that the user can no longer set kato to be arbitrarily long when we now introduce failover dependency on it.

We need to set a sane maximum value that will failover in a reasonable timeframe. In other words, kato cannot be allowed to be set by the user to 60 minutes. While we didn't care about it before, now it means that failover may take 60+ minutes.

Hence, my request to set kato to a max absolute value of seconds. My vote was 10 (2x of the default), but we can also go with 30.

#16 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 34 lines · Sagi Grimberg
Quoted 12 lines · Daniel Wagner
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 8 lines
drivers/nvme/host/core.c +8 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+  unsigned long delay;
+
+  if (ctrl->cqt)
+          delay = msecs_to_jiffies(ctrl->cqt);
+  else
+          delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

Quoted 4 lines
Quoted 3 lines

This was said before, but if we are going to always start waiting for kato for failover purposes,
we first need a patch that prevent kato from being arbitrarily long.

That should be addressed with the cross controller reset (CCR).

CCR is a better solution as it is explicit, and faster.

Quoted 3 lines

The KATO*n
+ CQT is the upper limit for the target recovery. As soon we have CCR, the recovery delay is reduced to the time the CCR exchange takes.

What I meant was that the user can no longer set kato to be arbitrarily long when we
now introduce failover dependency on it.

We need to set a sane maximum value that will failover in a reasonable timeframe.
In other words, kato cannot be allowed to be set by the user to 60 minutes. While we didn't
care about it before, now it means that failover may take 60+ minutes.

Quoted 3 lines

Hence, my request to set kato to a max absolute value of seconds. My vote was 10 (2x of the default),
but we can also go with 30.

Adding a maximum value for KATO makes a lot of sense to me. This will help keep us away from a hung task timeout when the full delay is taken into account. 30 makes sense to me from the perspective that the maximum should be long enough to handle non-ideal situations functionally, but not a value that you expect people to use regularly.

I think CQT should have a maximum allowed value for similar reasons. If we do clamp down on the CQT, we could be opening ourselves to the target not completely cleaning up, but it keeps us from a hung task timeout, and _any_ delay will help most of the time.

CCR will not address arbitrarily long times for either because:
1. It is optional.
2. It may fail.
3. We still need a ceiling on the recovery time we can handle.

Sincerely,
Randy Jennings

maintainer
#17 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 20 lines
Quoted 11 lines

What I meant was that the user can no longer set kato to be arbitrarily long when we
now introduce failover dependency on it.

We need to set a sane maximum value that will failover in a reasonable timeframe.
In other words, kato cannot be allowed to be set by the user to 60 minutes. While we didn't
care about it before, now it means that failover may take 60+ minutes.

Quoted 3 lines

Hence, my request to set kato to a max absolute value of seconds. My vote was 10 (2x of the default),
but we can also go with 30.

Adding a maximum value for KATO makes a lot of sense to me. This will help keep us away from a hung task timeout when the full delay is taken into account. 30 makes sense to me from the perspective that the maximum should be long enough to handle non-ideal situations functionally, but not a value that you expect people to use regularly.

Quoted 4 lines

I think CQT should have a maximum allowed value for similar reasons. If we do clamp down on the CQT, we could be opening ourselves to the target not completely cleaning up, but it keeps us from a hung task timeout, and _any_ delay will help most of the time.

CQT comes from the controller, and if it is high, it effectively means that the controller cannot handle faster failover reliably. So I think we should leave it as is. It is the vendor problem.

Quoted 5 lines

CCR will not address arbitrarily long times for either because:
1. It is optional.
2. It may fail.
3. We still need a ceiling on the recovery time we can handle.

Yes, makes sense. if it fails, we need to wait until something expires, which would be CQT.

#18 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 26 lines · Sagi Grimberg
Quoted 21 lines
Quoted 12 lines

What I meant was that the user can no longer set kato to be arbitrarily long when we
now introduce failover dependency on it.

We need to set a sane maximum value that will failover in a reasonable timeframe.
In other words, kato cannot be allowed to be set by the user to 60 minutes. While we didn't
care about it before, now it means that failover may take 60+ minutes.

Quoted 3 lines

Hence, my request to set kato to a max absolute value of seconds. My vote was 10 (2x of the default),
but we can also go with 30.

Adding a maximum value for KATO makes a lot of sense to me. This will help keep us away from a hung task timeout when the full delay is taken into account. 30 makes sense to me from the perspective that the maximum should be long enough to handle non-ideal situations functionally, but not a value that you expect people to use regularly.

Quoted 4 lines

I think CQT should have a maximum allowed value for similar reasons. If we do clamp down on the CQT, we could be opening ourselves to the target not completely cleaning up, but it keeps us from a hung task timeout, and _any_ delay will help most of the time.

Quoted 5 lines

CQT comes from the controller, and if it is high, it effectively means that the
controller cannot handle faster failover reliably. So I think we should leave it
as is. It is the vendor problem.

Okay, that is one way to approach it. However, because of the hung task issue, we would be allowing the vendor to panic the initiator with a hung task. Until CCR, and without implementing other checks (for events which might not happen), this hung task would happen on every messy disconnect with that vendor/array.

Sincerely,
Randy Jennings

maintainer
#19 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 11 lines
Quoted 6 lines

CQT comes from the controller, and if it is high, it effectively means that the
controller cannot handle faster failover reliably. So I think we should leave it
as is. It is the vendor problem.

Okay, that is one way to approach it. However, because of the hung task issue, we would be allowing the vendor to panic the initiator with a hung task. Until CCR, and without implementing other checks (for events which might not happen), this hung task would happen on every messy disconnect with that vendor/array.

Its kind of pick your poison situation I guess.
We can log an error for controllers that expose overly long CQT...

Not sure we'll see a hung task here tho, its not like there is a kthread blocking on this, its a delayed work so I think the watchdog won't complain about it...

#20 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 7 lines · Sagi Grimberg
Quoted 7 lines
Quoted 7 lines

CQT comes from the controller, and if it is high, it effectively means that the
controller cannot handle faster failover reliably. So I think we should leave it
as is. It is the vendor problem.

Quoted 8 lines
Quoted 5 lines

Okay, that is one way to approach it. However, because of the hung task issue, we would be allowing the vendor to panic the initiator with a hung task. Until CCR, and without implementing other checks (for events which might not happen), this hung task would happen on every messy disconnect with that vendor/array.

Its kind of pick your poison situation I guess.
We can log an error for controllers that expose overly long CQT...

That sounds like a good idea.

Quoted 4 lines

Not sure we'll see a hung task here tho, its not like there is a kthread blocking
on this, its a delayed work so I think the watchdog won't complain about it...

That is probably true with this patch set.

I believe controller reset (for instance, requested through sysfs) is not supposed to finish until all the requests are no longer being processed (at least if it should have the semantics of a controller level reset from the spec). This patch set may not tie the two together on a disconnected controller, but I think it should. Also, if reconnection in error recovery is tied to this delay, as it is in the patches Mohamed posted (https://lkml.org/lkml/2025/3/24/1136),
there were other things waiting on error recovery finishing. Delaying reconnection in error recovery until the requests are dead makes a lot of sense to me.

Sincerely,
Randy Jennings

#21 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Mohamed Khalfella
Quoted 4 lines · Mohamed Khalfella
Quoted 4 lines
drivers/nvme/host/core.c +4 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
Quoted 8 lines

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 4 lines
drivers/nvme/host/core.c +4 -0
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

The failover schedule delay is the additional amount of time we have to wait for the target to cleanup (CQT). If the CTQ is not valid I thought the spec said to wait for a KATO. Possible I got that wrong.

The factor 3 or 2 is relavant for the timeout value for the KATO command we schedule. The failover schedule timeout is ontop of the command timeout value.

Quoted 28 lines
Quoted 15 lines
drivers/nvme/host/multipath.c +2 -0
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
 void nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
 	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
 	unsigned long flags;
 	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
 	nvme_mpath_clear_current_path(ns);
 

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 128

Quoted 9 lines
drivers/nvme/host/multipath.c +3 -3
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
 	blk_steal_bios(&ns->head->requeue_list, req);
 	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);

I see this is the only place where held requests are added to failover_list.

Quoted 1 line

- Will this hold admin requests in failover_list?

Yes.

Quoted 3 lines

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

Quoted 4 lines

- In case of controller reset or delete if nvme_disable_ctrl() successfully disables the controller, then we do not want to add canceled requests to failover_list, right? Does this implementation consider this case?

Not sure. I've tested a few things but I am pretty sure this RFC is far from being complete.

#22 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 7 lines · Daniel Wagner
Quoted 6 lines · Mohamed Khalfella
Quoted 5 lines
drivers/nvme/host/core.c +5 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+   if (ctrl->cqt)
+           delay = msecs_to_jiffies(ctrl->cqt);
+   else
+           delay = ctrl->kato * HZ;
Quoted 7 lines
Quoted 3 lines

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

The failover schedule delay is the additional amount of time we have to wait for the target to cleanup (CQT). If the CTQ is not valid I thought the spec said to wait for a KATO. Possible I got that wrong.

This is correct (according to the spec, if CQT is 0, wait for KATO instead of 0). I would treat that as a suggestion, though.

Quoted 3 lines

The factor 3 or 2 is relavant for the timeout value for the KATO command we schedule. The failover schedule timeout is ontop of the command timeout value.

3 or 2 is not related to the timeout value of the KATO command. The timeout value of the KATO command is whatever the host wants it to be (with some values being more productive than others). The target is supposed to respond to the KATO command as soon as the target receives it (roughly). The host timeout value should account for network delays and getting-around-to-it delays on the target.

2*/3*KATO is from when the host has detected a loss of communication to when the host knows (given some assumptions) that the target has detected a loss of communication. A command timeout on the host is a fine time for the host to decide that it has lost communication with the target, but there are other events. At the time the host has detected a loss of communication, it needs to tear down the association (which includes stopping KATO & starting disconnect on _all_ the connections in the association). CQT does not start until the host knows that the target has detected a loss of communication. So, Mohamed's delay is correct.

Quoted 6 lines
Quoted 3 lines

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

This is not correct according to the spec, and, I believe, not a good implementation choice. The driver (in the spec) is instructed _not_ to return an error for any request until it knows (given some assumptions) that the target is no longer processing the request (or that processing does not matter; as in the case of a READ).

Sincerely,
Randy Jennings

#23 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 10 lines · Randy Jennings

2*/3*KATO is from when the host has detected a loss of communication to when the host knows (given some assumptions) that the target has detected a loss of communication. A command timeout on the host is a fine time for the host to decide that it has lost communication with the target, but there are other events. At the time the host has detected a loss of communication, it needs to tear down the association (which includes stopping KATO & starting disconnect on _all_ the connections in the association). CQT does not start until the host knows that the target has detected a loss of communication. So, Mohamed's delay is correct.

I was answering the question why I have chosen the values nvme_schedule_failover and wanted to point out these values are not related to the timeout detection.

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

This is not correct according to the spec, and, I believe, not a good implementation choice. The driver (in the spec) is instructed _not_ to return an error for any request until it knows (given some assumptions) that the target is no longer processing the request (or that processing does not matter; as in the case of a READ).

I was not precise enough: admin commands should fail.

I suggest to keep this topic separated for the time being. It makes this discussion even more complex.

#24 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner
Quoted 50 lines · Daniel Wagner
Quoted 4 lines · Mohamed Khalfella
Quoted 4 lines
drivers/nvme/host/core.c +4 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
Quoted 8 lines

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 4 lines
drivers/nvme/host/core.c +4 -0
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

The failover schedule delay is the additional amount of time we have to wait for the target to cleanup (CQT). If the CTQ is not valid I thought the spec said to wait for a KATO. Possible I got that wrong.

The factor 3 or 2 is relavant for the timeout value for the KATO command we schedule. The failover schedule timeout is ontop of the command timeout value.

Quoted 28 lines
Quoted 15 lines
drivers/nvme/host/multipath.c +2 -0
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
 void nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
 	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
 	unsigned long flags;
 	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
 	nvme_mpath_clear_current_path(ns);
 

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 128

Quoted 9 lines
drivers/nvme/host/multipath.c +3 -3
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
 	blk_steal_bios(&ns->head->requeue_list, req);
 	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);

I see this is the only place where held requests are added to failover_list.

Quoted 3 lines
Quoted 1 line

- Will this hold admin requests in failover_list?

Yes.

Help me see this:

- nvme_failover_req() is the only place reqs are added to failover_list.
- nvme_decide_disposition() returns FAILOVER only if req has REQ_NVME_MPATH set.

How/where do admin requests get REQ_NVME_MPATH set?

Quoted 8 lines
Quoted 4 lines

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

Quoted 7 lines
Quoted 4 lines

- In case of controller reset or delete if nvme_disable_ctrl() successfully disables the controller, then we do not want to add canceled requests to failover_list, right? Does this implementation consider this case?

Not sure. I've tested a few things but I am pretty sure this RFC is far from being complete.

I think it does not, and maybe it should honor this. Otherwise every controller reset/delete will end up holding requests unnecessarily.

#25 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 6 lines · Mohamed Khalfella

Help me see this:

- nvme_failover_req() is the only place reqs are added to failover_list.
- nvme_decide_disposition() returns FAILOVER only if req has REQ_NVME_MPATH set.

How/where do admin requests get REQ_NVME_MPATH set?

Admin commands don't set REQ_NVME_MPATH. This is what the current code does and I have deliberately decided not to touch this with this RFC.

Given how much discussion the CQT/CCR feature triggers, I don't think it's a good idea to add this topic to this discussion.

Quoted 12 lines
Quoted 12 lines
Quoted 3 lines

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

Quoted 4 lines

- In case of controller reset or delete if nvme_disable_ctrl() successfully disables the controller, then we do not want to add canceled requests to failover_list, right? Does this implementation consider this case?

Quoted 5 lines
Quoted 2 lines

Not sure. I've tested a few things but I am pretty sure this RFC is far from being complete.

I think it does not, and maybe it should honor this. Otherwise every controller reset/delete will end up holding requests unnecessarily.

Yes, this is one of the problems with the failover queue. It could be solved by really starting to track the delay timeout for each commands. But this is a lot of logic code and complexity. Thus during the discussion at LSFMM everyone including me, said failover queue idea should not be our first choice.

#26 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 6 lines · Daniel Wagner
Quoted 5 lines · Mohamed Khalfella

Help me see this:

- nvme_failover_req() is the only place reqs are added to failover_list.
- nvme_decide_disposition() returns FAILOVER only if req has REQ_NVME_MPATH set.

Quoted 8 lines
Quoted 1 line

How/where do admin requests get REQ_NVME_MPATH set?

Admin commands don't set REQ_NVME_MPATH. This is what the current code does and I have deliberately decided not to touch this with this RFC.

Given how much discussion the CQT/CCR feature triggers, I don't think it's a good idea to add this topic to this discussion.

The point is that holding requests at nvme_failover_req() does not cover admin requests. Do you plan to add support for holding admin requests in the next revision of these patches?

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 3 lines

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

Quoted 4 lines

- In case of controller reset or delete if nvme_disable_ctrl() successfully disables the controller, then we do not want to add canceled requests to failover_list, right? Does this implementation consider this case?

Not sure. I've tested a few things but I am pretty sure this RFC is far from being complete.

Quoted 8 lines
Quoted 2 lines

I think it does not, and maybe it should honor this. Otherwise every controller reset/delete will end up holding requests unnecessarily.

Yes, this is one of the problems with the failover queue. It could be solved by really starting to track the delay timeout for each commands. But this is a lot of logic code and complexity. Thus during the discussion at LSFMM everyone including me, said failover queue idea should not be our first choice.

Got it. I assume this will be addressed in the next revision?

#27 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner
Quoted 52 lines · Daniel Wagner
Quoted 4 lines · Mohamed Khalfella
Quoted 4 lines
drivers/nvme/host/core.c +4 -0
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
Quoted 8 lines

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 4 lines
drivers/nvme/host/core.c +4 -0
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

I thought that delay = m * ctrl->kato + ctrl->cqt where m = ctrl->ctratt & NVME_CTRL_ATTR_TBKAS ? 3 : 2 no?

The failover schedule delay is the additional amount of time we have to wait for the target to cleanup (CQT). If the CTQ is not valid I thought the spec said to wait for a KATO. Possible I got that wrong.

The factor 3 or 2 is relavant for the timeout value for the KATO command we schedule. The failover schedule timeout is ontop of the command timeout value.

Quoted 28 lines
Quoted 15 lines
drivers/nvme/host/multipath.c +2 -0
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
 void nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
 	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
 	unsigned long flags;
 	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
 	nvme_mpath_clear_current_path(ns);
 

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 128

Quoted 9 lines
drivers/nvme/host/multipath.c +3 -3
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
 	blk_steal_bios(&ns->head->requeue_list, req);
 	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);

I see this is the only place where held requests are added to failover_list.

Quoted 1 line

- Will this hold admin requests in failover_list?

Quoted 8 lines

Yes.

Quoted 3 lines

- What about requests that do not go through nvme_failover_req(), like passthrough requests, do we not want to hold these requests until it is safe for them to be retried?

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

I see your point. Unless I am missing something these requests should be held equally to bio requests from multipath layer. Let us say app submitted write a request that got canceled immediately, how does the app know when it is safe to retry the write request?

Holding requests like write until it is safe to be retried is the whole point of this work, right?

Quoted 8 lines
Quoted 5 lines

- In case of controller reset or delete if nvme_disable_ctrl() successfully disables the controller, then we do not want to add canceled requests to failover_list, right? Does this implementation consider this case?

Not sure. I've tested a few things but I am pretty sure this RFC is far from being complete.

#28 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 8 lines · Mohamed Khalfella
Quoted 2 lines · Daniel Wagner

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

I see your point. Unless I am missing something these requests should be held equally to bio requests from multipath layer. Let us say app submitted write a request that got canceled immediately, how does the app know when it is safe to retry the write request?

Good question, but nothing new as far I can tell. If the kernel doesn't start to retry passthru IO commands, we have to figure out how to pass additional information to the userland.

Quoted 2 lines

Holding requests like write until it is safe to be retried is the whole point of this work, right?

My first goal was to address the IO commands submitted via the block layer. I didn't had the IO passthru interface on my radar. I agree, getting the IO passthru path correct is also good idea.

Anyway, I don't have a lot of experience with the IO passthru interface. A quick test to fitgure out what the status quo is: 'nvme read' results in an EINTR or 'Resource temporarily unavailable' (EAGAIN or EWOULDBLOCK) when a tcp connection between host and target is blocked (*), though it comes from an admin command. It looks like I have to dive into this a bit more.

(*) I think this might be the same problem as the systemd folks have reported.

#29 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 5 lines · Daniel Wagner
Quoted 4 lines · Mohamed Khalfella
Quoted 2 lines · Daniel Wagner

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

Quoted 9 lines
Quoted 4 lines

I see your point. Unless I am missing something these requests should be held equally to bio requests from multipath layer. Let us say app submitted write a request that got canceled immediately, how does the app know when it is safe to retry the write request?

Good question, but nothing new as far I can tell. If the kernel doesn't start to retry passthru IO commands, we have to figure out how to pass additional information to the userland.

nvme multipath does not retry passthru commands. That is said, there is nothing prevents userspace from retrying canceled command immediately resulting in the unwanted behavior these very patches try to address.

Quoted 6 lines
Quoted 2 lines

Holding requests like write until it is safe to be retried is the whole point of this work, right?

My first goal was to address the IO commands submitted via the block layer. I didn't had the IO passthru interface on my radar. I agree, getting the IO passthru path correct is also good idea.

Okay. This will be addressed in the next revision, right?

Quoted 10 lines

Anyway, I don't have a lot of experience with the IO passthru interface. A quick test to fitgure out what the status quo is: 'nvme read' results in an EINTR or 'Resource temporarily unavailable' (EAGAIN or EWOULDBLOCK) when a tcp connection between host and target is blocked (*), though it comes from an admin command. It looks like I have to dive into this a bit more.

(*) I think this might be the same problem as the systemd folks have reported.

maintainer
#30 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 9 lines · Mohamed Khalfella
Quoted 8 lines · Daniel Wagner
Quoted 7 lines · Mohamed Khalfella
Quoted 2 lines · Daniel Wagner

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

I see your point. Unless I am missing something these requests should be held equally to bio requests from multipath layer. Let us say app submitted write a request that got canceled immediately, how does the app know when it is safe to retry the write request?

Quoted 7 lines
Quoted 4 lines

Good question, but nothing new as far I can tell. If the kernel doesn't start to retry passthru IO commands, we have to figure out how to pass additional information to the userland.

nvme multipath does not retry passthru commands. That is said, there is nothing prevents userspace from retrying canceled command immediately resulting in the unwanted behavior these very patches try to address.

userspace can read the controller cqt and implement the retry logic on its own. If it doesn't/can't, it should use normal fs io. the driver does not handle passthru retries.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Holding requests like write until it is safe to be retried is the whole point of this work, right?

My first goal was to address the IO commands submitted via the block layer. I didn't had the IO passthru interface on my radar. I agree, getting the IO passthru path correct is also good idea.

Okay. This will be addressed in the next revision, right?

I don't think it should. passthru IO requires the issuer to understand the nvme device, and CQT falls under this definition.

#31 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 16 lines · Sagi Grimberg
Quoted 13 lines · Mohamed Khalfella
Quoted 12 lines · Daniel Wagner
Quoted 7 lines · Mohamed Khalfella
Quoted 2 lines · Daniel Wagner

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

I see your point. Unless I am missing something these requests should be held equally to bio requests from multipath layer. Let us say app submitted write a request that got canceled immediately, how does the app know when it is safe to retry the write request?

Good question, but nothing new as far I can tell. If the kernel doesn't start to retry passthru IO commands, we have to figure out how to pass additional information to the userland.

Quoted 8 lines
Quoted 3 lines

nvme multipath does not retry passthru commands. That is said, there is nothing prevents userspace from retrying canceled command immediately resulting in the unwanted behavior these very patches try to address.

userspace can read the controller cqt and implement the retry logic on its own.
If it doesn't/can't, it should use normal fs io. the driver does not handle passthru retries.

passthru requests are not very different from normal IO. If the driver holds normal IO requests to prevent corruption, it should hold passthru requests too, for the same reason, no?

IMO, keeping the request holding logic in the driver makes more sense than implementing it in userspace. One reason is that CCR can help release requests held requests faster.

Quoted 12 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

Holding requests like write until it is safe to be retried is the whole point of this work, right?

My first goal was to address the IO commands submitted via the block layer. I didn't had the IO passthru interface on my radar. I agree, getting the IO passthru path correct is also good idea.

Okay. This will be addressed in the next revision, right?

I don't think it should. passthru IO requires the issuer to understand the nvme
device, and CQT falls under this definition.

#32 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 26 lines · Mohamed Khalfella
Quoted 16 lines · Sagi Grimberg
Quoted 13 lines · Mohamed Khalfella
Quoted 12 lines · Daniel Wagner
Quoted 7 lines · Mohamed Khalfella
Quoted 2 lines · Daniel Wagner

Pasthrough commands should fail immediately. Userland is in charge here, not the kernel. At least this what should happen here.

I see your point. Unless I am missing something these requests should be held equally to bio requests from multipath layer. Let us say app submitted write a request that got canceled immediately, how does the app know when it is safe to retry the write request?

Good question, but nothing new as far I can tell. If the kernel doesn't start to retry passthru IO commands, we have to figure out how to pass additional information to the userland.

Quoted 8 lines
Quoted 3 lines

nvme multipath does not retry passthru commands. That is said, there is nothing prevents userspace from retrying canceled command immediately resulting in the unwanted behavior these very patches try to address.

userspace can read the controller cqt and implement the retry logic on its own.
If it doesn't/can't, it should use normal fs io. the driver does not handle passthru retries.

Quoted 8 lines

passthru requests are not very different from normal IO. If the driver holds normal IO requests to prevent corruption, it should hold passthru requests too, for the same reason, no?

IMO, keeping the request holding logic in the driver makes more sense than implementing it in userspace. One reason is that CCR can help release requests held requests faster.

One thing to keep in mind: We cannot hold requests during controller reset. Requests are an index into a statically allocated array from the request queue, which gets deleted when the request queue is
removed during controller teardown.

So I _really_ would like to exclude handling of admin and passthrough commands for now, as there are extremely few commands which are not idempotent. If we really care we can just error them out upon submission until error recovery is done.
But I'm not sure if it's worth the hassle; at this time we don't even handle admin commands correctly (admin commands should not be affected by the ANA status, yet they are).

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#33 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner

Hi Daniel,

I just noticed that your patchset addresses a similar issue to the one I'm trying to solve with my recently submitted patchset [1]. Compared to your approach, mine differs in a few key aspects:

1. Only aborted requests are delayed for retry. In the current implementation, nvmf_complete_timed_out_request and nvme_cancel_request set the request status to NVME_SC_HOST_ABORTED_CMD. These requests are usually already sent to the target, but may have timed out or been canceled before a response is received. Since the target may still be processing them, the host needs to delay retrying to ensure the target has completed or cleaned up the stale requests. On the other hand, requests that are not aborted - such as those that never got submitted due to no usable path (e.g., from nvme_ns_head_submit_bio), or those that already received an ANA error from the target - do not need delayed retry.

2. The host explicitly disconnects and stops KeepAlive before delay scheduling retrying requests. This aligns with Section 9.6 "Communication Loss Handling" of the NVMe Base Specification 2.1. Once the host disconnects, the target may take up to the KATO interval to detect the lost connection and begin cleaning up any remaining requests.

@@ -2178,6 +2180,7 @@ static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
nvme_quiesce_admin_queue(&ctrl->ctrl);
nvme_disable_ctrl(&ctrl->ctrl, shutdown); nvme_rdma_teardown_admin_queue(ctrl, shutdown);
+ nvme_delay_kick_retry_lists(&ctrl->ctrl); <<< delay kick retry after teardown all queues
}

3. Delayed retry of aborted requests is supported across multiple scenarios, including error recovery work, controller reset, controller deletion, and controller reconnect failure handling. Here's the relevant code for reference.

drivers/nvme/host/tcp.c +2 -0
index 9109d5476417..f07b3960df7c 100644
@@ -2449,6 +2449,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 destroy_admin:
 	nvme_stop_keep_alive(ctrl);
 	nvme_tcp_teardown_admin_queue(ctrl, new);
+	nvme_delay_kick_retry_lists(ctrl); <<< requests may be timed out when ctrl reconnects
 	return ret;
 }
 
@@ -2494,6 +2495,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 	nvme_tcp_teardown_admin_queue(ctrl, false);
 	nvme_unquiesce_admin_queue(ctrl);
 	nvme_auth_stop(ctrl);
+	nvme_delay_kick_retry_lists(ctrl); <<< retry_lists may contain timed out or cancelled requests
 
 	if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING)) {
 		/* state change failure is ok if we started ctrl delete */

@@ -2513,6 +2515,7 @@ static void nvme_tcp_teardown_ctrl(struct nvme_ctrl *ctrl, bool shutdown)
nvme_quiesce_admin_queue(ctrl);
nvme_disable_ctrl(ctrl, shutdown);
nvme_tcp_teardown_admin_queue(ctrl, shutdown);
+ nvme_delay_kick_retry_lists(ctrl); <<< retry_lists may contain timed out or cancelled requests when ctrl reset or delete
}

Besides, in nvme_tcp_error_recovery_work, the delayed retry must occur after nvme_tcp_teardown_io_queues, because the teardown cancels requests that may need to be retried too.

One limitation of my patchset is that it does not yet include full CQT support, and due to testing environment constraints, only nvme_tcp and nvme_rdma are currently covered.

I'd be happy to discuss the pros and cons of both approaches - perhaps we can combine the best aspects.

Looking forward to your thoughts.

Thanks,
Jiewei

[1] https://lore.kernel.org/linux-nvme/[email protected]/

#34 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner

Hi Daniel,

I just noticed that your patchset addresses a similar issue to the one I'm trying to solve with my recently submitted patchset [1]. Compared to your approach, mine differs in a few key aspects:

1. Only aborted requests are delayed for retry. In the current implementation, nvmf_complete_timed_out_request and nvme_cancel_request set the request status to NVME_SC_HOST_ABORTED_CMD. These requests are usually already sent to the target, but may have timed out or been canceled before a response is received. Since the target may still be processing them, the host needs to delay retrying to ensure the target has completed or cleaned up the stale requests. On the other hand, requests that are not aborted - such as those that never got submitted due to no usable path (e.g., from nvme_ns_head_submit_bio), or those that already received an ANA error from the target - do not need delayed retry.

2. The host explicitly disconnects and stops KeepAlive before delay scheduling retrying requests. This aligns with Section 9.6 "Communication Loss Handling" of the NVMe Base Specification 2.1. Once the host disconnects, the target may take up to the KATO interval to detect the lost connection and begin cleaning up any remaining requests.

@@ -2178,6 +2180,7 @@ static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
nvme_quiesce_admin_queue(&ctrl->ctrl);
nvme_disable_ctrl(&ctrl->ctrl, shutdown); nvme_rdma_teardown_admin_queue(ctrl, shutdown);
+ nvme_delay_kick_retry_lists(&ctrl->ctrl); <<< delay kick retry after teardown all queues
}

3. Delayed retry of aborted requests is supported across multiple scenarios, including error recovery work, controller reset, controller deletion, and controller reconnect failure handling. Here's the relevant code for reference.

drivers/nvme/host/tcp.c +2 -0
index 9109d5476417..f07b3960df7c 100644
@@ -2449,6 +2449,7 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new)
 destroy_admin:
 	nvme_stop_keep_alive(ctrl);
 	nvme_tcp_teardown_admin_queue(ctrl, new);
+	nvme_delay_kick_retry_lists(ctrl); <<< requests may be timed out when ctrl reconnects
 	return ret;
 }
 
@@ -2494,6 +2495,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 	nvme_tcp_teardown_admin_queue(ctrl, false);
 	nvme_unquiesce_admin_queue(ctrl);
 	nvme_auth_stop(ctrl);
+	nvme_delay_kick_retry_lists(ctrl); <<< retry_lists may contain timed out or cancelled requests
 
 	if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING)) {
 		/* state change failure is ok if we started ctrl delete */

@@ -2513,6 +2515,7 @@ static void nvme_tcp_teardown_ctrl(struct nvme_ctrl *ctrl, bool shutdown)
nvme_quiesce_admin_queue(ctrl);
nvme_disable_ctrl(ctrl, shutdown);
nvme_tcp_teardown_admin_queue(ctrl, shutdown);
+ nvme_delay_kick_retry_lists(ctrl); <<< retry_lists may contain timed out or cancelled requests when ctrl reset or delete
}

Besides, in nvme_tcp_error_recovery_work, the delayed retry must occur after nvme_tcp_teardown_io_queues, because the teardown cancels requests that may need to be retried too.

One limitation of my patchset is that it does not yet include full CQT support, and due to testing environment constraints, only nvme_tcp and nvme_rdma are currently covered.

I'd be happy to discuss the pros and cons of both approaches - perhaps we can combine the best aspects.

Looking forward to your thoughts.

Thanks,
Jiewei

[1] https://lore.kernel.org/linux-nvme/[email protected]/

maintainer
#35 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner
Quoted 20 lines · Daniel Wagner

The TP4129 mendates that the failover should be delayed by CQT. Thus when nvme_decide_disposition returns FAILOVER do not immediately re-queue it on the namespace level instead queue it on the ctrl's request_list and moved later to the namespace's requeue_list.

---
  drivers/nvme/host/core.c      | 19 ++++++++++++++++
  drivers/nvme/host/fc.c        |  4 ++++
  drivers/nvme/host/multipath.c | 52 ++++++++++++++++++++++++++++++++++++++++---
  drivers/nvme/host/nvme.h      | 15 +++++++++++++
  drivers/nvme/host/rdma.c      |  2 ++
  drivers/nvme/host/tcp.c       |  1 +
  6 files changed, 90 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +0 -0
index 135045528ea1c79eac0d6d47d5f7f05a7c98acc4..f3155c7735e75e06c4359c26db8931142c067e1d 100644

drivers/nvme/host/core.c static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl) · line 242

Quoted 4 lines
drivers/nvme/host/core.c +1 -0
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 
  	flush_work(&ctrl->reset_work);
  	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);

This will terminate all pending failvoer commands - this is the intended behavior?

Quoted 5 lines
drivers/nvme/host/core.c +0 -0
  	nvme_remove_namespaces(ctrl);
  	ctrl->ops->delete_ctrl(ctrl);
  	nvme_uninit_ctrl(ctrl);
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
  	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1321

Quoted 10 lines
drivers/nvme/host/core.c +8 -0
  }
 
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

This delay was there before?

Quoted 73 lines
drivers/nvme/host/core.c +10 -0
+
+	queue_delayed_work(nvme_wq, &ctrl->failover_work, delay);
+}
+EXPORT_SYMBOL_GPL(nvme_schedule_failover);
+
  static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
  						 blk_status_t status)
  {
@@ -1336,6 +1350,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
  		dev_err(ctrl->device,
  			"failed nvme_keep_alive_end_io error=%d\n",
  				status);
+
+		nvme_schedule_failover(ctrl);
  		return RQ_END_IO_NONE;
  	}
 
@@ -4716,6 +4732,7 @@ EXPORT_SYMBOL_GPL(nvme_remove_io_tag_set);
 
  void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
  {
+	nvme_schedule_failover(ctrl);
  	nvme_mpath_stop(ctrl);
  	nvme_auth_stop(ctrl);
  	nvme_stop_failfast_work(ctrl);
@@ -4842,6 +4859,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
  	INIT_DELAYED_WORK(&ctrl->ka_work, nvme_keep_alive_work);
  	INIT_DELAYED_WORK(&ctrl->failfast_work, nvme_failfast_work);
+	INIT_DELAYED_WORK(&ctrl->failover_work, nvme_failover_work);
+	INIT_LIST_HEAD(&ctrl->failover_list);
  	memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd));
  	ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive;
  	ctrl->ka_last_check_time = jiffies;
drivers/nvme/host/fc.c +4 -0
index cdc1ba277a5c23ef1afd26e6911b082f3d12b215..bd897b29cd286008b781bbcb4230e08019da6b6b 100644
@@ -2553,6 +2553,8 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
  {
  	enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
  	/*
  	 * if an error (io timeout, etc) while (re)connecting, the remote
  	 * port requested terminating of the association (disconnect_ls)
@@ -3378,6 +3380,8 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
  	/* will block will waiting for io to terminate */
  	nvme_fc_delete_association(ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
  	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
  		dev_err(ctrl->ctrl.device,
  			"NVME-FC{%d}: error_recovery: Couldn't change state "
drivers/nvme/host/multipath.c +2 -0
index 2a7635565083046c575efe1793362ae10581defd..a14b055796b982df96609f53174a5d1334c1c0c4 100644
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
  void nvme_failover_req(struct request *req)
  {
  	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
  	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
  	unsigned long flags;
  	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
  	nvme_mpath_clear_current_path(ns);
 

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 128

Quoted 9 lines
drivers/nvme/host/multipath.c +3 -3
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
  	blk_steal_bios(&ns->head->requeue_list, req);
  	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);

So these request - which we stripped their bios, are now placed on a failover queue?

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 139

Quoted 11 lines
drivers/nvme/host/multipath.c +11 -0
+
+	if (state == NVME_CTRL_DELETING) {
+		/*
+		 * request which fail over in the DELETING state were
+		 * canceled and blk_mq_tagset_wait_completed_request will
+		 * block until they have been proceed. Though
+		 * nvme_failover_work is already stopped. Thus schedule
+		 * a failover; it's still necessary to delay these commands
+		 * by CQT.
+		 */
+		nvme_schedule_failover(ctrl);

This condition is unclear. Isn't any request failing over should do this? Something is unclear here.

Quoted 102 lines
drivers/nvme/host/multipath.c +33 -0
+	}
+}
+
+void nvme_flush_failover(struct nvme_ctrl *ctrl)
+{
+	LIST_HEAD(failover_list);
+	struct request *rq;
+	bool kick = false;
+
+	spin_lock_irq(&ctrl->lock);
+	list_splice_init(&ctrl->failover_list, &failover_list);
+	spin_unlock_irq(&ctrl->lock);
+
+	while (!list_empty(&failover_list)) {
+		rq = list_entry(failover_list.next,
+				struct request, queuelist);
+		list_del_init(&rq->queuelist);
+
+		nvme_req(rq)->status = 0;
+		nvme_end_req(rq);
+		kick = true;
+	}
+
+	if (kick)
+		nvme_kick_requeue_lists(ctrl);
+}
+
+void nvme_failover_work(struct work_struct *work)
+{
+	struct nvme_ctrl *ctrl = container_of(to_delayed_work(work),
+					struct nvme_ctrl, failover_work);
+
+	nvme_flush_failover(ctrl);
  }
 
  void nvme_mpath_start_request(struct request *rq)
drivers/nvme/host/nvme.h +15 -0
index 7563332b5b7b76fc6165ec8c6f2d144737d4fe85..10eb323bdaf139526959180c1e66ab4579bb145d 100644
@@ -411,6 +411,9 @@ struct nvme_ctrl {
 
  	enum nvme_ctrl_type cntrltype;
  	enum nvme_dctype dctype;
+
+	struct delayed_work failover_work;
+	struct list_head failover_list;
  };
 
  static inline enum nvme_ctrl_state nvme_ctrl_state(struct nvme_ctrl *ctrl)
@@ -954,6 +957,9 @@ void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
  void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
  void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys);
  void nvme_failover_req(struct request *req);
+void nvme_failover_work(struct work_struct *work);
+void nvme_schedule_failover(struct nvme_ctrl *ctrl);
+void nvme_flush_failover(struct nvme_ctrl *ctrl);
  void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
  int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl,struct nvme_ns_head *head);
  void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid);
@@ -996,6 +1002,15 @@ static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
  static inline void nvme_failover_req(struct request *req)
  {
  }
+static inline void nvme_failover_work(struct work_struct *work)
+{
+}
+static inline void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+}
+static inline void nvme_flush_failover(struct nvme_ctrl *ctrl)
+{
+}
  static inline void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl)
  {
  }
drivers/nvme/host/rdma.c +2 -0
index 86a2891d9bcc7a990cd214a7fe93fa5c55b292c7..9bee376f881b4c3ebe5502abf23a8e76829780ff 100644
@@ -1127,6 +1127,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 
  	nvme_stop_keep_alive(&ctrl->ctrl);
  	flush_work(&ctrl->ctrl.async_event_work);
+	nvme_schedule_failover(&ctrl->ctrl);
  	nvme_rdma_teardown_io_queues(ctrl, false);
  	nvme_unquiesce_io_queues(&ctrl->ctrl);
  	nvme_rdma_teardown_admin_queue(ctrl, false);
@@ -2153,6 +2154,7 @@ static const struct blk_mq_ops nvme_rdma_admin_mq_ops = {
 
  static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown)
  {
+	nvme_schedule_failover(&ctrl->ctrl);
  	nvme_rdma_teardown_io_queues(ctrl, shutdown);
  	nvme_quiesce_admin_queue(&ctrl->ctrl);
  	nvme_disable_ctrl(&ctrl->ctrl, shutdown);
drivers/nvme/host/tcp.c +0 -0
index d0023bcfd8a79a193adf2807a24481c8c164a174..3a6c1d3febaf233996e4dcf684793327b5d1412f 100644

drivers/nvme/host/tcp.c static void nvme_tcp_error_recovery_work(struct work_struct *work) · line 2351

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -0
@@ -2345,6 +2345,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 
  	nvme_stop_keep_alive(ctrl);
  	flush_work(&ctrl->async_event_work);
+	nvme_schedule_failover(ctrl);
  	nvme_tcp_teardown_io_queues(ctrl, false);
  	/* unquiesce to fail fast pending requests */
  	nvme_unquiesce_io_queues(ctrl);

What is the reason for rdma and tcp not being identical?

#36 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 9 lines · Sagi Grimberg

drivers/nvme/host/core.c static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl) · line 241

Quoted 6 lines
drivers/nvme/host/core.c +1 -0
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
  	flush_work(&ctrl->reset_work);
  	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);

This will terminate all pending failvoer commands - this is the intended behavior?

Yes it will, I don't think so. From the feedback I gather so far, it seems the correct way (spec) is to handle each command individually.

FWIW, we are shipping

https://lore.kernel.org/linux-nvme/[email protected]/

as stop-gap solution for a while and our customer wasn't able to reproduce the ghost writes anymore. And as far I can tell it does also failover all pending commands.

Quoted 17 lines
Quoted 5 lines
drivers/nvme/host/core.c +0 -0
  	nvme_remove_namespaces(ctrl);
  	ctrl->ops->delete_ctrl(ctrl);
  	nvme_uninit_ctrl(ctrl);

drivers/nvme/host/core.c static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) · line 1320

Quoted 10 lines
drivers/nvme/host/core.c +8 -0
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
  	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
  }
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;

This delay was there before?

No, this is function returns the additional time the host should wait for it starts to failover. So this is the CQT value, after the KATO timeout protocol and this timeout is not present in the nvme subsystem.

Quoted 13 lines
Quoted 13 lines
drivers/nvme/host/core.c +2 -1
  void nvme_failover_req(struct request *req)
  {
  	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
  	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
  	unsigned long flags;
  	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
  	nvme_mpath_clear_current_path(ns);
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
  	blk_steal_bios(&ns->head->requeue_list, req);
  	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
-	nvme_req(req)->status = 0;
Quoted 8 lines

drivers/nvme/host/core.c void nvme_failover_req(struct request *req) · line 127

Quoted 5 lines
drivers/nvme/host/core.c +3 -2
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);

So these request - which we stripped their bios, are now placed on a failover queue?

Yes, this is the idea.

Quoted 6 lines
Quoted 6 lines
drivers/nvme/host/core.c +6 -0
+
+	if (state == NVME_CTRL_DELETING) {
+		/*
+		 * request which fail over in the DELETING state were
+		 * canceled and blk_mq_tagset_wait_completed_request will
+		 * block until they have been proceed. Though
Quoted 8 lines

drivers/nvme/host/core.c void nvme_failover_req(struct request *req) · line 138

Quoted 5 lines
drivers/nvme/host/core.c +5 -0
+		 * nvme_failover_work is already stopped. Thus schedule
+		 * a failover; it's still necessary to delay these commands
+		 * by CQT.
+		 */
+		nvme_schedule_failover(ctrl);

This condition is unclear. Isn't any request failing over should do this? Something is unclear here.

Ye, the comment is not very clear. Sorry about that. I observed that blk_mq_tagset_wait_completed_request would block when the controller was already in DELETING state and request were canceled, IIRC. The commands would be queued on the failover queue but never processed nvme_failover_work.

Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/core.c +1 -0
+	nvme_schedule_failover(&ctrl->ctrl);
  	nvme_rdma_teardown_io_queues(ctrl, shutdown);
  	nvme_quiesce_admin_queue(&ctrl->ctrl);
  	nvme_disable_ctrl(&ctrl->ctrl, shutdown);
drivers/nvme/host/tcp.c +0 -0
index d0023bcfd8a79a193adf2807a24481c8c164a174..3a6c1d3febaf233996e4dcf684793327b5d1412f 100644
@@ -2345,6 +2345,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
  	nvme_stop_keep_alive(ctrl);
Quoted 8 lines

drivers/nvme/host/tcp.c static void nvme_tcp_error_recovery_work(struct work_struct *work)

Quoted 6 lines
drivers/nvme/host/tcp.c +1 -0
  	flush_work(&ctrl->async_event_work);
+	nvme_schedule_failover(ctrl);
  	nvme_tcp_teardown_io_queues(ctrl, false);
  	/* unquiesce to fail fast pending requests */
  	nvme_unquiesce_io_queues(ctrl);
 

What is the reason for rdma and tcp not being identical?

Sorry, can't remember. But I really want to start moving this code into one place.

#37 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

In reply to Daniel Wagner
Quoted 112 lines · Daniel Wagner

The TP4129 mendates that the failover should be delayed by CQT. Thus when nvme_decide_disposition returns FAILOVER do not immediately re-queue it on the namespace level instead queue it on the ctrl's request_list and moved later to the namespace's requeue_list.

---
 drivers/nvme/host/core.c      | 19 ++++++++++++++++
 drivers/nvme/host/fc.c        |  4 ++++
 drivers/nvme/host/multipath.c | 52 ++++++++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h      | 15 +++++++++++++
 drivers/nvme/host/rdma.c      |  2 ++
 drivers/nvme/host/tcp.c       |  1 +
 6 files changed, 90 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +19 -0
index 135045528ea1c79eac0d6d47d5f7f05a7c98acc4..f3155c7735e75e06c4359c26db8931142c067e1d 100644
@@ -239,6 +239,7 @@ static void nvme_do_delete_ctrl(struct nvme_ctrl *ctrl)
 
 	flush_work(&ctrl->reset_work);
 	nvme_stop_ctrl(ctrl);
+	nvme_flush_failover(ctrl);
 	nvme_remove_namespaces(ctrl);
 	ctrl->ops->delete_ctrl(ctrl);
 	nvme_uninit_ctrl(ctrl);
@@ -1310,6 +1311,19 @@ static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
 	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
 }
 
+void nvme_schedule_failover(struct nvme_ctrl *ctrl)
+{
+	unsigned long delay;
+
+	if (ctrl->cqt)
+		delay = msecs_to_jiffies(ctrl->cqt);
+	else
+		delay = ctrl->kato * HZ;
+
+	queue_delayed_work(nvme_wq, &ctrl->failover_work, delay);
+}
+EXPORT_SYMBOL_GPL(nvme_schedule_failover);
+
 static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 						 blk_status_t status)
 {
@@ -1336,6 +1350,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 		dev_err(ctrl->device,
 			"failed nvme_keep_alive_end_io error=%d\n",
 				status);
+
+		nvme_schedule_failover(ctrl);
 		return RQ_END_IO_NONE;
 	}
 
@@ -4716,6 +4732,7 @@ EXPORT_SYMBOL_GPL(nvme_remove_io_tag_set);
 
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl)
 {
+	nvme_schedule_failover(ctrl);
 	nvme_mpath_stop(ctrl);
 	nvme_auth_stop(ctrl);
 	nvme_stop_failfast_work(ctrl);
@@ -4842,6 +4859,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
 
 	INIT_DELAYED_WORK(&ctrl->ka_work, nvme_keep_alive_work);
 	INIT_DELAYED_WORK(&ctrl->failfast_work, nvme_failfast_work);
+	INIT_DELAYED_WORK(&ctrl->failover_work, nvme_failover_work);
+	INIT_LIST_HEAD(&ctrl->failover_list);
 	memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd));
 	ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive;
 	ctrl->ka_last_check_time = jiffies;
drivers/nvme/host/fc.c +4 -0
index cdc1ba277a5c23ef1afd26e6911b082f3d12b215..bd897b29cd286008b781bbcb4230e08019da6b6b 100644
@@ -2553,6 +2553,8 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
 {
 	enum nvme_ctrl_state state = nvme_ctrl_state(&ctrl->ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	/*
 	 * if an error (io timeout, etc) while (re)connecting, the remote
 	 * port requested terminating of the association (disconnect_ls)
@@ -3378,6 +3380,8 @@ nvme_fc_reset_ctrl_work(struct work_struct *work)
 	/* will block will waiting for io to terminate */
 	nvme_fc_delete_association(ctrl);
 
+	nvme_schedule_failover(&ctrl->ctrl);
+
 	if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING))
 		dev_err(ctrl->ctrl.device,
 			"NVME-FC{%d}: error_recovery: Couldn't change state "
drivers/nvme/host/multipath.c +2 -0
index 2a7635565083046c575efe1793362ae10581defd..a14b055796b982df96609f53174a5d1334c1c0c4 100644
@@ -86,9 +86,11 @@ void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
 void nvme_failover_req(struct request *req)
 {
 	struct nvme_ns *ns = req->q->queuedata;
+	struct nvme_ctrl *ctrl = nvme_req(req)->ctrl;
 	u16 status = nvme_req(req)->status & NVME_SCT_SC_MASK;
 	unsigned long flags;
 	struct bio *bio;
+	enum nvme_ctrl_state state = nvme_ctrl_state(ctrl);
 
 	nvme_mpath_clear_current_path(ns);
 

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 129

Quoted 10 lines
drivers/nvme/host/multipath.c +4 -3
@@ -121,9 +123,53 @@ void nvme_failover_req(struct request *req)
 	blk_steal_bios(&ns->head->requeue_list, req);
 	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
 
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);
+

In case the delay in nvme_schedule_failover() is larget than request timeout, is it possible for timeout callback to be called while a request is sitting in failover_list?

Is there any guarantee to prevent this from happening? I understand from the patch that we do not want this to happen, right?

#38 lore raw

Re: [PATCH RFC 3/3] nvme: delay failover by command quiesce timeout

Quoted 11 lines · Mohamed Khalfella

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 129

Quoted 7 lines
drivers/nvme/host/multipath.c +4 -3
-	nvme_req(req)->status = 0;
-	nvme_end_req(req);
-	kblockd_schedule_work(&ns->head->requeue_work);
+	spin_lock_irqsave(&ctrl->lock, flags);
+	list_add_tail(&req->queuelist, &ctrl->failover_list);
+	spin_unlock_irqrestore(&ctrl->lock, flags);
+

In case the delay in nvme_schedule_failover() is larget than request timeout, is it possible for timeout callback to be called while a request is sitting in failover_list?

Yes this can happen.

Quoted 2 lines

Is there any guarantee to prevent this from happening? I understand from the patch that we do not want this to happen, right?

As already said, I think with failover queue we need to handle timeouts on command basis.