Skip to contentSkip to conversation

keepalive bugfixes

PATCH Superseded Uday Shankar 3 participants 10 messages Apr 17, 2023 – Apr 20, 2023 NVMe

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover keepalive bugfixes

While reviewing the Linux KATO implementation in an attempt to better understand the current NVMe Keep Alive specification, we found a few issues in the host implementation that could contribute to spurious Keep Alive timeouts being detected by controllers.

Uday Shankar (3):
nvme: double KA polling frequency to avoid KATO with TBKAS on
nvme: check IO start time when deciding to defer KA
nvme: improve handling of long keep alives

 drivers/nvme/host/core.c | 38 +++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/nvme.h |  5 +++--
 2 files changed, 38 insertions(+), 5 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/3 nvme: double KA polling frequency to avoid KATO with TBKAS on

With TBKAS on, the completion of one command can defer sending a keep alive for up to twice the delay between successive runs of nvme_keep_alive_work. The current delay of KATO / 2 thus makes it possible for one command to defer sending a keep alive for up to KATO, which can result in the controller detecting a KATO. The following trace demonstrates the issue, taking KATO = 8 for simplicity:

1. t = 0: run nvme_keep_alive_work, no keep-alive sent
2. t = ε: I/O completion seen, set comp_seen = true
3. t = 4: run nvme_keep_alive_work, see comp_seen == true, skip sending keep-alive, set comp_seen = false
4. t = 8: run nvme_keep_alive_work, see comp_seen == false, send a keep-alive command.

Here, there is a delay of 8 - ε between receiving a command completion and sending the next command. With ε small, the controller is likely to detect a keep alive timeout.

Fix this by running nvme_keep_alive_work with a delay of KATO / 4 whenever TBKAS is on. Going through the above trace now gives us a worst-case delay of 4 - ε, which is in line with the recommendation of sending a command every KATO / 2 in the NVMe specification.

---
 drivers/nvme/host/core.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +7 -1
index 6c1e7d6709e0..1298c7b9bffb 100644
@@ -1150,10 +1150,16 @@ EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, NVME_TARGET_PASSTHRU);
  * 
  *   The host should send Keep Alive commands at half of the Keep Alive Timeout
  *   accounting for transport roundtrip times [..].
+ * 
+ * When TBKAS is on, we need to run nvme_keep_alive_work at twice this
+ * frequency, as one command completion can postpone sending a keep alive
+ * command by up to twice the delay between runs.
  */
 static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
 {
-	queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ / 2);
+	unsigned long delay = (ctrl->ctratt & NVME_CTRL_ATTR_TBKAS) ?
+		ctrl->kato * HZ / 4 : ctrl->kato * HZ / 2;
+	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
 }
 
 static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
Signature · 1 line
2.25.1
maintainer
#2 lore raw

Re: [PATCH 1/3] nvme: double KA polling frequency to avoid KATO with TBKAS on

Quoted 44 lines

With TBKAS on, the completion of one command can defer sending a keep alive for up to twice the delay between successive runs of nvme_keep_alive_work. The current delay of KATO / 2 thus makes it possible for one command to defer sending a keep alive for up to KATO, which can result in the controller detecting a KATO. The following trace demonstrates the issue, taking KATO = 8 for simplicity:

1. t = 0: run nvme_keep_alive_work, no keep-alive sent
2. t = ε: I/O completion seen, set comp_seen = true
3. t = 4: run nvme_keep_alive_work, see comp_seen == true, skip sending keep-alive, set comp_seen = false
4. t = 8: run nvme_keep_alive_work, see comp_seen == false, send a keep-alive command.

Here, there is a delay of 8 - ε between receiving a command completion and sending the next command. With ε small, the controller is likely to detect a keep alive timeout.

Fix this by running nvme_keep_alive_work with a delay of KATO / 4 whenever TBKAS is on. Going through the above trace now gives us a worst-case delay of 4 - ε, which is in line with the recommendation of sending a command every KATO / 2 in the NVMe specification.

---
  drivers/nvme/host/core.c | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +4 -0
index 6c1e7d6709e0..1298c7b9bffb 100644
@@ -1150,10 +1150,16 @@ EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, NVME_TARGET_PASSTHRU);
   *
   *   The host should send Keep Alive commands at half of the Keep Alive Timeout
   *   accounting for transport roundtrip times [..].
+ *
+ * When TBKAS is on, we need to run nvme_keep_alive_work at twice this
+ * frequency, as one command completion can postpone sending a keep alive
+ * command by up to twice the delay between runs.

drivers/nvme/host/core.c EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, NVME_TARGET_PASSTHRU); · line 1165

Quoted 10 lines
drivers/nvme/host/core.c +3 -1
   */
  static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
  {
-	queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ / 2);
+	unsigned long delay = (ctrl->ctratt & NVME_CTRL_ATTR_TBKAS) ?
+		ctrl->kato * HZ / 4 : ctrl->kato * HZ / 2;
+	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
  }
 
  static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,

This looks fine to me, the only thing that is a bit concerning is that we may excessively send keep-alive too frequently (default kato is 10 divide by 4 gives every 2 seconds).

#3 lore raw

Re: [PATCH 1/3] nvme: double KA polling frequency to avoid KATO with TBKAS on

Quoted 51 lines · Sagi Grimberg
Quoted 51 lines

With TBKAS on, the completion of one command can defer sending a keep alive for up to twice the delay between successive runs of nvme_keep_alive_work. The current delay of KATO / 2 thus makes it possible for one command to defer sending a keep alive for up to KATO, which can result in the controller detecting a KATO. The following trace demonstrates the issue, taking KATO = 8 for simplicity:

1. t = 0: run nvme_keep_alive_work, no keep-alive sent
2. t = ε: I/O completion seen, set comp_seen = true
3. t = 4: run nvme_keep_alive_work, see comp_seen == true, skip sending keep-alive, set comp_seen = false
4. t = 8: run nvme_keep_alive_work, see comp_seen == false, send a keep-alive command.

Here, there is a delay of 8 - ε between receiving a command completion and sending the next command. With ε small, the controller is likely to detect a keep alive timeout.

Fix this by running nvme_keep_alive_work with a delay of KATO / 4 whenever TBKAS is on. Going through the above trace now gives us a worst-case delay of 4 - ε, which is in line with the recommendation of sending a command every KATO / 2 in the NVMe specification.

---
  drivers/nvme/host/core.c | 8 +++++++-

  1 file changed, 7 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +3 -0
index 6c1e7d6709e0..1298c7b9bffb 100644
@@ -1150,10 +1150,16 @@ EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, 
 NVME_TARGET_PASSTHRU);
    *
    *   The host should send Keep Alive commands at half of the Keep Alive Timeout
    *   accounting for transport roundtrip times [..].
+ *
+ * When TBKAS is on, we need to run nvme_keep_alive_work at twice this
+ * frequency, as one command completion can postpone sending a keep 

alive
+ * command by up to twice the delay between runs.
   */
  static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl) {
-    queue_delayed_work(nvme_wq, &ctrl->ka_work, ctrl->kato * HZ / 2); +    unsigned long delay = (ctrl->ctratt & NVME_CTRL_ATTR_TBKAS) ?

Quoted 8 lines
Quoted 4 lines

+        ctrl->kato * HZ / 4 : ctrl->kato * HZ / 2;
+    queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
  }
  static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,

This looks fine to me, the only thing that is a bit concerning is that we may excessively send keep-alive too frequently (default kato is 10 divide by 4 gives every 2 seconds).

Well, this is with TBKAS on, so we're sending keep-alives only if there is no other traffic on the wire. And then sending a keep-alive every two seconds is hardly exacting.

Cheers,

Hannes

#4 lore raw

PATCH 2/3 nvme: check IO start time when deciding to defer KA

In reply to Uday Shankar

When a command completes, we set a flag which will skip sending a keep alive at the next run of nvme_keep_alive_work when TBKAS is on. However, if the command was submitted long ago, it's possible that the controller may have also restarted its keep alive timer (as a result of receiving the command) long ago. The following trace demonstrates the issue, assuming TBKAS is on and KATO = 8 for simplicity:

1. t = 0: submit I/O commands A, B, C, D, E
2. t = 0.5: commands A, B, C, D, E reach controller, restart its keep alive timer
3. t = 1: A completes
4. t = 2: run nvme_keep_alive_work, see recent completion, do nothing
5. t = 3: B completes
6. t = 4: run nvme_keep_alive_work, see recent completion, do nothing
7. t = 5: C completes
8. t = 6: run nvme_keep_alive_work, see recent completion, do nothing
9. t = 7: D completes
10. t = 8: run nvme_keep_alive_work, see recent completion, do nothing
11. t = 9: E completes

At this point, 8.5 seconds have passed without restarting the controller's keep alive timer, so the controller will detect a keep alive timeout.

Fix this by checking the IO start time when deciding to defer sending a keep alive command. Only set comp_seen if the command started after the most recent run of nvme_keep_alive_work. With this change, the completions of B, C, and D will not set comp_seen and the run of nvme_keep_alive_work at t = 4 will send a keep alive.

---
 drivers/nvme/host/core.c | 14 +++++++++++++-
 drivers/nvme/host/nvme.h |  5 +++--
 2 files changed, 16 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +13 -1
index 1298c7b9bffb..8a63051d7b5e 100644
@@ -397,7 +397,14 @@ void nvme_complete_rq(struct request *req)
 	trace_nvme_complete_rq(req);
 	nvme_cleanup_cmd(req);
 
-	if (ctrl->kas)
+	/*
+	 * Completions of long-running commands should not be able to
+	 * defer sending of periodic keep alives, since the controller
+	 * may have completed processing such commands a long time ago
+	 * (arbitrarily close to command submission time).
+	 */
+	if (ctrl->kas && !ctrl->comp_seen
+		      && nvme_req(req)->start_time >= ctrl->ka_last_check_time)
 		ctrl->comp_seen = true;
 
 	switch (nvme_decide_disposition(req)) {
@@ -1178,6 +1185,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 		return RQ_END_IO_NONE;
 	}
 
+	WRITE_ONCE(ctrl->ka_last_check_time, jiffies);
+	smp_wmb();
 	ctrl->comp_seen = false;
 	spin_lock_irqsave(&ctrl->lock, flags);
 	if (ctrl->state == NVME_CTRL_LIVE ||
@@ -1196,6 +1205,9 @@ static void nvme_keep_alive_work(struct work_struct *work)
 	bool comp_seen = ctrl->comp_seen;
 	struct request *rq;
 
+	WRITE_ONCE(ctrl->ka_last_check_time, jiffies);
+	smp_wmb();
+
 	if ((ctrl->ctratt & NVME_CTRL_ATTR_TBKAS) && comp_seen) {
 		dev_dbg(ctrl->device,
 			"reschedule traffic based keep-alive timer\n");
drivers/nvme/host/nvme.h +3 -2
index bf46f122e9e1..f044ffb9ce10 100644
@@ -162,9 +162,7 @@ struct nvme_request {
 	u8			retries;
 	u8			flags;
 	u16			status;
-#ifdef CONFIG_NVME_MULTIPATH
 	unsigned long		start_time;
-#endif
 	struct nvme_ctrl	*ctrl;
 };
 
@@ -323,6 +321,7 @@ struct nvme_ctrl {
 	struct delayed_work ka_work;
 	struct delayed_work failfast_work;
 	struct nvme_command ka_cmd;
+	unsigned long ka_last_check_time;
 	struct work_struct fw_act_work;
 	unsigned long events;
 
@@ -1028,6 +1027,8 @@ static inline void nvme_start_request(struct request *rq)
 {
 	if (rq->cmd_flags & REQ_NVME_MPATH)
 		nvme_mpath_start_request(rq);
+	else
+		nvme_req(rq)->start_time = jiffies;
 	blk_mq_start_request(rq);
 }
 
Signature · 1 line
2.25.1
maintainer
#5 lore raw

Re: [PATCH 2/3] nvme: check IO start time when deciding to defer KA

Quoted 48 lines · Uday Shankar

When a command completes, we set a flag which will skip sending a keep alive at the next run of nvme_keep_alive_work when TBKAS is on. However, if the command was submitted long ago, it's possible that the controller may have also restarted its keep alive timer (as a result of receiving the command) long ago. The following trace demonstrates the issue, assuming TBKAS is on and KATO = 8 for simplicity:

1. t = 0: submit I/O commands A, B, C, D, E
2. t = 0.5: commands A, B, C, D, E reach controller, restart its keep alive timer
3. t = 1: A completes
4. t = 2: run nvme_keep_alive_work, see recent completion, do nothing
5. t = 3: B completes
6. t = 4: run nvme_keep_alive_work, see recent completion, do nothing
7. t = 5: C completes
8. t = 6: run nvme_keep_alive_work, see recent completion, do nothing
9. t = 7: D completes
10. t = 8: run nvme_keep_alive_work, see recent completion, do nothing
11. t = 9: E completes

At this point, 8.5 seconds have passed without restarting the controller's keep alive timer, so the controller will detect a keep alive timeout.

Fix this by checking the IO start time when deciding to defer sending a keep alive command. Only set comp_seen if the command started after the most recent run of nvme_keep_alive_work. With this change, the completions of B, C, and D will not set comp_seen and the run of nvme_keep_alive_work at t = 4 will send a keep alive.

---
  drivers/nvme/host/core.c | 14 +++++++++++++-
  drivers/nvme/host/nvme.h |  5 +++--
  2 files changed, 16 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +0 -0
index 1298c7b9bffb..8a63051d7b5e 100644
@@ -397,7 +397,14 @@ void nvme_complete_rq(struct request *req)
  	trace_nvme_complete_rq(req);
  	nvme_cleanup_cmd(req);
 

drivers/nvme/host/core.c void nvme_complete_rq(struct request *req) · line 408

Quoted 10 lines
drivers/nvme/host/core.c +8 -1
-	if (ctrl->kas)
+	/*
+	 * Completions of long-running commands should not be able to
+	 * defer sending of periodic keep alives, since the controller
+	 * may have completed processing such commands a long time ago
+	 * (arbitrarily close to command submission time).
+	 */
+	if (ctrl->kas && !ctrl->comp_seen
+		      && nvme_req(req)->start_time >= ctrl->ka_last_check_time)
  		ctrl->comp_seen = true;

indentation is wrong here.

Quoted 44 lines
drivers/nvme/host/core.c +5 -0
  	switch (nvme_decide_disposition(req)) {
@@ -1178,6 +1185,8 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
  		return RQ_END_IO_NONE;
  	}
 
+	WRITE_ONCE(ctrl->ka_last_check_time, jiffies);
+	smp_wmb();
  	ctrl->comp_seen = false;
  	spin_lock_irqsave(&ctrl->lock, flags);
  	if (ctrl->state == NVME_CTRL_LIVE ||
@@ -1196,6 +1205,9 @@ static void nvme_keep_alive_work(struct work_struct *work)
  	bool comp_seen = ctrl->comp_seen;
  	struct request *rq;
 
+	WRITE_ONCE(ctrl->ka_last_check_time, jiffies);
+	smp_wmb();
+
  	if ((ctrl->ctratt & NVME_CTRL_ATTR_TBKAS) && comp_seen) {
  		dev_dbg(ctrl->device,
  			"reschedule traffic based keep-alive timer\n");
drivers/nvme/host/nvme.h +1 -2
index bf46f122e9e1..f044ffb9ce10 100644
@@ -162,9 +162,7 @@ struct nvme_request {
  	u8			retries;
  	u8			flags;
  	u16			status;
-#ifdef CONFIG_NVME_MULTIPATH
  	unsigned long		start_time;
-#endif
  	struct nvme_ctrl	*ctrl;
  };
 
@@ -323,6 +321,7 @@ struct nvme_ctrl {
  	struct delayed_work ka_work;
  	struct delayed_work failfast_work;
  	struct nvme_command ka_cmd;
+	unsigned long ka_last_check_time;
  	struct work_struct fw_act_work;
  	unsigned long events;
 

drivers/nvme/host/nvme.h static inline void nvme_start_request(struct request *rq) · line 1031

Quoted 5 lines
drivers/nvme/host/nvme.h +2 -0
@@ -1028,6 +1027,8 @@ static inline void nvme_start_request(struct request *rq)
  {
  	if (rq->cmd_flags & REQ_NVME_MPATH)
  		nvme_mpath_start_request(rq);
+	else
+		nvme_req(rq)->start_time = jiffies;

nvme_mpath_start_request may not set the start_time if stats are disabled...

#6 lore raw

Re: [PATCH 2/3] nvme: check IO start time when deciding to defer KA

Quoted 6 lines · Sagi Grimberg

drivers/nvme/host/core.c void nvme_complete_rq(struct request *req) · line 408

Quoted 3 lines
drivers/nvme/host/core.c +2 -0
+	if (ctrl->kas && !ctrl->comp_seen
+		      && nvme_req(req)->start_time >= ctrl->ka_last_check_time)
  		ctrl->comp_seen = true;

indentation is wrong here.

Quoted 9 lines

drivers/nvme/host/nvme.h static inline void nvme_start_request(struct request *rq) · line 1031

Quoted 6 lines
drivers/nvme/host/nvme.h +2 -0
@@ -1028,6 +1027,8 @@ static inline void nvme_start_request(struct request *rq)
  {
  	if (rq->cmd_flags & REQ_NVME_MPATH)
  		nvme_mpath_start_request(rq);
+	else
+		nvme_req(rq)->start_time = jiffies;

nvme_mpath_start_request may not set the start_time if stats are disabled...

Thanks for catching these, I'll fix them up in a v2.

#7 lore raw

PATCH 3/3 nvme: improve handling of long keep alives

In reply to Uday Shankar

Upon keep alive completion, nvme_keep_alive_work is scheduled with the same delay every time. If keep alive commands are completing slowly, this may cause a keep alive timeout. The following trace illustrates the issue, taking KATO = 8 and TBKAS off for simplicity:

1. t = 0: run nvme_keep_alive_work, send keep alive
2. t = ε: keep alive reaches controller, controller restarts its keep alive timer
3. t = 4: host receives keep alive completion, schedules nvme_keep_alive_work with delay 4
4. t = 8: run nvme_keep_alive_work, send keep alive

Here, a keep alive having RTT of 4 causes a delay of at least 8 - ε between the controller receiving successive keep alives. With ε small, the controller is likely to detect a keep alive timeout.

Fix this by calculating the RTT of the keep alive command, and adjusting the scheduling delay of the next keep alive work accordingly.

---
 drivers/nvme/host/core.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
drivers/nvme/host/core.c +17 -3
index 8a63051d7b5e..fbb8b2f41fe4 100644
@@ -1162,10 +1162,15 @@ EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, NVME_TARGET_PASSTHRU);
  * frequency, as one command completion can postpone sending a keep alive
  * command by up to twice the delay between runs.
  */
+static unsigned long nvme_keep_alive_work_period(struct nvme_ctrl *ctrl)
+{
+	return (ctrl->ctratt & NVME_CTRL_ATTR_TBKAS) ?
+		(ctrl->kato * HZ / 4) : (ctrl->kato * HZ / 2);
+}
+
 static void nvme_queue_keep_alive_work(struct nvme_ctrl *ctrl)
 {
-	unsigned long delay = (ctrl->ctratt & NVME_CTRL_ATTR_TBKAS) ?
-		ctrl->kato * HZ / 4 : ctrl->kato * HZ / 2;
+	unsigned long delay = nvme_keep_alive_work_period(ctrl);
 	queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
 }
 
@@ -1175,6 +1180,15 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 	struct nvme_ctrl *ctrl = rq->end_io_data;
 	unsigned long flags;
 	bool startka = false;
+	unsigned long rtt = jiffies - nvme_req(rq)->start_time;
+	unsigned long delay = nvme_keep_alive_work_period(ctrl);
+
+	/* Subtract off the keepalive RTT so nvme_keep_alive_work runs
+	 * at the desired frequency. */
+	if (rtt <= delay)
+		delay -= rtt;
+	else
+		delay = 0;
 
 	blk_mq_free_request(rq);
 
@@ -1194,7 +1208,7 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
 		startka = true;
 	spin_unlock_irqrestore(&ctrl->lock, flags);
 	if (startka)
-		nvme_queue_keep_alive_work(ctrl);
+		queue_delayed_work(nvme_wq, &ctrl->ka_work, delay);
 	return RQ_END_IO_NONE;
 }
 
Signature · 1 line
2.25.1
maintainer
#8 lore raw

Re: [PATCH 3/3] nvme: improve handling of long keep alives

Quoted 12 lines · Uday Shankar

Upon keep alive completion, nvme_keep_alive_work is scheduled with the same delay every time. If keep alive commands are completing slowly, this may cause a keep alive timeout. The following trace illustrates the issue, taking KATO = 8 and TBKAS off for simplicity:

1. t = 0: run nvme_keep_alive_work, send keep alive
2. t = ε: keep alive reaches controller, controller restarts its keep alive timer
3. t = 4: host receives keep alive completion, schedules nvme_keep_alive_work with delay 4
4. t = 8: run nvme_keep_alive_work, send keep alive

Quoted 6 lines

Here, a keep alive having RTT of 4 causes a delay of at least 8 - ε between the controller receiving successive keep alives. With ε small, the controller is likely to detect a keep alive timeout.

Fix this by calculating the RTT of the keep alive command, and adjusting the scheduling delay of the next keep alive work accordingly.

Is this something that was met in reality?

it is surprising that host->ctrl is super fast and ctrl->host is super slow to the extent that this situation exists in reality...

#9 lore raw

Re: [PATCH 3/3] nvme: improve handling of long keep alives

Quoted 5 lines · Sagi Grimberg

Is this something that was met in reality?

it is surprising that host->ctrl is super fast and ctrl->host is super slow to the extent that this situation exists in reality...

We haven't seen this exact issue in reality, but we more generally have observed significant delays in host-side keep alive processing with controllers that do not support TBKA. This patch should help with that issue.