keepalive bugfixes
v6 cover keepalive bugfixes
While reviewing the Linux KATO implementation in an attempt to better understand the current NVMe Keep Alive specification, we found a few issues in the host implementation that could contribute to spurious Keep Alive timeouts being detected by controllers.
Changes v5-v6 (https://lore.kernel.org/linux-nvme/[email protected]/):
- Patch 2: fix line length and extra empty line, add comment about
req->deadline - req->timeout
Changes v4-v5 (https://lore.kernel.org/linux-nvme/[email protected]/):
- Patch 2: revert changes to start_time; use req->deadline -
req->timeout instead
Changes v3-v4 (https://lore.kernel.org/linux-nvme/[email protected]/):
- Patch 1: fix style, improve readability, include helper from patch 3
- Patch 2: refactor to remove need for WRITE_ONCE/barriers
- Patch 3: fix comment style, move helper to patch 1
Changes v2-v3 (https://lore.kernel.org/linux-nvme/[email protected]/):
- Patch 3: add a warning log line for long keepalive RTT
Changes v1-v2 (https://lore.kernel.org/linux-nvme/[email protected]/):
- Patch 2: fix indentation, set start_time even when stats disabled
Uday Shankar (3):
nvme: double KA polling frequency to avoid KATO with TBKAS on
nvme: check IO start time when deciding to defer KA
nvme: improve handling of long keep alives
drivers/nvme/host/core.c | 48 +++++++++++++++++++++++++++++++++++++--- drivers/nvme/host/nvme.h | 1 + 2 files changed, 46 insertions(+), 3 deletions(-)
Signature · 1 line
2.25.1
PATCH v6 1/3 nvme: double KA polling frequency to avoid KATO with TBKAS on
With TBKAS on, the completion of one command can defer sending a keep alive for up to twice the delay between successive runs of nvme_keep_alive_work. The current delay of KATO / 2 thus makes it possible for one command to defer sending a keep alive for up to KATO, which can result in the controller detecting a KATO. The following trace demonstrates the issue, taking KATO = 8 for simplicity:
1. t = 0: run nvme_keep_alive_work, no keep-alive sent
2. t = ε: I/O completion seen, set comp_seen = true
3. t = 4: run nvme_keep_alive_work, see comp_seen == true, skip sending keep-alive, set comp_seen = false
4. t = 8: run nvme_keep_alive_work, see comp_seen == false, send a keep-alive command.
Here, there is a delay of 8 - ε between receiving a command completion and sending the next command. With ε small, the controller is likely to detect a keep alive timeout.
Fix this by running nvme_keep_alive_work with a delay of KATO / 4 whenever TBKAS is on. Going through the above trace now gives us a worst-case delay of 4 - ε, which is in line with the recommendation of sending a command every KATO / 2 in the NVMe specification.
--- drivers/nvme/host/core.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +17 -1
@@ -1161,9 +1161,25 @@ EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, NVME_TARGET_PASSTHRU);* The host should send Keep Alive commands at half of the Keep Alive Timeout * accounting for transport roundtrip times [..]. */ +static unsigned long nvme_keep_alive_work_period (struct nvme_ctrl *ctrl) +{ +unsigned long delay = ctrl->kato *HZ /2 ; + +/* +* When using Traffic Based Keep Alive, we need to run +* nvme_keep_alive_work at twice the normal frequency, as one +* command completion can postpone sending a keep alive command +* by up to twice the delay between runs. +*/ +if (ctrl->ctratt &NVME_CTRL_ATTR_TBKAS ) + delay /=2 ; +return delay; +} +static void nvme_queue_keep_alive_work (struct nvme_ctrl *ctrl) { -queue_delayed_work (nvme_wq, &ctrl->ka_work, ctrl->kato *HZ /2 ); +queue_delayed_work (nvme_wq, &ctrl->ka_work, +nvme_keep_alive_work_period (ctrl)); } static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,
Signature · 1 line
2.25.1
PATCH v6 2/3 nvme: check IO start time when deciding to defer KA
In reply to Uday ShankarWhen a command completes, we set a flag which will skip sending a keep alive at the next run of nvme_keep_alive_work when TBKAS is on. However, if the command was submitted long ago, it's possible that the controller may have also restarted its keep alive timer (as a result of receiving the command) long ago. The following trace demonstrates the issue, assuming TBKAS is on and KATO = 8 for simplicity:
1. t = 0: submit I/O commands A, B, C, D, E
2. t = 0.5: commands A, B, C, D, E reach controller, restart its keep alive timer
3. t = 1: A completes
4. t = 2: run nvme_keep_alive_work, see recent completion, do nothing
5. t = 3: B completes
6. t = 4: run nvme_keep_alive_work, see recent completion, do nothing
7. t = 5: C completes
8. t = 6: run nvme_keep_alive_work, see recent completion, do nothing
9. t = 7: D completes
10. t = 8: run nvme_keep_alive_work, see recent completion, do nothing
11. t = 9: E completes
At this point, 8.5 seconds have passed without restarting the controller's keep alive timer, so the controller will detect a keep alive timeout.
Fix this by checking the IO start time when deciding to defer sending a keep alive command. Only set comp_seen if the command started after the most recent run of nvme_keep_alive_work. With this change, the completions of B, C, and D will not set comp_seen and the run of nvme_keep_alive_work at t = 4 will send a keep alive.
--- drivers/nvme/host/core.c | 14 +++++++++++++- drivers/nvme/host/nvme.h | 1 + 2 files changed, 14 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +13 -1
@@ -397,7 +397,16 @@ void nvme_complete_rq(struct request *req)trace_nvme_complete_rq (req);nvme_cleanup_cmd (req); -if (ctrl->kas) +/* +* Completions of long-running commands should not be able to +* defer sending of periodic keep alives, since the controller +* may have completed processing such commands a long time ago +* (arbitrarily close to command submission time). +* req->deadline - req->timeout is the command submission time +* in jiffies. +*/ +if (ctrl->kas && + req->deadline - req->timeout >= ctrl->ka_last_check_time) ctrl->comp_seen = true;switch (nvme_decide_disposition (req)) {
@@ -1198,6 +1207,7 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,return RQ_END_IO_NONE ; } + ctrl->ka_last_check_time = jiffies; ctrl->comp_seen = false;spin_lock_irqsave (&ctrl->lock, flags); if (ctrl->state == NVME_CTRL_LIVE ||
@@ -1216,6 +1226,8 @@ static void nvme_keep_alive_work(struct work_struct *work)bool comp_seen = ctrl->comp_seen;struct request *rq; + ctrl->ka_last_check_time = jiffies; +if ((ctrl->ctratt &NVME_CTRL_ATTR_TBKAS ) && comp_seen) {dev_dbg (ctrl->device,"reschedule traffic based keep-alive timer\n" );
drivers/nvme/host/nvme.h +1 -0
@@ -323,6 +323,7 @@ struct nvme_ctrl {struct delayed_work ka_work;struct delayed_work failfast_work;struct nvme_command ka_cmd; +unsigned long ka_last_check_time;struct work_struct fw_act_work;unsigned long events;
Signature · 1 line
2.25.1
Re: [PATCH v6 2/3] nvme: check IO start time when deciding to defer KA
Looks good:
PATCH v6 3/3 nvme: improve handling of long keep alives
In reply to Uday ShankarUpon keep alive completion, nvme_keep_alive_work is scheduled with the same delay every time. If keep alive commands are completing slowly, this may cause a keep alive timeout. The following trace illustrates the issue, taking KATO = 8 and TBKAS off for simplicity:
1. t = 0: run nvme_keep_alive_work, send keep alive
2. t = ε: keep alive reaches controller, controller restarts its keep alive timer
3. t = 4: host receives keep alive completion, schedules nvme_keep_alive_work with delay 4
4. t = 8: run nvme_keep_alive_work, send keep alive
Here, a keep alive having RTT of 4 causes a delay of at least 8 - ε between the controller receiving successive keep alives. With ε small, the controller is likely to detect a keep alive timeout.
Fix this by calculating the RTT of the keep alive command, and adjusting the scheduling delay of the next keep alive work accordingly.
--- drivers/nvme/host/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +15 -1
@@ -1197,6 +1197,20 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq,struct nvme_ctrl *ctrl = rq->end_io_data;unsigned long flags;bool startka = false; +unsigned long rtt = jiffies -nvme_req (rq)->start_time; +unsigned long delay =nvme_keep_alive_work_period (ctrl); + +/* +* Subtract off the keepalive RTT so nvme_keep_alive_work runs +* at the desired frequency. +*/ +if (rtt <= delay) { + delay -= rtt; + }else { +dev_warn (ctrl->device,"long keepalive RTT (%u ms)\n" , +jiffies_to_msecs (rtt)); + delay =0 ; + }blk_mq_free_request (rq);
@@ -1215,7 +1229,7 @@ static enum rq_end_io_ret nvme_keep_alive_end_io(struct request *rq, startka = true;spin_unlock_irqrestore (&ctrl->lock, flags);if (startka) -nvme_queue_keep_alive_work (ctrl); +queue_delayed_work (nvme_wq, &ctrl->ka_work, delay);return RQ_END_IO_NONE ; }
Signature · 1 line
2.25.1
Thanks, applied for nvme-6.4.