nvme-tcp: Do not terminate commands when in RESETTING
PATCH 1/1 nvme-tcp: Do not terminate commands when in RESETTING
From: Hannes Reinecke <[email protected]>
Terminating commands from the timeout handler might lead to a data corruption as the timeout might trigger before KATO expired.
This is the case when several commands have been started before the keep-alive command and the command timeouts trigger just after the keep-alive command has been sent. Then the first command will trigger an error recovery, but all the other commands will be aborted directly and immediately retried.
So return BLK_EH_RESET_TIMER for I/O commands when error recovery has been started to ensure that the commands will be retried only after the KATO interval.
--- drivers/nvme/host/tcp.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/nvme/host/tcp.c +12 -0
@@ -2429,6 +2429,18 @@ static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) rq->tag,nvme_cid (rq), pdu->hdr.type, opc, nvme_opcode_str(qid, opc, fctype), qid); +/* +* If the error recovery is started we should ignore all +* I/O commands as they'll be aborted once error recovery starts. +* Otherwise they'll be failed over immediately and might +* cause data corruption. +*/ +if (ctrl->state ==NVME_CTRL_RESETTING && qid >0 ) { +/* Avoid interfering with firmware download */ +if (!WARN_ON (work_pending (&ctrl->fw_act_work))) +return BLK_EH_RESET_TIMER ; + } +if (ctrl->state !=NVME_CTRL_LIVE ) {/* * If we are resetting, connecting or deleting we should
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerQuoted 26 lines
From: Hannes Reinecke <[email protected]>
Terminating commands from the timeout handler might lead to a data corruption as the timeout might trigger before KATO expired.
This is the case when several commands have been started before the keep-alive command and the command timeouts trigger just after the keep-alive command has been sent. Then the first command will trigger an error recovery, but all the other commands will be aborted directly and immediately retried.
So returnBLK_EH_RESET_TIMERfor I/O commands when error recovery has been started to ensure that the commands will be retried only after the KATO interval.Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/tcp.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2436
Quoted 8 lines
drivers/nvme/host/tcp.c +5
-0@@ -2429,6 +2429,18 @@ static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) rq->tag,nvme_cid (rq), pdu->hdr.type, opc, nvme_opcode_str(qid, opc, fctype), qid); +/* +* If the error recovery is started we should ignore all +* I/O commands as they'll be aborted once error recovery starts. +* Otherwise they'll be failed over immediately and might +* cause data corruption.
I think that we can settle with a simpler comment:
/*
* If the controller is in state RESETTING, the error
* recovery handler will cancel all inflight commands soon
* which in turn may failover to a different path. */
The error handler will start promptly anyways.
drivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2438
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+*/ +if (ctrl->state ==NVME_CTRL_RESETTING && qid >0 ) {
What would be the issue doing this for admin commands as well?
drivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2440
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+/* Avoid interfering with firmware download */ +if (!WARN_ON (work_pending (&ctrl->fw_act_work)))
Let's ignore the fw_act_work, it's not a thing in fabrics.
drivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2446
Quoted 6 lines
drivers/nvme/host/tcp.c +3
-0+return BLK_EH_RESET_TIMER ; + } +if (ctrl->state !=NVME_CTRL_LIVE ) {/* * If we are resetting, connecting or deleting we should
I think that the comment here needs an update as well.
BTW, what happens when deleting a controller? Won't the state be DELETING and the situation you are trying to prevent happen as well?
Quoted 42 lines · Sagi Grimberg
Quoted 26 lines
From: Hannes Reinecke <[email protected]>
Terminating commands from the timeout handler might lead to a data corruption as the timeout might trigger before KATO expired.
This is the case when several commands have been started before the keep-alive command and the command timeouts trigger just after the keep-alive command has been sent. Then the first command will trigger an error recovery, but all the other commands will be aborted directly and immediately retried.
So returnBLK_EH_RESET_TIMERfor I/O commands when error recovery has been started to ensure that the commands will be retried only after the KATO interval.Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 12 ++++++++++++1 file changed, 12 insertions(+)drivers/nvme/host/tcp.c +0
-0drivers/nvme/host/tcp.c static enum blk_eh_timer_return · line 2436
Quoted 8 lines
drivers/nvme/host/tcp.c +5
-0@@ -2429,6 +2429,18 @@ static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) rq->tag, nvme_cid(rq), pdu->hdr.type, opc, nvme_opcode_str(qid, opc, fctype), qid); + /* +* If the error recovery is started we should ignore all +* I/O commands as they'll be aborted once error recovery starts. +* Otherwise they'll be failed over immediately and might +* cause data corruption. I think that we can settle with a simpler comment:
/*
* If the controller is in state RESETTING, the error
* recovery handler will cancel all inflight commands soon
* which in turn may failover to a different path. */
Quoted 2 lines
The error handler will start promptly anyways.
Ok.
Quoted 5 lines
drivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2438
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+*/ + if (ctrl->state == NVME_CTRL_RESETTING && qid > 0) {What would be the issue doing this for admin commands as well?
There are several commands sent during the 'connect' workflow before the queue is switched to 'LIVE', and we need to terminate them correctly such that the 'connect' workflow can make progress (and we're not deadlocking waiting for the commands to complete).
(Cf the comment after the 'LIVE' check later in the code)
Plus all commands on the admin queue do not change the state of the data, so it doesn't matter if they got re-issued to a different controller. Well, it _might_ matter, but then we'll be getting an NVMe status back and we'll be notified in the message log.
Core issue with this patch is that we want to avoid a silent failover of data I/O before KATO expires.
Quoted 5 lines
drivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2440
Quoted 2 lines
drivers/nvme/host/tcp.c +2
-0+ /* Avoid interfering with firmware download */ + if (!WARN_ON(work_pending(&ctrl->fw_act_work)))Let's ignore the
fw_act_work, it's not a thing in fabrics.
Wasn't sure, but if you say so, ok.
Quoted 9 lines
drivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2446
Quoted 6 lines
drivers/nvme/host/tcp.c +3
-0+ return BLK_EH_RESET_TIMER; + } + if (ctrl->state != NVME_CTRL_LIVE) { /** If we are resetting, connecting or deleting we should I think that the comment here needs an update as well.
Oh, I thought that I had modified that, too. But yes, of course.
Quoted 2 lines
BTW, what happens when deleting a controller? Won't the state be DELETING and the situation you are trying to prevent happen as well?
Not sure. Haven't really tested that path, and it'll be hard to trigger anyway (you would have to trigger controller deletion _just_ before the I/O timeout hits).
But yeah, I could check for DELETING, too.
Cheers,
Hannes
Keith Busch
maintainerdrivers/nvme/host/tcp.c static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) · line 2444
Quoted 7 lines · On Thu, Jan 11, 2024 at 08:29:29AM +0100, [email protected] wrote:
drivers/nvme/host/tcp.c +6
-0+if (ctrl->state ==NVME_CTRL_RESETTING && qid >0 ) { +/* Avoid interfering with firmware download */ +if (!WARN_ON (work_pending (&ctrl->fw_act_work))) +return BLK_EH_RESET_TIMER ; + } +if (ctrl->state !=NVME_CTRL_LIVE ) {
Unfortunately I submitted a fix late in 6.7 that changes all the ctrl->state checks to use the READ_ONCE helper. Let's wait for block tree's 6.8 to sync back to mainline, then redo this patch with those updates included to avoid the conflicts.