nvme: fixes for command retry
v2 cover nvme: fixes for command retry
Hello,
This patch series is V2 for Advanced Command Retry Enable(ACRE) feature.
The first patch checks all CRDT(Command Retry Delay Time) [0..2] to figure out whether CRDT is configured or not.
The other patch has been updated to take internal flag in the controller instance instead of the module parameter which has been posted in the previous series [1]. This is based on the review by Chao: Thanks Chao!
Please review.
Thanks,
Since V1:
- The first patch has been applied (Christoph). Removed from this series.
- Update the last patch to take acre flag from the controller instance instead of module parameter by updating nvme_decide_disposition(). (Chao)
[1] https://lore.kernel.org/linux-nvme/[email protected]/T/#t
Minwoo Im (2):
nvme: check all retry delay times in Identify Controller
nvme: retry commands based on ACRE result
drivers/nvme/host/core.c | 18 +++++++++++++++--- drivers/nvme/host/nvme.h | 1 + 2 files changed, 16 insertions(+), 3 deletions(-)
Signature · 7 lines
2.17.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 1/2 nvme: check all retry delay times in Identify Controller
If none of Command Retry Delay Time(CRDT)s are set, then we can simply skip to configure Advanced Command Retry Enable(ACRE).
CRDT selection will be made in Completion Queue Entry that controller has responsibility to decide.
This patch checks all CRDT[0, 1, and 2] in Identify Controller data structrure instead of checking the first one: CRDT[0] for cases that:
- CRDT[0] == 0, but CRDT[1 or 2] != 0
--- drivers/nvme/host/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +1 -1
@@ -2499,7 +2499,7 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)int ret;/* Don't bother enabling the feature if retry delay is not reported */ -if (!ctrl->crdt[0 ]) +if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ; host =kzalloc (sizeof (*host),GFP_KERNEL );
Signature · 7 lines
2.17.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 2/2 nvme: retry commands based on ACRE result
In reply to Minwoo ImIntroduce acre flag for Advanced Command Retry Enable to controller instance to decide whether to retry or not in error cases. This flag is set if Set Features for Host Behavior Support with ACRE bit set is successfully done during reset_work.
This patch also fixes nvme_decide_disposition() because all the nvme requests are initialized with REQ_FAILFAST_DRIVER to req->cmd_flags so that blk_noretry_request(req) is always true. Check ctrl->acre first and if it's not host path error, then it will be retried.
--- drivers/nvme/host/core.c | 16 ++++++++++++++-- drivers/nvme/host/nvme.h | 1 + 2 files changed, 15 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +14 -2
@@ -303,8 +303,7 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)if (likely (nvme_req (req)->status ==0 ))return COMPLETE ; -if (blk_noretry_request (req) || - (nvme_req (req)->status &NVME_SC_DNR ) || +if ((nvme_req (req)->status &NVME_SC_DNR ) ||nvme_req (req)->retries >= nvme_max_retries)return COMPLETE ;
@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre) { +if (!nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ; + }else if (blk_noretry_request (req)) +return COMPLETE ; +return RETRY ; }
@@ -2498,6 +2504,8 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)struct nvme_feat_host_behavior *host;int ret; + ctrl->acre = false; +/* Don't bother enabling the feature if retry delay is not reported */ if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ;
@@ -2509,6 +2517,10 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl) host->acre =NVME_ENABLE_ACRE ; ret =nvme_set_features (ctrl,NVME_FEAT_HOST_BEHAVIOR ,0 , host,sizeof (*host),NULL ); + +if (!ret) + ctrl->acre = true; +kfree (host);return ret; }
drivers/nvme/host/nvme.h +1 -0
@@ -270,6 +270,7 @@ struct nvme_ctrl {#ifdef CONFIG_BLK_DEV_ZONED u32 max_zone_append;#endif +bool acre;u16 crdt[3 ];u16 oncs;u16 oacs;
Signature · 7 lines
2.17.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH V2 2/2] nvme: retry commands based on ACRE result
Quoted 7 lines
Introduce acre flag for Advanced Command Retry Enable to controller instance to decide whether to retry or not in error cases. This flag is set if Set Features for Host Behavior Support with ACRE bit set is successfully done during
reset_work.This patch also fixes
nvme_decide_disposition()because all the nvme requests are initialized withREQ_FAILFAST_DRIVERtoreq->cmd_flags
You mean for passthru commands, nvme doesn't control this for other commands.
Quoted 3 lines
so
thatblk_noretry_request(req)is always true. Checkctrl->acrefirst and if it's not host path error, then it will be retried.
I suggest to split out this part to its own patch.
Quoted 62 lines
Cc Chao Leng <[email protected]>Signed-off-by Minwoo Im <[email protected]>--- drivers/nvme/host/core.c | 16 ++++++++++++++-- drivers/nvme/host/nvme.h | 1 + 2 files changed, 15 insertions(+), 2 deletions(-)drivers/nvme/host/core.c +14
-2@@ -303,8 +303,7 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)if (likely (nvme_req (req)->status ==0 ))return COMPLETE ; -if (blk_noretry_request (req) || - (nvme_req (req)->status &NVME_SC_DNR ) || +if ((nvme_req (req)->status &NVME_SC_DNR ) ||nvme_req (req)->retries >= nvme_max_retries)return COMPLETE ;@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre) { +if (!nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ; + }else if (blk_noretry_request (req)) +return COMPLETE ; +return RETRY ; }@@ -2498,6 +2504,8 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)struct nvme_feat_host_behavior *host;int ret; + ctrl->acre = false; +/* Don't bother enabling the feature if retry delay is not reported */ if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ;@@ -2509,6 +2517,10 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl) host->acre =NVME_ENABLE_ACRE ; ret =nvme_set_features (ctrl,NVME_FEAT_HOST_BEHAVIOR ,0 , host,sizeof (*host),NULL ); + +if (!ret) + ctrl->acre = true; +kfree (host);return ret; }drivers/nvme/host/nvme.h +0
-0
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 276
Quoted 7 lines
drivers/nvme/host/nvme.h +1
-0@@ -270,6 +270,7 @@ struct nvme_ctrl {#ifdef CONFIG_BLK_DEV_ZONED u32 max_zone_append;#endif +bool acre;u16 crdt[3 ];u16 oncs;u16 oacs;
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V2 2/2] nvme: retry commands based on ACRE result
Hello,
Thanks for your time!
Quoted 11 lines · Sagi Grimberg
Quoted 7 lines
Introduce acre flag for Advanced Command Retry Enable to controller instance to decide whether to retry or not in error cases. This flag is set if Set Features for Host Behavior Support with ACRE bit set is successfully done during
reset_work.This patch also fixes
nvme_decide_disposition()because all the nvme requests are initialized withREQ_FAILFAST_DRIVERtoreq->cmd_flagsYou mean for passthru commands, nvme doesn't control this for other commands.
Oh, I forgot to mention... Thanks for catching this.
Quoted 6 lines
Quoted 3 lines
so
thatblk_noretry_request(req)is always true. Checkctrl->acrefirst and if it's not host path error, then it will be retried.I suggest to split out this part to its own patch.
Okay. I will split it up to two patches:
- Introducing ACRE flag to controller
- Check ACRE flag in nvme_decide_disposition()
Thanks!
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V2 2/2] nvme: retry commands based on ACRE result
In reply to Minwoo ImQuoted 32 lines · Minwoo Im
Introduce acre flag for Advanced Command Retry Enable to controller instance to decide whether to retry or not in error cases. This flag is set if Set Features for Host Behavior Support with ACRE bit set is successfully done during
reset_work.This patch also fixes
nvme_decide_disposition()because all the nvme requests are initialized withREQ_FAILFAST_DRIVERtoreq->cmd_flagsso thatblk_noretry_request(req)is always true. Checkctrl->acrefirst and if it's not host path error, then it will be retried.Cc Chao Leng <[email protected]>Signed-off-by Minwoo Im <[email protected]>--- drivers/nvme/host/core.c | 16 ++++++++++++++-- drivers/nvme/host/nvme.h | 1 + 2 files changed, 15 insertions(+), 2 deletions(-)drivers/nvme/host/core.c +1
-2@@ -303,8 +303,7 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)if (likely (nvme_req (req)->status ==0 ))return COMPLETE ; -if (blk_noretry_request (req) || - (nvme_req (req)->status &NVME_SC_DNR ) || +if ((nvme_req (req)->status &NVME_SC_DNR ) ||nvme_req (req)->retries >= nvme_max_retries)return COMPLETE ;
drivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 324
Quoted 9 lines
drivers/nvme/host/core.c +6
-0@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre) { +if (!nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ; + }else if (blk_noretry_request (req)) +return COMPLETE ;
If acre is true and nvme_req(req)->status is nvme path error and blk_noretry_request(req) is true, we need return complete for REQ_FAILFAST_XX. Otherwise, connection process may takes long time in abnormal scenarios. We can do like this:
+ if (nvme_req(req)->ctrl->acre &&
+ !nvme_is_path_error(nvme_req(req)->status) && + !blk_queue_dying(req->q)) + return RETRY;
+
+ if (blk_noretry_request(req))
+ return COMPLETE;
Quoted 30 lines
drivers/nvme/host/core.c +7
-0+return RETRY ; }@@ -2498,6 +2504,8 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)struct nvme_feat_host_behavior *host;int ret; + ctrl->acre = false; +/* Don't bother enabling the feature if retry delay is not reported */ if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ;@@ -2509,6 +2517,10 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl) host->acre =NVME_ENABLE_ACRE ; ret =nvme_set_features (ctrl,NVME_FEAT_HOST_BEHAVIOR ,0 , host,sizeof (*host),NULL ); + +if (!ret) + ctrl->acre = true; +kfree (host);return ret; }drivers/nvme/host/nvme.h +0
-0
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 276
Quoted 7 lines
drivers/nvme/host/nvme.h +1
-0@@ -270,6 +270,7 @@ struct nvme_ctrl {#ifdef CONFIG_BLK_DEV_ZONED u32 max_zone_append;#endif +bool acre;u16 crdt[3 ];u16 oncs;u16 oacs;
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V2 2/2] nvme: retry commands based on ACRE result
Hello,
Review
Quoted 12 lines
If acre is true and nvme_req(req)->status is nvme path error and
blk_noretry_request(req)is true, we need return complete forREQ_FAILFAST_XX. Otherwise, connection process may takes long time in abnormal scenarios. We can do like this:(quoted diff) +7
-0+ if (nvme_req(req)->ctrl->acre && + !nvme_is_path_error(nvme_req(req)->status) && + !blk_queue_dying(req->q)) + return RETRY; + + if (blk_noretry_request(req)) + return COMPLETE;
Ah.. You're right.. I really appreciate for your review! I will prepare the next patches.
Thank you!
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCH V2 2/2] nvme: retry commands based on ACRE result
In reply to Minwoo Imdrivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 327
Quoted 13 lines · Minwoo Im
drivers/nvme/host/core.c +7
-0@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre) { +if (!nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ; + }else if (blk_noretry_request (req)) +return COMPLETE ; +return RETRY ; }
This is returning RETRY for path related errors when blk_notry_request() is true. That will deadlock initialization on a command timeout. The check should be something like this:
if (nvme_req(req)->ctrl->acre && !nvme_is_path_error(nvme_req(req)->status) && !blk_queue_dying(req->q)) return RETRY; else if (blk_noretry_request(req)) return COMPLETE;
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V2 2/2] nvme: retry commands based on ACRE result
Hello Keith,
Quoted 19 lines · Keith Busch
drivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 327
Quoted 13 lines · Minwoo Im
drivers/nvme/host/core.c +7
-0@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre) { +if (!nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ; + }else if (blk_noretry_request (req)) +return COMPLETE ; +return RETRY ; }This is returning RETRY for path related errors when
blk_notry_request()is true. That will deadlock initialization on a command timeout. The check should be something like this:
Quoted 8 lines
if (nvme_req(req)->ctrl->acre && !nvme_is_path_error(nvme_req(req)->status) && !blk_queue_dying(req->q)) return RETRY; else if (blk_noretry_request(req)) return COMPLETE;
Thank you for chatching this! As Chao pointed it out, I will prepare the next version of this patch. I really appreciate it.
Thanks!
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme