nvme: fixes for command retry
v3 cover nvme: fixes for command retry
Hello,
Here's V3 patch series for Advanced Command Retery Enable (ACRE). This series enables command retry feature based on a flag introduced to nvme controller instance in this series.
This patchset placed ACRE flag check first than REQ_FAILFAST_* cmd_flags checking. It makes Set Feature for Host Bevior in reset_work more meaningful.
Please review.
Thanks!
Since V2:
- Update commit description more informative. (Sagi)
- Split [2/2] patch into two patches. (Sagi)
- Fix condition to return COMPLETE in [2/2]. (Chao and Keith)
Since V1:
- The first patch has been applied (Christoph). Removed from this series.
- Update the last patch to take acre flag from the controller instance instead of module parameter by updating nvme_decide_disposition(). (Chao)
Minwoo Im (3):
nvme: check all retry delay times in Identify Controller
nvme: introduce acre flag in nvme_ctrl
nvme: retry commands based on ACRE flag
drivers/nvme/host/core.c | 18 +++++++++++++++--- drivers/nvme/host/nvme.h | 1 + 2 files changed, 16 insertions(+), 3 deletions(-)
Signature · 7 lines
2.17.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v3 1/3 nvme: check all retry delay times in Identify Controller
If none of Command Retry Delay Time(CRDT)s are set, then we can simply skip to configure Advanced Command Retry Enable(ACRE).
CRDT selection will be made in Completion Queue Entry that controller has responsibility to decide.
This patch checks all CRDT[0, 1, and 2] in Identify Controller data structrure instead of checking the first one: CRDT[0] for cases that:
- CRDT[0] == 0, but CRDT[1 or 2] != 0
--- drivers/nvme/host/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +1 -1
@@ -2499,7 +2499,7 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)int ret;/* Don't bother enabling the feature if retry delay is not reported */ -if (!ctrl->crdt[0 ]) +if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ; host =kzalloc (sizeof (*host),GFP_KERNEL );
Signature · 7 lines
2.17.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V3 1/3] nvme: check all retry delay times in Identify Controller
Quoted 32 lines · Minwoo Im
If none of Command Retry Delay Time(CRDT)s are set, then we can simply skip to configure Advanced Command Retry Enable(ACRE).
CRDT selection will be made in Completion Queue Entry that controller has responsibility to decide.
This patch checks all CRDT[0, 1, and 2] in Identify Controller data structrure instead of checking the first one: CRDT[0] for cases that:
- CRDT[0] == 0, but CRDT[1 or 2] != 0
Signed-off-by Minwoo Im <[email protected]>--- drivers/nvme/host/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/core.c +1
-1@@ -2499,7 +2499,7 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)int ret;/* Don't bother enabling the feature if retry delay is not reported */ -if (!ctrl->crdt[0 ]) +if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ; host =kzalloc (sizeof (*host),GFP_KERNEL );Signature · 1 line
2.17.1
Hello Christoph, and Keith,
May I get your feedback on this? Is there any background reason not to configure ACRE without checking CRDT2[1] and CRDT3[2] ?
Thanks!
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v3 2/3 nvme: introduce acre flag in nvme_ctrl
In reply to Minwoo ImAdvanced Command Retry Enable (ACRE) flag is added in nvme_ctrl instance to indicate that Set Features for Host Behavior Support with ACRE enabled is whether successfully done or not during reset_work.
This flag will be used to decide to retry commands that fail or not in the followed patch.
--- drivers/nvme/host/core.c | 6 ++++++ drivers/nvme/host/nvme.h | 1 + 2 files changed, 7 insertions(+)
drivers/nvme/host/core.c +6 -0
@@ -2498,6 +2498,8 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)struct nvme_feat_host_behavior *host;int ret; + ctrl->acre = false; +/* Don't bother enabling the feature if retry delay is not reported */ if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ;
@@ -2509,6 +2511,10 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl) host->acre =NVME_ENABLE_ACRE ; ret =nvme_set_features (ctrl,NVME_FEAT_HOST_BEHAVIOR ,0 , host,sizeof (*host),NULL ); + +if (!ret) + ctrl->acre = true; +kfree (host);return ret; }
drivers/nvme/host/nvme.h +1 -0
@@ -270,6 +270,7 @@ struct nvme_ctrl {#ifdef CONFIG_BLK_DEV_ZONED u32 max_zone_append;#endif +bool acre;u16 crdt[3 ];u16 oncs;u16 oacs;
Signature · 7 lines
2.17.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V3 2/3] nvme: introduce acre flag in nvme_ctrl
Quoted 43 lines · Minwoo Im
Advanced Command Retry Enable (ACRE) flag is added in
nvme_ctrlinstance to indicate that Set Features for Host Behavior Support with ACRE enabled is whether successfully done or not duringreset_work.This flag will be used to decide to retry commands that fail or not in the followed patch.
Signed-off-by Minwoo Im <[email protected]>--- drivers/nvme/host/core.c | 6 ++++++ drivers/nvme/host/nvme.h | 1 + 2 files changed, 7 insertions(+)drivers/nvme/host/core.c +6
-0@@ -2498,6 +2498,8 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl)struct nvme_feat_host_behavior *host;int ret; + ctrl->acre = false; +/* Don't bother enabling the feature if retry delay is not reported */ if (!ctrl->crdt[0 ] && !ctrl->crdt[1 ] && !ctrl->crdt[2 ])return 0 ;@@ -2509,6 +2511,10 @@ static int nvme_configure_acre(struct nvme_ctrl *ctrl) host->acre =NVME_ENABLE_ACRE ; ret =nvme_set_features (ctrl,NVME_FEAT_HOST_BEHAVIOR ,0 , host,sizeof (*host),NULL ); + +if (!ret) + ctrl->acre = true; +kfree (host);return ret; }drivers/nvme/host/nvme.h +0
-0
drivers/nvme/host/nvme.h struct nvme_ctrl { · line 276
Quoted 7 lines
drivers/nvme/host/nvme.h +1
-0@@ -270,6 +270,7 @@ struct nvme_ctrl {#ifdef CONFIG_BLK_DEV_ZONED u32 max_zone_append;#endif +bool acre;u16 crdt[3 ];u16 oncs;u16 oacs;
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v3 3/3 nvme: retry commands based on ACRE flag
In reply to Minwoo ImAllow nvme_decide_disposition() to retry commands with error by checking Advanced Command Retry Enable (ACRE) flag of controller first than REQ_FAILFAST_* cmd_flags. But, we just can't allow all command failures to be retried due to reasons like connecting may take too long due to the retries. This patch only allows non-host path error commands to be retried.
--- drivers/nvme/host/core.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +8 -2
@@ -303,8 +303,7 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)if (likely (nvme_req (req)->status ==0 ))return COMPLETE ; -if (blk_noretry_request (req) || - (nvme_req (req)->status &NVME_SC_DNR ) || +if ((nvme_req (req)->status &NVME_SC_DNR ) ||nvme_req (req)->retries >= nvme_max_retries)return COMPLETE ;
@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre && + !nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ; +else if (blk_noretry_request (req)) +return COMPLETE ; +return RETRY ; }
Signature · 7 lines
2.17.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V3 3/3] nvme: retry commands based on ACRE flag
Quoted 34 lines · Minwoo Im
Allow
nvme_decide_disposition()to retry commands with error by checking Advanced Command Retry Enable (ACRE) flag of controller first than REQ_FAILFAST_*cmd_flags. But, we just can't allow all command failures to be retried due to reasons like connecting may take too long due to the retries. This patch only allows non-host path error commands to be retried.Cc Chao Leng <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Minwoo Im <[email protected]>--- drivers/nvme/host/core.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/nvme/host/core.c +1
-2@@ -303,8 +303,7 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)if (likely (nvme_req (req)->status ==0 ))return COMPLETE ; -if (blk_noretry_request (req) || - (nvme_req (req)->status &NVME_SC_DNR ) || +if ((nvme_req (req)->status &NVME_SC_DNR ) ||nvme_req (req)->retries >= nvme_max_retries)return COMPLETE ;@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; }
drivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 328
Quoted 10 lines
drivers/nvme/host/core.c +7
-0+if (nvme_req (req)->ctrl->acre && + !nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ; +else if (blk_noretry_request (req)) +return COMPLETE ; +return RETRY ; }
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCH V3 3/3] nvme: retry commands based on ACRE flag
In reply to Minwoo Imdrivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 322
Quoted 8 lines · Minwoo Im
drivers/nvme/host/core.c +4
-0@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre && + !nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ;
Is there really any reason to tie this to acre?
drivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 324
Quoted 2 lines
drivers/nvme/host/core.c +2
-0+else if (blk_noretry_request (req)) +return COMPLETE ;
I am not sure we should ignore the FAILFAST for non-path errors. If we need retryable admin commands, we should let the driver provide a way for callers to dispatch requests without that flag.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V3 3/3] nvme: retry commands based on ACRE flag
Quoted 12 lines · Keith Busch
drivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 322
Quoted 8 lines · Minwoo Im
drivers/nvme/host/core.c +4
-0@@ -317,6 +316,13 @@ static inline enum nvme_disposition nvme_decide_disposition(struct request *req)return COMPLETE ; } +if (nvme_req (req)->ctrl->acre && + !nvme_is_path_error (nvme_req (req)->status) && + !blk_queue_dying (req->q)) +return RETRY ;Is there really any reason to tie this to acre?
Quoted 6 lines
drivers/nvme/host/core.c static inline enum nvme_disposition nvme_decide_disposition(struct request *req) · line 324
Quoted 2 lines
drivers/nvme/host/core.c +2
-0+else if (blk_noretry_request (req)) +return COMPLETE ;I am not sure we should ignore the FAILFAST for non-path errors. If we need retryable admin commands, we should let the driver provide a way for callers to dispatch requests without that flag.
Understood. I thought the opposite way about FAILFAST in case with acre, if device is enabled with acre, all commands would be retried regardless to FAILFAST... Thanks for pointing that out!
How do you think which one is right choice to go with if a user-space application(e.g., nvme-cli) wants a command to be retired in case of ACRE && Error && !DNR:
- User-space application should figure out !DNR and retry the command. (Maybe we are not able to easily figure out exact status code from the user-space application by the return value).
- Driver should retry the command right before putting result up to the user-space even it's a FAILFAST request.
Thanks,
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerRe: [PATCH V3 3/3] nvme: retry commands based on ACRE flag
Quoted 12 lines · Minwoo Im
Quoted 3 lines
I am not sure we should ignore the FAILFAST for non-path errors. If we need retryable admin commands, we should let the driver provide a way for callers to dispatch requests without that flag.
Understood. I thought the opposite way about FAILFAST in case with acre, if device is enabled with acre, all commands would be retried regardless to FAILFAST... Thanks for pointing that out!
How do you think which one is right choice to go with if a user-space
application(e.g., nvme-cli)wants a command to be retired in case of ACRE && Error && !DNR:
Quoted 8 lines
- User-space application should figure out !DNR and retry the command. (Maybe we are not able to easily figure out exact status code from the user-space application by the return value).
- Driver should retry the command right before putting result up to the user-space even it's a FAILFAST request.
Thanks,
We could come up with a version of the ioctls that use the normal retry mechanisms. nvme_passthru_cmd64 has two reserved fields we could use for UAPI flags like this.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V3 3/3] nvme: retry commands based on ACRE flag
Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Minwoo Im
Quoted 3 lines
I am not sure we should ignore the FAILFAST for non-path errors. If we need retryable admin commands, we should let the driver provide a way for callers to dispatch requests without that flag.
Understood. I thought the opposite way about FAILFAST in case with acre, if device is enabled with acre, all commands would be retried regardless to FAILFAST... Thanks for pointing that out!
How do you think which one is right choice to go with if a user-space
application(e.g., nvme-cli)wants a command to be retired in case of ACRE && Error && !DNR:- User-space application should figure out !DNR and retry the command. (Maybe we are not able to easily figure out exact status code from the user-space application by the return value).
Quoted 8 lines
Quoted 4 lines
- Driver should retry the command right before putting result up to the user-space even it's a FAILFAST request.
Thanks,
We could come up with a version of the ioctls that use the normal retry mechanisms.
nvme_passthru_cmd64has two reserved fields we could use for UAPI flags like this.
Thank you for your feedback on this!
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerRe: [PATCH V3 3/3] nvme: retry commands based on ACRE flag
In reply to Christoph HellwigQuoted 17 lines · Christoph Hellwig
Quoted 16 lines · Minwoo Im
Quoted 3 lines
I am not sure we should ignore the FAILFAST for non-path errors. If we need retryable admin commands, we should let the driver provide a way for callers to dispatch requests without that flag.
Understood. I thought the opposite way about FAILFAST in case with acre, if device is enabled with acre, all commands would be retried regardless to FAILFAST... Thanks for pointing that out!
How do you think which one is right choice to go with if a user-space
application(e.g., nvme-cli)wants a command to be retired in case of ACRE && Error && !DNR:- User-space application should figure out !DNR and retry the command. (Maybe we are not able to easily figure out exact status code from the user-space application by the return value).
Quoted 8 lines
Quoted 4 lines
- Driver should retry the command right before putting result up to the user-space even it's a FAILFAST request.
Thanks,
We could come up with a version of the ioctls that use the normal retry mechanisms.
nvme_passthru_cmd64has two reserved fields we could use for UAPI flags like this.
Yeah, I'm totally okay with adding driver control flags for this sort of behavior.
For the moment, the ioctl interface has always been 1:1 for commands submitted to completions received, so all user space applications using this should have been prepared to receive unfiltered status back and handle retries on their own.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH V3 3/3] nvme: retry commands based on ACRE flag
Quoted 27 lines · Keith Busch
Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Minwoo Im
Quoted 3 lines
I am not sure we should ignore the FAILFAST for non-path errors. If we need retryable admin commands, we should let the driver provide a way for callers to dispatch requests without that flag.
Understood. I thought the opposite way about FAILFAST in case with acre, if device is enabled with acre, all commands would be retried regardless to FAILFAST... Thanks for pointing that out!
How do you think which one is right choice to go with if a user-space
application(e.g., nvme-cli)wants a command to be retired in case of ACRE && Error && !DNR:- User-space application should figure out !DNR and retry the command. (Maybe we are not able to easily figure out exact status code from the user-space application by the return value).
Quoted 8 lines
Quoted 4 lines
- Driver should retry the command right before putting result up to the user-space even it's a FAILFAST request.
Thanks,
We could come up with a version of the ioctls that use the normal retry mechanisms.
nvme_passthru_cmd64has two reserved fields we could use for UAPI flags like this.
Quoted 7 lines
Yeah, I'm totally okay with adding driver control flags for this sort of behavior.
For the moment, the ioctl interface has always been 1:1 for commands submitted to completions received, so all user space applications using this should have been prepared to receive unfiltered status back and handle retries on their own.
Thanks Keith for your feedback!
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme