nvme controller reset and namespace scan work race conditions
Subsystems (2)
Sagi Grimberg
maintainerv3 cover nvme controller reset and namespace scan work race conditions
Hey Hannes and Keith,
This is the third attempt to handle the reset and scanning race saga.
The approach is to have the relevant admin commands return a proper status code that reflects that we had a transport error and not remove the namepsace if that is indeed the case.
This should be a reliable way to know if the revalidate_disk failed due to a transport error or not.
I am able to reproduce this race with the following command (using
tcp/rdma):
for j in seq 50; do for i in seq 50; do nvme reset /dev/nvme0; done ; nvme disconnect-all; nvme connect-all; done
With this patch set (plus two more tcp/rdma transport specific patches that address a other issues) I was able to pass the test without reproducing the hang that you hannes reported.
Changes from v2:
- added fc patch from James (can you test please?)
- made nvme_identify_ns return id or PTR_ERR (Hannes)
Changes from v1:
- different approach
James Smart (1):
nvme-fc: Fail transport errors with NVME_SC_HOST_PATH
Sagi Grimberg (6):
nvme: fail cancelled commands with NVME_SC_HOST_PATH_ERROR
nvme: return nvme_error_status for sync commands failure
nvme: make nvme_identify_ns propagate errors back
nvme: make nvme_report_ns_ids propagate error back
nvme-tcp: fail command with NVME_SC_HOST_PATH_ERROR send failed
nvme: don't remove namespace if revalidate failed because of a
transport error
drivers/nvme/host/core.c | 49 +++++++++++++++++++++++++++------------- drivers/nvme/host/fc.c | 37 ++++++++++++++++++++++++------ drivers/nvme/host/tcp.c | 2 +- 3 files changed, 64 insertions(+), 24 deletions(-)
Signature · 1 line
2.17.1
Sagi Grimberg
maintainerPATCH v3 1/7 nvme: fail cancelled commands with NVME_SC_HOST_PATH_ERROR
NVME_SC_ABORT_REQ means that the request was aborted due to an abort command received. In our case, this is a transport cancellation, so host pathing error is much more appropriate.
--- drivers/nvme/host/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +1 -1
@@ -294,7 +294,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)dev_dbg_ratelimited (((struct nvme_ctrl *) data)->device,"Cancelling I/O %d" , req->tag); -nvme_req (req)->status =NVME_SC_ABORT_REQ ; +nvme_req (req)->status =NVME_SC_HOST_PATH_ERROR ;blk_mq_complete_request_sync (req);return true; }
Signature · 1 line
2.17.1
PATCH v3 1/7 nvme: fail cancelled commands with NVME_SC_HOST_PATH_ERROR
Quoted 16 lines · Sagi Grimberg
NVME_SC_ABORT_REQmeans that the request was aborted due to an abort command received. In our case, this is a transport cancellation, so host pathing error is much more appropriate.Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 300
Quoted 8 lines
drivers/nvme/host/core.c +1
-1@@ -294,7 +294,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)dev_dbg_ratelimited (((struct nvme_ctrl *) data)->device,"Cancelling I/O %d" , req->tag); -nvme_req (req)->status =NVME_SC_ABORT_REQ ; +nvme_req (req)->status =NVME_SC_HOST_PATH_ERROR ;blk_mq_complete_request_sync (req);return true; }
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v3 2/7 nvme: return nvme_error_status for sync commands failure
In reply to Sagi Grimbergcallers should not rely on raw nvme status, blk_status_t is more appropriate.
--- drivers/nvme/host/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +2 -2
@@ -803,7 +803,7 @@ int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,if (nvme_req (req)->flags &NVME_REQ_CANCELLED ) ret = -EINTR ;else - ret =nvme_req (req)->status; + ret =nvme_error_status (req); out:blk_mq_free_request (req);return ret;
@@ -894,7 +894,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,if (nvme_req (req)->flags &NVME_REQ_CANCELLED ) ret = -EINTR ;else - ret =nvme_req (req)->status; + ret =nvme_error_status (req);if (result) *result =le32_to_cpu (nvme_req (req)->result.u32);if (meta && !ret && !write) {
Signature · 1 line
2.17.1
PATCH v3 2/7 nvme: return nvme_error_status for sync commands failure
Quoted 13 lines · Sagi Grimberg
callers should not rely on raw nvme status,
blk_status_tis more appropriate.Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd, · line 809
Quoted 8 lines
drivers/nvme/host/core.c +1
-1@@ -803,7 +803,7 @@ int __nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,if (nvme_req (req)->flags &NVME_REQ_CANCELLED ) ret = -EINTR ;else - ret =nvme_req (req)->status; + ret =nvme_error_status (req); out:blk_mq_free_request (req);return ret;
Please update the description for this function, too.
drivers/nvme/host/core.c static int nvme_submit_user_cmd(struct request_queue *q, · line 900
Quoted 10 lines
drivers/nvme/host/core.c +1
-1@@ -894,7 +894,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,if (nvme_req (req)->flags &NVME_REQ_CANCELLED ) ret = -EINTR ;else - ret =nvme_req (req)->status; + ret =nvme_error_status (req);if (result) *result =le32_to_cpu (nvme_req (req)->result.u32);if (meta && !ret && !write) {
But even with this change we'll be leaking blk_status_t error code to things like nvme_ioctl(). I'd rather have it mapped to posix error codes here by calling blk_status_to_errno().
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v3 3/7 nvme: make nvme_identify_ns propagate errors back
In reply to Sagi Grimbergright now callers of nvme_identify_ns only know that it failed, but don't know why. Make nvme_identify_ns propagate the error back.
--- drivers/nvme/host/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/nvme/host/core.c +6 -6
@@ -1108,13 +1108,13 @@ static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl, id =kmalloc (sizeof (*id),GFP_KERNEL );if (!id) -return NULL ; +return ERR_PTR (-ENOMEM ); error =nvme_submit_sync_cmd (ctrl->admin_q, &c, id,sizeof (*id));if (error) {dev_warn (ctrl->device,"Identify namespace failed (%d)\n" , error);kfree (id); -return NULL ; +return ERR_PTR (error); }return id;
@@ -1748,8 +1748,8 @@ static int nvme_revalidate_disk(struct gendisk *disk) } id =nvme_identify_ns (ctrl, ns->head->ns_id); -if (!id) -return -ENODEV ; +if (IS_ERR (id)) +return PTR_ERR (id);if (id->ncap ==0 ) { ret = -ENODEV ;
@@ -3337,8 +3337,8 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)nvme_set_queue_limits (ctrl, ns->queue); id =nvme_identify_ns (ctrl, nsid); -if (!id) { - ret = -EIO ; +if (IS_ERR (id)) { + ret =PTR_ERR (id); goto out_free_queue; }
Signature · 1 line
2.17.1
Keith Busch
maintainerPATCH v3 3/7 nvme: make nvme_identify_ns propagate errors back
Quoted 16 lines · Sagi Grimberg
right now callers of
nvme_identify_nsonly know that it failed, but don't know why. Makenvme_identify_nspropagate the error back.Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)drivers/nvme/host/core.c +0
-0@@ -1108,13 +1108,13 @@ static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl, id =kmalloc (sizeof (*id),GFP_KERNEL );
drivers/nvme/host/core.c static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl, · line 1117
Quoted 10 lines
drivers/nvme/host/core.c +2
-2if (!id) -return NULL ; +return ERR_PTR (-ENOMEM ); error =nvme_submit_sync_cmd (ctrl->admin_q, &c, id,sizeof (*id));if (error) {dev_warn (ctrl->device,"Identify namespace failed (%d)\n" , error);kfree (id); -return NULL ; +return ERR_PTR (error);
The previous patch has nvme_submit_sync_cmd() return a BLK_STS_* value in some cases, but ERR_PTR requires -errno's in order for IS_ERR to detect errors. I think you need:
if (error > 0) return ERR_PTR(blk_status_to_errno(error)); else return ERR_PTR(error);
Sagi Grimberg
maintainerPATCH v3 3/7 nvme: make nvme_identify_ns propagate errors back
Quoted 32 lines
Quoted 17 lines
right now callers of
nvme_identify_nsonly know that it failed, but don't know why. Makenvme_identify_nspropagate the error back.Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)drivers/nvme/host/core.c +0
-0@@ -1108,13 +1108,13 @@ static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl, id =kmalloc (sizeof (*id),GFP_KERNEL );drivers/nvme/host/core.c static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl, · line 1117
Quoted 10 lines
drivers/nvme/host/core.c +2
-2if (!id) -return NULL ; +return ERR_PTR (-ENOMEM ); error =nvme_submit_sync_cmd (ctrl->admin_q, &c, id,sizeof (*id));if (error) {dev_warn (ctrl->device,"Identify namespace failed (%d)\n" , error);kfree (id); -return NULL ; +return ERR_PTR (error);The previous patch has
nvme_submit_sync_cmd()return a BLK_STS_* value in some cases, butERR_PTRrequires -errno's in order forIS_ERRto detect errors. I think you need:
Quoted 5 lines
if (error > 0) return ERR_PTR(blk_status_to_errno(error)); else return ERR_PTR(error);
You are right. sending a respin soon
PATCH v3 3/7 nvme: make nvme_identify_ns propagate errors back
In reply to Sagi GrimbergQuoted 42 lines · Sagi Grimberg
right now callers of
nvme_identify_nsonly know that it failed, but don't know why. Makenvme_identify_nspropagate the error back.Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)drivers/nvme/host/core.c +4
-4@@ -1108,13 +1108,13 @@ static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl, id =kmalloc (sizeof (*id),GFP_KERNEL );if (!id) -return NULL ; +return ERR_PTR (-ENOMEM ); error =nvme_submit_sync_cmd (ctrl->admin_q, &c, id,sizeof (*id));if (error) {dev_warn (ctrl->device,"Identify namespace failed (%d)\n" , error);kfree (id); -return NULL ; +return ERR_PTR (error); }return id;@@ -1748,8 +1748,8 @@ static int nvme_revalidate_disk(struct gendisk *disk) } id =nvme_identify_ns (ctrl, ns->head->ns_id); -if (!id) -return -ENODEV ; +if (IS_ERR (id)) +return PTR_ERR (id);if (id->ncap ==0 ) { ret = -ENODEV ;
drivers/nvme/host/core.c static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3344
Quoted 10 lines
drivers/nvme/host/core.c +2
-2@@ -3337,8 +3337,8 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)nvme_set_queue_limits (ctrl, ns->queue); id =nvme_identify_ns (ctrl, nsid); -if (!id) { - ret = -EIO ; +if (IS_ERR (id)) { + ret =PTR_ERR (id); goto out_free_queue; }
See my comments to the previous patch. This will only work if we return POSIX error codes only; it'll probably fall flat on its face when returning blk_status_t codes.
Once we have converted nvme_submit_sync_cmd() to return POSIX error codes this is fine.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v3 4/7 nvme: make nvme_report_ns_ids propagate error back
In reply to Sagi GrimbergAnd make the callers check the return status and propagate back accordingly.
--- drivers/nvme/host/core.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +17 -5
@@ -1603,9 +1603,11 @@ static void nvme_config_write_zeroes(struct gendisk *disk, struct nvme_ns *ns)blk_queue_max_write_zeroes_sectors (disk->queue, max_sectors); } -static void nvme_report_ns_ids (struct nvme_ctrl *ctrl,unsigned int nsid, +static int nvme_report_ns_ids (struct nvme_ctrl *ctrl,unsigned int nsid,struct nvme_id_ns *id,struct nvme_ns_ids *ids) { +int ret =0 ; +memset (ids,0 ,sizeof (*ids));if (ctrl->vs >=NVME_VS (1 ,1 ,0 ))
@@ -1616,10 +1618,12 @@ static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,/* Don't treat error as fatal we potentially * already have a NGUID or EUI-64 */ -if (nvme_identify_ns_descs (ctrl, nsid, ids)) + ret =nvme_identify_ns_descs (ctrl, nsid, ids); +if (ret)dev_warn (ctrl->device,"%s: Identify Descriptors failed\n" , __func__); } +return ret; }static bool nvme_ns_ids_valid (struct nvme_ns_ids *ids)
@@ -1757,7 +1761,10 @@ static int nvme_revalidate_disk(struct gendisk *disk) }__nvme_revalidate_disk (disk, id); -nvme_report_ns_ids (ctrl, ns->head->ns_id, id, &ids); + ret =nvme_report_ns_ids (ctrl, ns->head->ns_id, id, &ids); +if (ret) + goto out; +if (!nvme_ns_ids_equal (&ns->head->ids, &ids)) {dev_err (ctrl->device,"identifiers changed for nsid %d\n" , ns->head->ns_id);
@@ -3182,7 +3189,9 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl, head->ns_id = nsid;kref_init (&head->ref); -nvme_report_ns_ids (ctrl, nsid, id, &head->ids); + ret =nvme_report_ns_ids (ctrl, nsid, id, &head->ids); +if (ret) + goto out_cleanup_srcu; ret =__nvme_check_ids (ctrl->subsys, head);if (ret) {
@@ -3230,7 +3239,10 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid, }else {struct nvme_ns_ids ids; -nvme_report_ns_ids (ctrl, nsid, id, &ids); + ret =nvme_report_ns_ids (ctrl, nsid, id, &ids); +if (ret) + goto out_unlock; +if (!nvme_ns_ids_equal (&head->ids, &ids)) { dev_err(ctrl->device, "IDs don't match for shared namespace %d\n",
Signature · 1 line
2.17.1
PATCH v3 4/7 nvme: make nvme_report_ns_ids propagate error back
Quoted 65 lines · Sagi Grimberg
And make the callers check the return status and propagate back accordingly.
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)drivers/nvme/host/core.c +13
-4@@ -1603,9 +1603,11 @@ static void nvme_config_write_zeroes(struct gendisk *disk, struct nvme_ns *ns)blk_queue_max_write_zeroes_sectors (disk->queue, max_sectors); } -static void nvme_report_ns_ids (struct nvme_ctrl *ctrl,unsigned int nsid, +static int nvme_report_ns_ids (struct nvme_ctrl *ctrl,unsigned int nsid,struct nvme_id_ns *id,struct nvme_ns_ids *ids) { +int ret =0 ; +memset (ids,0 ,sizeof (*ids));if (ctrl->vs >=NVME_VS (1 ,1 ,0 ))@@ -1616,10 +1618,12 @@ static void nvme_report_ns_ids(struct nvme_ctrl *ctrl, unsigned int nsid,/* Don't treat error as fatal we potentially * already have a NGUID or EUI-64 */ -if (nvme_identify_ns_descs (ctrl, nsid, ids)) + ret =nvme_identify_ns_descs (ctrl, nsid, ids); +if (ret)dev_warn (ctrl->device,"%s: Identify Descriptors failed\n" , __func__); } +return ret; }static bool nvme_ns_ids_valid (struct nvme_ns_ids *ids)@@ -1757,7 +1761,10 @@ static int nvme_revalidate_disk(struct gendisk *disk) }__nvme_revalidate_disk (disk, id); -nvme_report_ns_ids (ctrl, ns->head->ns_id, id, &ids); + ret =nvme_report_ns_ids (ctrl, ns->head->ns_id, id, &ids); +if (ret) + goto out; +if (!nvme_ns_ids_equal (&ns->head->ids, &ids)) {dev_err (ctrl->device,"identifiers changed for nsid %d\n" , ns->head->ns_id);@@ -3182,7 +3189,9 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl, head->ns_id = nsid;kref_init (&head->ref); -nvme_report_ns_ids (ctrl, nsid, id, &head->ids); + ret =nvme_report_ns_ids (ctrl, nsid, id, &head->ids); +if (ret) + goto out_cleanup_srcu; ret =__nvme_check_ids (ctrl->subsys, head);if (ret) {@@ -3230,7 +3239,10 @@ static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid, }else {
drivers/nvme/host/core.c static int nvme_init_ns_head(struct nvme_ns *ns, unsigned nsid, · line 3248
Quoted 10 lines
drivers/nvme/host/core.c +4
-1struct nvme_ns_ids ids; -nvme_report_ns_ids (ctrl, nsid, id, &ids); + ret =nvme_report_ns_ids (ctrl, nsid, id, &ids); +if (ret) + goto out_unlock; +if (!nvme_ns_ids_equal (&head->ids, &ids)) { dev_err(ctrl->device, "IDs don't match for shared namespace %d\n",
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v3 5/7 nvme-tcp: fail command with NVME_SC_HOST_PATH_ERROR send failed
In reply to Sagi GrimbergThis is a more appropriate error status for a transport error detected by us (the host).
--- drivers/nvme/host/tcp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/tcp.c +1 -1
@@ -842,7 +842,7 @@ static inline void nvme_tcp_done_send_req(struct nvme_tcp_queue *queue)static void nvme_tcp_fail_request (struct nvme_tcp_request *req) { -nvme_tcp_end_request (blk_mq_rq_from_pdu (req),NVME_SC_DATA_XFER_ERROR ); +nvme_tcp_end_request (blk_mq_rq_from_pdu (req),NVME_SC_HOST_PATH_ERROR ); }static int nvme_tcp_try_send_data (struct nvme_tcp_request *req)
Signature · 1 line
2.17.1
PATCH v3 5/7 nvme-tcp: fail command with NVME_SC_HOST_PATH_ERROR send failed
Quoted 14 lines · Sagi Grimberg
This is a more appropriate error status for a transport error detected by us (the host).
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/tcp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static inline void nvme_tcp_done_send_req(struct nvme_tcp_queue *queue) · line 848
Quoted 8 lines
drivers/nvme/host/tcp.c +1
-1@@ -842,7 +842,7 @@ static inline void nvme_tcp_done_send_req(struct nvme_tcp_queue *queue)static void nvme_tcp_fail_request (struct nvme_tcp_request *req) { -nvme_tcp_end_request (blk_mq_rq_from_pdu (req),NVME_SC_DATA_XFER_ERROR ); +nvme_tcp_end_request (blk_mq_rq_from_pdu (req),NVME_SC_HOST_PATH_ERROR ); }static int nvme_tcp_try_send_data (struct nvme_tcp_request *req)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v3 6/7 nvme-fc: Fail transport errors with NVME_SC_HOST_PATH
In reply to Sagi GrimbergFrom: James Smart <[email protected]>
NVME_SC_INTERNAL should indicate an internal controller errors and not host transport errors. These errors will propagate to upper layers (essentially nvme core) and be interpereted as transport errors which should not be taken into account for namespace state or condition.
--- drivers/nvme/host/fc.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-)
drivers/nvme/host/fc.c +30 -7
@@ -1608,9 +1608,13 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) sizeof(op->rsp_iu), DMA_FROM_DEVICE);if (opstate ==FCPOP_STATE_ABORTED ) - status =cpu_to_le16 (NVME_SC_ABORT_REQ <<1 ); -else if (freq->status) - status =cpu_to_le16 (NVME_SC_INTERNAL <<1 ); + status =cpu_to_le16 (NVME_SC_HOST_PATH_ERROR <<1 ); +else if (freq->status) { + status =cpu_to_le16 (NVME_SC_HOST_PATH_ERROR <<1 ); +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: io failed due to lldd error %d\n" , + ctrl->cnum, freq->status); + }/* * For the linux implementation, if we have an unsuccesful
@@ -1637,8 +1641,13 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)* no payload in the CQE by the transport. */ if (freq->transferred_length != -be32_to_cpu (op->cmd_iu.data_len)) { - status =cpu_to_le16 (NVME_SC_INTERNAL <<1 ); +be32_to_cpu (op->cmd_iu.data_len)) { + status =cpu_to_le16 (NVME_SC_HOST_PATH_ERROR <<1 ); +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: io failed due to bad transfer " +"length: %d vs expected %d\n" , + ctrl->cnum, freq->transferred_length, +be32_to_cpu (op->cmd_iu.data_len)); goto done; } result.u64 =0 ;
@@ -1655,7 +1664,17 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) freq->transferred_length || op->rsp_iu.status_code || sqe->common.command_id != cqe->command_id)) { - status =cpu_to_le16 (NVME_SC_INTERNAL <<1 ); + status =cpu_to_le16 (NVME_SC_HOST_PATH_ERROR <<1 ); +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: io failed due to bad NVMe_ERSP: " +"iu len %d, xfr len %d vs %d, status code " +"%d, cmdid %d vs %d\n" , + ctrl->cnum,be16_to_cpu (op->rsp_iu.iu_len), +be32_to_cpu (op->rsp_iu.xfrd_len), + freq->transferred_length, + op->rsp_iu.status_code, + sqe->common.command_id, + cqe->command_id); goto done; } result = cqe->result;
@@ -1663,7 +1682,11 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)break ;default : - status =cpu_to_le16 (NVME_SC_INTERNAL <<1 ); + status =cpu_to_le16 (NVME_SC_HOST_PATH_ERROR <<1 ); +dev_info (ctrl->ctrl.device, +"NVME-FC{%d}: io failed due to odd NVMe_xRSP iu " +"len %d\n" , + ctrl->cnum, freq->rcv_rsplen); goto done; }
Signature · 1 line
2.17.1
PATCH v3 6/7 nvme-fc: Fail transport errors with NVME_SC_HOST_PATH
Quoted 8 lines · Sagi Grimberg
From: James Smart <james.smart at broadcom.com>
NVME_SC_INTERNALshould indicate an internal controller errors and not host transport errors. These errors will propagate to upper layers (essentially nvme core) and be interpereted as transport errors which should not be taken into account for namespace state or condition.
Quoted 5 lines
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/fc.c | 37 ++++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v3 7/7 nvme: don't remove namespace if revalidate failed because of a transport error
In reply to Sagi GrimbergIf a controller reset is racing with a namespace revalidation, the revalidation (admin) I/O will surely fail, but we should not remove the namespace as we will execute the I/O when the controller is back up.
Fix this by checking the specific error code that revalidate_disk returns, and if it is a transport related error, do not remove the namespace as it will either recover when the controller is back up and schedule a subsequent scan, or the controller is going away and the namespaces will be removed anyways.
This fixes a hang namespace scanning racing with a controller reset and also sporious I/O errors in path failover coditions where the controller reset is racing with the namespace scan work with multipath enabled.
--- drivers/nvme/host/core.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +7 -2
@@ -228,6 +228,8 @@ static blk_status_t nvme_error_status(struct request *req)return BLK_STS_PROTECTION ;case NVME_SC_RESERVATION_CONFLICT :return BLK_STS_NEXUS ; +case NVME_SC_HOST_PATH_ERROR : +return BLK_STS_TRANSPORT ;default :return BLK_STS_IOERR ; }
@@ -3452,8 +3454,11 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid) ns =nvme_find_get_ns (ctrl, nsid);if (ns) { -if (ns->disk &&revalidate_disk (ns->disk)) -nvme_ns_remove (ns); +if (ns->disk) { +blk_status_t sts =revalidate_disk (ns->disk); +if (sts && sts !=BLK_STS_TRANSPORT ) +nvme_ns_remove (ns); + }nvme_put_ns (ns); }else nvme_alloc_ns (ctrl, nsid);
Signature · 1 line
2.17.1
PATCH v3 7/7 nvme: don't remove namespace if revalidate failed because of a transport error
Quoted 39 lines · Sagi Grimberg
If a controller reset is racing with a namespace revalidation, the revalidation (admin) I/O will surely fail, but we should not remove the namespace as we will execute the I/O when the controller is back up.
Fix this by checking the specific error code that
revalidate_diskreturns, and if it is a transport related error, do not remove the namespace as it will either recover when the controller is back up and schedule a subsequent scan, or the controller is going away and the namespaces will be removed anyways.This fixes a hang namespace scanning racing with a controller reset and also sporious I/O errors in path failover coditions where the controller reset is racing with the namespace scan work with multipath enabled.
Reported-by Hannes Reinecke <hare at suse.de>Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/nvme/host/core.c +2
-0@@ -228,6 +228,8 @@ static blk_status_t nvme_error_status(struct request *req)return BLK_STS_PROTECTION ;case NVME_SC_RESERVATION_CONFLICT :return BLK_STS_NEXUS ; +case NVME_SC_HOST_PATH_ERROR : +return BLK_STS_TRANSPORT ;default :return BLK_STS_IOERR ; }@@ -3452,8 +3454,11 @@ static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid) ns =nvme_find_get_ns (ctrl, nsid);if (ns) {
drivers/nvme/host/core.c static void nvme_validate_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3464
Quoted 10 lines
drivers/nvme/host/core.c +5
-2-if (ns->disk &&revalidate_disk (ns->disk)) -nvme_ns_remove (ns); +if (ns->disk) { +blk_status_t sts =revalidate_disk (ns->disk); +if (sts && sts !=BLK_STS_TRANSPORT ) +nvme_ns_remove (ns); + }nvme_put_ns (ns); }else nvme_alloc_ns (ctrl, nsid);
This is a neat solution, yet I'm not convinced that it'll solve the problem entirely. I'll give it a spin.
But in any case, it's definitely a step in the right direction. So:
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v3 7/7 nvme: don't remove namespace if revalidate failed because of a transport error
Quoted 3 lines
This is a neat solution, yet I'm not convinced that it'll solve the problem entirely. I'll give it a spin.
Thanks Hannes,
Note that with tcp/rdma, there is another patch needed to pass this test.
See:
[PATCH rfc] nvme: make all fabrics command run on a separate request queue
The reason is, that because we have admin commands making their way to the target between we disable the controller and (re)enable it as part of the reset process. Because of that, we have the target failing some of the scan commands that are wrongly made it through (which cause the scan work to wrongly remove the ns and hang).
So I would recommend running it with that patch as well (and also get your feedback on it ;))
v3 cover nvme controller reset and namespace scan work race conditions
In reply to Sagi GrimbergQuoted 1 line · Sagi Grimberg
for j in
seq 50; do for i inseq 50; do nvme reset/dev/nvme0; done ; nvme disconnect-all; nvme connect-all; done
Has anyone already volunteered to convert this into a patch for the blktests project?
Thanks,
Bart.
Sagi Grimberg
maintainerv3 cover nvme controller reset and namespace scan work race conditions
Quoted 6 lines
Quoted 3 lines
for j in
seq 50; do for i inseq 50; do nvme reset/dev/nvme0; done ; nvme disconnect-all; nvme connect-all; doneHas anyone already volunteered to convert this into a patch for the blktests project?
I'll see if it can reliably reproduce it for loop. Or perhaps I can run this with tcp on 127.0.0.1?
v3 cover nvme controller reset and namespace scan work race conditions
Quoted 9 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines
for j in
seq 50; do for i inseq 50; do nvme reset/dev/nvme0; done ; nvme disconnect-all; nvme connect-all; doneHas anyone already volunteered to convert this into a patch for the blktests project?
I'll see if it can reliably reproduce it for loop. Or perhaps I can run this with tcp on 127.0.0.1?
There are more tests in blktests that use loopback over a local network interface, e.g. the SRP tests. You may want to have a look at get_ipv4_addr(), get_ipv6_addr() and other functions in common/multipath-over-rdma.
Bart.