scsi: core: fix race between handling STS_RESOURCE and completion
PATCH 1/1 scsi: core: fix race between handling STS_RESOURCE and completion
When queuing IO request to LLD, STS_RESOURCE may be returned because:
- host in recovery or blocked
- target queue throttling or blocked
- LLD rejection
Any one of the above doesn't happen frequently enough.
BLK_STS_DEV_RESOURCE is returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', so BLK_STS_DEV_RESOURCE is returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for either scsi_run_queue_async() or blk-mq's RESTART.
Fix the issue by not returning BLK_STS_DEV_RESOURCE in this situation.
--- drivers/scsi/scsi_lib.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +1 -2
@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,break ;case BLK_STS_RESOURCE :case BLK_STS_ZONE_RESOURCE : -if (atomic_read (&sdev->device_busy) || -scsi_device_blocked (sdev)) +if (scsi_device_blocked (sdev)) ret =BLK_STS_DEV_RESOURCE ;break ;default :
Signature · 1 line
2.28.0
Quoted 36 lines · Ming Lei
When queuing IO request to LLD,
STS_RESOURCEmay be returned because:- host in recovery or blocked
- target queue throttling or blocked
- LLD rejectionAny one of the above doesn't happen frequently enough.
BLK_STS_DEV_RESOURCEis returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', soBLK_STS_DEV_RESOURCEis returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for eitherscsi_run_queue_async()or blk-mq's RESTART.Fix the issue by not returning
BLK_STS_DEV_RESOURCEin this situation.Cc Hannes Reinecke <[email protected]>Cc Sumit Saxena <[email protected]>Cc Kashyap Desai <[email protected]>Cc Bart Van Assche <[email protected]>Cc Ewan Milne <[email protected]>Cc Long Li <[email protected]>Tested-by "chenxiang (M)" <[email protected]>Reported-by John Garry <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/scsi/scsi_lib.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)drivers/scsi/scsi_lib.c +0
-0
drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709
Quoted 9 lines
drivers/scsi/scsi_lib.c +1
-2@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,break ;case BLK_STS_RESOURCE :case BLK_STS_ZONE_RESOURCE : -if (atomic_read (&sdev->device_busy) || -scsi_device_blocked (sdev)) +if (scsi_device_blocked (sdev)) ret =BLK_STS_DEV_RESOURCE ;break ;default :
Since this patch modifies code introduced in commit 86ff7c2a80cd ("blk-mq: introduce BLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes: tag?
Thanks,
Bart.
Quoted 42 lines · Bart Van Assche
Quoted 41 lines · Ming Lei
When queuing IO request to LLD,
STS_RESOURCEmay be returned because:- host in recovery or blocked
- target queue throttling or blocked
- LLD rejectionAny one of the above doesn't happen frequently enough.
BLK_STS_DEV_RESOURCEis returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', soBLK_STS_DEV_RESOURCEis returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for eitherscsi_run_queue_async()or blk-mq's RESTART.Fix the issue by not returning
BLK_STS_DEV_RESOURCEin this situation.Cc Hannes Reinecke <[email protected]>Cc Sumit Saxena <[email protected]>Cc Kashyap Desai <[email protected]>Cc Bart Van Assche <[email protected]>Cc Ewan Milne <[email protected]>Cc Long Li <[email protected]>Tested-by "chenxiang (M)" <[email protected]>Reported-by John Garry <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/scsi/scsi_lib.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)drivers/scsi/scsi_lib.c +0
-2@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,break ;case BLK_STS_RESOURCE :case BLK_STS_ZONE_RESOURCE : -if (atomic_read (&sdev->device_busy) || -scsi_device_blocked (sdev))
Quoted 8 lines
drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709
Quoted 4 lines
drivers/scsi/scsi_lib.c +1
-0+if (scsi_device_blocked (sdev)) ret =BLK_STS_DEV_RESOURCE ;break ;default :Since this patch modifies code introduced in commit
86ff7c2a80cd("blk-mq:
introduceBLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes:
tag?
This same race exists before commit 86ff7c2a80cd, so I think the 'Fixes:' tag is misleading.
Thanks,
Ming
Quoted 50 lines · Ming Lei
Quoted 41 lines · Bart Van Assche
Quoted 40 lines · Ming Lei
When queuing IO request to LLD,
STS_RESOURCEmay be returned because:- host in recovery or blocked
- target queue throttling or blocked
- LLD rejectionAny one of the above doesn't happen frequently enough.
BLK_STS_DEV_RESOURCEis returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', soBLK_STS_DEV_RESOURCEis returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for eitherscsi_run_queue_async()or blk-mq's RESTART.Fix the issue by not returning
BLK_STS_DEV_RESOURCEin this situation.Cc Hannes Reinecke <[email protected]>Cc Sumit Saxena <[email protected]>Cc Kashyap Desai <[email protected]>Cc Bart Van Assche <[email protected]>Cc Ewan Milne <[email protected]>Cc Long Li <[email protected]>Tested-by "chenxiang (M)" <[email protected]>Reported-by John Garry <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/scsi/scsi_lib.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)drivers/scsi/scsi_lib.c +0
-1@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,break ;case BLK_STS_RESOURCE :case BLK_STS_ZONE_RESOURCE : - if (atomic_read(&sdev->device_busy) ||Quoted 8 lines
drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709
Quoted 5 lines
drivers/scsi/scsi_lib.c +1
-1- scsi_device_blocked(sdev)) +if (scsi_device_blocked (sdev)) ret =BLK_STS_DEV_RESOURCE ;break ;default :Since this patch modifies code introduced in commit
86ff7c2a80cd("blk-mq:
introduceBLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes:
tag?
Quoted 2 lines
This same race exists before commit
86ff7c2a80cd, so I think the 'Fixes:' tag is misleading.
When reverted the patch "scsi: core: Only re-run queue in scsi_end_request() if device queue is busy", it also solves the issue.
Does the issue is brought by the patch? If so, maybe adding fixes("Fixes: ed5dd6a67d5e ("scsi: core: Only re-run queue in scsi_end_request() if device queue is busy")") is more accuratte.
Quoted 9 lines
Thanks,
Ming.
Quoted 53 lines · chenxiang (M)
Quoted 50 lines · Ming Lei
Quoted 49 lines · Bart Van Assche
Quoted 36 lines · Ming Lei
When queuing IO request to LLD,
STS_RESOURCEmay be returned because:- host in recovery or blocked
- target queue throttling or blocked
- LLD rejectionAny one of the above doesn't happen frequently enough.
BLK_STS_DEV_RESOURCEis returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', soBLK_STS_DEV_RESOURCEis returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for eitherscsi_run_queue_async()or blk-mq's RESTART.Fix the issue by not returning
BLK_STS_DEV_RESOURCEin this situation.Cc Hannes Reinecke <[email protected]>Cc Sumit Saxena <[email protected]>Cc Kashyap Desai <[email protected]>Cc Bart Van Assche <[email protected]>Cc Ewan Milne <[email protected]>Cc Long Li <[email protected]>Tested-by "chenxiang (M)" <[email protected]>Reported-by John Garry <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/scsi/scsi_lib.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)drivers/scsi/scsi_lib.c +0
-0drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709
Quoted 9 lines
drivers/scsi/scsi_lib.c +1
-2@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,break ;case BLK_STS_RESOURCE :case BLK_STS_ZONE_RESOURCE : -if (atomic_read (&sdev->device_busy) || -scsi_device_blocked (sdev)) +if (scsi_device_blocked (sdev)) ret =BLK_STS_DEV_RESOURCE ;break ;default :Since this patch modifies code introduced in commit
86ff7c2a80cd("blk-mq:
introduceBLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes:
tag?
Quoted 8 lines
Quoted 2 lines
This same race exists before commit
86ff7c2a80cd, so I think the 'Fixes:' tag is misleading.When reverted the patch "scsi: core: Only re-run queue in
scsi_end_request()if device queue is busy", it also solves the issue.
Does the issue is brought by the patch? If so, maybe adding fixes("Fixes:ed5dd6a67d5e("scsi: core: Only re-run queue inscsi_end_request()if device queue is busy")") is more accuratte.
The debugfs log shows that the issue isn't related with restart or out of budget because the single request stays in hctx->dispatch, and the flag of RQF_DONTPREP is set too.
The scsi's restart(scsi_run_queue_async) from request completion is just for handling requests in scheduler queue.
Thanks,
Ming
Martin K. Petersen
AppliedQuoted 9 lines · Ming Lei
When queuing IO request to LLD,
STS_RESOURCEmay be returned because:- host in recovery or blocked
- target queue throttling or blocked
- LLD rejectionAny one of the above doesn't happen frequently enough.
[...]
Applied to 5.10/scsi-fixes, thanks!
[1/1] scsi: core: fix race between handling STS_RESOURCE and completion
https://git.kernel.org/mkp/scsi/c/673235f91531
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering