Skip to contentSkip to conversation

scsi: core: fix race between handling STS_RESOURCE and completion

PATCH Applied Ming Lei 4 participants 6 messages Dec 2, 2020 – Dec 8, 2020 SCSI

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 scsi: core: fix race between handling STS_RESOURCE and completion

When queuing IO request to LLD, STS_RESOURCE may be returned because:

- host in recovery or blocked
- target queue throttling or blocked
- LLD rejection

Any one of the above doesn't happen frequently enough.

BLK_STS_DEV_RESOURCE is returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', so BLK_STS_DEV_RESOURCE is returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for either scsi_run_queue_async() or blk-mq's RESTART.

Fix the issue by not returning BLK_STS_DEV_RESOURCE in this situation.

Cc Hannes Reinecke <[email protected]>
Cc Sumit Saxena <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Ewan Milne <[email protected]>
Cc Long Li <[email protected]>
---
 drivers/scsi/scsi_lib.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +1 -2
index 60c7a7d74852..03c6d0620bfd 100644
@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
 		break;
 	case BLK_STS_RESOURCE:
 	case BLK_STS_ZONE_RESOURCE:
-		if (atomic_read(&sdev->device_busy) ||
-		    scsi_device_blocked(sdev))
+		if (scsi_device_blocked(sdev))
 			ret = BLK_STS_DEV_RESOURCE;
 		break;
 	default:
Signature · 1 line
2.28.0
#1 lore raw
Quoted 36 lines · Ming Lei

When queuing IO request to LLD, STS_RESOURCE may be returned because:

- host in recovery or blocked
- target queue throttling or blocked
- LLD rejection

Any one of the above doesn't happen frequently enough.

BLK_STS_DEV_RESOURCE is returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', so BLK_STS_DEV_RESOURCE is returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for either scsi_run_queue_async() or blk-mq's RESTART.

Fix the issue by not returning BLK_STS_DEV_RESOURCE in this situation.

Cc Hannes Reinecke <[email protected]>
Cc Sumit Saxena <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Ewan Milne <[email protected]>
Cc Long Li <[email protected]>
---
  drivers/scsi/scsi_lib.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +0 -0
index 60c7a7d74852..03c6d0620bfd 100644

drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709

Quoted 9 lines
drivers/scsi/scsi_lib.c +1 -2
@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
  		break;
  	case BLK_STS_RESOURCE:
  	case BLK_STS_ZONE_RESOURCE:
-		if (atomic_read(&sdev->device_busy) ||
-		    scsi_device_blocked(sdev))
+		if (scsi_device_blocked(sdev))
  			ret = BLK_STS_DEV_RESOURCE;
  		break;
  	default:

Since this patch modifies code introduced in commit 86ff7c2a80cd ("blk-mq: introduce BLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes: tag?

Thanks,

Bart.

author
#2 lore raw
Quoted 42 lines · Bart Van Assche
Quoted 41 lines · Ming Lei

When queuing IO request to LLD, STS_RESOURCE may be returned because:

- host in recovery or blocked
- target queue throttling or blocked
- LLD rejection

Any one of the above doesn't happen frequently enough.

BLK_STS_DEV_RESOURCE is returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', so BLK_STS_DEV_RESOURCE is returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for either scsi_run_queue_async() or blk-mq's RESTART.

Fix the issue by not returning BLK_STS_DEV_RESOURCE in this situation.

Cc Hannes Reinecke <[email protected]>
Cc Sumit Saxena <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Ewan Milne <[email protected]>
Cc Long Li <[email protected]>
---
  drivers/scsi/scsi_lib.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +0 -2
index 60c7a7d74852..03c6d0620bfd 100644
@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
  		break;
  	case BLK_STS_RESOURCE:
  	case BLK_STS_ZONE_RESOURCE:
-		if (atomic_read(&sdev->device_busy) ||
-		    scsi_device_blocked(sdev))
Quoted 8 lines

drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709

Quoted 4 lines
drivers/scsi/scsi_lib.c +1 -0
+		if (scsi_device_blocked(sdev))
  			ret = BLK_STS_DEV_RESOURCE;
  		break;
  	default:

Since this patch modifies code introduced in commit 86ff7c2a80cd ("blk-mq:
introduce BLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes:
tag?

This same race exists before commit 86ff7c2a80cd, so I think the 'Fixes:' tag is misleading.

Thanks,
Ming

#3 lore raw
Quoted 50 lines · Ming Lei
Quoted 41 lines · Bart Van Assche
Quoted 40 lines · Ming Lei

When queuing IO request to LLD, STS_RESOURCE may be returned because:

- host in recovery or blocked
- target queue throttling or blocked
- LLD rejection

Any one of the above doesn't happen frequently enough.

BLK_STS_DEV_RESOURCE is returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', so BLK_STS_DEV_RESOURCE is returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for either scsi_run_queue_async() or blk-mq's RESTART.

Fix the issue by not returning BLK_STS_DEV_RESOURCE in this situation.

Cc Hannes Reinecke <[email protected]>
Cc Sumit Saxena <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Ewan Milne <[email protected]>
Cc Long Li <[email protected]>
---
   drivers/scsi/scsi_lib.c | 3 +--
   1 file changed, 1 insertion(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +0 -1
index 60c7a7d74852..03c6d0620bfd 100644
@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
   		break;
   	case BLK_STS_RESOURCE:
   	case BLK_STS_ZONE_RESOURCE:
-		if (atomic_read(&sdev->device_busy) ||
Quoted 8 lines

drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709

Quoted 5 lines
drivers/scsi/scsi_lib.c +1 -1
-		    scsi_device_blocked(sdev))
+		if (scsi_device_blocked(sdev))
   			ret = BLK_STS_DEV_RESOURCE;
   		break;
   	default:

Since this patch modifies code introduced in commit 86ff7c2a80cd ("blk-mq:
introduce BLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes:
tag?

Quoted 2 lines

This same race exists before commit 86ff7c2a80cd, so I think the 'Fixes:' tag is misleading.

When reverted the patch "scsi: core: Only re-run queue in scsi_end_request() if device queue is busy", it also solves the issue.
Does the issue is brought by the patch? If so, maybe adding fixes("Fixes: ed5dd6a67d5e ("scsi: core: Only re-run queue in scsi_end_request() if device queue is busy")") is more accuratte.

Quoted 9 lines

Thanks,
Ming

.

author
#4 lore raw
Quoted 53 lines · chenxiang (M)
Quoted 50 lines · Ming Lei
Quoted 49 lines · Bart Van Assche
Quoted 36 lines · Ming Lei

When queuing IO request to LLD, STS_RESOURCE may be returned because:

- host in recovery or blocked
- target queue throttling or blocked
- LLD rejection

Any one of the above doesn't happen frequently enough.

BLK_STS_DEV_RESOURCE is returned to block layer for avoiding unnecessary re-run queue, and it is just one small optimization. However, all in-flight requests originated from this scsi device may be completed just after reading 'sdev->device_busy', so BLK_STS_DEV_RESOURCE is returned to block layer. And the current failed IO won't get chance to be queued any more, since it is invisible at that time for either scsi_run_queue_async() or blk-mq's RESTART.

Fix the issue by not returning BLK_STS_DEV_RESOURCE in this situation.

Cc Hannes Reinecke <[email protected]>
Cc Sumit Saxena <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Ewan Milne <[email protected]>
Cc Long Li <[email protected]>
---
   drivers/scsi/scsi_lib.c | 3 +--
   1 file changed, 1 insertion(+), 2 deletions(-)
drivers/scsi/scsi_lib.c +0 -0
index 60c7a7d74852..03c6d0620bfd 100644

drivers/scsi/scsi_lib.c static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1709

Quoted 9 lines
drivers/scsi/scsi_lib.c +1 -2
@@ -1703,8 +1703,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
   		break;
   	case BLK_STS_RESOURCE:
   	case BLK_STS_ZONE_RESOURCE:
-		if (atomic_read(&sdev->device_busy) ||
-		    scsi_device_blocked(sdev))
+		if (scsi_device_blocked(sdev))
   			ret = BLK_STS_DEV_RESOURCE;
   		break;
   	default:

Since this patch modifies code introduced in commit 86ff7c2a80cd ("blk-mq:
introduce BLK_STS_DEV_RESOURCE"), does this patch perhaps needs a Fixes:
tag?

Quoted 8 lines
Quoted 2 lines

This same race exists before commit 86ff7c2a80cd, so I think the 'Fixes:' tag is misleading.

When reverted the patch "scsi: core: Only re-run queue in scsi_end_request() if device queue is busy", it also solves the issue.
Does the issue is brought by the patch? If so, maybe adding fixes("Fixes:
ed5dd6a67d5e ("scsi: core: Only re-run queue in scsi_end_request() if device queue is busy")") is more accuratte.

The debugfs log shows that the issue isn't related with restart or out of budget because the single request stays in hctx->dispatch, and the flag of RQF_DONTPREP is set too.

The scsi's restart(scsi_run_queue_async) from request completion is just for handling requests in scheduler queue.

Thanks,
Ming

#5 lore raw
In reply to Ming Lei
Quoted 9 lines · Ming Lei

When queuing IO request to LLD, STS_RESOURCE may be returned because:

- host in recovery or blocked
- target queue throttling or blocked
- LLD rejection

Any one of the above doesn't happen frequently enough.

[...]

Applied to 5.10/scsi-fixes, thanks!

[1/1] scsi: core: fix race between handling STS_RESOURCE and completion

      https://git.kernel.org/mkp/scsi/c/673235f91531
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering