Skip to contentSkip to conversation

(unknown)

PATCH New [email protected] 2 participants 5 messages Oct 9, 2013 – Oct 10, 2013 SCSI

lore
Share & tools
lore
#0 lore raw

Hi Aaron,

I found a race condition with the block layer runtime PM due to which the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns -EBUSY.

The issue is easily reproduced with a scsi disk with disabled tagged command queuing

sd_probe_async() ->

	add_disk() ->
		disk_add_event() ->
			schedule(disk_events_workfn)
	sd_revalidate_disk()
	blk_pm_runtime_init()

return;

Let's say the disk_events_workfn() calls sd_check_events() which tries to send test_unit_ready() and because of sd_revalidate_disk() trying to send another commands the test_unit_ready() might be re-queued as the tagged command queuing is disabled.

So the race condition is -

Thread 1 | Thread 2
sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
blk_runtime_pm_init() | blk_pm_requeue_request() ->

				  |	nr_pending = -1 since
				  |	q->dev != NULL

Do you have any suggestions on how to fix this issue?

Signature · 2 lines
Regards,
Sujit
#1 lore raw

Re: Race condition in block layer runtime PM init and scsi disk driver

Quoted 39 lines

Hi Aaron,

I found a race condition with the block layer runtime PM due to which the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns -EBUSY.

The issue is easily reproduced with a scsi disk with disabled tagged command queuing

sd_probe_async() ->

	add_disk() ->
		disk_add_event() ->
			schedule(disk_events_workfn)
	sd_revalidate_disk()
	blk_pm_runtime_init()

return;

Let's say the disk_events_workfn() calls sd_check_events() which tries to send test_unit_ready() and because of sd_revalidate_disk() trying to send another commands the test_unit_ready() might be re-queued as the tagged command queuing is disabled.

So the race condition is -

Thread 1 | Thread 2
sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
blk_runtime_pm_init() | blk_pm_requeue_request() ->

				  |	nr_pending = -1 since
				  |	q->dev != NULL

Do you have any suggestions on how to fix this issue?

--
Regards,
Sujit

#2 lore raw

Re: Race condition in block layer runtime PM init and scsi disk driver

Quoted 25 lines · On 10/09/2013 04:32 PM, [email protected] wrote:
Quoted 25 lines

Hi Aaron,

I found a race condition with the block layer runtime PM due to which the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns -EBUSY.

The issue is easily reproduced with a scsi disk with disabled tagged command queuing

sd_probe_async() ->

	add_disk() ->
		disk_add_event() ->
			schedule(disk_events_workfn)
	sd_revalidate_disk()
	blk_pm_runtime_init()

return;

Let's say the disk_events_workfn() calls sd_check_events() which tries to send test_unit_ready() and because of sd_revalidate_disk() trying to send another commands the test_unit_ready() might be re-queued as the tagged command queuing is disabled.

So the race condition is -

Quoted 8 lines
Quoted 8 lines

Thread 1 | Thread 2
sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
blk_runtime_pm_init() | blk_pm_requeue_request() ->

				  |	nr_pending = -1 since
				  |	q->dev != NULL

Do you have any suggestions on how to fix this issue?

Thanks for the report. I wonder if the following patch helps?

Do the runtime init related work before add_disk, so that every request is counted properly.

drivers/scsi/sd.c +1 -1
index e62d17d..5693f6d 100644
@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
 		gd->events |= DISK_EVENT_MEDIA_CHANGE;
 	}
 
+	blk_pm_runtime_init(sdp->request_queue, dev);
 	add_disk(gd);
 	if (sdkp->capacity)
 		sd_dif_config_host(sdkp);
@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
 
 	sd_printk(KERN_NOTICE, sdkp, "Attached SCSI %sdisk\n",
 		  sdp->removable ? "removable " : "");
-	blk_pm_runtime_init(sdp->request_queue, dev);
 	scsi_autopm_put_device(sdp);
 	put_device(&sdkp->dev);
 }

Thanks,
Aaron

Quoted 8 lines
Quoted 6 lines

--
Regards,
Sujit

author Tested-by
#3 lore raw

Re: Race condition in block layer runtime PM init and scsi disk driver

Quoted 33 lines · On 10/09/2013 04:32 PM, [email protected] wrote:
Quoted 32 lines
Quoted 32 lines

Hi Aaron,

I found a race condition with the block layer runtime PM due to which the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns -EBUSY.

The issue is easily reproduced with a scsi disk with disabled tagged command queuing

sd_probe_async() ->

	add_disk() ->
		disk_add_event() ->
			schedule(disk_events_workfn)
	sd_revalidate_disk()
	blk_pm_runtime_init()

return;

Let's say the disk_events_workfn() calls sd_check_events() which tries to send test_unit_ready() and because of sd_revalidate_disk() trying to send another commands the test_unit_ready() might be re-queued as the tagged command queuing is disabled.

So the race condition is -

Thread 1 | Thread 2
sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
blk_runtime_pm_init() | blk_pm_requeue_request() ->

				  |	nr_pending = -1 since
				  |	q->dev != NULL
Quoted 3 lines
Quoted 1 line
Quoted 1 line

Do you have any suggestions on how to fix this issue?

Thanks for the report. I wonder if the following patch helps?

Thanks it works. Would you like to send formal patch for this? You can add my tested-by ack.

Quoted 40 lines

Do the runtime init related work before add_disk, so that every request is counted properly.

drivers/scsi/sd.c +1 -0
index e62d17d..5693f6d 100644
@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data,
 async_cookie_t cookie)
 		gd->events |= DISK_EVENT_MEDIA_CHANGE;
 	}
 
+	blk_pm_runtime_init(sdp->request_queue, dev);
 	add_disk(gd);
 	if (sdkp->capacity)
drivers/scsi/sd.c +0 -1
 		sd_dif_config_host(sdkp);
@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data,
 async_cookie_t cookie)
 
 	sd_printk(KERN_NOTICE, sdkp, "Attached SCSI %sdisk\n",
 		  sdp->removable ? "removable " : "");
-	blk_pm_runtime_init(sdp->request_queue, dev);
 	scsi_autopm_put_device(sdp);
 	put_device(&sdkp->dev);

}

Thanks,
Aaron

Quoted 8 lines
Quoted 6 lines

--
Regards,
Sujit

#4 lore raw

PATCH 1/1 sd: call blk_pm_runtime_init before add_disk

Sujit has found a race condition that would make q->nr_pending unbalanced, it occurs as Sujit explained:

"
sd_probe_async() ->

	add_disk() ->
		disk_add_event() ->
			schedule(disk_events_workfn)
	sd_revalidate_disk()
	blk_pm_runtime_init()

return;

Let's say the disk_events_workfn() calls sd_check_events() which tries to send test_unit_ready() and because of sd_revalidate_disk() trying to send another commands the test_unit_ready() might be re-queued as the tagged command queuing is disabled.

So the race condition is -

Thread 1 | Thread 2
sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()
blk_runtime_pm_init() | blk_pm_requeue_request() ->

				  |	nr_pending = -1 since
				  |	q->dev != NULL

"

The problem is, the test_unit_ready request doesn't get counted the first time it is queued, so the later decrement of q->nr_pending in blk_pm_requeue_request makes it unbalanced.

Fix this by calling blk_pm_runtime_init before add_disk so that all requests initiated there will all be counted.

---
 drivers/scsi/sd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/scsi/sd.c +1 -1
index e62d17d..5693f6d 100644
@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
 		gd->events |= DISK_EVENT_MEDIA_CHANGE;
 	}
 
+	blk_pm_runtime_init(sdp->request_queue, dev);
 	add_disk(gd);
 	if (sdkp->capacity)
 		sd_dif_config_host(sdkp);
@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie)
 
 	sd_printk(KERN_NOTICE, sdkp, "Attached SCSI %sdisk\n",
 		  sdp->removable ? "removable " : "");
-	blk_pm_runtime_init(sdp->request_queue, dev);
 	scsi_autopm_put_device(sdp);
 	put_device(&sdkp->dev);
 }
Signature · 1 line
1.8.4.12.g2ea3df6