(unknown)
Hi Aaron,
I found a race condition with the block layer runtime PM due to which the q->nr_pending is decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns -EBUSY.
The issue is easily reproduced with a scsi disk with disabled tagged command queuing
sd_probe_async() ->
add_disk() -> disk_add_event() -> schedule(disk_events_workfn) sd_revalidate_disk() blk_pm_runtime_init()
return;
Let's say the disk_events_workfn() calls sd_check_events() which tries to send test_unit_ready() and because of sd_revalidate_disk() trying to send another commands the test_unit_ready() might be re-queued as the tagged command queuing is disabled.
So the race condition is -
Thread 1 | Thread 2sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()blk_runtime_pm_init() | blk_pm_requeue_request() ->
| nr_pending = -1 since | q->dev != NULL
Do you have any suggestions on how to fix this issue?
Signature · 2 lines
Regards, Sujit
Re: Race condition in block layer runtime PM init and scsi disk driver
Quoted 39 lines
Hi Aaron,
I found a race condition with the block layer runtime PM due to which the
q->nr_pendingis decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns-EBUSY.The issue is easily reproduced with a scsi disk with disabled tagged command queuing
sd_probe_async()->add_disk() -> disk_add_event() -> schedule(disk_events_workfn) sd_revalidate_disk() blk_pm_runtime_init()return;
Let's say the
disk_events_workfn()callssd_check_events()which tries to sendtest_unit_ready()and because ofsd_revalidate_disk()trying to send another commands thetest_unit_ready()might be re-queued as the tagged command queuing is disabled.So the race condition is -
Thread 1 | Thread 2
sd_revalidate_disk()|sd_check_events()
...nr_pending = 0 asq->dev= NULL|scsi_queue_insert()blk_runtime_pm_init()|blk_pm_requeue_request()->| nr_pending = -1 since | q->dev != NULLDo you have any suggestions on how to fix this issue?
--
Regards,
Sujit
Re: Race condition in block layer runtime PM init and scsi disk driver
Quoted 25 lines · On 10/09/2013 04:32 PM, [email protected] wrote:
Quoted 25 lines
Hi Aaron,
I found a race condition with the block layer runtime PM due to which the
q->nr_pendingis decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns-EBUSY.The issue is easily reproduced with a scsi disk with disabled tagged command queuing
sd_probe_async()->add_disk() -> disk_add_event() -> schedule(disk_events_workfn) sd_revalidate_disk() blk_pm_runtime_init()return;
Let's say the
disk_events_workfn()callssd_check_events()which tries to sendtest_unit_ready()and because ofsd_revalidate_disk()trying to send another commands thetest_unit_ready()might be re-queued as the tagged command queuing is disabled.So the race condition is -
Quoted 8 lines
Quoted 8 lines
Thread 1 | Thread 2
sd_revalidate_disk()|sd_check_events()
...nr_pending = 0 asq->dev= NULL|scsi_queue_insert()blk_runtime_pm_init()|blk_pm_requeue_request()->| nr_pending = -1 since | q->dev != NULLDo you have any suggestions on how to fix this issue?
Thanks for the report. I wonder if the following patch helps?
Do the runtime init related work before add_disk, so that every request is counted properly.
drivers/scsi/sd.c +1 -1
@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie) gd->events |=DISK_EVENT_MEDIA_CHANGE ; } +blk_pm_runtime_init (sdp->request_queue, dev);add_disk (gd);if (sdkp->capacity)sd_dif_config_host (sdkp);
@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie)sd_printk (KERN_NOTICE , sdkp,"Attached SCSI %sdisk\n" , sdp->removable ?"removable " :"" ); -blk_pm_runtime_init (sdp->request_queue, dev);scsi_autopm_put_device (sdp);put_device (&sdkp->dev); }
Thanks,
Aaron
Quoted 8 lines
Quoted 6 lines
--
Regards,
Sujit
[email protected]
Tested-byRe: Race condition in block layer runtime PM init and scsi disk driver
Quoted 33 lines · On 10/09/2013 04:32 PM, [email protected] wrote:
Quoted 32 lines
Quoted 32 lines
Hi Aaron,
I found a race condition with the block layer runtime PM due to which the
q->nr_pendingis decremented to less than zero (0xFFFF_FFFF (-1)) and hence the blk pre-runtime suspend always returns-EBUSY.The issue is easily reproduced with a scsi disk with disabled tagged command queuing
sd_probe_async()->add_disk() -> disk_add_event() -> schedule(disk_events_workfn) sd_revalidate_disk() blk_pm_runtime_init()return;
Let's say the
disk_events_workfn()callssd_check_events()which tries to sendtest_unit_ready()and because ofsd_revalidate_disk()trying to send another commands thetest_unit_ready()might be re-queued as the tagged command queuing is disabled.So the race condition is -
Thread 1 | Thread 2
sd_revalidate_disk()|sd_check_events()
...nr_pending = 0 asq->dev= NULL|scsi_queue_insert()blk_runtime_pm_init()|blk_pm_requeue_request()->| nr_pending = -1 since | q->dev != NULL
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Do you have any suggestions on how to fix this issue?
Thanks for the report. I wonder if the following patch helps?
Thanks it works. Would you like to send formal patch for this? You can add my tested-by ack.
Quoted 40 lines
Do the runtime init related work before
add_disk, so that every request is counted properly.drivers/scsi/sd.c +1
-0@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie) gd->events |= DISK_EVENT_MEDIA_CHANGE; } +blk_pm_runtime_init (sdp->request_queue, dev);add_disk (gd);if (sdkp->capacity)drivers/scsi/sd.c +0
-1sd_dif_config_host (sdkp);@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie) sd_printk(KERN_NOTICE, sdkp, "Attached SCSI %sdisk\n", sdp->removable ? "removable " : ""); - blk_pm_runtime_init(sdp->request_queue, dev); scsi_autopm_put_device(sdp); put_device(&sdkp->dev);}
Thanks,
AaronQuoted 8 lines
Quoted 6 lines
--
Regards,
Sujit
PATCH 1/1 sd: call blk_pm_runtime_init before add_disk
Sujit has found a race condition that would make q->nr_pending unbalanced, it occurs as Sujit explained:
"sd_probe_async() ->
add_disk() -> disk_add_event() -> schedule(disk_events_workfn) sd_revalidate_disk() blk_pm_runtime_init()
return;
Let's say the disk_events_workfn() calls sd_check_events() which tries to send test_unit_ready() and because of sd_revalidate_disk() trying to send another commands the test_unit_ready() might be re-queued as the tagged command queuing is disabled.
So the race condition is -
Thread 1 | Thread 2sd_revalidate_disk() | sd_check_events()
...nr_pending = 0 as q->dev = NULL| scsi_queue_insert()blk_runtime_pm_init() | blk_pm_requeue_request() ->
| nr_pending = -1 since | q->dev != NULL
"
The problem is, the test_unit_ready request doesn't get counted the first time it is queued, so the later decrement of q->nr_pending in blk_pm_requeue_request makes it unbalanced.
Fix this by calling blk_pm_runtime_init before add_disk so that all requests initiated there will all be counted.
--- drivers/scsi/sd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/scsi/sd.c +1 -1
@@ -2854,6 +2854,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie) gd->events |=DISK_EVENT_MEDIA_CHANGE ; } +blk_pm_runtime_init (sdp->request_queue, dev);add_disk (gd);if (sdkp->capacity)sd_dif_config_host (sdkp);
@@ -2862,7 +2863,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie)sd_printk (KERN_NOTICE , sdkp,"Attached SCSI %sdisk\n" , sdp->removable ?"removable " :"" ); -blk_pm_runtime_init (sdp->request_queue, dev);scsi_autopm_put_device (sdp);put_device (&sdkp->dev); }
Signature · 1 line
1.8.4.12.g2ea3df6