block: Fix use-after-free issue while accessing ioscheduler lock
PATCH v2 1/1 block: Fix use-after-free issue while accessing ioscheduler lock
Observes below crash while accessing (use-after-free) lock member of bfq data.
context#1 context#2
process_one_work()kthread() blk_mq_run_work_fn()worker_thread() ->__blk_mq_run_hw_queue()process_one_work() ->blk_mq_sched_dispatch_requests()__blk_release_queue() ->blk_mq_do_dispatch_sched()
->__elevator_exit()
->blk_mq_exit_sched()
->exit_sched()
->kfree()
->bfq_dispatch_request()
->spin_unlock_irq(&bfqd->lock)This is because of the kblockd delayed work that might got scheduled around blk_release_queue() and accessed use-after-free member of bfq_data.
240.212359: <2> Unable to handle kernel paging request at
virtual address ffffffee2e33ad70
...
240.212637: <2> Workqueue: kblockd blk_mq_run_work_fn 240.212649: <2> pstate: 00c00085 (nzcv daIf +PAN +UAO) 240.212666: <2> pc : queued_spin_lock_slowpath+0x10c/0x2e0 240.212677: <2> lr : queued_spin_lock_slowpath+0x84/0x2e0 ... Call trace: 240.212865: <2> queued_spin_lock_slowpath+0x10c/0x2e0 240.212876: <2> do_raw_spin_lock+0xf0/0xf4 240.212890: <2> _raw_spin_lock_irq+0x74/0x94 240.212906: <2> bfq_dispatch_request+0x4c/0xd60 240.212918: <2> blk_mq_do_dispatch_sched+0xe0/0x1f0 240.212927: <2> blk_mq_sched_dispatch_requests+0x130/0x194 240.212940: <2> __blk_mq_run_hw_queue+0x100/0x158 240.212950: <2> blk_mq_run_work_fn+0x1c/0x28 240.212963: <2> process_one_work+0x280/0x460 240.212973: <2> worker_thread+0x27c/0x4dc 240.212986: <2> kthread+0x160/0x170
Fix this by cancelling the delayed work if any before elevator exits.
Changes since V1:
- Moved the logic into blk_cleanup_queue() as per Ming comments.
--- block/blk-mq.c | 1 + 1 file changed, 1 insertion(+)
block/blk-mq.c +1 -0
@@ -2598,6 +2598,7 @@ static void blk_mq_exit_hw_queues(struct request_queue *q,break ;blk_mq_debugfs_unregister_hctx (hctx);blk_mq_exit_hctx (q, set, hctx, i); +cancel_delayed_work_sync (&hctx->run_work); } }
Signature · 1 line
2.7.4
Quoted 9 lines · Pradeep P V K
Observes below crash while accessing (use-after-free) lock member of bfq data.
context#1 context#2
process_one_work()kthread()blk_mq_run_work_fn()worker_thread()->__blk_mq_run_hw_queue()process_one_work()->blk_mq_sched_dispatch_requests()__blk_release_queue()->blk_mq_do_dispatch_sched()
Just found __blk_release_queue killed in v5.9 cycle.
Quoted 6 lines
->__elevator_exit()
->blk_mq_exit_sched()->exit_sched() ->kfree() ->bfq_dispatch_request() ->spin_unlock_irq(&bfqd->lock)
Actually not sure if the above race is easy to trigger in recent kernel, because we do call cancel_delayed_work_sync() in blk_mq_hw_sysfs_release(), which is usually called before __elevator_exit() from blk_exit_queue()/blk_release_queue().
So can you share your kernel version in which the issue is reproduced? And can you reproduce this issue on v5.8 or v5.9-rc5?
Quoted 41 lines
This is because of the kblockd delayed work that might got scheduled around
blk_release_queue()and accessed use-after-free member ofbfq_data.240.212359: <2> Unable to handle kernel paging request at
virtual addressffffffee2e33ad70
...240.212637: <2> Workqueue: kblockd blk_mq_run_work_fn 240.212649: <2> pstate: 00c00085 (nzcv daIf +PAN +UAO) 240.212666: <2> pc : queued_spin_lock_slowpath+0x10c/0x2e0 240.212677: <2> lr : queued_spin_lock_slowpath+0x84/0x2e0 ... Call trace: 240.212865: <2> queued_spin_lock_slowpath+0x10c/0x2e0 240.212876: <2> do_raw_spin_lock+0xf0/0xf4 240.212890: <2> _raw_spin_lock_irq+0x74/0x94 240.212906: <2> bfq_dispatch_request+0x4c/0xd60 240.212918: <2> blk_mq_do_dispatch_sched+0xe0/0x1f0 240.212927: <2> blk_mq_sched_dispatch_requests+0x130/0x194 240.212940: <2> __blk_mq_run_hw_queue+0x100/0x158 240.212950: <2> blk_mq_run_work_fn+0x1c/0x28 240.212963: <2> process_one_work+0x280/0x460 240.212973: <2> worker_thread+0x27c/0x4dc 240.212986: <2> kthread+0x160/0x170Fix this by cancelling the delayed work if any before elevator exits.
Changes since V1:
- Moved the logic intoblk_cleanup_queue()as per Ming comments.Signed-off-by Pradeep P V K <[email protected]>--- block/blk-mq.c | 1 + 1 file changed, 1 insertion(+)block/blk-mq.c +0
-0
block/blk-mq.c static void blk_mq_exit_hw_queues(struct request_queue *q, · line 2603
Quoted 6 lines
block/blk-mq.c +1
-0@@ -2598,6 +2598,7 @@ static void blk_mq_exit_hw_queues(struct request_queue *q,break ;blk_mq_debugfs_unregister_hctx (hctx);blk_mq_exit_hctx (q, set, hctx, i); +cancel_delayed_work_sync (&hctx->run_work); } }
It should be better to move cancel_delayed_work_sync() into blk_mq_exit_hctx(), exactly before adding hctx into unused list.
Thanks,
Ming
Quoted 4 lines · Ming Lei
Quoted 3 lines · Pradeep P V K
Observes below crash while accessing (use-after-free) lock member of bfq data.
Quoted 9 lines
Quoted 6 lines
context#1 context#2
process_one_work()kthread()blk_mq_run_work_fn()worker_thread()->__blk_mq_run_hw_queue()process_one_work()->blk_mq_sched_dispatch_requests()__blk_release_queue()->blk_mq_do_dispatch_sched()Just found
__blk_release_queuekilled in v5.9 cycle.
Yes on v5.9 blk_release_queue() will be called directly by q->kobj when request_queue ref. goes zero but
where as on older kernel versions (< 5.9), blk_release_queue() will schedule a work to invoke/call "__blk_release_queue()".
Quoted 7 lines
Quoted 6 lines
->__elevator_exit()
->blk_mq_exit_sched()->exit_sched() ->kfree() ->bfq_dispatch_request() ->spin_unlock_irq(&bfqd->lock)
Quoted 5 lines
Actually not sure if the above race is easy to trigger in recent kernel, because we do call
cancel_delayed_work_sync()inblk_mq_hw_sysfs_release(), which is usually called before__elevator_exit()from blk_exit_queue()/blk_release_queue().
blk_mq_hw_sysfs_release() will be called from blk_mq_release() i.e. with kobject_put(hctx->kobj), which is after __elevator_exit()
__elevator_exit() is called from blk_exit_queue() which is prior to blk_mq_release().
Quoted 3 lines
So can you share your kernel version in which the issue is reproduced? And can you reproduce this issue on v5.8 or v5.9-rc5?
This issue is seen on v5.4 stable and it is very easy to reproduce on v5.4. sorry, i don't have a resource with v5.8 or with latest kernel. I can help you to get tested on v5.4. From the issue prospective, both v5.4 kernel and latest kernels calls blk_mq_release() -> blk_mq_hw_sysfs_release() after __elevator_exit(). So, i think it wont matter much here.
Quoted 36 lines
Quoted 36 lines
This is because of the kblockd delayed work that might got scheduled around
blk_release_queue()and accessed use-after-free member ofbfq_data.240.212359: <2> Unable to handle kernel paging request at
virtual addressffffffee2e33ad70
...240.212637: <2> Workqueue: kblockd blk_mq_run_work_fn 240.212649: <2> pstate: 00c00085 (nzcv daIf +PAN +UAO) 240.212666: <2> pc : queued_spin_lock_slowpath+0x10c/0x2e0 240.212677: <2> lr : queued_spin_lock_slowpath+0x84/0x2e0 ... Call trace: 240.212865: <2> queued_spin_lock_slowpath+0x10c/0x2e0 240.212876: <2> do_raw_spin_lock+0xf0/0xf4 240.212890: <2> _raw_spin_lock_irq+0x74/0x94 240.212906: <2> bfq_dispatch_request+0x4c/0xd60 240.212918: <2> blk_mq_do_dispatch_sched+0xe0/0x1f0 240.212927: <2> blk_mq_sched_dispatch_requests+0x130/0x194 240.212940: <2> __blk_mq_run_hw_queue+0x100/0x158 240.212950: <2> blk_mq_run_work_fn+0x1c/0x28 240.212963: <2> process_one_work+0x280/0x460 240.212973: <2> worker_thread+0x27c/0x4dc 240.212986: <2> kthread+0x160/0x170Fix this by cancelling the delayed work if any before elevator exits.
Changes since V1:
- Moved the logic intoblk_cleanup_queue()as per Ming comments.Signed-off-by Pradeep P V K <[email protected]>--- block/blk-mq.c | 1 + 1 file changed, 1 insertion(+)
Quoted 16 lines
block/blk-mq.c static void blk_mq_exit_hw_queues(struct · line 2604
Quoted 12 lines
block/blk-mq.c +1
-0@@ -2598,6 +2598,7 @@ static void blk_mq_exit_hw_queues(structrequest_queue *q, break;blk_mq_debugfs_unregister_hctx (hctx);blk_mq_exit_hctx (q, set, hctx, i); +cancel_delayed_work_sync (&hctx->run_work); } }It should be better to move
cancel_delayed_work_sync()intoblk_mq_exit_hctx(), exactly before adding hctx into unused list.
Sure. i will do it in my next patch series.
Quoted 3 lines
Thanks,
Ming
Thanks and Regards,
Pradeep
Quoted 100 lines · On Tue, Sep 15, 2020 at 05:50:33PM +0530, [email protected] wrote:
Quoted 4 lines · Ming Lei
Quoted 3 lines · Pradeep P V K
Observes below crash while accessing (use-after-free) lock member of bfq data.
Quoted 9 lines
Quoted 6 lines
context#1 context#2
process_one_work()kthread()blk_mq_run_work_fn()worker_thread()->__blk_mq_run_hw_queue()process_one_work()->blk_mq_sched_dispatch_requests()__blk_release_queue()->blk_mq_do_dispatch_sched()Just found
__blk_release_queuekilled in v5.9 cycle.Yes on v5.9
blk_release_queue()will be called directly byq->kobjwhenrequest_queueref. goes zero but
where as on older kernel versions (< 5.9),blk_release_queue()will schedule a work to invoke/call "__blk_release_queue()".Quoted 7 lines
Quoted 6 lines
->__elevator_exit()
->blk_mq_exit_sched()->exit_sched() ->kfree() ->bfq_dispatch_request() ->spin_unlock_irq(&bfqd->lock)Quoted 6 lines
Actually not sure if the above race is easy to trigger in recent kernel, because we do call
cancel_delayed_work_sync()inblk_mq_hw_sysfs_release(),
which is usually called before__elevator_exit()from blk_exit_queue()/blk_release_queue().
blk_mq_hw_sysfs_release()will be called fromblk_mq_release()i.e. withkobject_put(hctx->kobj), which is after__elevator_exit()
__elevator_exit()is called fromblk_exit_queue()which is prior toblk_mq_release().Quoted 3 lines
So can you share your kernel version in which the issue is reproduced? And can you reproduce this issue on v5.8 or v5.9-rc5?
This issue is seen on v5.4 stable and it is very easy to reproduce on v5.4. sorry, i don't have a resource with v5.8 or with latest kernel. I can help you
to get tested on v5.4. From the issue prospective, both v5.4 kernel and latest kernels callsblk_mq_release()->blk_mq_hw_sysfs_release()after__elevator_exit(). So, i think it wont matter much here.Quoted 36 lines
Quoted 36 lines
This is because of the kblockd delayed work that might got scheduled around
blk_release_queue()and accessed use-after-free member ofbfq_data.240.212359: <2> Unable to handle kernel paging request at
virtual addressffffffee2e33ad70
...240.212637: <2> Workqueue: kblockd blk_mq_run_work_fn 240.212649: <2> pstate: 00c00085 (nzcv daIf +PAN +UAO) 240.212666: <2> pc : queued_spin_lock_slowpath+0x10c/0x2e0 240.212677: <2> lr : queued_spin_lock_slowpath+0x84/0x2e0 ... Call trace: 240.212865: <2> queued_spin_lock_slowpath+0x10c/0x2e0 240.212876: <2> do_raw_spin_lock+0xf0/0xf4 240.212890: <2> _raw_spin_lock_irq+0x74/0x94 240.212906: <2> bfq_dispatch_request+0x4c/0xd60 240.212918: <2> blk_mq_do_dispatch_sched+0xe0/0x1f0 240.212927: <2> blk_mq_sched_dispatch_requests+0x130/0x194 240.212940: <2> __blk_mq_run_hw_queue+0x100/0x158 240.212950: <2> blk_mq_run_work_fn+0x1c/0x28 240.212963: <2> process_one_work+0x280/0x460 240.212973: <2> worker_thread+0x27c/0x4dc 240.212986: <2> kthread+0x160/0x170Fix this by cancelling the delayed work if any before elevator exits.
Changes since V1:
- Moved the logic intoblk_cleanup_queue()as per Ming comments.Signed-off-by Pradeep P V K <[email protected]>--- block/blk-mq.c | 1 + 1 file changed, 1 insertion(+)Quoted 16 lines
block/blk-mq.c static void blk_mq_exit_hw_queues(struct · line 2604
Quoted 12 lines
block/blk-mq.c +1
-0@@ -2598,6 +2598,7 @@ static void blk_mq_exit_hw_queues(structrequest_queue *q, break;blk_mq_debugfs_unregister_hctx (hctx);blk_mq_exit_hctx (q, set, hctx, i); +cancel_delayed_work_sync (&hctx->run_work); } }It should be better to move
cancel_delayed_work_sync()intoblk_mq_exit_hctx(), exactly before adding hctx into unused list.
Quoted 1 line
Sure. i will do it in my next patch series.
Thinking of further, looks your 1st post is right, because hw queue won't be run when refcount of the queue drops to zero.
Sorry for the noise.
thanks,
Ming
Quoted 101 lines · Ming Lei
Quoted 100 lines · On Tue, Sep 15, 2020 at 05:50:33PM +0530, [email protected] wrote:
Quoted 4 lines · Ming Lei
Quoted 3 lines · Pradeep P V K
Observes below crash while accessing (use-after-free) lock member of bfq data.
Quoted 9 lines
Quoted 6 lines
context#1 context#2
process_one_work()kthread()blk_mq_run_work_fn()worker_thread()->__blk_mq_run_hw_queue()process_one_work()->blk_mq_sched_dispatch_requests()__blk_release_queue()->blk_mq_do_dispatch_sched()Just found
__blk_release_queuekilled in v5.9 cycle.Yes on v5.9
blk_release_queue()will be called directly byq->kobjwhenrequest_queueref. goes zero but
where as on older kernel versions (< 5.9),blk_release_queue()will schedule a work to invoke/call "__blk_release_queue()".Quoted 7 lines
Quoted 6 lines
->__elevator_exit()
->blk_mq_exit_sched()->exit_sched() ->kfree() ->bfq_dispatch_request() ->spin_unlock_irq(&bfqd->lock)Quoted 6 lines
Actually not sure if the above race is easy to trigger in recent kernel, because we do call
cancel_delayed_work_sync()inblk_mq_hw_sysfs_release(),
which is usually called before__elevator_exit()from blk_exit_queue()/blk_release_queue().
blk_mq_hw_sysfs_release()will be called fromblk_mq_release()i.e. withkobject_put(hctx->kobj), which is after__elevator_exit()
__elevator_exit()is called fromblk_exit_queue()which is prior toblk_mq_release().Quoted 3 lines
So can you share your kernel version in which the issue is reproduced? And can you reproduce this issue on v5.8 or v5.9-rc5?
This issue is seen on v5.4 stable and it is very easy to reproduce on v5.4. sorry, i don't have a resource with v5.8 or with latest kernel. I can help you
to get tested on v5.4. From the issue prospective, both v5.4 kernel and latest kernels callsblk_mq_release()->blk_mq_hw_sysfs_release()after__elevator_exit(). So, i think it wont matter much here.Quoted 52 lines
Quoted 41 lines
This is because of the kblockd delayed work that might got scheduled around
blk_release_queue()and accessed use-after-free member ofbfq_data.240.212359: <2> Unable to handle kernel paging request at
virtual addressffffffee2e33ad70
...240.212637: <2> Workqueue: kblockd blk_mq_run_work_fn 240.212649: <2> pstate: 00c00085 (nzcv daIf +PAN +UAO) 240.212666: <2> pc : queued_spin_lock_slowpath+0x10c/0x2e0 240.212677: <2> lr : queued_spin_lock_slowpath+0x84/0x2e0 ... Call trace: 240.212865: <2> queued_spin_lock_slowpath+0x10c/0x2e0 240.212876: <2> do_raw_spin_lock+0xf0/0xf4 240.212890: <2> _raw_spin_lock_irq+0x74/0x94 240.212906: <2> bfq_dispatch_request+0x4c/0xd60 240.212918: <2> blk_mq_do_dispatch_sched+0xe0/0x1f0 240.212927: <2> blk_mq_sched_dispatch_requests+0x130/0x194 240.212940: <2> __blk_mq_run_hw_queue+0x100/0x158 240.212950: <2> blk_mq_run_work_fn+0x1c/0x28 240.212963: <2> process_one_work+0x280/0x460 240.212973: <2> worker_thread+0x27c/0x4dc 240.212986: <2> kthread+0x160/0x170Fix this by cancelling the delayed work if any before elevator exits.
Changes since V1:
- Moved the logic intoblk_cleanup_queue()as per Ming comments.Signed-off-by Pradeep P V K <[email protected]>--- block/blk-mq.c | 1 + 1 file changed, 1 insertion(+)block/blk-mq.c +0
-0block/blk-mq.c static void blk_mq_exit_hw_queues(struct · line 2604
Quoted 7 lines
block/blk-mq.c +1
-0@@ -2598,6 +2598,7 @@ static void blk_mq_exit_hw_queues(structrequest_queue *q, break;blk_mq_debugfs_unregister_hctx (hctx);blk_mq_exit_hctx (q, set, hctx, i); +cancel_delayed_work_sync (&hctx->run_work); } }It should be better to move
cancel_delayed_work_sync()intoblk_mq_exit_hctx(), exactly before adding hctx into unused list.
Quoted 5 lines
Quoted 1 line
Sure. i will do it in my next patch series.
Thinking of further, looks your 1st post is right, because hw queue won't be run when refcount of the queue drops to zero.
ok. Can you help add your Review-by sign off on my V1 patch.
Quoted 2 lines
Sorry for the noise.
never mind - it's ok.
Quoted 2 lines
thanks,
Ming
Thanks and Regards,
Pradeep