bfq: Check if bfqq is NULL in bfq_insert_request
Subsystems (2)
PATCH 1/1 bfq: Check if bfqq is NULL in bfq_insert_request
In bfq_insert_request(), bfqq is initialized with:
bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;
Indeed, rq->elv.icq can be NULL if the memory allocation in create_task_io_context() failed.
A comment in bfq_insert_request() suggests that bfqq is supposed to be non-NULL if 'at_head || blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.
This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80
The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd /sys/kernel/debug/failslab
echo 10 > probability
echo 300 > times
Avoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
--- block/bfq-iosched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/bfq-iosched.c +1 -1
@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else
Signature · 1 line
2.7.4
Quoted 8 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.
The above function should have been ioc_create_icq(), sorry.
Guenter
Quoted 63 lines
A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>--- block/bfq-iosched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
Quoted 10 lines · Guenter Roeck
Quoted 8 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.
Quoted 41 lines
The above function should have been
ioc_create_icq(), sorry.Guenter
Quoted 37 lines
A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
What about other place use blk_init_rq()?
E.g in bfq_request_merged():
1897 struct bfq_queue *bfqq = bfq_init_rq(req);
1898 struct bfq_data *bfqd = bfqq->bfqd;
Which may have same Null-pointer bug?
-Bob
Quoted 26 lines
Quoted 26 lines
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>--- block/bfq-iosched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
Quoted 54 lines · Bob Liu
Quoted 10 lines · Guenter Roeck
Quoted 8 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.Quoted 41 lines
The above function should have been
ioc_create_icq(), sorry.Guenter
Quoted 37 lines
A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
Quoted 7 lines
What about other place use
blk_init_rq()?
E.g inbfq_request_merged():
1897 structbfq_queue*bfqq =bfq_init_rq(req);
1898 structbfq_data*bfqd =bfqq->bfqd;Which may have same Null-pointer bug?
Good question. Doug asked it as well. My understanding is that the code won't hit those places since bfq is essentially bypassed. Of course, I may be completely wrong, so we should wait for Paolo to confirm if I my understanding is correct (or wrong).
Thanks,
Guenter
Quoted 29 lines
-Bob
Quoted 26 lines
Quoted 26 lines
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>--- block/bfq-iosched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
ping ... just in case this patch got lost in Paolo's queue.
Guenter
Quoted 71 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>--- block/bfq-iosched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
Hi Guenter,
sorry for the delay (Dolomiti's fault).
I didn't consider that rq->elv-icq might have been NULL also because of OOM. Thanks for spotting this issue.
As for the other places where the return value of bfq_init_rq is used, unfortunately I think they matter too. Those other places are related to request merging, which is the alternative destiny of requests (instead of being just inserted). But, regardless of whether a request is to be merged or inserted, that request may be destined to a bfq_queue (possibly merged with a request already in a bfq_queue), and a NULL return value by bfq_init_rq leads to a crash. I guess you can reproduce your failure also for the merge case, by generating sequential, direct I/O with queue depth > 1, and of course by enabling failslab.
If my considerations above are correct, do you want to propose a complete fix yourself?
Thanks,
Paolo
Quoted 78 lines
Il giorno 28 lug 2019, alle ore 17:19, Guenter Roeck <[email protected]> ha scritto:
ping ... just in case this patch got lost in Paolo's queue.
Guenter
Quoted 71 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>---
block/bfq-iosched.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
Quoted 6 lines · Paolo Valente
Hi Guenter,
sorry for the delay (Dolomiti's fault).I didn't consider that rq->elv-icq might have been NULL also because of OOM. Thanks for spotting this issue.
Quoted 11 lines
As for the other places where the return value of
bfq_init_rqis used, unfortunately I think they matter too. Those other places are related to request merging, which is the alternative destiny of requests (instead of being just inserted). But, regardless of whether a request is to be merged or inserted, that request may be destined to abfq_queue(possibly merged with a request already in abfq_queue), and a NULL return value bybfq_init_rqleads to a crash. I guess you can reproduce your failure also for the merge case, by generating sequential, direct I/O with queue depth > 1, and of course by enabling failslab.
My assumption was that requests would only be merged if they are associated with the same io context. In that case, that IO context isn't reallocated with ioc_create_icq() but reused, and icq would thus never be NULL. I guess that assumption was wrong.
Quoted 3 lines
If my considerations above are correct, do you want to propose a complete fix yourself?
Sure, I'll send an updated patch.
Thanks,
Guenter
Quoted 83 lines
Thanks,
PaoloQuoted 78 lines
Il giorno 28 lug 2019, alle ore 17:19, Guenter Roeck <[email protected]> ha scritto:
ping ... just in case this patch got lost in Paolo's queue.
Guenter
Quoted 71 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>---
block/bfq-iosched.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
Quoted 19 lines
Il giorno 30 lug 2019, alle ore 15:35, Guenter Roeck <[email protected]> ha scritto:
Quoted 6 lines · Paolo Valente
Hi Guenter,
sorry for the delay (Dolomiti's fault).
I didn't consider that rq->elv-icq might have been NULL also because of OOM. Thanks for spotting this issue.
As for the other places where the return value ofbfq_init_rqis used, unfortunately I think they matter too. Those other places are relatedQuoted 8 lines
to request merging, which is the alternative destiny of requests (instead of being just inserted). But, regardless of whether a request is to be merged or inserted, that request may be destined to a
bfq_queue(possibly merged with a request already in abfq_queue), and a NULL return value bybfq_init_rqleads to a crash. I guess you can reproduce your failure also for the merge case, by generating sequential, direct I/O with queue depth > 1, and of course by enabling failslab.
Quoted 4 lines
My assumption was that requests would only be merged if they are associated with the same io context. In that case, that IO context isn't reallocated with
ioc_create_icq()but reused, and icq would thus never be NULL. I guess that assumption was wrong.
I don't remember such a filtering. I had a look again, but didn't find anything relevant. However, more competent people see these emails. Maybe someone can give us better advice. Otherwise, to stay on the safe side, I'd propose to handle any possible NULL return.
And I'll manage it, as per your request.
Thanks,
Paolo
Quoted 90 lines
Quoted 3 lines
If my considerations above are correct, do you want to propose a complete fix yourself?
Sure, I'll send an updated patch.
Thanks,
GuenterQuoted 80 lines
Thanks,
PaoloQuoted 78 lines
Il giorno 28 lug 2019, alle ore 17:19, Guenter Roeck <[email protected]> ha scritto:
ping ... just in case this patch got lost in Paolo's queue.
Guenter
Quoted 71 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>---
block/bfq-iosched.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
Quoted 19 lines · Paolo Valente
Quoted 19 lines
Il giorno 30 lug 2019, alle ore 15:35, Guenter Roeck <[email protected]> ha scritto:
Quoted 14 lines · Paolo Valente
Hi Guenter,
sorry for the delay (Dolomiti's fault).
I didn't consider that rq->elv-icq might have been NULL also because of OOM. Thanks for spotting this issue.
As for the other places where the return value ofbfq_init_rqis used, unfortunately I think they matter too. Those other places are related to request merging, which is the alternative destiny of requests (instead of being just inserted). But, regardless of whether a request is to be merged or inserted, that request may be destined to abfq_queue(possibly merged with a request already in abfq_queue), and a NULL return value bybfq_init_rqleads to a crash. I guess you can reproduce your failure also for the merge case, by generating sequential, direct I/O with queue depth > 1, and of course by enabling failslab.
Quoted 7 lines
Quoted 4 lines
My assumption was that requests would only be merged if they are associated with the same io context. In that case, that IO context isn't reallocated with
ioc_create_icq()but reused, and icq would thus never be NULL. I guess that assumption was wrong.I don't remember such a filtering. I had a look again, but didn't find anything relevant. However, more competent people see these
Me not either, when I had a closer look yesterday. My conclusion was that your analysis is correct.
Thanks,
Guenter
Quoted 100 lines
emails. Maybe someone can give us better advice. Otherwise, to stay on the safe side, I'd propose to handle any possible NULL return.
And I'll manage it, as per your request.
Thanks,
PaoloQuoted 90 lines
Quoted 3 lines
If my considerations above are correct, do you want to propose a complete fix yourself?
Sure, I'll send an updated patch.
Thanks,
GuenterQuoted 80 lines
Thanks,
PaoloQuoted 78 lines
Il giorno 28 lug 2019, alle ore 17:19, Guenter Roeck <[email protected]> ha scritto:
ping ... just in case this patch got lost in Paolo's queue.
Guenter
Quoted 71 lines · Guenter Roeck
In
bfq_insert_request(), bfqq is initialized with:bfqq = bfq_init_rq(rq); In bfq_init_rq(), we find: if (unlikely(!rq->elv.icq)) return NULL;Indeed,
rq->elv.icqcan be NULL if the memory allocation increate_task_io_context()failed.A comment in
bfq_insert_request()suggests that bfqq is supposed to be non-NULL if 'at_head||blk_rq_is_passthrough(rq)' is false. Yet, as debugging and practical experience shows, this is not the case in the above situation.This results in the following crash.
Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b0 ... Call trace: bfq_setup_cooperator+0x44/0x134 bfq_insert_requests+0x10c/0x630 blk_mq_sched_insert_requests+0x60/0xb4 blk_mq_flush_plug_list+0x290/0x2d4 blk_flush_plug_list+0xe0/0x230 blk_finish_plug+0x30/0x40 generic_writepages+0x60/0x94 blkdev_writepages+0x24/0x30 do_writepages+0x74/0xac __filemap_fdatawrite_range+0x94/0xc8 file_write_and_wait_range+0x44/0xa0 blkdev_fsync+0x38/0x68 vfs_fsync_range+0x68/0x80 do_fsync+0x44/0x80The problem is relatively easy to reproduce by running an image with failslab enabled, such as with:
cd
/sys/kernel/debug/failslab
echo 10 > probability
echo 300 > timesAvoid the problem by checking if bfqq is NULL before using it. With the NULL check in place, requests with missing io context are queued immediately, and the crash is no longer seen.
Fixes 18e5a57d79878 ("block, bfq: postpone rq preparation to insert or merge")Reported-by Hsin-Yi Wang <[email protected]>Cc Hsin-Yi Wang <[email protected]>Cc Nicolas Boichat <[email protected]>Cc Doug Anderson <[email protected]>Signed-off-by Guenter Roeck <[email protected]>---
block/bfq-iosched.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)block/bfq-iosched.c +1
-1@@ -5417,7 +5417,7 @@ static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,spin_lock_irq (&bfqd->lock); bfqq =bfq_init_rq (rq); -if (at_head ||blk_rq_is_passthrough (rq)) { +if (!bfqq || at_head ||blk_rq_is_passthrough (rq)) {if (at_head)list_add (&rq->queuelist, &bfqd->dispatch);else Signature · 1 line
2.7.4
Paolo,
Quoted 17 lines · Paolo Valente
Hi Guenter,
sorry for the delay (Dolomiti's fault).I didn't consider that rq->elv-icq might have been NULL also because of OOM. Thanks for spotting this issue.
As for the other places where the return value of
bfq_init_rqis used, unfortunately I think they matter too. Those other places are related to request merging, which is the alternative destiny of requests (instead of being just inserted). But, regardless of whether a request is to be merged or inserted, that request may be destined to abfq_queue(possibly merged with a request already in abfq_queue), and a NULL return value bybfq_init_rqleads to a crash. I guess you can reproduce your failure also for the merge case, by generating sequential, direct I/O with queue depth > 1, and of course by enabling failslab.
Quoted 3 lines
If my considerations above are correct, do you want to propose a complete fix yourself?
I had another look into the code. Unfortunately, both bfq_request_merged() and bfq_requests_merged() simply assume that bfq_init_rq() never returns NULL, and don't give me an idea for a path of action if it returns NULL after all. I'll have to pass the problem off to you for a fix.
Thanks,
Guenter