refscale: Fix use of uninitalized wait_queue_head_t
Subsystems (2)
PATCH v2 1/1 refscale: Fix use of uninitalized wait_queue_head_t
It was found that running the refscale test might crash the kernel once in a while with the following error:
[ 8569.952896] BUG: unable to handle page fault for address: ffffffffffffffe8 [ 8569.952900] #PF: supervisor read access in kernel mode [ 8569.952902] #PF: error_code(0x0000) - not-present page [ 8569.952904] PGD c4b048067 P4D c4b049067 PUD c4b04b067 PMD 0 [ 8569.952910] Oops: 0000 [#1] PREEMPT_RT SMP NOPTI [ 8569.952916] Hardware name: Dell Inc. PowerEdge R750/0WMWCR, BIOS 1.2.4 05/28/2021 [ 8569.952917] RIP: 0010:prepare_to_wait_event+0x101/0x190 : [ 8569.952940] Call Trace: [ 8569.952941] <TASK> [ 8569.952944] ref_scale_reader+0x380/0x4a0 [refscale] [ 8569.952959] kthread+0x10e/0x130 [ 8569.952966] ret_from_fork+0x1f/0x30 [ 8569.952973] </TASK>
This is likely caused by the fact that init_waitqueue_head() is called after the ref_scale_reader kthread is created. The kthread can potentially try to use the waitqueue head before it is properly initialized. The crash happened at
static inline void __add_wait_queue(...)
{
:
if (!(wq->flags & WQ_FLAG_PRIORITY)) <=== Crash hereThe offset of flags from list_head entry in wait_queue_entry is -0x18. If reader_tasks[i].wq.head.next is NULL as allocated reader_task structure is zero initialized, the instruction will try to access address 0xffffffffffffffe8 which is the fault address listed above.
Fix this by initializing the waitqueue head first before kthread creation.
--- kernel/rcu/refscale.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/rcu/refscale.c +1 -2
@@ -1107,12 +1107,11 @@ ref_scale_init(void)VERBOSE_SCALEOUT ("Starting %d reader threads" , nreaders);for (i =0 ; i < nreaders; i++) { +init_waitqueue_head (&reader_tasks[i].wq); firsterr =torture_create_kthread (ref_scale_reader, (void *)i, reader_tasks[i].task);if (torture_init_error (firsterr)) goto unwind; - -init_waitqueue_head (&(reader_tasks[i].wq)); }// Main Task
Signature · 1 line
2.31.1
Paul E. McKenney
maintainer Reviewed-by Qiuxu Zhuo Reviewed-by Davidlohr Bueso Acked-by Joel Fernandes (Google) on v2 1/1 AppliedQuoted 33 lines · Waiman Long
It was found that running the refscale test might crash the kernel once in a while with the following error:
[ 8569.952896] BUG: unable to handle page fault for address: ffffffffffffffe8 [ 8569.952900] #PF: supervisor read access in kernel mode [ 8569.952902] #PF: error_code(0x0000) - not-present page [ 8569.952904] PGD c4b048067 P4D c4b049067 PUD c4b04b067 PMD 0 [ 8569.952910] Oops: 0000 [#1] PREEMPT_RT SMP NOPTI [ 8569.952916] Hardware name: Dell Inc. PowerEdge R750/0WMWCR, BIOS 1.2.4 05/28/2021 [ 8569.952917] RIP: 0010:prepare_to_wait_event+0x101/0x190 : [ 8569.952940] Call Trace: [ 8569.952941] <TASK> [ 8569.952944] ref_scale_reader+0x380/0x4a0 [refscale] [ 8569.952959] kthread+0x10e/0x130 [ 8569.952966] ret_from_fork+0x1f/0x30 [ 8569.952973] </TASK>This is likely caused by the fact that
init_waitqueue_head()is called after theref_scale_readerkthread is created. The kthread can potentially try to use the waitqueue head before it is properly initialized. The crash happened atstatic inline void __add_wait_queue(...) { : if (!(wq->flags & WQ_FLAG_PRIORITY)) <=== Crash hereThe offset of flags from
list_headentry inwait_queue_entryis -0x18. If reader_tasks[i].wq.head.next is NULL as allocatedreader_taskstructure is zero initialized, the instruction will try to access address0xffffffffffffffe8which is the fault address listed above.
Quoted 7 lines
Fix this by initializing the waitqueue head first before kthread creation.
Fixes 653ed64b01dc ("refperf: Add a test to measure performance of read-side synchronization")Signed-off-by Waiman Long <[email protected]>Reviewed-by Qiuxu Zhuo <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>
Queued and pushed, thank you all!
As always, I could not resist wordsmithing the commit log, please see below.
Thanx, Paul
------------------------------------------------------------------------
commit 933d3bf8f96d7cedf78081030e004d23aee2b56c
Author: Waiman Long <[email protected]> Date: Fri Jul 7 13:53:55 2023 -0400
refscale: Fix uninitalized use of wait_queue_head_t
Running the refscale test occasionally crashes the kernel with the following error:
[ 8569.952896] BUG: unable to handle page fault for address: ffffffffffffffe8 [ 8569.952900] #PF: supervisor read access in kernel mode
[ 8569.952902] #PF: error_code(0x0000) - not-present page
[ 8569.952904] PGD c4b048067 P4D c4b049067 PUD c4b04b067 PMD 0
[ 8569.952910] Oops: 0000 [#1] PREEMPT_RT SMP NOPTI
[ 8569.952916] Hardware name: Dell Inc. PowerEdge R750/0WMWCR, BIOS 1.2.4 05/28/2021 [ 8569.952917] RIP: 0010:prepare_to_wait_event+0x101/0x190
:
[ 8569.952940] Call Trace:
[ 8569.952941] <TASK>
[ 8569.952944] ref_scale_reader+0x380/0x4a0 [refscale]
[ 8569.952959] kthread+0x10e/0x130
[ 8569.952966] ret_from_fork+0x1f/0x30
[ 8569.952973] </TASK>
The likely cause is that init_waitqueue_head() is called after the call to the torture_create_kthread() function that creates the ref_scale_reader kthread. Although this init_waitqueue_head() call will very likely complete before this kthread is created and starts running, it is possible that the calling kthread will be delayed between the calls to torture_create_kthread() and init_waitqueue_head(). In this case, the new kthread will use the waitqueue head before it is properly initialized, which is not good for the kernel's health and well-being.
The above crash happened here:
static inline void __add_wait_queue(...)
{
:
if (!(wq->flags & WQ_FLAG_PRIORITY)) <=== Crash here
The offset of flags from list_head entry in wait_queue_entry is -0x18. If reader_tasks[i].wq.head.next is NULL as allocated reader_task structure is zero initialized, the instruction will try to access address 0xffffffffffffffe8, which is exactly the fault address listed above.
This commit therefore invokes init_waitqueue_head() before creating the kthread.
kernel/rcu/refscale.c +1 -2
@@ -1107,12 +1107,11 @@ ref_scale_init(void)VERBOSE_SCALEOUT ("Starting %d reader threads" , nreaders);for (i =0 ; i < nreaders; i++) { +init_waitqueue_head (&reader_tasks[i].wq); firsterr =torture_create_kthread (ref_scale_reader, (void *)i, reader_tasks[i].task);if (torture_init_error (firsterr)) goto unwind; - -init_waitqueue_head (&(reader_tasks[i].wq)); }// Main Task
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Waiman Long
It was found that running the refscale test might crash the kernel once in a while with the following error:
[ 8569.952896] BUG: unable to handle page fault for address: ffffffffffffffe8 [ 8569.952900] #PF: supervisor read access in kernel mode [ 8569.952902] #PF: error_code(0x0000) - not-present page [ 8569.952904] PGD c4b048067 P4D c4b049067 PUD c4b04b067 PMD 0 [ 8569.952910] Oops: 0000 [#1] PREEMPT_RT SMP NOPTI [ 8569.952916] Hardware name: Dell Inc. PowerEdge R750/0WMWCR, BIOS 1.2.4 05/28/2021 [ 8569.952917] RIP: 0010:prepare_to_wait_event+0x101/0x190 : [ 8569.952940] Call Trace: [ 8569.952941] <TASK> [ 8569.952944] ref_scale_reader+0x380/0x4a0 [refscale] [ 8569.952959] kthread+0x10e/0x130 [ 8569.952966] ret_from_fork+0x1f/0x30 [ 8569.952973] </TASK>This is likely caused by the fact that
init_waitqueue_head()is called after theref_scale_readerkthread is created. The kthread can potentially try to use the waitqueue head before it is properly initialized. The crash happened atstatic inline void __add_wait_queue(...) { : if (!(wq->flags & WQ_FLAG_PRIORITY)) <=== Crash hereThe offset of flags from
list_headentry inwait_queue_entryis -0x18. If reader_tasks[i].wq.head.next is NULL as allocatedreader_taskstructure is zero initialized, the instruction will try to access address0xffffffffffffffe8which is the fault address listed above.Fix this by initializing the waitqueue head first before kthread creation.
Quoted 8 lines
Quoted 4 lines
Fixes 653ed64b01dc ("refperf: Add a test to measure performance of read-side synchronization")Signed-off-by Waiman Long <[email protected]>Reviewed-by Qiuxu Zhuo <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Queued and pushed, thank you all!
As always, I could not resist wordsmithing the commit log, please see below.
Thanks for the wordsmithing. I don't mind at all.
Cheers,
Longman
Quoted 78 lines
Thanx, Paul
------------------------------------------------------------------------
commit
933d3bf8f96d7cedf78081030e004d23aee2b56cAuthor: Waiman Long <[email protected]> Date: Fri Jul 7 13:53:55 2023 -0400refscale: Fix uninitalized use of
wait_queue_head_tRunning the refscale test occasionally crashes the kernel with the following error:
[ 8569.952896] BUG: unable to handle page fault for address:
ffffffffffffffe8[ 8569.952900] #PF: supervisor read access in kernel mode
[ 8569.952902] #PF:error_code(0x0000)- not-present page
[ 8569.952904] PGDc4b048067P4Dc4b049067PUDc4b04b067PMD 0
[ 8569.952910] Oops: 0000 [#1]PREEMPT_RTSMP NOPTI
[ 8569.952916] Hardware name: Dell Inc. PowerEdge R750/0WMWCR, BIOS 1.2.4 05/28/2021 [ 8569.952917] RIP: 0010:prepare_to_wait_event+0x101/0x190
:
[ 8569.952940] Call Trace:
[ 8569.952941] <TASK>
[ 8569.952944]ref_scale_reader+0x380/0x4a0[refscale]
[ 8569.952959]kthread+0x10e/0x130
[ 8569.952966]ret_from_fork+0x1f/0x30
[ 8569.952973] </TASK>The likely cause is that
init_waitqueue_head()is called after the call to thetorture_create_kthread()function that creates theref_scale_readerkthread. Although thisinit_waitqueue_head()call will very likely complete before this kthread is created and starts running, it is possible that the calling kthread will be delayed between the calls totorture_create_kthread()andinit_waitqueue_head(). In this case, the new kthread will use the waitqueue head before it is properly initialized, which is not good for the kernel's health and well-being.The above crash happened here:
static inline void
__add_wait_queue(...)
{
:
if (!(wq->flags &WQ_FLAG_PRIORITY)) <=== Crash hereThe offset of flags from
list_headentry inwait_queue_entryis -0x18. If reader_tasks[i].wq.head.next is NULL as allocatedreader_taskstructure is zero initialized, the instruction will try to access address0xffffffffffffffe8, which is exactly the fault address listed above.This commit therefore invokes
init_waitqueue_head()before creating the kthread.Fixes 653ed64b01dc ("refperf: Add a test to measure performance of read-side synchronization")Signed-off-by Waiman Long <[email protected]>Reviewed-by Qiuxu Zhuo <[email protected]>Reviewed-by Davidlohr Bueso <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>kernel/rcu/refscale.c +1
-2@@ -1107,12 +1107,11 @@ ref_scale_init(void)VERBOSE_SCALEOUT ("Starting %d reader threads" , nreaders);for (i =0 ; i < nreaders; i++) { +init_waitqueue_head (&reader_tasks[i].wq); firsterr =torture_create_kthread (ref_scale_reader, (void *)i, reader_tasks[i].task);if (torture_init_error (firsterr)) goto unwind; - -init_waitqueue_head (&(reader_tasks[i].wq)); }// Main Task