rcu: Return early if callback is not specified
Subsystems (2)
Uladzislau Rezki (Sony)
maintainerPATCH 1/3 rcu: Return early if callback is not specified
Currently the call_rcu() API does not check whether a callback pointer is NULL. If NULL is passed, rcu_core() will try to invoke it, resulting in NULL pointer dereference and a kernel crash.
To prevent this and improve debuggability, this patch adds a check for NULL and emits a kernel stack trace to help identify a faulty caller.
--- kernel/rcu/tree.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/rcu/tree.c +4 -0
@@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)/* Misaligned rcu_head! */ WARN_ON_ONCE ((unsigned long )head & (sizeof (void *) -1 )); +/* Avoid NULL dereference if callback is NULL. */ +if (WARN_ON_ONCE (!func)) +return ; +if (debug_rcu_head_queue (head)) {/* * Probable double call_rcu(), so leak the callback.
Signature · 1 line
2.39.5
Uladzislau Rezki (Sony)
maintainerPATCH 2/3 rcu: Enable rcu_normal_wake_from_gp on small systems
Automatically enable the rcu_normal_wake_from_gp parameter on systems with a small number of CPUs. The activation threshold is set to 16 CPUs.
This helps to reduce a latency of normal synchronize_rcu() API by waking up GP-waiters earlier and decoupling synchronize_rcu() callers from regular callback handling.
A benchmark running 64 parallel jobs invoking synchronize_rcu() demonstrates a notable latency reduction with the setting enabled.
Latency distribution (microseconds):
<default>
0 - 9999 : 1 10000 - 19999 : 4 20000 - 29999 : 399 30000 - 39999 : 3197 40000 - 49999 : 10428 50000 - 59999 : 17363 60000 - 69999 : 15529 70000 - 79999 : 9287 80000 - 89999 : 4249 90000 - 99999 : 1915
100000 - 109999 : 922
110000 - 119999 : 390
120000 - 129999 : 187
...
<default>
<rcu_normal_wake_from_gp>
0 - 9999 : 1 10000 - 19999 : 234 20000 - 29999 : 6678 30000 - 39999 : 33463 40000 - 49999 : 20669 50000 - 59999 : 2766 60000 - 69999 : 183
...
<rcu_normal_wake_from_gp>
--- kernel/rcu/tree.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +6 -1
@@ -1625,7 +1625,9 @@ static void rcu_sr_put_wait_head(struct llist_node *node)atomic_set_release (&sr_wn->inuse,0 ); } -/* Disabled by default. */ +/* Enable rcu_normal_wake_from_gp automatically on small systems. */ +#define WAKE_FROM_GP_CPU_THRESHOLD 16 +static int rcu_normal_wake_from_gp;module_param (rcu_normal_wake_from_gp, int,0644 );static struct workqueue_struct *sync_wq;
@@ -4847,6 +4849,9 @@ void __init rcu_init(void) sync_wq =alloc_workqueue ("sync_wq" ,WQ_MEM_RECLAIM ,0 );WARN_ON (!sync_wq); +if (num_possible_cpus () <=WAKE_FROM_GP_CPU_THRESHOLD ) +WRITE_ONCE (rcu_normal_wake_from_gp,1 ); +/* Fill in default value for rcutree.qovld boot parameter. */ /* -After- the rcu_node ->lock fields are initialized! */ if (qovld <0 )
Signature · 1 line
2.39.5
Joel Fernandes
maintainerRe: [PATCH 2/3] rcu: Enable rcu_normal_wake_from_gp on small systems
Quoted 62 lines · Uladzislau Rezki (Sony)
Automatically enable the
rcu_normal_wake_from_gpparameter on systems with a small number of CPUs. The activation threshold is set to 16 CPUs.This helps to reduce a latency of normal
synchronize_rcu()API by waking up GP-waiters earlier and decouplingsynchronize_rcu()callers from regular callback handling.A benchmark running 64 parallel jobs invoking
synchronize_rcu()demonstrates a notable latency reduction with the setting enabled.Latency distribution (microseconds):
<default>
0 - 9999 : 1 10000 - 19999 : 4 20000 - 29999 : 399 30000 - 39999 : 3197 40000 - 49999 : 10428 50000 - 59999 : 17363 60000 - 69999 : 15529 70000 - 79999 : 9287 80000 - 89999 : 4249 90000 - 99999 : 1915100000 - 109999 : 922
110000 - 119999 : 390
120000 - 129999 : 187
...
<default><
rcu_normal_wake_from_gp>0 - 9999 : 1 10000 - 19999 : 234 20000 - 29999 : 6678 30000 - 39999 : 33463 40000 - 49999 : 20669 50000 - 59999 : 2766 60000 - 69999 : 183...
<rcu_normal_wake_from_gp>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- kernel/rcu/tree.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)kernel/rcu/tree.c +3
-1@@ -1625,7 +1625,9 @@ static void rcu_sr_put_wait_head(struct llist_node *node)atomic_set_release (&sr_wn->inuse,0 ); } -/* Disabled by default. */ +/* Enable rcu_normal_wake_from_gp automatically on small systems. */ +#define WAKE_FROM_GP_CPU_THRESHOLD 16 +static int rcu_normal_wake_from_gp;module_param (rcu_normal_wake_from_gp, int,0644 );static struct workqueue_struct *sync_wq;
kernel/rcu/tree.c void __init rcu_init(void) · line 4854
Quoted 6 lines
kernel/rcu/tree.c +3
-0@@ -4847,6 +4849,9 @@ void __init rcu_init(void) sync_wq =alloc_workqueue ("sync_wq" ,WQ_MEM_RECLAIM ,0 );WARN_ON (!sync_wq); +if (num_possible_cpus () <=WAKE_FROM_GP_CPU_THRESHOLD ) +WRITE_ONCE (rcu_normal_wake_from_gp,1 ); +
I think this will get weird if user explictly specifies rcutree.rcu_normal_wake_from_gp=0 ? Then we're silently overriding the param. Maybe, initialize it to -1, and then if it was set 0 by user, don't override it. But otherwise, set it to 1. Per your third patch, '1' is a default, not a forced value.
thanks,
- Joel
Uladzislau Rezki
maintainerRe: [PATCH 2/3] rcu: Enable rcu_normal_wake_from_gp on small systems
Quoted 68 lines · Joel Fernandes
Quoted 65 lines · Uladzislau Rezki (Sony)
Automatically enable the
rcu_normal_wake_from_gpparameter on systems with a small number of CPUs. The activation threshold is set to 16 CPUs.This helps to reduce a latency of normal
synchronize_rcu()API by waking up GP-waiters earlier and decouplingsynchronize_rcu()callers from regular callback handling.A benchmark running 64 parallel jobs invoking
synchronize_rcu()demonstrates a notable latency reduction with the setting enabled.Latency distribution (microseconds):
<default>
0 - 9999 : 1 10000 - 19999 : 4 20000 - 29999 : 399 30000 - 39999 : 3197 40000 - 49999 : 10428 50000 - 59999 : 17363 60000 - 69999 : 15529 70000 - 79999 : 9287 80000 - 89999 : 4249 90000 - 99999 : 1915100000 - 109999 : 922
110000 - 119999 : 390
120000 - 129999 : 187
...
<default><
rcu_normal_wake_from_gp>0 - 9999 : 1 10000 - 19999 : 234 20000 - 29999 : 6678 30000 - 39999 : 33463 40000 - 49999 : 20669 50000 - 59999 : 2766 60000 - 69999 : 183...
<rcu_normal_wake_from_gp>Signed-off-by Uladzislau Rezki (Sony) <[email protected]>--- kernel/rcu/tree.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)kernel/rcu/tree.c +3
-1@@ -1625,7 +1625,9 @@ static void rcu_sr_put_wait_head(struct llist_node *node)atomic_set_release (&sr_wn->inuse,0 ); } -/* Disabled by default. */ +/* Enable rcu_normal_wake_from_gp automatically on small systems. */ +#define WAKE_FROM_GP_CPU_THRESHOLD 16 +static int rcu_normal_wake_from_gp;module_param (rcu_normal_wake_from_gp, int,0644 );static struct workqueue_struct *sync_wq;@@ -4847,6 +4849,9 @@ void __init rcu_init(void) sync_wq =alloc_workqueue ("sync_wq" ,WQ_MEM_RECLAIM ,0 );WARN_ON (!sync_wq);
Quoted 9 lines
kernel/rcu/tree.c void __init rcu_init(void) · line 4854
Quoted 3 lines
kernel/rcu/tree.c +3
-0+if (num_possible_cpus () <=WAKE_FROM_GP_CPU_THRESHOLD ) +WRITE_ONCE (rcu_normal_wake_from_gp,1 ); +I think this will get weird if user explictly specifies rcutree.rcu_normal_wake_from_gp=0 ? Then we're silently overriding the param. Maybe, initialize it to -1, and then if it was set 0 by user, don't override it. But otherwise, set it to 1. Per your third patch, '1' is a default, not a forced value.
Right. That case should be fixed.
--
Uladzislau Rezki
Uladzislau Rezki (Sony)
maintainerPATCH 3/3 Documentation/kernel-parameters: Update rcu_normal_wake_from_gp doc
In reply to Uladzislau Rezki (Sony)Update the documentation about rcu_normal_wake_from_gp parameter.
--- Documentation/admin-guide/kernel-parameters.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
@@ -5485,7 +5485,7 @@ echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1" - Default is 0. + Default is 1 if num_possible_cpus() <= 16 otherwise 0. rcuscale.gp_async= [KNL] Measure performance of asynchronous
Signature · 1 line
2.39.5
Quoted 9 lines · Uladzislau Rezki (Sony)
Currently the
call_rcu()API does not check whether a callback pointer is NULL. If NULL is passed,rcu_core()will try to invoke it, resulting in NULL pointer dereference and a kernel crash.To prevent this and improve debuggability, this patch adds a check for NULL and emits a kernel stack trace to help identify a faulty caller.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>
I will add this first one (only this one since we're discussing the others) to a new rcu/fixes-for-6.16 branch, but let me know if any objections.
Will push that branch out during -rc2 or -rc3 after sufficient testing.
thanks,
- Joel
Quoted 19 lines
--- kernel/rcu/tree.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/rcu/tree.c +4
-0@@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)/* Misaligned rcu_head! */ WARN_ON_ONCE ((unsigned long )head & (sizeof (void *) -1 )); +/* Avoid NULL dereference if callback is NULL. */ +if (WARN_ON_ONCE (!func)) +return ; +if (debug_rcu_head_queue (head)) {/* * Probable double call_rcu(), so leak the callback.
Uladzislau Rezki
maintainerQuoted 11 lines · Joel Fernandes
Quoted 8 lines · Uladzislau Rezki (Sony)
Currently the
call_rcu()API does not check whether a callback pointer is NULL. If NULL is passed,rcu_core()will try to invoke it, resulting in NULL pointer dereference and a kernel crash.To prevent this and improve debuggability, this patch adds a check for NULL and emits a kernel stack trace to help identify a faulty caller.
Quoted 9 lines
Quoted 1 line
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>Reviewed-by Joel Fernandes <[email protected]>I will add this first one (only this one since we're discussing the others) to a new rcu/fixes-for-6.16 branch, but let me know if any objections.
Will push that branch out during -rc2 or -rc3 after sufficient testing.
Yep, that sounds good to me about rc-2/3 releases.
--
Uladzislau Rezki
Boqun Feng
Reviewed-byQuoted 5 lines · Joel Fernandes
Quoted 4 lines · Uladzislau Rezki (Sony)
Currently the
call_rcu()API does not check whether a callback pointer is NULL. If NULL is passed,rcu_core()will try to invoke it, resulting in NULL pointer dereference and a kernel crash.
Quoted 8 lines
Quoted 5 lines
To prevent this and improve debuggability, this patch adds a check for NULL and emits a kernel stack trace to help identify a faulty caller.
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>Reviewed-by Joel Fernandes <[email protected]>
Quoted 3 lines
I will add this first one (only this one since we're discussing the others) to a new rcu/fixes-for-6.16 branch, but let me know if any objections.
Not sure it’s urgent enough given the current evidence.
Regards,
Boqun
Quoted 25 lines
Will push that branch out during -rc2 or -rc3 after sufficient testing.
thanks,
- Joel
Quoted 19 lines
--- kernel/rcu/tree.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/rcu/tree.c +4
-0@@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)/* Misaligned rcu_head! */ WARN_ON_ONCE ((unsigned long )head & (sizeof (void *) -1 )); +/* Avoid NULL dereference if callback is NULL. */ +if (WARN_ON_ONCE (!func)) +return ; +if (debug_rcu_head_queue (head)) {/* * Probable double call_rcu(), so leak the callback.
Quoted 10 lines · Boqun Feng
Quoted 9 lines · Joel Fernandes
Quoted 8 lines · Uladzislau Rezki (Sony)
Currently the
call_rcu()API does not check whether a callback pointer is NULL. If NULL is passed,rcu_core()will try to invoke it, resulting in NULL pointer dereference and a kernel crash.To prevent this and improve debuggability, this patch adds a check for NULL and emits a kernel stack trace to help identify a faulty caller.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>Reviewed-by Joel Fernandes <[email protected]>Reviewed-by Boqun Feng <[email protected]>
(Accidentally sent from another email account, FWIW, this is Boqun Feng and I approve this message, sorry couldn’t resist :))
Regards,
Boqun
Quoted 34 lines
Quoted 3 lines
I will add this first one (only this one since we're discussing the others) to a new rcu/fixes-for-6.16 branch, but let me know if any objections.
Not sure it’s urgent enough given the current evidence.
Regards,
BoqunQuoted 25 lines
Will push that branch out during -rc2 or -rc3 after sufficient testing.
thanks,
- Joel
Quoted 19 lines
--- kernel/rcu/tree.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/rcu/tree.c +4
-0@@ -3072,6 +3072,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)/* Misaligned rcu_head! */ WARN_ON_ONCE ((unsigned long )head & (sizeof (void *) -1 )); +/* Avoid NULL dereference if callback is NULL. */ +if (WARN_ON_ONCE (!func)) +return ; +if (debug_rcu_head_queue (head)) {/* * Probable double call_rcu(), so leak the callback.
Uladzislau Rezki
maintainerQuoted 12 lines · Boqun Feng
Quoted 9 lines · Joel Fernandes
Quoted 8 lines · Uladzislau Rezki (Sony)
Currently the
call_rcu()API does not check whether a callback pointer is NULL. If NULL is passed,rcu_core()will try to invoke it, resulting in NULL pointer dereference and a kernel crash.To prevent this and improve debuggability, this patch adds a check for NULL and emits a kernel stack trace to help identify a faulty caller.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>Reviewed-by Joel Fernandes <[email protected]>Reviewed-by Boqun Feng <[email protected]>
Thank you for review, Boqun!
Quoted 6 lines
Quoted 3 lines
I will add this first one (only this one since we're discussing the others) to a new rcu/fixes-for-6.16 branch, but let me know if any objections.
Not sure it’s urgent enough given the current evidence.
Let me clarify it a bit. My point is that, we get a kernel crash in a subsystem we are responsible for, i.e. no matter if there are faulty users of it(third party applications), the point is users can crash it.
The kernel robot reports it and it is already a strong indication that the subsystem is not hardened against invalid inputs:
"BUG: unable to handle kernel NULL pointer dereference in rcu_core (3)"
so this in the rcu_core() which is part of RCU.
But, anyway Joel should decide. I shared my opinion :)
--
Uladzislau Rezki
Quoted 26 lines · Uladzislau Rezki
Quoted 12 lines · Boqun Feng
Quoted 9 lines · Joel Fernandes
Quoted 8 lines · Uladzislau Rezki (Sony)
Currently the
call_rcu()API does not check whether a callback pointer is NULL. If NULL is passed,rcu_core()will try to invoke it, resulting in NULL pointer dereference and a kernel crash.To prevent this and improve debuggability, this patch adds a check for NULL and emits a kernel stack trace to help identify a faulty caller.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Uladzislau Rezki (Sony) <[email protected]>Reviewed-by Joel Fernandes <[email protected]>Reviewed-by Boqun Feng <[email protected]>Thank you for review, Boqun!
Quoted 4 lines
Quoted 3 lines
I will add this first one (only this one since we're discussing the others) to a new rcu/fixes-for-6.16 branch, but let me know if any objections.
Quoted 6 lines
Quoted 2 lines
Not sure it´s urgent enough given the current evidence.
Let me clarify it a bit. My point is that, we get a kernel crash in a subsystem we are responsible for, i.e. no matter if there are faulty users of it(third party applications), the point is users can crash it.
Fair enough.
Quoted 9 lines
The kernel robot reports it and it is already a strong indication that the subsystem is not hardened against invalid inputs:
"BUG: unable to handle kernel NULL pointer dereference in
rcu_core(3)"so this in the
rcu_core()which is part of RCU.But, anyway Joel should decide. I shared my opinion :)
Of course, my point is that the urgency is not high enough so we have to put it in rcu/fixes, but it's a fix, and if Joel had the time to do it, feel free. Joel's decision.
Regards,
Boqun
Quoted 2 lines
--
Uladzislau Rezki
Joel Fernandes
maintainerOn 6/12/2025 5:03 PM, Boqun Feng wrote:
[..]
Quoted 6 lines
Quoted 6 lines
The kernel robot reports it and it is already a strong indication that the subsystem is not hardened against invalid inputs:
"BUG: unable to handle kernel NULL pointer dereference in
rcu_core(3)"
Quoted 9 lines
Quoted 4 lines
so this in the
rcu_core()which is part of RCU.But, anyway Joel should decide. I shared my opinion :)
Of course, my point is that the urgency is not high enough so we have to put it in rcu/fixes, but it's a fix, and if Joel had the time to do it, feel free. Joel's decision.
Yeah I feel Vlad's fix for a crash is important so I'll send this up to Linus for 6.16 after some testing this week. For the other 2 patches, since that is triggered by a trace point, I'll just let Neeraj take them for 6.17.
thanks,
- Joel