rcu: Explain why rcu_all_qs() is a stub in preemptible TREE RCU
Frederic Weisbecker
maintainerPATCH 1/2 rcu: Explain why rcu_all_qs() is a stub in preemptible TREE RCU
cond_resched() reports an RCU quiescent state only in non-preemptible TREE RCU implementation. Provide an explanation for the different behaviour in CONFIG_PREEMPT_RCU=y.
--- kernel/sched/core.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/sched/core.c +13 -0
@@ -7780,6 +7780,19 @@ int __sched __cond_resched(void)preempt_schedule_common ();return 1 ; } +/* +* A process spending a long time in the kernel space might +* have too few opportunities to report quiescent states +* when CONFIG_PREEMPT_RCU=n because then the tick can't know +* if it's interrupting an RCU read side critical section. In the +* absence of voluntary sleeps, the last resort resides in tracking +* calls to cond_resched() which always imply quiescent states. +* +* On the other hand, preemptible RCU has a real RCU read side +* tracking that allows the tick for reporting interrupted quiescent +* states or, in the worst case, deferred quiescent states after +* rcu_read_unlock(). +*/ #ifndef CONFIG_PREEMPT_RCU rcu_all_qs ();#endif
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 2/2 rcu: Remove needless preemption disablement in rcu_all_qs()
The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use __this_cpu_write() directly, although that path is mostly used when CONFIG_PREEMPT=n.
--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -871,7 +871,7 @@ void rcu_all_qs(void)preempt_enable ();return ; } -this_cpu_write (rcu_data.rcu_urgent_qs, false); +__this_cpu_write (rcu_data.rcu_urgent_qs, false);if (unlikely (raw_cpu_read (rcu_data.rcu_need_heavy_qs))) {local_irq_save (flags);rcu_momentary_dyntick_idle ();
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerRe: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()
Quoted 18 lines · Frederic Weisbecker
The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use
__this_cpu_write()directly, although that path is mostly used whenCONFIG_PREEMPT=n.Signed-off-by Frederic Weisbecker <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Joel Fernandes <[email protected]>Cc Uladzislau Rezki <[email protected]>Cc Boqun Feng <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree_plugin.h +0
-0
kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -871,7 +871,7 @@ void rcu_all_qs(void)preempt_enable ();return ; } -this_cpu_write (rcu_data.rcu_urgent_qs, false); +__this_cpu_write (rcu_data.rcu_urgent_qs, false);
There's another subtle difference between this_cpu_write() and __this_cpu_write() aside from preempt. this_cpu_write() is also IRQ-safe, while __this_cpu_write() is not.
I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.
Frederic Weisbecker
maintainerRe: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()
Quoted 29 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker
The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use
__this_cpu_write()directly, although that path is mostly used whenCONFIG_PREEMPT=n.Signed-off-by Frederic Weisbecker <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Joel Fernandes <[email protected]>Cc Uladzislau Rezki <[email protected]>Cc Boqun Feng <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree_plugin.h +0
-0kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -871,7 +871,7 @@ void rcu_all_qs(void)preempt_enable ();return ; } -this_cpu_write (rcu_data.rcu_urgent_qs, false); +__this_cpu_write (rcu_data.rcu_urgent_qs, false);There's another subtle difference between
this_cpu_write()and__this_cpu_write()aside from preempt.this_cpu_write()is also IRQ-safe, while__this_cpu_write()is not.
Quoted 3 lines
I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.
You're right, I missed that. I see this rcu_urgent_qs is set by RCU TASKS from rcu_tasks_wait_gp() (did I missed another path?). Not sure if this is called from IRQ nor if it actually matters to protect against IRQs for that single write.
I'm not quite used to rcu_tasks. Paul?
Re: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()
Quoted 30 lines · Frederic Weisbecker
Quoted 29 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker
The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use
__this_cpu_write()directly, although that path is mostly used whenCONFIG_PREEMPT=n.Signed-off-by Frederic Weisbecker <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Joel Fernandes <[email protected]>Cc Uladzislau Rezki <[email protected]>Cc Boqun Feng <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree_plugin.h +0
-0kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -871,7 +871,7 @@ void rcu_all_qs(void)preempt_enable ();return ; } -this_cpu_write (rcu_data.rcu_urgent_qs, false); +__this_cpu_write (rcu_data.rcu_urgent_qs, false);There's another subtle difference between
this_cpu_write()and__this_cpu_write()aside from preempt.this_cpu_write()is also IRQ-safe, while__this_cpu_write()is not.
Quoted 8 lines
Quoted 3 lines
I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.
You're right, I missed that. I see this
rcu_urgent_qsis set by RCU TASKS fromrcu_tasks_wait_gp()(did I missed another path?). Not sure if this is called from IRQ nor if it actually matters to protect against IRQs for that single write.
I think __this_cpu_write() being IRQ-unsafe means it may overwrite percpu writes to other bytes in the same word? Let's say the rcu_urgent_qs is the lowest byte in the word, the pseduo asm code of __this_cpu_write() may be:
__this_cpu_write(ptr, v): long tmp = *ptr; tmp &= ~(0xff); tmp |= v; *ptr = tmp;
and the following sequence introduces an overwrite:
__this_cpu_write(ptr, v): // v is 0, and *ptr is 1 long tmp = *ptr; // tmp is 1 <interrupted> this_cpu_write() // modify another byte of *ptr, make it // 0xff01 <ret from interrupt> tmp &= ~(0xff) // tmp is 0 tmp |=v; // tmp is 0 *ptr = tmp; // *ptr is 0, overwrite a percpu write on // another field.
I know that many archs have byte-wise store, so compilers don't really have the reason to generate code as above, but __this_cpu_write() is just a normal write, nothing prevents this from happenning, unless I'm missing something here?
Regards,
Boqun
Quoted 2 lines
I'm not quite used to
rcu_tasks. Paul?
Paul E. McKenney
maintainerRe: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()
Quoted 64 lines · Boqun Feng
Quoted 30 lines · Frederic Weisbecker
Quoted 29 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker
The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use
__this_cpu_write()directly, although that path is mostly used whenCONFIG_PREEMPT=n.Signed-off-by Frederic Weisbecker <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Joel Fernandes <[email protected]>Cc Uladzislau Rezki <[email protected]>Cc Boqun Feng <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree_plugin.h +0
-0kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -871,7 +871,7 @@ void rcu_all_qs(void)preempt_enable ();return ; } -this_cpu_write (rcu_data.rcu_urgent_qs, false); +__this_cpu_write (rcu_data.rcu_urgent_qs, false);There's another subtle difference between
this_cpu_write()and__this_cpu_write()aside from preempt.this_cpu_write()is also IRQ-safe, while__this_cpu_write()is not.Quoted 8 lines
Quoted 3 lines
I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.
You're right, I missed that. I see this
rcu_urgent_qsis set by RCU TASKS fromrcu_tasks_wait_gp()(did I missed another path?). Not sure if this is called from IRQ nor if it actually matters to protect against IRQs for that single write.I think
__this_cpu_write()being IRQ-unsafe means it may overwrite percpu writes to other bytes in the same word? Let's say thercu_urgent_qsis the lowest byte in the word, the pseduo asm code of__this_cpu_write()may be:__this_cpu_write(ptr, v): long tmp = *ptr; tmp &= ~(0xff); tmp |= v; *ptr = tmp;and the following sequence introduces an overwrite:
__this_cpu_write(ptr, v): // v is 0, and *ptr is 1 long tmp = *ptr; // tmp is 1 <interrupted> this_cpu_write() // modify another byte of *ptr, make it // 0xff01 <ret from interrupt> tmp &= ~(0xff) // tmp is 0 tmp |=v; // tmp is 0 *ptr = tmp; // *ptr is 0, overwrite a percpu write on // another field.
Quoted 4 lines
I know that many archs have byte-wise store, so compilers don't really have the reason to generate code as above, but
__this_cpu_write()is just a normal write, nothing prevents this from happenning, unless I'm missing something here?
There can indeed be writes to .rcu_urgent_qs from interrupt handlers, for example in CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels from rcu_read_unlock(). More conventionally, the RCU_SOFTIRQ handler can run on the back of an interrupts, and can invoke rcu_check_quiescent_state(), which invokes rcu_report_qs_rdp(), which invokes rcu_disable_urgency_upon_qs(), which writes to ->rcu_urgent_qs.
RCU takes a strict view of data races, so this wants the existing this_cpu_write().
However, RCU very likely has this_cpu_write() calls that should instead be __this_cpu_write() calls and vice versa, so please do continue treating any that you see with an appropriate level of suspicion.
Thanx, Paul
Quoted 5 lines
Regards,
BoqunQuoted 2 lines
I'm not quite used to
rcu_tasks. Paul?
Quoted 29 lines · Frederic Weisbecker
cond_resched()reports an RCU quiescent state only in non-preemptible TREE RCU implementation. Provide an explanation for the different behaviour inCONFIG_PREEMPT_RCU=y.Signed-off-by Frederic Weisbecker <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Joel Fernandes <[email protected]>Cc Uladzislau Rezki <[email protected]>Cc Boqun Feng <[email protected]>Cc Peter Zijlstra (Intel) <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/sched/core.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)kernel/sched/core.c +6
-0@@ -7780,6 +7780,19 @@ int __sched __cond_resched(void)preempt_schedule_common ();return 1 ; } +/* +* A process spending a long time in the kernel space might +* have too few opportunities to report quiescent states +* when CONFIG_PREEMPT_RCU=n because then the tick can't know +* if it's interrupting an RCU read side critical section. In the +* absence of voluntary sleeps, the last resort resides in tracking
kernel/sched/core.c int __sched __cond_resched(void) · line 7798
Quoted 10 lines
kernel/sched/core.c +7
-0+* calls to cond_resched() which always imply quiescent states. +* +* On the other hand, preemptible RCU has a real RCU read side +* tracking that allows the tick for reporting interrupted quiescent +* states or, in the worst case, deferred quiescent states after +* rcu_read_unlock(). +*/ #ifndef CONFIG_PREEMPT_RCU rcu_all_qs ();#endif
Looks like a good addition! I have wordsmithed the comment and commit log a bit. If Peter wants to take either version of this:
In the meantime, I have queued it for v5.15.
Thanx, Paul
------------------------------------------------------------------------
commit 41a95363ac9f020cc0e4fcc4b73015c60b6620f0
Author: Frederic Weisbecker <[email protected]> Date: Tue Jul 6 01:43:43 2021 +0200
rcu: Explain why rcu_all_qs() is a stub in preemptible TREE RCU
The cond_resched() function reports an RCU quiescent state only in non-preemptible TREE RCU implementation. This commit therefore adds a comment explaining why cond_resched() does nothing in preemptible kernels.
kernel/sched/core.c +11 -0
@@ -7780,6 +7780,17 @@ int __sched __cond_resched(void)preempt_schedule_common ();return 1 ; } +/* +* In preemptible kernels, ->rcu_read_lock_nesting tells the tick +* whether the current CPU is in an RCU read-side critical section, +* so the tick can report quiescent states even for CPUs looping +* in kernel context. In contrast, in non-preemptible kernels, +* RCU readers leave no in-memory hints, which means that CPU-bound +* processes executing in kernel context might never report an +* RCU quiescent state. Therefore, the following code causes +* cond_resched() to report a quiescent state, but only when RCU +* is in urgent need of one. +*/ #ifndef CONFIG_PREEMPT_RCU rcu_all_qs ();#endif