Skip to contentSkip to conversation

rcu: Explain why rcu_all_qs() is a stub in preemptible TREE RCU

PATCH In review Frederic Weisbecker 4 participants 7 messages Jul 5, 2021 – Jul 6, 2021

Subsystems (2)
lore
Share & tools
lore

Frederic Weisbecker

author maintainer
#0 lore raw

PATCH 1/2 rcu: Explain why rcu_all_qs() is a stub in preemptible TREE RCU

cond_resched() reports an RCU quiescent state only in non-preemptible TREE RCU implementation. Provide an explanation for the different behaviour in CONFIG_PREEMPT_RCU=y.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Peter Zijlstra (Intel) <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/sched/core.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/sched/core.c +13 -0
index cf16f8fda9a6..db374cb38eb2 100644
@@ -7780,6 +7780,19 @@ int __sched __cond_resched(void)
 		preempt_schedule_common();
 		return 1;
 	}
+	/*
+	 * A process spending a long time in the kernel space might
+	 * have too few opportunities to report quiescent states
+	 * when CONFIG_PREEMPT_RCU=n because then the tick can't know
+	 * if it's interrupting an RCU read side critical section. In the
+	 * absence of voluntary sleeps, the last resort resides in tracking
+	 * calls to cond_resched() which always imply quiescent states.
+	 *
+	 * On the other hand, preemptible RCU has a real RCU read side
+	 * tracking that allows the tick for reporting interrupted quiescent
+	 * states or, in the worst case, deferred quiescent states after
+	 * rcu_read_unlock().
+	 */
 #ifndef CONFIG_PREEMPT_RCU
 	rcu_all_qs();
 #endif
Signature · 1 line
2.25.1
author maintainer
#1 lore raw

PATCH 2/2 rcu: Remove needless preemption disablement in rcu_all_qs()

The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use __this_cpu_write() directly, although that path is mostly used when CONFIG_PREEMPT=n.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
---
 kernel/rcu/tree_plugin.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
index 27b74352cccf..38b3d01424d7 100644
@@ -871,7 +871,7 @@ void rcu_all_qs(void)
 		preempt_enable();
 		return;
 	}
-	this_cpu_write(rcu_data.rcu_urgent_qs, false);
+	__this_cpu_write(rcu_data.rcu_urgent_qs, false);
 	if (unlikely(raw_cpu_read(rcu_data.rcu_need_heavy_qs))) {
 		local_irq_save(flags);
 		rcu_momentary_dyntick_idle();
Signature · 1 line
2.25.1
maintainer
#2 lore raw

Re: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()

Quoted 18 lines · Frederic Weisbecker

The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use __this_cpu_write() directly, although that path is mostly used when CONFIG_PREEMPT=n.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
---
 kernel/rcu/tree_plugin.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +0 -0
index 27b74352cccf..38b3d01424d7 100644

kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874

Quoted 5 lines
kernel/rcu/tree_plugin.h +1 -1
@@ -871,7 +871,7 @@ void rcu_all_qs(void)
 		preempt_enable();
 		return;
 	}
-	this_cpu_write(rcu_data.rcu_urgent_qs, false);
+	__this_cpu_write(rcu_data.rcu_urgent_qs, false);

There's another subtle difference between this_cpu_write() and __this_cpu_write() aside from preempt. this_cpu_write() is also IRQ-safe, while __this_cpu_write() is not.

I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.

author maintainer
#3 lore raw

Re: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()

Quoted 29 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker

The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use __this_cpu_write() directly, although that path is mostly used when CONFIG_PREEMPT=n.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
---
 kernel/rcu/tree_plugin.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +0 -0
index 27b74352cccf..38b3d01424d7 100644

kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874

Quoted 5 lines
kernel/rcu/tree_plugin.h +1 -1
@@ -871,7 +871,7 @@ void rcu_all_qs(void)
 		preempt_enable();
 		return;
 	}
-	this_cpu_write(rcu_data.rcu_urgent_qs, false);
+	__this_cpu_write(rcu_data.rcu_urgent_qs, false);

There's another subtle difference between this_cpu_write() and __this_cpu_write() aside from preempt. this_cpu_write() is also IRQ-safe, while __this_cpu_write() is not.

Quoted 3 lines

I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.

You're right, I missed that. I see this rcu_urgent_qs is set by RCU TASKS from rcu_tasks_wait_gp() (did I missed another path?). Not sure if this is called from IRQ nor if it actually matters to protect against IRQs for that single write.

I'm not quite used to rcu_tasks. Paul?

#4 lore raw

Re: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()

Quoted 30 lines · Frederic Weisbecker
Quoted 29 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker

The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use __this_cpu_write() directly, although that path is mostly used when CONFIG_PREEMPT=n.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
---
 kernel/rcu/tree_plugin.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +0 -0
index 27b74352cccf..38b3d01424d7 100644

kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874

Quoted 5 lines
kernel/rcu/tree_plugin.h +1 -1
@@ -871,7 +871,7 @@ void rcu_all_qs(void)
 		preempt_enable();
 		return;
 	}
-	this_cpu_write(rcu_data.rcu_urgent_qs, false);
+	__this_cpu_write(rcu_data.rcu_urgent_qs, false);

There's another subtle difference between this_cpu_write() and __this_cpu_write() aside from preempt. this_cpu_write() is also IRQ-safe, while __this_cpu_write() is not.

Quoted 8 lines
Quoted 3 lines

I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.

You're right, I missed that. I see this rcu_urgent_qs is set by RCU TASKS from rcu_tasks_wait_gp() (did I missed another path?). Not sure if this is called from IRQ nor if it actually matters to protect against IRQs for that single write.

I think __this_cpu_write() being IRQ-unsafe means it may overwrite percpu writes to other bytes in the same word? Let's say the rcu_urgent_qs is the lowest byte in the word, the pseduo asm code of __this_cpu_write() may be:

	__this_cpu_write(ptr, v):
		long tmp = *ptr;
		tmp &= ~(0xff);
		tmp |= v;
		*ptr = tmp;

and the following sequence introduces an overwrite:

	__this_cpu_write(ptr, v): // v is 0, and *ptr is 1
		long tmp = *ptr; // tmp is 1
		<interrupted>
		this_cpu_write() // modify another byte of *ptr, make it
				 // 0xff01
		<ret from interrupt>
		tmp &= ~(0xff) // tmp is 0
		tmp |=v;       // tmp is 0
		*ptr = tmp;    // *ptr is 0, overwrite a percpu write on
			       // another field.

I know that many archs have byte-wise store, so compilers don't really have the reason to generate code as above, but __this_cpu_write() is just a normal write, nothing prevents this from happenning, unless I'm missing something here?

Regards,
Boqun

Quoted 2 lines

I'm not quite used to rcu_tasks. Paul?

maintainer
#5 lore raw

Re: [PATCH 2/2] rcu: Remove needless preemption disablement in rcu_all_qs()

Quoted 64 lines · Boqun Feng
Quoted 30 lines · Frederic Weisbecker
Quoted 29 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker

The preemption is already disabled when we write rcu_data.rcu_urgent_qs. We can use __this_cpu_write() directly, although that path is mostly used when CONFIG_PREEMPT=n.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
---
 kernel/rcu/tree_plugin.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +0 -0
index 27b74352cccf..38b3d01424d7 100644

kernel/rcu/tree_plugin.h void rcu_all_qs(void) · line 874

Quoted 5 lines
kernel/rcu/tree_plugin.h +1 -1
@@ -871,7 +871,7 @@ void rcu_all_qs(void)
 		preempt_enable();
 		return;
 	}
-	this_cpu_write(rcu_data.rcu_urgent_qs, false);
+	__this_cpu_write(rcu_data.rcu_urgent_qs, false);

There's another subtle difference between this_cpu_write() and __this_cpu_write() aside from preempt. this_cpu_write() is also IRQ-safe, while __this_cpu_write() is not.

Quoted 8 lines
Quoted 3 lines

I've not looked at the usage here to see if that is relevant, but the Changelog only mentioned the preempt side of things, and that argument is incomplete in general.

You're right, I missed that. I see this rcu_urgent_qs is set by RCU TASKS from rcu_tasks_wait_gp() (did I missed another path?). Not sure if this is called from IRQ nor if it actually matters to protect against IRQs for that single write.

I think __this_cpu_write() being IRQ-unsafe means it may overwrite percpu writes to other bytes in the same word? Let's say the rcu_urgent_qs is the lowest byte in the word, the pseduo asm code of __this_cpu_write() may be:

	__this_cpu_write(ptr, v):
		long tmp = *ptr;
		tmp &= ~(0xff);
		tmp |= v;
		*ptr = tmp;

and the following sequence introduces an overwrite:

	__this_cpu_write(ptr, v): // v is 0, and *ptr is 1
		long tmp = *ptr; // tmp is 1
		<interrupted>
		this_cpu_write() // modify another byte of *ptr, make it
				 // 0xff01
		<ret from interrupt>
		tmp &= ~(0xff) // tmp is 0
		tmp |=v;       // tmp is 0
		*ptr = tmp;    // *ptr is 0, overwrite a percpu write on
			       // another field.
Quoted 4 lines

I know that many archs have byte-wise store, so compilers don't really have the reason to generate code as above, but __this_cpu_write() is just a normal write, nothing prevents this from happenning, unless I'm missing something here?

There can indeed be writes to .rcu_urgent_qs from interrupt handlers, for example in CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels from rcu_read_unlock(). More conventionally, the RCU_SOFTIRQ handler can run on the back of an interrupts, and can invoke rcu_check_quiescent_state(), which invokes rcu_report_qs_rdp(), which invokes rcu_disable_urgency_upon_qs(), which writes to ->rcu_urgent_qs.

RCU takes a strict view of data races, so this wants the existing this_cpu_write().

However, RCU very likely has this_cpu_write() calls that should instead be __this_cpu_write() calls and vice versa, so please do continue treating any that you see with an appropriate level of suspicion.

Thanx, Paul

Quoted 5 lines

Regards,
Boqun

Quoted 2 lines

I'm not quite used to rcu_tasks. Paul?

maintainer Acked-by on v1 1/2
#6 lore raw
In reply to Frederic Weisbecker
Quoted 29 lines · Frederic Weisbecker

cond_resched() reports an RCU quiescent state only in non-preemptible TREE RCU implementation. Provide an explanation for the different behaviour in CONFIG_PREEMPT_RCU=y.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Peter Zijlstra (Intel) <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/sched/core.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/sched/core.c +6 -0
index cf16f8fda9a6..db374cb38eb2 100644
@@ -7780,6 +7780,19 @@ int __sched __cond_resched(void)
 		preempt_schedule_common();
 		return 1;
 	}
+	/*
+	 * A process spending a long time in the kernel space might
+	 * have too few opportunities to report quiescent states
+	 * when CONFIG_PREEMPT_RCU=n because then the tick can't know
+	 * if it's interrupting an RCU read side critical section. In the
+	 * absence of voluntary sleeps, the last resort resides in tracking

kernel/sched/core.c int __sched __cond_resched(void) · line 7798

Quoted 10 lines
kernel/sched/core.c +7 -0
+	 * calls to cond_resched() which always imply quiescent states.
+	 *
+	 * On the other hand, preemptible RCU has a real RCU read side
+	 * tracking that allows the tick for reporting interrupted quiescent
+	 * states or, in the worst case, deferred quiescent states after
+	 * rcu_read_unlock().
+	 */
 #ifndef CONFIG_PREEMPT_RCU
 	rcu_all_qs();
 #endif

Looks like a good addition! I have wordsmithed the comment and commit log a bit. If Peter wants to take either version of this:

In the meantime, I have queued it for v5.15.

Thanx, Paul

------------------------------------------------------------------------

commit 41a95363ac9f020cc0e4fcc4b73015c60b6620f0

Author: Frederic Weisbecker <[email protected]>
Date:   Tue Jul 6 01:43:43 2021 +0200

rcu: Explain why rcu_all_qs() is a stub in preemptible TREE RCU

The cond_resched() function reports an RCU quiescent state only in non-preemptible TREE RCU implementation. This commit therefore adds a comment explaining why cond_resched() does nothing in preemptible kernels.

Cc Neeraj Upadhyay <[email protected]>
Cc Joel Fernandes <[email protected]>
Cc Uladzislau Rezki <[email protected]>
Cc Boqun Feng <[email protected]>
Cc Peter Zijlstra (Intel) <[email protected]>
Cc Ingo Molnar <[email protected]>
kernel/sched/core.c +11 -0
index cf16f8fda9a6..eae24fdf3820 100644
@@ -7780,6 +7780,17 @@ int __sched __cond_resched(void)
 		preempt_schedule_common();
 		return 1;
 	}
+	/*
+	 * In preemptible kernels, ->rcu_read_lock_nesting tells the tick
+	 * whether the current CPU is in an RCU read-side critical section,
+	 * so the tick can report quiescent states even for CPUs looping
+	 * in kernel context.  In contrast, in non-preemptible kernels,
+	 * RCU readers leave no in-memory hints, which means that CPU-bound
+	 * processes executing in kernel context might never report an
+	 * RCU quiescent state.  Therefore, the following code causes
+	 * cond_resched() to report a quiescent state, but only when RCU
+	 * is in urgent need of one.
+	 */
 #ifndef CONFIG_PREEMPT_RCU
 	rcu_all_qs();
 #endif