Skip to contentSkip to conversation

rcu: Allow a CPU to leave and reenter NOCB state

PATCH In review Frederic Weisbecker 3 participants 57 messages May 13, 2020 – Jun 11, 2020

Subsystems (2)
lore
Share & tools
lore

Frederic Weisbecker

author maintainer
#0 lore raw

cover rcu: Allow a CPU to leave and reenter NOCB state

This is a necessary step toward making nohz_full controllable through cpuset. Next step should be to allow a CPU to be nocb even if it wasn't part of the nocb set on boot.

The core design of this set is mostly based on suggestions from Paul of course.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
rcu/nohz

HEAD: 31cb4ee9da4e9cc6314498ff22d83f0d872b1a88

Thanks,
Frederic
---

Frederic Weisbecker (10):
rcu: Directly lock rdp->nocb_lock on nocb code entrypoints
rcu: Use direct rdp->nocb_lock operations on local calls
rcu: Make locking explicit in do_nocb_deferred_wakeup_common()
rcu: Implement rcu_segcblist_is_offloaded() config dependent
rcu: Remove useless conditional nocb unlock
rcu: Make nocb_cb kthread parkable
rcu: Temporarily assume that nohz full CPUs might not be NOCB
rcu: Allow to deactivate nocb on a CPU
rcu: Allow to re-offload a CPU that used to be nocb
rcu: Nocb (de)activate through sysfs

 include/linux/rcu_segcblist.h |   2 +
 include/linux/rcupdate.h      |   4 ++
 kernel/cpu.c                  |  23 +++++++
 kernel/rcu/rcu_segcblist.c    |   6 +-
 kernel/rcu/rcu_segcblist.h    |   8 ++-
 kernel/rcu/tree.c             |  24 +++----
 kernel/rcu/tree.h             |   2 +-
 kernel/rcu/tree_plugin.h      | 149 ++++++++++++++++++++++++++++++++++--------
 8 files changed, 172 insertions(+), 46 deletions(-)
author maintainer
#1 lore raw

PATCH 1/10 rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) can unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This also prepare for toggling CPUs to/from callback's offloaded mode where the offloaded state will possibly change when rdp->nocb_lock isn't taken. We'll still want the entrypoints to lock the rdp in any case.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +7 -7
index 097635c41135..523570469864 100644
@@ -1909,7 +1909,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_bypass_timer);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
@@ -1942,7 +1942,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	 */
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
-		rcu_nocb_lock_irqsave(rdp, flags);
+		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -1951,7 +1951,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 			(void)rcu_nocb_try_flush_bypass(rdp, j);
 			bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 			continue; /* No callbacks here, try next. */
 		}
 		if (bypass_ncbs) {
@@ -1996,7 +1996,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 		} else {
 			needwake = false;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake) {
 			swake_up_one(&rdp->nocb_cb_wq);
 			gotcbs = true;
@@ -2084,7 +2084,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	rcu_do_batch(rdp);
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
 	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
 	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
@@ -2092,7 +2092,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
 	if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake_gp)
 			rcu_gp_kthread_wake();
 		return;
@@ -2100,7 +2100,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
 	WRITE_ONCE(rdp->nocb_cb_sleep, true);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
Signature · 1 line
2.25.0
Reviewed-by on v1 1/10
#2 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 8 lines · Frederic Weisbecker

Pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) can unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This also prepare for toggling CPUs to/from callback's offloaded mode where the offloaded state will possibly change when rdp->nocb_lock isn't taken. We'll still want the entrypoints to lock the rdp in any case.

Suggested rewrite for change log:
<wordsmithing>
Make pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This prepares for future toggling of CPUs to/from callback's offloaded mode where the offloaded state can change when rdp->nocb_lock is not held. We'll still want the entrypoints to lock the rdp in any case. </wordsmithing>

Also, can we inline rcu_nocb_lock_irqsave() into do_nocb_deferred_wakeup_common() since that's the only user, and then delete rcu_nocb_lock_irqsave() and the corresponding unlock? That would also remove confusion about which API to use for nocb locking (i.e. whether to directly acquire lock or call rcu_nocb_lock_irqsave()).

thanks,

- Joel

Quoted 82 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +7 -7
index 097635c41135..523570469864 100644
@@ -1909,7 +1909,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_bypass_timer);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
@@ -1942,7 +1942,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	 */
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
-		rcu_nocb_lock_irqsave(rdp, flags);
+		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -1951,7 +1951,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 			(void)rcu_nocb_try_flush_bypass(rdp, j);
 			bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 			continue; /* No callbacks here, try next. */
 		}
 		if (bypass_ncbs) {
@@ -1996,7 +1996,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 		} else {
 			needwake = false;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake) {
 			swake_up_one(&rdp->nocb_cb_wq);
 			gotcbs = true;
@@ -2084,7 +2084,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	rcu_do_batch(rdp);
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
 	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
 	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
@@ -2092,7 +2092,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
 	if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake_gp)
 			rcu_gp_kthread_wake();
 		return;
@@ -2100,7 +2100,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
 	WRITE_ONCE(rdp->nocb_cb_sleep, true);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
Signature · 1 line
2.25.0
maintainer
#3 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 21 lines · Joel Fernandes
Quoted 8 lines · Frederic Weisbecker

Pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) can unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This also prepare for toggling CPUs to/from callback's offloaded mode where the offloaded state will possibly change when rdp->nocb_lock isn't taken. We'll still want the entrypoints to lock the rdp in any case.

Suggested rewrite for change log:
<wordsmithing>
Make pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This prepares for future toggling of CPUs to/from callback's offloaded mode where the offloaded state can change when rdp->nocb_lock is not held. We'll still want the entrypoints to lock the rdp in any case. </wordsmithing>

Quoted 7 lines

Also, can we inline rcu_nocb_lock_irqsave() into do_nocb_deferred_wakeup_common() since that's the only user, and then delete rcu_nocb_lock_irqsave() and the corresponding unlock? That would also remove confusion about which API to use for nocb locking (i.e. whether to directly acquire lock or call rcu_nocb_lock_irqsave()).

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing? If it is feasible, that would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road. And because irqs are disabled while holding the lock, it should be possible to keep state on a per-CPU basis.

The ugliest scenario is callback adoption, where there are two ->cblist structures in need of being locked. In that case, changes are excluded (because that is in CPU hotplug code), but is it possible to take advantage of that reasonably?

Maybe these changes are the best we can do, but it would be good to if the same primitive locked a ->cblist regardless of context.

Can that be made to work reasonably?

Thanx, Paul

Quoted 87 lines

thanks,

- Joel

Quoted 82 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +7 -7
index 097635c41135..523570469864 100644
@@ -1909,7 +1909,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_bypass_timer);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
@@ -1942,7 +1942,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	 */
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
-		rcu_nocb_lock_irqsave(rdp, flags);
+		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -1951,7 +1951,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 			(void)rcu_nocb_try_flush_bypass(rdp, j);
 			bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 			continue; /* No callbacks here, try next. */
 		}
 		if (bypass_ncbs) {
@@ -1996,7 +1996,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 		} else {
 			needwake = false;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake) {
 			swake_up_one(&rdp->nocb_cb_wq);
 			gotcbs = true;
@@ -2084,7 +2084,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	rcu_do_batch(rdp);
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
 	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
 	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
@@ -2092,7 +2092,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
 	if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake_gp)
 			rcu_gp_kthread_wake();
 		return;
@@ -2100,7 +2100,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
 	WRITE_ONCE(rdp->nocb_cb_sleep, true);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
Signature · 1 line
2.25.0
#4 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Hi Paul,

Quoted 28 lines · Paul E. McKenney
Quoted 27 lines · Joel Fernandes
Quoted 8 lines · Frederic Weisbecker

Pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) can unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This also prepare for toggling CPUs to/from callback's offloaded mode where the offloaded state will possibly change when rdp->nocb_lock isn't taken. We'll still want the entrypoints to lock the rdp in any case.

Suggested rewrite for change log:
<wordsmithing>
Make pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This prepares for future toggling of CPUs to/from callback's offloaded mode where the offloaded state can change when rdp->nocb_lock is not held. We'll still want the entrypoints to lock the rdp in any case. </wordsmithing>

Also, can we inline rcu_nocb_lock_irqsave() into do_nocb_deferred_wakeup_common() since that's the only user, and then delete rcu_nocb_lock_irqsave() and the corresponding unlock? That would also remove confusion about which API to use for nocb locking (i.e. whether to directly acquire lock or call rcu_nocb_lock_irqsave()).

Quoted 6 lines
Quoted 1 line

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing?

One way to prevent things from changing could be to employ Steven's poor-man's RCU to mediate the "changing state" itself assuming the switch from the fast-path to the slow-path does not happen often. :) So whichever path is changing the state needs to wait for that poor-man's GP.

Any case, are you concerned with the performance issues with the unconditional locking and that's why you suggest still keeping it conditional?

Also, coming back to my point of inline the helper into the last caller - do_nocb_deferred_wakeup_common(). I think since we don't need to check if the rdp is offloaded and do any conditional locking. The timer can be called only with offloaded rdp. So we can directly do the unconditional spinlock instead of using the rcu_nocb_lock helper there.

Quoted 3 lines

would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road. And because irqs are disabled while holding the lock, it should be possible to keep state on a per-CPU basis.

Agreed, that would be nice. Though if we could keep simple, that'd be nice too.

Quoted 4 lines

The ugliest scenario is callback adoption, where there are two ->cblist structures in need of being locked. In that case, changes are excluded (because that is in CPU hotplug code), but is it possible to take advantage of that reasonably?

Could you describe this a bit more? Thanks.

Quoted 2 lines

Maybe these changes are the best we can do, but it would be good to if the same primitive locked a ->cblist regardless of context.

Here you are comparing 2 primitives. Do you mean that just IRQs being disabled is another primitive, and rcu_nocb_lock is another one?

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

thanks,

- Joel

Quoted 91 lines

Can that be made to work reasonably?

Thanx, Paul

Quoted 87 lines

thanks,

- Joel

Quoted 82 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +7 -7
index 097635c41135..523570469864 100644
@@ -1909,7 +1909,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_bypass_timer);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
@@ -1942,7 +1942,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	 */
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
-		rcu_nocb_lock_irqsave(rdp, flags);
+		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -1951,7 +1951,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 			(void)rcu_nocb_try_flush_bypass(rdp, j);
 			bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 			continue; /* No callbacks here, try next. */
 		}
 		if (bypass_ncbs) {
@@ -1996,7 +1996,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 		} else {
 			needwake = false;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake) {
 			swake_up_one(&rdp->nocb_cb_wq);
 			gotcbs = true;
@@ -2084,7 +2084,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	rcu_do_batch(rdp);
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
 	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
 	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
@@ -2092,7 +2092,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
 	if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake_gp)
 			rcu_gp_kthread_wake();
 		return;
@@ -2100,7 +2100,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
 	WRITE_ONCE(rdp->nocb_cb_sleep, true);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
Signature · 1 line
2.25.0
maintainer
#5 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 35 lines · Joel Fernandes

Hi Paul,

Quoted 32 lines · Paul E. McKenney
Quoted 21 lines · Joel Fernandes
Quoted 8 lines · Frederic Weisbecker

Pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) can unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This also prepare for toggling CPUs to/from callback's offloaded mode where the offloaded state will possibly change when rdp->nocb_lock isn't taken. We'll still want the entrypoints to lock the rdp in any case.

Suggested rewrite for change log:
<wordsmithing>
Make pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This prepares for future toggling of CPUs to/from callback's offloaded mode where the offloaded state can change when rdp->nocb_lock is not held. We'll still want the entrypoints to lock the rdp in any case. </wordsmithing>

Quoted 7 lines

Also, can we inline rcu_nocb_lock_irqsave() into do_nocb_deferred_wakeup_common() since that's the only user, and then delete rcu_nocb_lock_irqsave() and the corresponding unlock? That would also remove confusion about which API to use for nocb locking (i.e. whether to directly acquire lock or call rcu_nocb_lock_irqsave()).

Thank you for looking this over, Joel!

Quoted 7 lines
Quoted 2 lines

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing?

One way to prevent things from changing could be to employ Steven's poor-man's RCU to mediate the "changing state" itself assuming the switch from the fast-path to the slow-path does not happen often. :) So whichever path is changing the state needs to wait for that poor-man's GP.

That should not be needed, given that acquiring ->nocb_lock on the CPU corresponding to that lock suffices in both cases. The trick is making sure that the release matches the acquire.

Quoted 2 lines

Any case, are you concerned with the performance issues with the unconditional locking and that's why you suggest still keeping it conditional?

My concerns are more about maintainability.

Quoted 5 lines

Also, coming back to my point of inline the helper into the last caller - do_nocb_deferred_wakeup_common(). I think since we don't need to check if the rdp is offloaded and do any conditional locking. The timer can be called only with offloaded rdp. So we can directly do the unconditional spinlock instead of using the rcu_nocb_lock helper there.

Indeed we can. But should we?

Quoted 6 lines
Quoted 3 lines

would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road. And because irqs are disabled while holding the lock, it should be possible to keep state on a per-CPU basis.

Agreed, that would be nice. Though if we could keep simple, that'd be nice too.

Having one set of functions/macros that are always used to protect the ->cblist, no matter what the context, is a very attractive form of simple. ;-)

Quoted 6 lines
Quoted 4 lines

The ugliest scenario is callback adoption, where there are two ->cblist structures in need of being locked. In that case, changes are excluded (because that is in CPU hotplug code), but is it possible to take advantage of that reasonably?

Could you describe this a bit more? Thanks.

Right now, callbacks are merged directly from the outgoing CPU's ->cblist to the surviving CPU's ->cblist. This means that both ->cblist structures must be locked at the same time, which would require additional state. After all, if only the one ->cblist were to be locked at a given time, a per-CPU variable could be used to track what method should be used to unlock the ->cblist.

This could be restructured to use an intermediate private ->cblist, but care would be required with the counts, as it is a very bad idea for a large group of callbacks to become invisible. (This is not a problem for rcu_barrier(), which excludes CPU hotplug, but it could be a serious problem for callback-flood-mitigation mechanisms. Yes, they are heuristic, but...)

Quoted 5 lines
Quoted 2 lines

Maybe these changes are the best we can do, but it would be good to if the same primitive locked a ->cblist regardless of context.

Here you are comparing 2 primitives. Do you mean that just IRQs being disabled is another primitive, and rcu_nocb_lock is another one?

I am not sure what this question means, but I am advocating looking into retaining a single wrapper that decides instead of direct use of the underlying primitives.

Or are you instead asking why there are two different methods of protecting the ->cblist structures? (If so, because call_rcu() happens often enough that we don't want lock-acquisition overhead unless we absolutely need it, which we do on nohz_full CPUs but not otherwise.)

Quoted 4 lines

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

I have used similar approaches within RCU, but on the other hand the
scheduler often has tighter latency constraints than RCU does. So I
think that is a better question for the scheduler maintainers than it is for me. ;-)

Thanx, Paul

Quoted 96 lines

thanks,

- Joel

Quoted 91 lines

Can that be made to work reasonably?

Thanx, Paul

Quoted 87 lines

thanks,

- Joel

Quoted 82 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +7 -7
index 097635c41135..523570469864 100644
@@ -1909,7 +1909,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_bypass_timer);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
@@ -1942,7 +1942,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	 */
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
-		rcu_nocb_lock_irqsave(rdp, flags);
+		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -1951,7 +1951,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 			(void)rcu_nocb_try_flush_bypass(rdp, j);
 			bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 			continue; /* No callbacks here, try next. */
 		}
 		if (bypass_ncbs) {
@@ -1996,7 +1996,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 		} else {
 			needwake = false;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake) {
 			swake_up_one(&rdp->nocb_cb_wq);
 			gotcbs = true;
@@ -2084,7 +2084,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	rcu_do_batch(rdp);
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
 	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
 	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
@@ -2092,7 +2092,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
 	if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake_gp)
 			rcu_gp_kthread_wake();
 		return;
@@ -2100,7 +2100,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
 	WRITE_ONCE(rdp->nocb_cb_sleep, true);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
Signature · 1 line
2.25.0
#6 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 39 lines · Paul E. McKenney
Quoted 38 lines · Joel Fernandes

Hi Paul,

Quoted 34 lines · Paul E. McKenney
Quoted 21 lines · Joel Fernandes
Quoted 8 lines · Frederic Weisbecker

Pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) can unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This also prepare for toggling CPUs to/from callback's offloaded mode where the offloaded state will possibly change when rdp->nocb_lock isn't taken. We'll still want the entrypoints to lock the rdp in any case.

Suggested rewrite for change log:
<wordsmithing>
Make pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This prepares for future toggling of CPUs to/from callback's offloaded mode where the offloaded state can change when rdp->nocb_lock is not held. We'll still want the entrypoints to lock the rdp in any case. </wordsmithing>

Quoted 7 lines

Also, can we inline rcu_nocb_lock_irqsave() into do_nocb_deferred_wakeup_common() since that's the only user, and then delete rcu_nocb_lock_irqsave() and the corresponding unlock? That would also remove confusion about which API to use for nocb locking (i.e. whether to directly acquire lock or call rcu_nocb_lock_irqsave()).

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing?

Quoted 8 lines
Quoted 4 lines

One way to prevent things from changing could be to employ Steven's poor-man's RCU to mediate the "changing state" itself assuming the switch from the fast-path to the slow-path does not happen often. :) So whichever path is changing the state needs to wait for that poor-man's GP.

That should not be needed, given that acquiring ->nocb_lock on the CPU corresponding to that lock suffices in both cases. The trick is making sure that the release matches the acquire.

Revisiting what you meant by "when things are changing", I'm assuming you meant a CPU is dynamically switched from non-offloaded mode to offloaded-mode. Please correct me if I'm wrong.

Assuming that's true, you asked how do we "do the right thing" in the lock/unlock APIs. I was also suggesting getting rid of them and directly acquiring/releasing the spinlock, like Frederic does. It sounds like that is not good enough and you want an API that can do conditional locking (and the said complexity is hidden behind the API). Allow me to read more code and see if I can understand that / how to do that.

Quoted 4 lines
Quoted 2 lines

Any case, are you concerned with the performance issues with the unconditional locking and that's why you suggest still keeping it conditional?

My concerns are more about maintainability.

Ok.

Quoted 7 lines
Quoted 5 lines

Also, coming back to my point of inline the helper into the last caller - do_nocb_deferred_wakeup_common(). I think since we don't need to check if the rdp is offloaded and do any conditional locking. The timer can be called only with offloaded rdp. So we can directly do the unconditional spinlock instead of using the rcu_nocb_lock helper there.

Indeed we can. But should we?

Yeah may be not, in the event that we could do conditional locking and benefit.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road. And because irqs are disabled while holding the lock, it should be possible to keep state on a per-CPU basis.

Agreed, that would be nice. Though if we could keep simple, that'd be nice too.

Having one set of functions/macros that are always used to protect the ->cblist, no matter what the context, is a very attractive form of simple. ;-)

I was thinking that API is already raw_spin_lock/unlock() but I'll revisit everything.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

The ugliest scenario is callback adoption, where there are two ->cblist structures in need of being locked. In that case, changes are excluded (because that is in CPU hotplug code), but is it possible to take advantage of that reasonably?

Quoted 8 lines
Quoted 1 line

Could you describe this a bit more? Thanks.

Right now, callbacks are merged directly from the outgoing CPU's ->cblist to the surviving CPU's ->cblist. This means that both ->cblist structures must be locked at the same time, which would require additional state. After all, if only the one ->cblist were to be locked at a given time, a per-CPU variable could be used to track what method should be used to unlock the ->cblist.

So you do mean conditional locking behind an API, and everyone call the API whether they want do the conditional locking or not. Ok.

Quoted 6 lines

This could be restructured to use an intermediate private ->cblist, but care would be required with the counts, as it is a very bad idea for a large group of callbacks to become invisible. (This is not a problem for rcu_barrier(), which excludes CPU hotplug, but it could be a serious problem for callback-flood-mitigation mechanisms. Yes, they are heuristic, but...)

Ok.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Maybe these changes are the best we can do, but it would be good to if the same primitive locked a ->cblist regardless of context.

Here you are comparing 2 primitives. Do you mean that just IRQs being disabled is another primitive, and rcu_nocb_lock is another one?

I am not sure what this question means, but I am advocating looking into retaining a single wrapper that decides instead of direct use of the underlying primitives.

Yep.

Quoted 4 lines

Or are you instead asking why there are two different methods of protecting the ->cblist structures? (If so, because call_rcu() happens often enough that we don't want lock-acquisition overhead unless we absolutely need it, which we do on nohz_full CPUs but not otherwise.)

Yeah that's what I was asking. About lock-acquisition overhead, I think its still uncontended overhead though because even if the nocb lock is taken when it was not needed, it is still to lock the local ->cblist. Correct me if I'm wrong though!

Quoted 9 lines
Quoted 4 lines

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

I have used similar approaches within RCU, but on the other hand the
scheduler often has tighter latency constraints than RCU does. So I
think that is a better question for the scheduler maintainers than it is for me. ;-)

Ok, it definitely keeps coming up in my radar first with the rcu_read_unlock_special() stuff, and now the nocb ;-). Perhaps it could also be good for a conference discussion!

thanks,

- Joel

Quoted 98 lines

Thanx, Paul

Quoted 96 lines

thanks,

- Joel

Quoted 91 lines

Can that be made to work reasonably?

Thanx, Paul

Quoted 87 lines

thanks,

- Joel

Quoted 82 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +7 -7
index 097635c41135..523570469864 100644
@@ -1909,7 +1909,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_bypass_timer);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
@@ -1942,7 +1942,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	 */
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
-		rcu_nocb_lock_irqsave(rdp, flags);
+		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -1951,7 +1951,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 			(void)rcu_nocb_try_flush_bypass(rdp, j);
 			bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 			continue; /* No callbacks here, try next. */
 		}
 		if (bypass_ncbs) {
@@ -1996,7 +1996,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 		} else {
 			needwake = false;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake) {
 			swake_up_one(&rdp->nocb_cb_wq);
 			gotcbs = true;
@@ -2084,7 +2084,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	rcu_do_batch(rdp);
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
 	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
 	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
@@ -2092,7 +2092,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
 	if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake_gp)
 			rcu_gp_kthread_wake();
 		return;
@@ -2100,7 +2100,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
 	WRITE_ONCE(rdp->nocb_cb_sleep, true);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
Signature · 1 line
2.25.0
maintainer
#7 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 45 lines · Joel Fernandes
Quoted 44 lines · Paul E. McKenney
Quoted 42 lines · Joel Fernandes

Hi Paul,

Quoted 28 lines · Paul E. McKenney
Quoted 27 lines · Joel Fernandes
Quoted 8 lines · Frederic Weisbecker

Pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) can unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This also prepare for toggling CPUs to/from callback's offloaded mode where the offloaded state will possibly change when rdp->nocb_lock isn't taken. We'll still want the entrypoints to lock the rdp in any case.

Suggested rewrite for change log:
<wordsmithing>
Make pure NOCB code entrypoints (nocb_cb kthread, nocb_gp kthread, nocb timers) unconditionally lock rdp->nocb_lock as they always execute in the context of an offloaded rdp.

This prepares for future toggling of CPUs to/from callback's offloaded mode where the offloaded state can change when rdp->nocb_lock is not held. We'll still want the entrypoints to lock the rdp in any case. </wordsmithing>

Also, can we inline rcu_nocb_lock_irqsave() into do_nocb_deferred_wakeup_common() since that's the only user, and then delete rcu_nocb_lock_irqsave() and the corresponding unlock? That would also remove confusion about which API to use for nocb locking (i.e. whether to directly acquire lock or call rcu_nocb_lock_irqsave()).

Quoted 6 lines
Quoted 1 line

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing?

One way to prevent things from changing could be to employ Steven's poor-man's RCU to mediate the "changing state" itself assuming the switch from the fast-path to the slow-path does not happen often. :) So whichever path is changing the state needs to wait for that poor-man's GP.

Quoted 7 lines
Quoted 3 lines

That should not be needed, given that acquiring ->nocb_lock on the CPU corresponding to that lock suffices in both cases. The trick is making sure that the release matches the acquire.

Revisiting what you meant by "when things are changing", I'm assuming you meant a CPU is dynamically switched from non-offloaded mode to offloaded-mode. Please correct me if I'm wrong.

Both. It does no good to acquire by disabling IRQs and then release by releasing a lock that you do not hold. Nor does it help to acquire the lock and then fail to release it, instead merely re-enabling IRQs.

Aside from the case where two locks are held, a per-CPU variable (as in another field in the rcu_data structure would work. And that case could probably be reworked to hold only one lock at a time.

Quoted 6 lines

Assuming that's true, you asked how do we "do the right thing" in the lock/unlock APIs. I was also suggesting getting rid of them and directly acquiring/releasing the spinlock, like Frederic does. It sounds like that is not good enough and you want an API that can do conditional locking (and the said complexity is hidden behind the API). Allow me to read more code and see if I can understand that / how to do that.

Indeed, the non-nohz_full code should not acquire the spinlock.

Quoted 13 lines
Quoted 4 lines
Quoted 2 lines

Any case, are you concerned with the performance issues with the unconditional locking and that's why you suggest still keeping it conditional?

My concerns are more about maintainability.

Ok.

Quoted 6 lines
Quoted 5 lines

Also, coming back to my point of inline the helper into the last caller - do_nocb_deferred_wakeup_common(). I think since we don't need to check if the rdp is offloaded and do any conditional locking. The timer can be called only with offloaded rdp. So we can directly do the unconditional spinlock instead of using the rcu_nocb_lock helper there.

Quoted 4 lines
Quoted 1 line

Indeed we can. But should we?

Yeah may be not, in the event that we could do conditional locking and benefit.

Underneath an API, as current mainline does.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road. And because irqs are disabled while holding the lock, it should be possible to keep state on a per-CPU basis.

Agreed, that would be nice. Though if we could keep simple, that'd be nice too.

Quoted 6 lines
Quoted 3 lines

Having one set of functions/macros that are always used to protect the ->cblist, no matter what the context, is a very attractive form of simple. ;-)

I was thinking that API is already raw_spin_lock/unlock() but I'll revisit everything.

It is the function/macro family that includes rcu_nocb_lock() and rcu_nocb_unlock(), as you can see from Frederic's first patch.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

The ugliest scenario is callback adoption, where there are two ->cblist structures in need of being locked. In that case, changes are excluded (because that is in CPU hotplug code), but is it possible to take advantage of that reasonably?

Quoted 8 lines
Quoted 1 line

Could you describe this a bit more? Thanks.

Right now, callbacks are merged directly from the outgoing CPU's ->cblist to the surviving CPU's ->cblist. This means that both ->cblist structures must be locked at the same time, which would require additional state. After all, if only the one ->cblist were to be locked at a given time, a per-CPU variable could be used to track what method should be used to unlock the ->cblist.

Quoted 2 lines

So you do mean conditional locking behind an API, and everyone call the API whether they want do the conditional locking or not. Ok.

Yes, as the code in mainline is now.

Quoted 26 lines
Quoted 6 lines

This could be restructured to use an intermediate private ->cblist, but care would be required with the counts, as it is a very bad idea for a large group of callbacks to become invisible. (This is not a problem for rcu_barrier(), which excludes CPU hotplug, but it could be a serious problem for callback-flood-mitigation mechanisms. Yes, they are heuristic, but...)

Ok.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Maybe these changes are the best we can do, but it would be good to if the same primitive locked a ->cblist regardless of context.

Here you are comparing 2 primitives. Do you mean that just IRQs being disabled is another primitive, and rcu_nocb_lock is another one?

I am not sure what this question means, but I am advocating looking into retaining a single wrapper that decides instead of direct use of the underlying primitives.

Yep.

Quoted 4 lines

Or are you instead asking why there are two different methods of protecting the ->cblist structures? (If so, because call_rcu() happens often enough that we don't want lock-acquisition overhead unless we absolutely need it, which we do on nohz_full CPUs but not otherwise.)

Quoted 4 lines

Yeah that's what I was asking. About lock-acquisition overhead, I think its still uncontended overhead though because even if the nocb lock is taken when it was not needed, it is still to lock the local ->cblist. Correct me if I'm wrong though!

It is uncontended, but it is unnecessary overhead. And call_rcu() can be invoked rather frequently.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

Quoted 8 lines
Quoted 4 lines

I have used similar approaches within RCU, but on the other hand the
scheduler often has tighter latency constraints than RCU does. So I
think that is a better question for the scheduler maintainers than it is for me. ;-)

Ok, it definitely keeps coming up in my radar first with the rcu_read_unlock_special() stuff, and now the nocb ;-). Perhaps it could also be good for a conference discussion!

Again, please understand that RCU has way looser latency constraints than the scheduler does. Adding half a jiffy to wakeup latency might not go over well, especially in the real-time application area.

But what did the scheduler maintainers say about this idea?

Thanx, Paul

Quoted 102 lines

thanks,

- Joel

Quoted 98 lines

Thanx, Paul

Quoted 96 lines

thanks,

- Joel

Quoted 91 lines

Can that be made to work reasonably?

Thanx, Paul

Quoted 87 lines

thanks,

- Joel

Quoted 82 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +7 -7
index 097635c41135..523570469864 100644
@@ -1909,7 +1909,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_bypass_timer);
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
@@ -1942,7 +1942,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	 */
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
-		rcu_nocb_lock_irqsave(rdp, flags);
+		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -1951,7 +1951,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 			(void)rcu_nocb_try_flush_bypass(rdp, j);
 			bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 			continue; /* No callbacks here, try next. */
 		}
 		if (bypass_ncbs) {
@@ -1996,7 +1996,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 		} else {
 			needwake = false;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake) {
 			swake_up_one(&rdp->nocb_cb_wq);
 			gotcbs = true;
@@ -2084,7 +2084,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	rcu_do_batch(rdp);
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
 	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
 	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
@@ -2092,7 +2092,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
 	if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		if (needwake_gp)
 			rcu_gp_kthread_wake();
 		return;
@@ -2100,7 +2100,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
 	WRITE_ONCE(rdp->nocb_cb_sleep, true);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
Signature · 1 line
2.25.0
#8 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

On Tue, May 26, 2020 at 02:09:47PM -0700, Paul E. McKenney wrote:
[...]

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

I have used similar approaches within RCU, but on the other hand the
scheduler often has tighter latency constraints than RCU does. So I
think that is a better question for the scheduler maintainers than it is for me. ;-)

Quoted 7 lines
Quoted 3 lines

Ok, it definitely keeps coming up in my radar first with the rcu_read_unlock_special() stuff, and now the nocb ;-). Perhaps it could also be good for a conference discussion!

Again, please understand that RCU has way looser latency constraints than the scheduler does. Adding half a jiffy to wakeup latency might not go over well, especially in the real-time application area.

Yeah, agreed that the "deadlock detection" code should be pretty light weight if/when it is written.

Quoted 1 line

But what did the scheduler maintainers say about this idea?

Last I remember when it came up during the rcu_read_unlock_special() deadlock discussions, there's no way to know for infra like RCU to know that it was invoked from the scheduler.

The idea I am bringing up now (about the scheduler itself detecting a recursion) was never brought up (not yet) with the sched maintainers (at least not by me).

thanks,

- Joel

maintainer
#9 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 16 lines · Joel Fernandes

On Tue, May 26, 2020 at 02:09:47PM -0700, Paul E. McKenney wrote:
[...]

Quoted 14 lines
Quoted 13 lines
Quoted 9 lines
Quoted 4 lines

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

I have used similar approaches within RCU, but on the other hand the
scheduler often has tighter latency constraints than RCU does. So I
think that is a better question for the scheduler maintainers than it is for me. ;-)

Ok, it definitely keeps coming up in my radar first with the rcu_read_unlock_special() stuff, and now the nocb ;-). Perhaps it could also be good for a conference discussion!

Quoted 6 lines
Quoted 3 lines

Again, please understand that RCU has way looser latency constraints than the scheduler does. Adding half a jiffy to wakeup latency might not go over well, especially in the real-time application area.

Yeah, agreed that the "deadlock detection" code should be pretty light weight if/when it is written.

In addition, to even stand a chance, you would need to use hrtimers. The half-jiffy (at a minimum) delay from any other deferral mechanism that I know of would be the kiss of death, especially from the viewpoint of the real-time guys.

Quoted 9 lines
Quoted 1 line

But what did the scheduler maintainers say about this idea?

Last I remember when it came up during the rcu_read_unlock_special() deadlock discussions, there's no way to know for infra like RCU to know that it was invoked from the scheduler.

The idea I am bringing up now (about the scheduler itself detecting a recursion) was never brought up (not yet) with the sched maintainers (at least not by me).

It might be good to bounce if off of them sooner rather than later.

Thanx, Paul

#10 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Hi Paul,

Quoted 22 lines · Paul E. McKenney
Quoted 20 lines · Joel Fernandes

On Tue, May 26, 2020 at 02:09:47PM -0700, Paul E. McKenney wrote:
[...]

Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

Quoted 8 lines
Quoted 4 lines

I have used similar approaches within RCU, but on the other hand the scheduler often has tighter latency constraints than RCU does. So I think that is a better question for the scheduler maintainers than it is for me. ;-)

Ok, it definitely keeps coming up in my radar first with the rcu_read_unlock_special() stuff, and now the nocb ;-). Perhaps it could also be good for a conference discussion!

Again, please understand that RCU has way looser latency constraints than the scheduler does. Adding half a jiffy to wakeup latency might not go over well, especially in the real-time application area.

Quoted 7 lines
Quoted 2 lines

Yeah, agreed that the "deadlock detection" code should be pretty light weight if/when it is written.

In addition, to even stand a chance, you would need to use hrtimers. The half-jiffy (at a minimum) delay from any other deferral mechanism that I know of would be the kiss of death, especially from the viewpoint of the real-time guys.

Just to make sure we are talking about the same kind of overhead - the deferring is only needed if the rq lock is already held (detected by trylocking). So there's no overhead in the common case other than the trylock possibly being slightly more expensive than the regular locking. Also, once the scheduler defers it, it uses the same kind of mechanism that other deferral mechanisms use to overcome this deadlock (timers, irq_work etc), so the overhead then would be no different than what he have now - the RT users would already have the wake up latency in current kernels without this idea implemented. Did I miss something?

Quoted 11 lines
Quoted 9 lines
Quoted 1 line

But what did the scheduler maintainers say about this idea?

Last I remember when it came up during the rcu_read_unlock_special() deadlock discussions, there's no way to know for infra like RCU to know that it was invoked from the scheduler.

The idea I am bringing up now (about the scheduler itself detecting a recursion) was never brought up (not yet) with the sched maintainers (at least not by me).

It might be good to bounce if off of them sooner rather than later.

Ok, I did that now over IRC. Thank you!

- Joel

maintainer
#11 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 33 lines · Joel Fernandes

Hi Paul,

Quoted 22 lines · Paul E. McKenney
Quoted 20 lines · Joel Fernandes

On Tue, May 26, 2020 at 02:09:47PM -0700, Paul E. McKenney wrote:
[...]

Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

BTW, I'm really itching to give it a try to make the scheduler more deadlock resilient (that is, if the scheduler wake up path detects a deadlock, then it defers the wake up using timers, or irq_work on its own instead of passing the burden of doing so to the callers). Thoughts?

Quoted 8 lines
Quoted 4 lines

I have used similar approaches within RCU, but on the other hand the scheduler often has tighter latency constraints than RCU does. So I think that is a better question for the scheduler maintainers than it is for me. ;-)

Ok, it definitely keeps coming up in my radar first with the rcu_read_unlock_special() stuff, and now the nocb ;-). Perhaps it could also be good for a conference discussion!

Again, please understand that RCU has way looser latency constraints than the scheduler does. Adding half a jiffy to wakeup latency might not go over well, especially in the real-time application area.

Quoted 7 lines
Quoted 2 lines

Yeah, agreed that the "deadlock detection" code should be pretty light weight if/when it is written.

In addition, to even stand a chance, you would need to use hrtimers. The half-jiffy (at a minimum) delay from any other deferral mechanism that I know of would be the kiss of death, especially from the viewpoint of the real-time guys.

Quoted 10 lines

Just to make sure we are talking about the same kind of overhead - the deferring is only needed if the rq lock is already held (detected by trylocking). So there's no overhead in the common case other than the trylock possibly being slightly more expensive than the regular locking. Also, once the scheduler defers it, it uses the same kind of mechanism that other deferral mechanisms use to overcome this deadlock (timers, irq_work etc), so the overhead then would be no different than what he have now - the RT users would already have the wake up latency in current kernels without this idea implemented. Did I miss something?

Aggressive real-time applications care deeply about the uncommon case.

Thanx, Paul

Quoted 15 lines
Quoted 11 lines
Quoted 9 lines
Quoted 1 line

But what did the scheduler maintainers say about this idea?

Last I remember when it came up during the rcu_read_unlock_special() deadlock discussions, there's no way to know for infra like RCU to know that it was invoked from the scheduler.

The idea I am bringing up now (about the scheduler itself detecting a recursion) was never brought up (not yet) with the sched maintainers (at least not by me).

It might be good to bounce if off of them sooner rather than later.

Ok, I did that now over IRC. Thank you!

- Joel

author maintainer
#12 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

In reply to Paul E. McKenney
Quoted 9 lines · Paul E. McKenney
Quoted 1 line · Joel Fernandes

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing? If it is feasible, that would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road.

This won't be pretty:

    locked = rcu_nocb_lock();
    rcu_nocb_unlock(locked);

And anyway we still want to unconditionally lock on many places, regardless of the offloaded state. I don't know how we could have a magic helper do the unconditional lock on some places and the conditional on others.

Also the point of turning the lock helpers into primitives is to make it clearer as to where we really need unconditional locking and where we allow it to be conditional. A gift to reviewers :-)

Thanks.

maintainer
#13 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines · Paul E. McKenney
Quoted 1 line · Joel Fernandes

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing? If it is feasible, that would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road.

Quoted 4 lines

This won't be pretty:

    locked = rcu_nocb_lock();
    rcu_nocb_unlock(locked);

I was thinking in terms of a bit in the rcu_data structure into which rcu_nocb_lock() and friends stored the status, and from which rcu_nocb_unlock() and friends retrieved that same status. Sort of like how preemptible RCU uses the ->rcu_read_lock_nesting field in task_struct.

As noted, this does require reworking the hotplug code to avoid the current holding of two such locks concurrently, which I am happy to do if that helps.

Or am I missing a subtle (or not-so-subtle) twist here?

Quoted 4 lines

And anyway we still want to unconditionally lock on many places, regardless of the offloaded state. I don't know how we could have a magic helper do the unconditional lock on some places and the conditional on others.

I was assuming (perhaps incorrectly) that an intermediate phase between not-offloaded and offloaded would take care of all of those cases.

Quoted 3 lines

Also the point of turning the lock helpers into primitives is to make it clearer as to where we really need unconditional locking and where we allow it to be conditional. A gift to reviewers :-)

Unless and until someone does a copy-pasta, thus unconditionally doing the wrong thing. ;-)

If we cannot avoid different spellings of ->cblist in different places, such is life, but I do want to make sure that we have fully considered the alternatives.

Thanx, Paul

author maintainer
#14 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 22 lines · Paul E. McKenney
Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines · Paul E. McKenney
Quoted 1 line · Joel Fernandes

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing? If it is feasible, that would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road.

Quoted 4 lines

This won't be pretty:

    locked = rcu_nocb_lock();
    rcu_nocb_unlock(locked);

I was thinking in terms of a bit in the rcu_data structure into which rcu_nocb_lock() and friends stored the status, and from which rcu_nocb_unlock() and friends retrieved that same status. Sort of like how preemptible RCU uses the ->rcu_read_lock_nesting field in task_struct.

Quoted 5 lines

As noted, this does require reworking the hotplug code to avoid the current holding of two such locks concurrently, which I am happy to do if that helps.

Or am I missing a subtle (or not-so-subtle) twist here?

So, during a CPU round, the nocb-gp kthread locks the corresponding rdp-nocb and then ignores if it is not offloaded. Surely there is a smarter ways to do that though. Well that's something we'll need to optimize at some point anyway.

Also we must then make sure that the nocb timers won't ever fire while we switch to offloaded state or they may fiddle with internals without locking nocb.

Quoted 8 lines
Quoted 5 lines

And anyway we still want to unconditionally lock on many places, regardless of the offloaded state. I don't know how we could have a magic helper do the unconditional lock on some places and the conditional on others.

I was assuming (perhaps incorrectly) that an intermediate phase between not-offloaded and offloaded would take care of all of those cases.

Perhaps partly but I fear that won't be enough.

Quoted 7 lines
Quoted 4 lines

Also the point of turning the lock helpers into primitives is to make it clearer as to where we really need unconditional locking and where we allow it to be conditional. A gift to reviewers :-)

Unless and until someone does a copy-pasta, thus unconditionally doing the wrong thing. ;-)

Yeah but the cost is more complicated code and synchronization to maintain the use of those all-in-one locking APIs everywhere. And current state is already not that simple :-)

Perhaps we should rename rcu_nocb_lock() into rcu_nocb_lock_cond() to prevent from accidents?

Also I've been thinking that rcu_nocb_lock() should meet any of these requirements:

* hotplug is locked
* rcu barrier is locked
* rnp is locked

Because checking the offloaded state (when nocb isn't locked yet) of an rdp without any of the above locks held is racy. And that should be easy to check and prevent from copy-pasta accidents.

What do you think?

maintainer
#15 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 29 lines · Frederic Weisbecker
Quoted 22 lines · Paul E. McKenney
Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines · Paul E. McKenney
Quoted 1 line · Joel Fernandes

Thank you for looking this over, Joel!

Is it feasible to make rcu_nocb_lock*() and rcu_nocb_unlock*() "do the right thing", even when things are changing? If it is feasible, that would prevent any number of "interesting" copy-pasta and "just now became common code" bugs down the road.

Quoted 4 lines

This won't be pretty:

    locked = rcu_nocb_lock();
    rcu_nocb_unlock(locked);

I was thinking in terms of a bit in the rcu_data structure into which rcu_nocb_lock() and friends stored the status, and from which rcu_nocb_unlock() and friends retrieved that same status. Sort of like how preemptible RCU uses the ->rcu_read_lock_nesting field in task_struct.

Quoted 5 lines

As noted, this does require reworking the hotplug code to avoid the current holding of two such locks concurrently, which I am happy to do if that helps.

Or am I missing a subtle (or not-so-subtle) twist here?

Quoted 7 lines

So, during a CPU round, the nocb-gp kthread locks the corresponding rdp-nocb and then ignores if it is not offloaded. Surely there is a smarter ways to do that though. Well that's something we'll need to optimize at some point anyway.

Also we must then make sure that the nocb timers won't ever fire while we switch to offloaded state or they may fiddle with internals without locking nocb.

Understood, the synchronization requirements during the switchover will be tricky. But that will be the case no matter how we set it up. If possible, it would be good if that trickiness could be concentrated rather than spread far and wide.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

And anyway we still want to unconditionally lock on many places, regardless of the offloaded state. I don't know how we could have a magic helper do the unconditional lock on some places and the conditional on others.

I was assuming (perhaps incorrectly) that an intermediate phase between not-offloaded and offloaded would take care of all of those cases.

Perhaps partly but I fear that won't be enough.

One approach is to rely on RCU read-side critical sections surrounding the lock acquisition and to stay in the intermediate phase until a grace period completes, preferably call_rcu() instead of synchronize_rcu().

This of course means refusing to do a transition if the CPU is still in the intermediate state from a prior transition.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Also the point of turning the lock helpers into primitives is to make it clearer as to where we really need unconditional locking and where we allow it to be conditional. A gift to reviewers :-)

Unless and until someone does a copy-pasta, thus unconditionally doing the wrong thing. ;-)

Quoted 6 lines

Yeah but the cost is more complicated code and synchronization to maintain the use of those all-in-one locking APIs everywhere. And current state is already not that simple :-)

Perhaps we should rename rcu_nocb_lock() into rcu_nocb_lock_cond() to prevent from accidents?

We do need something to avoid accidents. ;-)

Quoted 7 lines

Also I've been thinking that rcu_nocb_lock() should meet any of these requirements:

* hotplug is locked
* rcu barrier is locked
* rnp is locked

Quoted 5 lines

Because checking the offloaded state (when nocb isn't locked yet) of an rdp without any of the above locks held is racy. And that should be easy to check and prevent from copy-pasta accidents.

What do you think?

An RCU read-side critical section might be simpler.

Thanx, Paul

author maintainer
#16 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 8 lines · Paul E. McKenney
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

And anyway we still want to unconditionally lock on many places, regardless of the offloaded state. I don't know how we could have a magic helper do the unconditional lock on some places and the conditional on others.

I was assuming (perhaps incorrectly) that an intermediate phase between not-offloaded and offloaded would take care of all of those cases.

Quoted 8 lines
Quoted 1 line

Perhaps partly but I fear that won't be enough.

One approach is to rely on RCU read-side critical sections surrounding the lock acquisition and to stay in the intermediate phase until a grace period completes, preferably call_rcu() instead of synchronize_rcu().

This of course means refusing to do a transition if the CPU is still in the intermediate state from a prior transition.

That sounds good. But using synchronize_rcu() would be far easier. We need to keep the hotplug and rcu barrier locked during the transition.

Quoted 7 lines
Quoted 7 lines

Also I've been thinking that rcu_nocb_lock() should meet any of these requirements:

* hotplug is locked
* rcu barrier is locked
* rnp is locked

Quoted 7 lines
Quoted 5 lines

Because checking the offloaded state (when nocb isn't locked yet) of an rdp without any of the above locks held is racy. And that should be easy to check and prevent from copy-pasta accidents.

What do you think?

An RCU read-side critical section might be simpler.

Ok I think I can manage that.

Thanks.

maintainer
#17 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 32 lines · Frederic Weisbecker
Quoted 8 lines · Paul E. McKenney
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

And anyway we still want to unconditionally lock on many places, regardless of the offloaded state. I don't know how we could have a magic helper do the unconditional lock on some places and the conditional on others.

I was assuming (perhaps incorrectly) that an intermediate phase between not-offloaded and offloaded would take care of all of those cases.

Quoted 8 lines
Quoted 1 line

Perhaps partly but I fear that won't be enough.

One approach is to rely on RCU read-side critical sections surrounding the lock acquisition and to stay in the intermediate phase until a grace period completes, preferably call_rcu() instead of synchronize_rcu().

This of course means refusing to do a transition if the CPU is still in the intermediate state from a prior transition.

That sounds good. But using synchronize_rcu() would be far easier. We need to keep the hotplug and rcu barrier locked during the transition.

Quoted 11 lines
Quoted 11 lines

Also I've been thinking that rcu_nocb_lock() should meet any of these requirements:

* hotplug is locked
* rcu barrier is locked
* rnp is locked

Because checking the offloaded state (when nocb isn't locked yet) of an rdp without any of the above locks held is racy. And that should be easy to check and prevent from copy-pasta accidents.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

What do you think?

An RCU read-side critical section might be simpler.

Ok I think I can manage that.

And just to argue against myself...

Another approach is to maintain explicit multiple states for each ->cblist, perhaps something like this:

1. In softirq. Transition code advances to next.
2. To no-CB 1. Either GP or CB kthread for the transitioning
CPU advances to next. Note that the fact that the transition code runs on the transitioning CPU means that the RCU softirq handler doesn't need to be involved.
3. To no-CB 2. Either GP or CB kthread for the transitioning
CPU advances to next.
4. To no-CB 3. Transitioning code advances to next.
At this point, the no-CBs setup is fully functional.
5. No-CB. Transitioning code advances to next.
Again, the fact that the transitioning code is running on the transitioning CPU with interrupts disabled means that the RCU softirq handler need not be explicitly involved.
6. To softirq 1. The RCU softirq handler for the transitioning
CPU advances to next.
At this point, the RCU softirq handler is processing callbacks.
7. To softirq 2. Either GP or CB kthread for the transitioning
CPU advances to next.
At this point, the softirq handler is processing callbacks.
8. To softirq 3. Either GP or CB kthread for the transitioning
CPU advances to next.
At this point, the no-CBs setup is fully shut down.
9. To softirq 4. Transitioning code advances to next,
which is the first, "In softirq".
(This one -might- be unnecessary, but...)

All transitions are of course with the ->nocb_lock held.

When there is only one CPU during early boot near rcu_init() time, the transition from "In softirq" to "No-CB" can remain be instantaneous.

This has the advantage of not slowing things down just because there is an RCU callback flood in progress. It also uses an explicit protocol that should (give or take bugs) maintain full safety both in protection of ->cblist and in dealing with RCU callback floods.

Thoughts?

Thanx, Paul

author maintainer
#18 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 5 lines · Paul E. McKenney

And just to argue against myself...

Another approach is to maintain explicit multiple states for each ->cblist, perhaps something like this:

Quoted 7 lines

1. In softirq. Transition code advances to next.
2. To no-CB 1. Either GP or CB kthread for the transitioning
CPU advances to next. Note that the fact that the transition code runs on the transitioning CPU means that the RCU softirq handler doesn't need to be involved.
3. To no-CB 2. Either GP or CB kthread for the transitioning
CPU advances to next.

Just to clarify, if GP has set NO_CB2 in (2), we want CB to set NO_CB3 in 3), right? OTOH if CB has set NO_CB2 in (2), we want GP to set NO_CB3 in (3), right?

The point being to make sure that both threads acknowledge the transition?

Quoted 2 lines

4. To no-CB 3. Transitioning code advances to next.
At this point, the no-CBs setup is fully functional.

And softirq can stop processing callbacks from that point on.

Quoted 11 lines

5. No-CB. Transitioning code advances to next.
Again, the fact that the transitioning code is running on the transitioning CPU with interrupts disabled means that the RCU softirq handler need not be explicitly involved.
6. To softirq 1. The RCU softirq handler for the transitioning
CPU advances to next.
At this point, the RCU softirq handler is processing callbacks.
7. To softirq 2. Either GP or CB kthread for the transitioning
CPU advances to next.
At this point, the softirq handler is processing callbacks.

SOFTIRQ2 should be part of what happens in SOFTIRQ1. The transitioning CPU sets SOFTIRQ1, which is immediately visible by local softirqs, and wakes up CB/GP. CB/GP sets SOFTIRQ2, CB/GP sets SOFTIRQ3 and we go back to transitioning code that sets IN_SOFTIRQ.

Or did I miss something?

Quoted 12 lines

8. To softirq 3. Either GP or CB kthread for the transitioning
CPU advances to next.
At this point, the no-CBs setup is fully shut down.
9. To softirq 4. Transitioning code advances to next,
which is the first, "In softirq".
(This one -might- be unnecessary, but...)

All transitions are of course with the ->nocb_lock held.

When there is only one CPU during early boot near rcu_init() time, the transition from "In softirq" to "No-CB" can remain be instantaneous.

Quoted 6 lines

This has the advantage of not slowing things down just because there is an RCU callback flood in progress. It also uses an explicit protocol that should (give or take bugs) maintain full safety both in protection of ->cblist and in dealing with RCU callback floods.

Thoughts?

Agreed. And I really like that it details the whole process in a very explicit way.

Thanks!

maintainer
#19 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

Quoted 14 lines · Frederic Weisbecker
Quoted 5 lines · Paul E. McKenney

And just to argue against myself...

Another approach is to maintain explicit multiple states for each ->cblist, perhaps something like this:

Quoted 7 lines

1. In softirq. Transition code advances to next.
2. To no-CB 1. Either GP or CB kthread for the transitioning
CPU advances to next. Note that the fact that the transition code runs on the transitioning CPU means that the RCU softirq handler doesn't need to be involved.
3. To no-CB 2. Either GP or CB kthread for the transitioning
CPU advances to next.

Quoted 5 lines

Just to clarify, if GP has set NO_CB2 in (2), we want CB to set NO_CB3 in 3), right? OTOH if CB has set NO_CB2 in (2), we want GP to set NO_CB3 in (3), right?

The point being to make sure that both threads acknowledge the transition?

Exactly!

Quoted 4 lines
Quoted 2 lines

4. To no-CB 3. Transitioning code advances to next.
At this point, the no-CBs setup is fully functional.

And softirq can stop processing callbacks from that point on.

You got it!

Quoted 12 lines
Quoted 11 lines

5. No-CB. Transitioning code advances to next.
Again, the fact that the transitioning code is running on the transitioning CPU with interrupts disabled means that the RCU softirq handler need not be explicitly involved.
6. To softirq 1. The RCU softirq handler for the transitioning
CPU advances to next.
At this point, the RCU softirq handler is processing callbacks.
7. To softirq 2. Either GP or CB kthread for the transitioning
CPU advances to next.
At this point, the softirq handler is processing callbacks.

Quoted 6 lines

SOFTIRQ2 should be part of what happens in SOFTIRQ1. The transitioning CPU sets SOFTIRQ1, which is immediately visible by local softirqs, and wakes up CB/GP. CB/GP sets SOFTIRQ2, CB/GP sets SOFTIRQ3 and we go back to transitioning code that sets IN_SOFTIRQ.

Or did I miss something?

I was perhaps being overly paranoid. You might well be right.

Quoted 17 lines
Quoted 17 lines

8. To softirq 3. Either GP or CB kthread for the transitioning
CPU advances to next.
At this point, the no-CBs setup is fully shut down.
9. To softirq 4. Transitioning code advances to next,
which is the first, "In softirq".
(This one -might- be unnecessary, but...)

All transitions are of course with the ->nocb_lock held.

When there is only one CPU during early boot near rcu_init() time, the transition from "In softirq" to "No-CB" can remain be instantaneous.

This has the advantage of not slowing things down just because there is an RCU callback flood in progress. It also uses an explicit protocol that should (give or take bugs) maintain full safety both in protection of ->cblist and in dealing with RCU callback floods.

Quoted 6 lines
Quoted 1 line

Thoughts?

Agreed. And I really like that it details the whole process in a very explicit way.

Thanks!

Glad you like it! And of course please adjust it as needed, up to and including doing something completely different that works better. ;-)

Thanx, Paul

#20 lore raw

Re: [PATCH 01/10] rcu: Directly lock rdp->nocb_lock on nocb code entrypoints

On Wed, Jun 10, 2020 at 04:21:42PM -0700, Paul E. McKenney wrote:
[...]

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines

8. To softirq 3. Either GP or CB kthread for the transitioning
CPU advances to next.
At this point, the no-CBs setup is fully shut down.
9. To softirq 4. Transitioning code advances to next,
which is the first, "In softirq".
(This one -might- be unnecessary, but...)

All transitions are of course with the ->nocb_lock held.

When there is only one CPU during early boot near rcu_init() time, the transition from "In softirq" to "No-CB" can remain be instantaneous.

This has the advantage of not slowing things down just because there is an RCU callback flood in progress. It also uses an explicit protocol that should (give or take bugs) maintain full safety both in protection of ->cblist and in dealing with RCU callback floods.

Thoughts?

Quoted 7 lines
Quoted 4 lines

Agreed. And I really like that it details the whole process in a very explicit way.

Thanks!

Glad you like it! And of course please adjust it as needed, up to and including doing something completely different that works better. ;-)

Makes sense to me too, thanks!

- Joel

author maintainer
#21 lore raw

PATCH 2/10 rcu: Use direct rdp->nocb_lock operations on local calls

In reply to Frederic Weisbecker

Unconditionally lock rdp->nocb_lock on nocb code that is called after we verified that the rdp is offloaded:

This clarify the locking rules and expectations.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
kernel/rcu/tree_plugin.h +12 -12
index 523570469864..1d22b16c03e0 100644
@@ -1628,11 +1628,11 @@ static void wake_nocb_gp(struct rcu_data *rdp, bool force,
 	if (!READ_ONCE(rdp_gp->nocb_gp_kthread)) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
 				    TPS("AlreadyAwake"));
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		return;
 	}
 	del_timer(&rdp->nocb_timer);
-	rcu_nocb_unlock_irqrestore(rdp, flags);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
 	if (force || READ_ONCE(rdp_gp->nocb_gp_sleep)) {
 		WRITE_ONCE(rdp_gp->nocb_gp_sleep, false);
@@ -1753,7 +1753,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 
 	// Don't use ->nocb_bypass during early boot.
 	if (rcu_scheduler_active != RCU_SCHEDULER_RUNNING) {
-		rcu_nocb_lock(rdp);
+		raw_spin_lock(&rdp->nocb_lock);
 		WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass));
 		*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
 		return false;
@@ -1778,7 +1778,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 	// this jiffy, tell the caller to enqueue onto ->cblist.  But flush
 	// ->nocb_bypass first.
 	if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy) {
-		rcu_nocb_lock(rdp);
+		raw_spin_lock(&rdp->nocb_lock);
 		*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
 		if (*was_alldone)
 			trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
@@ -1792,7 +1792,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 	// flush ->nocb_bypass to ->cblist.
 	if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) ||
 	    ncbs >= qhimark) {
-		rcu_nocb_lock(rdp);
+		raw_spin_lock(&rdp->nocb_lock);
 		if (!rcu_nocb_flush_bypass(rdp, rhp, j)) {
 			*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
 			if (*was_alldone)
@@ -1807,7 +1807,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 			rcu_advance_cbs_nowake(rdp->mynode, rdp);
 			rdp->nocb_gp_adv_time = j;
 		}
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		return true; // Callback already enqueued.
 	}
 
@@ -1827,7 +1827,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 		local_irq_restore(flags);
 	} else {
 		// No-CBs GP kthread might be indefinitely asleep, if so, wake.
-		rcu_nocb_lock(rdp); // Rare during call_rcu() flood.
+		raw_spin_lock(&rdp->nocb_lock); // Rare during call_rcu() flood.
 		if (!rcu_segcblist_pend_cbs(&rdp->cblist)) {
 			trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
 					    TPS("FirstBQwake"));
@@ -1835,7 +1835,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 		} else {
 			trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
 					    TPS("FirstBQnoWake"));
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		}
 	}
 	return true; // Callback already enqueued.
@@ -1861,7 +1861,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
 	if (rcu_nocb_poll || !t) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
 				    TPS("WakeNotPoll"));
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		return;
 	}
 	// Need to actually to a wakeup.
@@ -1876,7 +1876,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
 		} else {
 			wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE,
 					   TPS("WakeEmptyIsDeferred"));
-			rcu_nocb_unlock_irqrestore(rdp, flags);
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		}
 	} else if (len > rdp->qlen_last_fqs_check + qhimark) {
 		/* ... or if many callbacks queued. */
@@ -1894,10 +1894,10 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
 		    !timer_pending(&rdp->nocb_bypass_timer))
 			wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE,
 					   TPS("WakeOvfIsDeferred"));
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	} else {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot"));
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 	}
 	return;
 }
Signature · 1 line
2.25.0
author maintainer
#22 lore raw

PATCH 3/10 rcu: Make locking explicit in do_nocb_deferred_wakeup_common()

In reply to Frederic Weisbecker

It can either be called inline (locally or CPU hotplug locked) when rdp->nocb_defer_wakeup is pending or from the nocb timer. In both cases the rdp is offlined and we want to take the nocb lock.

Clarify the locking rules and expectations.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +2 -2
index 1d22b16c03e0..1dd3fdd675a1 100644
@@ -2142,9 +2142,9 @@ static void do_nocb_deferred_wakeup_common(struct rcu_data *rdp)
 	unsigned long flags;
 	int ndw;
 
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (!rcu_nocb_need_deferred_wakeup(rdp)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		return;
 	}
 	ndw = READ_ONCE(rdp->nocb_defer_wakeup);
Signature · 1 line
2.25.0
#23 lore raw

Re: [PATCH 03/10] rcu: Make locking explicit in do_nocb_deferred_wakeup_common()

Quoted 5 lines · Frederic Weisbecker

It can either be called inline (locally or CPU hotplug locked) when rdp->nocb_defer_wakeup is pending or from the nocb timer. In both cases the rdp is offlined and we want to take the nocb lock.

Clarify the locking rules and expectations.

And you did inline the code in the locking helper already as I was suggesting in 01/10. :-) So that's settled ;-)

thanks,

- Joel

Quoted 31 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +2 -2
index 1d22b16c03e0..1dd3fdd675a1 100644
@@ -2142,9 +2142,9 @@ static void do_nocb_deferred_wakeup_common(struct rcu_data *rdp)
 	unsigned long flags;
 	int ndw;
 
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (!rcu_nocb_need_deferred_wakeup(rdp)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		return;
 	}
 	ndw = READ_ONCE(rdp->nocb_defer_wakeup);
Signature · 1 line
2.25.0
#24 lore raw

Re: [PATCH 03/10] rcu: Make locking explicit in do_nocb_deferred_wakeup_common()

In reply to Frederic Weisbecker
Quoted 3 lines · Frederic Weisbecker

It can either be called inline (locally or CPU hotplug locked) when rdp->nocb_defer_wakeup is pending or from the nocb timer. In both cases the rdp is offlined and we want to take the nocb lock.

s/offlined/offloaded/

thanks,

- Joel

Quoted 33 lines

Clarify the locking rules and expectations.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +2 -2
index 1d22b16c03e0..1dd3fdd675a1 100644
@@ -2142,9 +2142,9 @@ static void do_nocb_deferred_wakeup_common(struct rcu_data *rdp)
 	unsigned long flags;
 	int ndw;
 
-	rcu_nocb_lock_irqsave(rdp, flags);
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
 	if (!rcu_nocb_need_deferred_wakeup(rdp)) {
-		rcu_nocb_unlock_irqrestore(rdp, flags);
+		raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
 		return;
 	}
 	ndw = READ_ONCE(rdp->nocb_defer_wakeup);
Signature · 1 line
2.25.0
author maintainer
#25 lore raw

PATCH 4/10 rcu: Implement rcu_segcblist_is_offloaded() config dependent

In reply to Frederic Weisbecker

This simplify the usage of this API and avoid checking the kernel config from the callers.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcu_segcblist.h |  2 ++
 kernel/rcu/rcu_segcblist.c    |  2 ++
 kernel/rcu/rcu_segcblist.h    |  6 ++++++
 kernel/rcu/tree.c             | 21 +++++++--------------
 4 files changed, 17 insertions(+), 14 deletions(-)
include/linux/rcu_segcblist.h +2 -0
index b36afe7b22c9..0ced0a0ecbcf 100644
@@ -73,7 +73,9 @@ struct rcu_segcblist {
 	long len;
 #endif
 	u8 enabled;
+#ifdef CONFIG_RCU_NOCB_CPU
 	u8 offloaded;
+#endif
 };
 
 #define RCU_SEGCBLIST_INITIALIZER(n) \
kernel/rcu/rcu_segcblist.c +2 -0
index 9a0f66133b4b..d8ea2bef5574 100644
@@ -166,6 +166,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
 	rsclp->enabled = 0;
 }
 
+#ifdef CONFIG_RCU_NOCB_CPU
 /*
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.
@@ -174,6 +175,7 @@ void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
 {
 	rsclp->offloaded = 1;
 }
+#endif
 
 /*
  * Does the specified rcu_segcblist structure contain callbacks that
kernel/rcu/rcu_segcblist.h +6 -0
index 5c293afc07b8..4c1503a82492 100644
@@ -62,7 +62,11 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
 /* Is the specified rcu_segcblist offloaded?  */
 static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
 {
+#ifdef CONFIG_RCU_NOCB_CPU
 	return rsclp->offloaded;
+#else
+	return false;
+#endif
 }
 
 /*
@@ -78,7 +82,9 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)
 void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
 void rcu_segcblist_init(struct rcu_segcblist *rsclp);
 void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
+#ifdef CONFIG_RCU_NOCB_CPU
 void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
+#endif
 bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
 bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
 struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);
kernel/rcu/tree.c +7 -14
index d9a49cd6065a..804cf7dfff03 100644
@@ -1401,8 +1401,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
 {
 	bool ret = false;
 	bool need_qs;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
 
 	raw_lockdep_assert_held_rcu_node(rnp);
 
@@ -1790,8 +1789,7 @@ static void rcu_gp_cleanup(void)
 		needgp = true;
 	}
 	/* Advance CBs to reduce false positives below. */
-	offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-		    rcu_segcblist_is_offloaded(&rdp->cblist);
+	offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
 	if ((offloaded || !rcu_accelerate_cbs(rnp, rdp)) && needgp) {
 		WRITE_ONCE(rcu_state.gp_flags, RCU_GP_FLAG_INIT);
 		WRITE_ONCE(rcu_state.gp_req_activity, jiffies);
@@ -1985,8 +1983,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
 	unsigned long flags;
 	unsigned long mask;
 	bool needwake = false;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
 	struct rcu_node *rnp;
 
 	rnp = rdp->mynode;
@@ -2153,8 +2150,7 @@ int rcutree_dead_cpu(unsigned int cpu)
 static void rcu_do_batch(struct rcu_data *rdp)
 {
 	unsigned long flags;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
 	struct rcu_head *rhp;
 	struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
 	long bl, count;
@@ -2397,8 +2393,7 @@ static __latent_entropy void rcu_core(void)
 	unsigned long flags;
 	struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
 	struct rcu_node *rnp = rdp->mynode;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
 
 	if (cpu_is_offline(smp_processor_id()))
 		return;
@@ -2695,8 +2690,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
 				   rcu_segcblist_n_cbs(&rdp->cblist));
 
 	/* Go handle any RCU core processing required. */
-	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-	    unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
+	if (unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
 		__call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
 	} else {
 		__call_rcu_core(rdp, head, flags);
@@ -3243,8 +3237,7 @@ static int rcu_pending(int user)
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
 	if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
-	    (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
-	     !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
+	    !rcu_segcblist_is_offloaded(&rdp->cblist) &&
 	    !rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
 		return 1;
 
Signature · 1 line
2.25.0
maintainer
#26 lore raw

Re: [PATCH 04/10] rcu: Implement rcu_segcblist_is_offloaded() config dependent

Quoted 22 lines · Frederic Weisbecker

This simplify the usage of this API and avoid checking the kernel config from the callers.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcu_segcblist.h |  2 ++
 kernel/rcu/rcu_segcblist.c    |  2 ++
 kernel/rcu/rcu_segcblist.h    |  6 ++++++
 kernel/rcu/tree.c             | 21 +++++++--------------
 4 files changed, 17 insertions(+), 14 deletions(-)
include/linux/rcu_segcblist.h +0 -0
index b36afe7b22c9..0ced0a0ecbcf 100644

include/linux/rcu_segcblist.h struct rcu_segcblist { · line 78

Quoted 6 lines
include/linux/rcu_segcblist.h +2 -0
@@ -73,7 +73,9 @@ struct rcu_segcblist {
 	long len;
 #endif
 	u8 enabled;
+#ifdef CONFIG_RCU_NOCB_CPU
 	u8 offloaded;
+#endif

Given that this is only one byte and that removing it won't actually save any memory on most architectures, why not just leave it and adjust as shown below?

Quoted 16 lines
include/linux/rcu_segcblist.h +0 -0
 };
 
 #define RCU_SEGCBLIST_INITIALIZER(n) \
kernel/rcu/rcu_segcblist.c +1 -0
index 9a0f66133b4b..d8ea2bef5574 100644
@@ -166,6 +166,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
 	rsclp->enabled = 0;
 }
 
+#ifdef CONFIG_RCU_NOCB_CPU
 /*
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.

kernel/rcu/rcu_segcblist.c void rcu_segcblist_offload(struct rcu_segcblist *rsclp) · line 178

Quoted 4 lines
kernel/rcu/rcu_segcblist.c +1 -0
@@ -174,6 +175,7 @@ void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
 {
 	rsclp->offloaded = 1;
 }
+#endif

Leave this unconditional, as it is nowhere near a fastpath.

Quoted 7 lines
kernel/rcu/rcu_segcblist.c +0 -0
 /*
  * Does the specified rcu_segcblist structure contain callbacks that
kernel/rcu/rcu_segcblist.h +0 -0
index 5c293afc07b8..4c1503a82492 100644

kernel/rcu/rcu_segcblist.h static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp) · line 70

Quoted 9 lines
kernel/rcu/rcu_segcblist.h +4 -0
@@ -62,7 +62,11 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
 /* Is the specified rcu_segcblist offloaded?  */
 static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
 {
+#ifdef CONFIG_RCU_NOCB_CPU
 	return rsclp->offloaded;
+#else
+	return false;
+#endif
 }

Then this can just be:

	return IS_ENABLED(CONFIG_RCU_NOCB_CPU) && rsclp->offloaded;

kernel/rcu/rcu_segcblist.h static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg) · line 87

Quoted 8 lines
kernel/rcu/rcu_segcblist.h +2 -0
 /*
@@ -78,7 +82,9 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)
 void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
 void rcu_segcblist_init(struct rcu_segcblist *rsclp);
 void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
+#ifdef CONFIG_RCU_NOCB_CPU
 void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
+#endif

This can then remain unconditional.

Quoted 8 lines
kernel/rcu/rcu_segcblist.h +0 -0
 bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
 bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
 struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);
kernel/rcu/tree.c +0 -0
index d9a49cd6065a..804cf7dfff03 100644

kernel/rcu/tree.c static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) · line 1404

Quoted 6 lines
kernel/rcu/tree.c +1 -2
@@ -1401,8 +1401,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
 {
 	bool ret = false;
 	bool need_qs;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);

The adjustment to rcu_segcblist_is_offloaded() allows this (welcome!) simplification to remain.

kernel/rcu/tree.c static void rcu_gp_cleanup(void) · line 1792

Quoted 9 lines
kernel/rcu/tree.c +1 -2
 	raw_lockdep_assert_held_rcu_node(rnp);
 
@@ -1790,8 +1789,7 @@ static void rcu_gp_cleanup(void)
 		needgp = true;
 	}
 	/* Advance CBs to reduce false positives below. */
-	offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-		    rcu_segcblist_is_offloaded(&rdp->cblist);
+	offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);

And here as well.

kernel/rcu/tree.c rcu_report_qs_rdp(int cpu, struct rcu_data *rdp) · line 1986

Quoted 10 lines
kernel/rcu/tree.c +1 -2
 	if ((offloaded || !rcu_accelerate_cbs(rnp, rdp)) && needgp) {
 		WRITE_ONCE(rcu_state.gp_flags, RCU_GP_FLAG_INIT);
 		WRITE_ONCE(rcu_state.gp_req_activity, jiffies);
@@ -1985,8 +1983,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
 	unsigned long flags;
 	unsigned long mask;
 	bool needwake = false;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);

Ditto.

kernel/rcu/tree.c int rcutree_dead_cpu(unsigned int cpu) · line 2153

Quoted 10 lines
kernel/rcu/tree.c +1 -2
 	struct rcu_node *rnp;
 
 	rnp = rdp->mynode;
@@ -2153,8 +2150,7 @@ int rcutree_dead_cpu(unsigned int cpu)
 static void rcu_do_batch(struct rcu_data *rdp)
 {
 	unsigned long flags;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);

Ditto.

kernel/rcu/tree.c static __latent_entropy void rcu_core(void) · line 2396

Quoted 10 lines
kernel/rcu/tree.c +1 -2
 	struct rcu_head *rhp;
 	struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
 	long bl, count;
@@ -2397,8 +2393,7 @@ static __latent_entropy void rcu_core(void)
 	unsigned long flags;
 	struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
 	struct rcu_node *rnp = rdp->mynode;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);

Ditto.

kernel/rcu/tree.c __call_rcu(struct rcu_head *head, rcu_callback_t func) · line 2693

Quoted 10 lines
kernel/rcu/tree.c +1 -2
 	if (cpu_is_offline(smp_processor_id()))
 		return;
@@ -2695,8 +2690,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
 				   rcu_segcblist_n_cbs(&rdp->cblist));
 
 	/* Go handle any RCU core processing required. */
-	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-	    unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
+	if (unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {

Ditto.

kernel/rcu/tree.c static int rcu_pending(int user) · line 3240

Quoted 10 lines
kernel/rcu/tree.c +1 -2
 		__call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
 	} else {
 		__call_rcu_core(rdp, head, flags);
@@ -3243,8 +3237,7 @@ static int rcu_pending(int user)
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
 	if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
-	    (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
-	     !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
+	    !rcu_segcblist_is_offloaded(&rdp->cblist) &&

Ditto.

As in "Why didn't I do it that way to start with???" ;-)

Thanx, Paul

Quoted 6 lines
kernel/rcu/tree.c +0 -0
 	    !rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
 		return 1;
 
Signature · 1 line
2.25.0
author maintainer
#27 lore raw

Re: [PATCH 04/10] rcu: Implement rcu_segcblist_is_offloaded() config dependent

Quoted 18 lines · Paul E. McKenney
Quoted 17 lines · Frederic Weisbecker

This simplify the usage of this API and avoid checking the kernel config from the callers.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcu_segcblist.h |  2 ++
 kernel/rcu/rcu_segcblist.c    |  2 ++
 kernel/rcu/rcu_segcblist.h    |  6 ++++++
 kernel/rcu/tree.c             | 21 +++++++--------------
 4 files changed, 17 insertions(+), 14 deletions(-)
Quoted 15 lines

include/linux/rcu_segcblist.h struct rcu_segcblist { · line 78

Quoted 11 lines
include/linux/rcu_segcblist.h +2 -0
index b36afe7b22c9..0ced0a0ecbcf 100644
@@ -73,7 +73,9 @@ struct rcu_segcblist {
 	long len;
 #endif
 	u8 enabled;
+#ifdef CONFIG_RCU_NOCB_CPU
 	u8 offloaded;
+#endif

Given that this is only one byte and that removing it won't actually save any memory on most architectures, why not just leave it and adjust as shown below?

Right, the point was to make it private to that config and trigger a build error otherwise. But if we have an off case that's fine.

Quoted 12 lines
Quoted 12 lines
include/linux/rcu_segcblist.h +0 -0
 };
 
 #define RCU_SEGCBLIST_INITIALIZER(n) \
kernel/rcu/rcu_segcblist.c +0 -0
index 9a0f66133b4b..d8ea2bef5574 100644
@@ -166,6 +166,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
 	rsclp->enabled = 0;
 }
 
Quoted 11 lines

kernel/rcu/rcu_segcblist.c void rcu_segcblist_offload(struct rcu_segcblist *rsclp) · line 178

Quoted 9 lines
kernel/rcu/rcu_segcblist.c +2 -0
+#ifdef CONFIG_RCU_NOCB_CPU
 /*
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.
@@ -174,6 +175,7 @@ void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
 {
 	rsclp->offloaded = 1;
 }
+#endif

Leave this unconditional, as it is nowhere near a fastpath.

The point was to not raise false hopes to those who want to offload when it's not supported.

Let's perhaps have at least a WARN_ON_ONCE(1) if it is called when !CONFIG_RCU_NOCB_CPU ?

Quoted 20 lines
Quoted 8 lines
kernel/rcu/rcu_segcblist.c +0 -0
 /*
  * Does the specified rcu_segcblist structure contain callbacks that
kernel/rcu/rcu_segcblist.h +0 -0
index 5c293afc07b8..4c1503a82492 100644

kernel/rcu/rcu_segcblist.h static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp) · line 70

Quoted 9 lines
kernel/rcu/rcu_segcblist.h +4 -0
@@ -62,7 +62,11 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
 /* Is the specified rcu_segcblist offloaded?  */
 static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
 {
+#ifdef CONFIG_RCU_NOCB_CPU
 	return rsclp->offloaded;
+#else
+	return false;
+#endif
 }

Then this can just be:

Quoted 1 line
	return IS_ENABLED(CONFIG_RCU_NOCB_CPU) && rsclp->offloaded;

Ok.

Quoted 10 lines

kernel/rcu/rcu_segcblist.h static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) · line 1404

Quoted 7 lines
kernel/rcu/rcu_segcblist.h +1 -2
@@ -1401,8 +1401,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
 {
 	bool ret = false;
 	bool need_qs;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);

The adjustment to rcu_segcblist_is_offloaded() allows this (welcome!) simplification to remain.

Ok thanks!

Quoted 11 lines

kernel/rcu/rcu_segcblist.h static int rcu_pending(int user) · line 3240

Quoted 7 lines
kernel/rcu/rcu_segcblist.h +1 -2
@@ -3243,8 +3237,7 @@ static int rcu_pending(int user)
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
 	if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
-	    (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
-	     !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
+	    !rcu_segcblist_is_offloaded(&rdp->cblist) &&

Ditto.

As in "Why didn't I do it that way to start with???" ;-)

You say that to someone who's too lazy to script short commands typed 100 times a day ;-)

maintainer
#28 lore raw

Re: [PATCH 04/10] rcu: Implement rcu_segcblist_is_offloaded() config dependent

Quoted 62 lines · Frederic Weisbecker
Quoted 18 lines · Paul E. McKenney
Quoted 17 lines · Frederic Weisbecker

This simplify the usage of this API and avoid checking the kernel config from the callers.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcu_segcblist.h |  2 ++
 kernel/rcu/rcu_segcblist.c    |  2 ++
 kernel/rcu/rcu_segcblist.h    |  6 ++++++
 kernel/rcu/tree.c             | 21 +++++++--------------
 4 files changed, 17 insertions(+), 14 deletions(-)
Quoted 15 lines

include/linux/rcu_segcblist.h struct rcu_segcblist { · line 78

Quoted 11 lines
include/linux/rcu_segcblist.h +2 -0
index b36afe7b22c9..0ced0a0ecbcf 100644
@@ -73,7 +73,9 @@ struct rcu_segcblist {
 	long len;
 #endif
 	u8 enabled;
+#ifdef CONFIG_RCU_NOCB_CPU
 	u8 offloaded;
+#endif

Given that this is only one byte and that removing it won't actually save any memory on most architectures, why not just leave it and adjust as shown below?

Right, the point was to make it private to that config and trigger a build error otherwise. But if we have an off case that's fine.

Quoted 12 lines
Quoted 12 lines
include/linux/rcu_segcblist.h +0 -0
 };
 
 #define RCU_SEGCBLIST_INITIALIZER(n) \
kernel/rcu/rcu_segcblist.c +0 -0
index 9a0f66133b4b..d8ea2bef5574 100644
@@ -166,6 +166,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
 	rsclp->enabled = 0;
 }
 
Quoted 11 lines

kernel/rcu/rcu_segcblist.c void rcu_segcblist_offload(struct rcu_segcblist *rsclp) · line 178

Quoted 9 lines
kernel/rcu/rcu_segcblist.c +2 -0
+#ifdef CONFIG_RCU_NOCB_CPU
 /*
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.
@@ -174,6 +175,7 @@ void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
 {
 	rsclp->offloaded = 1;
 }
+#endif

Leave this unconditional, as it is nowhere near a fastpath.

Quoted 5 lines

The point was to not raise false hopes to those who want to offload when it's not supported.

Let's perhaps have at least a WARN_ON_ONCE(1) if it is called when !CONFIG_RCU_NOCB_CPU ?

Sounds like a good choice to me!

Quoted 46 lines
Quoted 19 lines
Quoted 7 lines
kernel/rcu/rcu_segcblist.c +0 -0
 /*
  * Does the specified rcu_segcblist structure contain callbacks that
kernel/rcu/rcu_segcblist.h +0 -0
index 5c293afc07b8..4c1503a82492 100644

kernel/rcu/rcu_segcblist.h static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp) · line 70

Quoted 9 lines
kernel/rcu/rcu_segcblist.h +4 -0
@@ -62,7 +62,11 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
 /* Is the specified rcu_segcblist offloaded?  */
 static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
 {
+#ifdef CONFIG_RCU_NOCB_CPU
 	return rsclp->offloaded;
+#else
+	return false;
+#endif
 }

Then this can just be:

Quoted 1 line
	return IS_ENABLED(CONFIG_RCU_NOCB_CPU) && rsclp->offloaded;

Ok.

Quoted 10 lines

kernel/rcu/rcu_segcblist.h static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) · line 1404

Quoted 7 lines
kernel/rcu/rcu_segcblist.h +1 -2
@@ -1401,8 +1401,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
 {
 	bool ret = false;
 	bool need_qs;
-	const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
-			       rcu_segcblist_is_offloaded(&rdp->cblist);
+	const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);

The adjustment to rcu_segcblist_is_offloaded() allows this (welcome!) simplification to remain.

Ok thanks!

Quoted 10 lines

kernel/rcu/rcu_segcblist.h static int rcu_pending(int user) · line 3240

Quoted 7 lines
kernel/rcu/rcu_segcblist.h +1 -2
@@ -3243,8 +3237,7 @@ static int rcu_pending(int user)
 
 	/* Has RCU gone idle with this CPU needing another grace period? */
 	if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
-	    (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
-	     !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
+	    !rcu_segcblist_is_offloaded(&rdp->cblist) &&

Ditto.

Quoted 4 lines
Quoted 1 line

As in "Why didn't I do it that way to start with???" ;-)

You say that to someone who's too lazy to script short commands typed 100 times a day ;-)

;-) ;-) ;-)

Thanx, Paul

author maintainer
#29 lore raw

PATCH 5/10 rcu: Remove useless conditional nocb unlock

In reply to Frederic Weisbecker

Not only is it in the bad order (rdp->nocb_lock should be unlocked after rnp) but it's also dead code as we are in the !offloaded path.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree.c | 1 -
 1 file changed, 1 deletion(-)
kernel/rcu/tree.c +0 -1
index 804cf7dfff03..cc95419f6491 100644
@@ -3680,7 +3680,6 @@ void rcutree_migrate_callbacks(int cpu)
 		raw_spin_unlock_rcu_node(my_rnp); /* irqs remain disabled. */
 		__call_rcu_nocb_wake(my_rdp, true, flags);
 	} else {
-		rcu_nocb_unlock(my_rdp); /* irqs remain disabled. */
 		raw_spin_unlock_irqrestore_rcu_node(my_rnp, flags);
 	}
 	if (needwake)
Signature · 1 line
2.25.0
author maintainer
#30 lore raw

PATCH 6/10 rcu: Make nocb_cb kthread parkable

In reply to Frederic Weisbecker

This will be necessary to correctly implement rdp de-offloading. We don't want rcu_do_batch() in nocb_cb kthread to race with local rcu_do_batch().

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree_plugin.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +5 -1
index 1dd3fdd675a1..43ecc047af26 100644
@@ -2104,7 +2104,9 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
-				 !READ_ONCE(rdp->nocb_cb_sleep));
+				    !READ_ONCE(rdp->nocb_cb_sleep) ||
+				    kthread_should_park());
+
 	if (!smp_load_acquire(&rdp->nocb_cb_sleep)) { /* VVV */
 		/* ^^^ Ensure CB invocation follows _sleep test. */
 		return;
@@ -2125,6 +2127,8 @@ static int rcu_nocb_cb_kthread(void *arg)
 	// if there are no more ready callbacks, waits for them.
 	for (;;) {
 		nocb_cb_wait(rdp);
+		if (kthread_should_park())
+			kthread_parkme();
 		cond_resched_tasks_rcu_qs();
 	}
 	return 0;
Signature · 1 line
2.25.0
Reviewed-by on v1 6/10
#31 lore raw

Re: [PATCH 06/10] rcu: Make nocb_cb kthread parkable

Quoted 11 lines · Frederic Weisbecker

This will be necessary to correctly implement rdp de-offloading. We don't want rcu_do_batch() in nocb_cb kthread to race with local rcu_do_batch().

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>

thanks,

- Joel

Quoted 31 lines
---
 kernel/rcu/tree_plugin.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +5 -1
index 1dd3fdd675a1..43ecc047af26 100644
@@ -2104,7 +2104,9 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	if (needwake_gp)
 		rcu_gp_kthread_wake();
 	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
-				 !READ_ONCE(rdp->nocb_cb_sleep));
+				    !READ_ONCE(rdp->nocb_cb_sleep) ||
+				    kthread_should_park());
+
 	if (!smp_load_acquire(&rdp->nocb_cb_sleep)) { /* VVV */
 		/* ^^^ Ensure CB invocation follows _sleep test. */
 		return;
@@ -2125,6 +2127,8 @@ static int rcu_nocb_cb_kthread(void *arg)
 	// if there are no more ready callbacks, waits for them.
 	for (;;) {
 		nocb_cb_wait(rdp);
+		if (kthread_should_park())
+			kthread_parkme();
 		cond_resched_tasks_rcu_qs();
 	}
 	return 0;
Signature · 1 line
2.25.0
author maintainer
#32 lore raw

PATCH 7/10 rcu: Temporarily assume that nohz full CPUs might not be NOCB

In reply to Frederic Weisbecker

So far nohz_full CPUs had to be nocb. This requirement may change temporarily as we are working on preparing RCU to be able to toggle the nocb state of a CPU. Once that is done and nohz_full can be toggled as well dynamically, we'll restore that initial requirement.

Thus for now as a temporary state, make rcu_nohz_full_cpu() aware of nohz_full CPUs that are not nocb so that they can handle the callbacks locally.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree.c        | 2 +-
 kernel/rcu/tree.h        | 2 +-
 kernel/rcu/tree_plugin.h | 7 ++++---
 3 files changed, 6 insertions(+), 5 deletions(-)
kernel/rcu/tree.c +1 -1
index cc95419f6491..74b6798309ef 100644
@@ -3223,7 +3223,7 @@ static int rcu_pending(int user)
 		return 1;
 
 	/* Is this a nohz_full CPU in userspace or idle?  (Ignore RCU if so.) */
-	if ((user || rcu_is_cpu_rrupt_from_idle()) && rcu_nohz_full_cpu())
+	if ((user || rcu_is_cpu_rrupt_from_idle()) && rcu_nohz_full_cpu(rdp))
 		return 0;
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
kernel/rcu/tree.h +1 -1
index 9dc2ec021da5..4b9643d9f5e0 100644
@@ -451,7 +451,7 @@ do {									\
 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
 
 static void rcu_bind_gp_kthread(void);
-static bool rcu_nohz_full_cpu(void);
+static bool rcu_nohz_full_cpu(struct rcu_data *rdp);
 static void rcu_dynticks_task_enter(void);
 static void rcu_dynticks_task_exit(void);
 
kernel/rcu/tree_plugin.h +4 -3
index 43ecc047af26..f19e81e0c691 100644
@@ -2532,13 +2532,14 @@ static void show_rcu_nocb_state(struct rcu_data *rdp)
  * The idea is to avoid waking up RCU core processing on such a
  * CPU unless the grace period has extended for too long.
  *
- * This code relies on the fact that all NO_HZ_FULL CPUs are also
- * CONFIG_RCU_NOCB_CPU CPUs.
+ * This code relies on the fact that NO_HZ_FULL CPUs might not
+ * be CONFIG_RCU_NOCB_CPU CPUs (temporary development state).
  */
-static bool rcu_nohz_full_cpu(void)
+static bool rcu_nohz_full_cpu(struct rcu_data *rdp)
 {
 #ifdef CONFIG_NO_HZ_FULL
 	if (tick_nohz_full_cpu(smp_processor_id()) &&
+	    rcu_segcblist_is_offloaded(&rdp->cblist) &&
 	    (!rcu_gp_in_progress() ||
 	     ULONG_CMP_LT(jiffies, READ_ONCE(rcu_state.gp_start) + HZ)))
 		return true;
Signature · 1 line
2.25.0
maintainer
#33 lore raw

Re: [PATCH 07/10] rcu: Temporarily assume that nohz full CPUs might not be NOCB

Quoted 4 lines · Frederic Weisbecker

So far nohz_full CPUs had to be nocb. This requirement may change temporarily as we are working on preparing RCU to be able to toggle the nocb state of a CPU. Once that is done and nohz_full can be toggled as well dynamically, we'll restore that initial requirement.

Would it simplify anything to make the CPU exit nohz_full first and then exit rcu_nocb and vice versa in the other direction? That way the assumption about nohz_full CPUs always being rcu_nocb could remain while still allowing runtime changes to both states.

Of course, given that setup, it would not be possible to cause a CPU to exit rcu_nocb state if it was still in nohz_full state.

My fear is that allowing a CPU to be in nohz_full state without also being in rcu_nocb state will cause needless confusion and bug reports.

Thanx, Paul

Quoted 68 lines

Thus for now as a temporary state, make rcu_nohz_full_cpu() aware of nohz_full CPUs that are not nocb so that they can handle the callbacks locally.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/rcu/tree.c        | 2 +-
 kernel/rcu/tree.h        | 2 +-
 kernel/rcu/tree_plugin.h | 7 ++++---
 3 files changed, 6 insertions(+), 5 deletions(-)
kernel/rcu/tree.c +1 -1
index cc95419f6491..74b6798309ef 100644
@@ -3223,7 +3223,7 @@ static int rcu_pending(int user)
 		return 1;
 
 	/* Is this a nohz_full CPU in userspace or idle?  (Ignore RCU if so.) */
-	if ((user || rcu_is_cpu_rrupt_from_idle()) && rcu_nohz_full_cpu())
+	if ((user || rcu_is_cpu_rrupt_from_idle()) && rcu_nohz_full_cpu(rdp))
 		return 0;
 
 	/* Is the RCU core waiting for a quiescent state from this CPU? */
kernel/rcu/tree.h +1 -1
index 9dc2ec021da5..4b9643d9f5e0 100644
@@ -451,7 +451,7 @@ do {									\
 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
 
 static void rcu_bind_gp_kthread(void);
-static bool rcu_nohz_full_cpu(void);
+static bool rcu_nohz_full_cpu(struct rcu_data *rdp);
 static void rcu_dynticks_task_enter(void);
 static void rcu_dynticks_task_exit(void);
 
kernel/rcu/tree_plugin.h +4 -3
index 43ecc047af26..f19e81e0c691 100644
@@ -2532,13 +2532,14 @@ static void show_rcu_nocb_state(struct rcu_data *rdp)
  * The idea is to avoid waking up RCU core processing on such a
  * CPU unless the grace period has extended for too long.
  *
- * This code relies on the fact that all NO_HZ_FULL CPUs are also
- * CONFIG_RCU_NOCB_CPU CPUs.
+ * This code relies on the fact that NO_HZ_FULL CPUs might not
+ * be CONFIG_RCU_NOCB_CPU CPUs (temporary development state).
  */
-static bool rcu_nohz_full_cpu(void)
+static bool rcu_nohz_full_cpu(struct rcu_data *rdp)
 {
 #ifdef CONFIG_NO_HZ_FULL
 	if (tick_nohz_full_cpu(smp_processor_id()) &&
+	    rcu_segcblist_is_offloaded(&rdp->cblist) &&
 	    (!rcu_gp_in_progress() ||
 	     ULONG_CMP_LT(jiffies, READ_ONCE(rcu_state.gp_start) + HZ)))
 		return true;
Signature · 1 line
2.25.0
author maintainer
#34 lore raw

Re: [PATCH 07/10] rcu: Temporarily assume that nohz full CPUs might not be NOCB

Quoted 10 lines · Paul E. McKenney
Quoted 4 lines · Frederic Weisbecker

So far nohz_full CPUs had to be nocb. This requirement may change temporarily as we are working on preparing RCU to be able to toggle the nocb state of a CPU. Once that is done and nohz_full can be toggled as well dynamically, we'll restore that initial requirement.

Would it simplify anything to make the CPU exit nohz_full first and then exit rcu_nocb and vice versa in the other direction? That way the assumption about nohz_full CPUs always being rcu_nocb could remain while still allowing runtime changes to both states.

That's the future plan but for now nohz_full can't even be exited yet. RCU is unlucky enough to be chosen as the starting point of this whole work :-)

Quoted 2 lines

Of course, given that setup, it would not be possible to cause a CPU to exit rcu_nocb state if it was still in nohz_full state.

Right.

Quoted 2 lines

My fear is that allowing a CPU to be in nohz_full state without also being in rcu_nocb state will cause needless confusion and bug reports.

Well, it should only be visible to those who work on it since there won't be a proper interface before we achieve the whole.

Thanks.

maintainer
#35 lore raw

Re: [PATCH 07/10] rcu: Temporarily assume that nohz full CPUs might not be NOCB

Quoted 7 lines · Frederic Weisbecker
Quoted 6 lines · Paul E. McKenney
Quoted 4 lines · Frederic Weisbecker

So far nohz_full CPUs had to be nocb. This requirement may change temporarily as we are working on preparing RCU to be able to toggle the nocb state of a CPU. Once that is done and nohz_full can be toggled as well dynamically, we'll restore that initial requirement.

Quoted 7 lines
Quoted 4 lines

Would it simplify anything to make the CPU exit nohz_full first and then exit rcu_nocb and vice versa in the other direction? That way the assumption about nohz_full CPUs always being rcu_nocb could remain while still allowing runtime changes to both states.

That's the future plan but for now nohz_full can't even be exited yet. RCU is unlucky enough to be chosen as the starting point of this whole work :-)

But testing could still start with CPUs marked rcu_nocb but not marked nohz_full, right? I must confess that I am a bit concerned about the increase in state space.

Quoted 10 lines
Quoted 2 lines

Of course, given that setup, it would not be possible to cause a CPU to exit rcu_nocb state if it was still in nohz_full state.

Right.

Quoted 2 lines

My fear is that allowing a CPU to be in nohz_full state without also being in rcu_nocb state will cause needless confusion and bug reports.

Well, it should only be visible to those who work on it since there won't be a proper interface before we achieve the whole.

Fair point, but I am also concerned about the welfare of the people working on it. ;-)

Thanx, Paul

author maintainer
#36 lore raw

Re: [PATCH 07/10] rcu: Temporarily assume that nohz full CPUs might not be NOCB

Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Paul E. McKenney
Quoted 4 lines · Frederic Weisbecker

So far nohz_full CPUs had to be nocb. This requirement may change temporarily as we are working on preparing RCU to be able to toggle the nocb state of a CPU. Once that is done and nohz_full can be toggled as well dynamically, we'll restore that initial requirement.

Would it simplify anything to make the CPU exit nohz_full first and then exit rcu_nocb and vice versa in the other direction? That way the assumption about nohz_full CPUs always being rcu_nocb could remain while still allowing runtime changes to both states.

Quoted 5 lines
Quoted 2 lines

That's the future plan but for now nohz_full can't even be exited yet. RCU is unlucky enough to be chosen as the starting point of this whole work :-)

But testing could still start with CPUs marked rcu_nocb but not marked nohz_full, right?

Ah! That makes sense indeed. I should indeed restrict de-offloading to CPUs that are not nohz_full.

Quoted 1 line

I must confess that I am a bit concerned about the increase in state space.

Yeah good point!

Quoted 2 lines

Fair point, but I am also concerned about the welfare of the people working on it. ;-)

Fair enough! :-)

author maintainer
#37 lore raw

PATCH 8/10 rcu: Allow to deactivate nocb on a CPU

In reply to Frederic Weisbecker

Allow a CPU's rdp to quit the callback offlined mode.
The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.

nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcupdate.h   |  2 ++
 kernel/rcu/rcu_segcblist.c |  4 +--
 kernel/rcu/rcu_segcblist.h |  2 +-
 kernel/rcu/tree_plugin.h   | 50 +++++++++++++++++++++++++++++++++++++-
 4 files changed, 54 insertions(+), 4 deletions(-)
include/linux/rcupdate.h +2 -0
index 2678a37c3169..1d3a4c37c3c1 100644
@@ -96,8 +96,10 @@ static inline void rcu_user_exit(void) { }
 
 #ifdef CONFIG_RCU_NOCB_CPU
 void rcu_init_nohz(void);
+void rcu_nocb_cpu_deoffload(int cpu);
 #else /* #ifdef CONFIG_RCU_NOCB_CPU */
 static inline void rcu_init_nohz(void) { }
+static inline void rcu_nocb_cpu_deoffload(int cpu) { }
 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
 
 /**
kernel/rcu/rcu_segcblist.c +2 -2
index d8ea2bef5574..4bed48da7702 100644
@@ -171,9 +171,9 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.
  */
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
 {
-	rsclp->offloaded = 1;
+	rsclp->offloaded = offload;
 }
 #endif
 
kernel/rcu/rcu_segcblist.h +1 -1
index 4c1503a82492..8f7c6c34cb1b 100644
@@ -83,7 +83,7 @@ void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
 void rcu_segcblist_init(struct rcu_segcblist *rsclp);
 void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
 #ifdef CONFIG_RCU_NOCB_CPU
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload);
 #endif
 bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
 bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
kernel/rcu/tree_plugin.h +49 -1

68 lines not on this page · raw message

Signature · 1 line
2.25.0
maintainer
#38 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 84 lines · Frederic Weisbecker

Allow a CPU's rdp to quit the callback offlined mode.
The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.

nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcupdate.h   |  2 ++
 kernel/rcu/rcu_segcblist.c |  4 +--
 kernel/rcu/rcu_segcblist.h |  2 +-
 kernel/rcu/tree_plugin.h   | 50 +++++++++++++++++++++++++++++++++++++-
 4 files changed, 54 insertions(+), 4 deletions(-)
include/linux/rcupdate.h +2 -0
index 2678a37c3169..1d3a4c37c3c1 100644
@@ -96,8 +96,10 @@ static inline void rcu_user_exit(void) { }
 
 #ifdef CONFIG_RCU_NOCB_CPU
 void rcu_init_nohz(void);
+void rcu_nocb_cpu_deoffload(int cpu);
 #else /* #ifdef CONFIG_RCU_NOCB_CPU */
 static inline void rcu_init_nohz(void) { }
+static inline void rcu_nocb_cpu_deoffload(int cpu) { }
 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
 
 /**
kernel/rcu/rcu_segcblist.c +2 -2
index d8ea2bef5574..4bed48da7702 100644
@@ -171,9 +171,9 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.
  */
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
 {
-	rsclp->offloaded = 1;
+	rsclp->offloaded = offload;
 }
 #endif
 
kernel/rcu/rcu_segcblist.h +1 -1
index 4c1503a82492..8f7c6c34cb1b 100644
@@ -83,7 +83,7 @@ void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
 void rcu_segcblist_init(struct rcu_segcblist *rsclp);
 void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
 #ifdef CONFIG_RCU_NOCB_CPU
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload);
 #endif
 bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
 bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
kernel/rcu/tree_plugin.h +4 -0
index f19e81e0c691..c74a4df8d5f2 100644
@@ -1943,6 +1943,10 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
 		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+		if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
+			continue;
+		}
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2189

Quoted 10 lines
kernel/rcu/tree_plugin.h +7 -0
@@ -2176,6 +2180,50 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)
 		do_nocb_deferred_wakeup_common(rdp);
 }
 
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+	unsigned long flags;
+	struct rcu_node *rnp = rdp->mynode;
+
+	printk("De-offloading %d\n", rdp->cpu);
+	kthread_park(rdp->nocb_cb_kthread);

I am a bit concerned about this, because from this point until the end of this function, no RCU callbacks can be invoked for this CPU. This could be a problem if there is a callback flood in progress, and such callback floods are in fact one reason that the sysadm might want to be switching from offloaded to non-offloaded. Is it possible to move this kthread_park() to the end of this function? Yes, this can result in concurrent invocation of different callbacks for this CPU, but because we have excluded rcu_barrier(), that should be OK.

Or is there some failure mode that I am failing to see? (Wouldn't be the first time...)

Quoted 7 lines
kernel/rcu/tree_plugin.h +7 -0
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+	rcu_nocb_flush_bypass(rdp, NULL, jiffies);
+	raw_spin_lock_rcu_node(rnp);
+	rcu_segcblist_offload(&rdp->cblist, false);
+	raw_spin_unlock_rcu_node(rnp);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
+}

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2207

Quoted 10 lines
kernel/rcu/tree_plugin.h +10 -0
+
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_deoffload(rdp);
+
+	return 0;
+}

For example, is the problem caused by invocations of this rcu_nocb_rdp_deoffload() function? But if so, do we really need to acquire rcu_state.barrier_mutex?

That aside, if it is possible to do the switch without interrupting callback invocation? Or is your idea that because we are always executing on the CPU being deoffloaded, that CPU has been prevented from posting callbacks in any case? If the latter, does that mean that it is not possible to deoffload offlined CPUs? (Not sure whether this restriction is a real problem, but figured that I should ask.)

Thanx, Paul

Quoted 33 lines
kernel/rcu/tree_plugin.h +19 -1
+void rcu_nocb_cpu_deoffload(int cpu)
+{
+	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
+
+	mutex_lock(&rcu_state.barrier_mutex);
+	cpus_read_lock();
+	if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
+		if (cpu_online(cpu)) {
+			work_on_cpu(cpu, rcu_nocb_rdp_deoffload, rdp);
+		} else {
+			__rcu_nocb_rdp_deoffload(rdp);
+		}
+		cpumask_clear_cpu(cpu, rcu_nocb_mask);
+	}
+	cpus_read_unlock();
+	mutex_unlock(&rcu_state.barrier_mutex);
+}
+
 void __init rcu_init_nohz(void)
 {
 	int cpu;
@@ -2218,7 +2266,7 @@ void __init rcu_init_nohz(void)
 		rdp = per_cpu_ptr(&rcu_data, cpu);
 		if (rcu_segcblist_empty(&rdp->cblist))
 			rcu_segcblist_init(&rdp->cblist);
-		rcu_segcblist_offload(&rdp->cblist);
+		rcu_segcblist_offload(&rdp->cblist, true);
 	}
 	rcu_organize_nocb_kthreads();
 }
Signature · 1 line
2.25.0
author maintainer
#39 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 18 lines · Paul E. McKenney

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2189

Quoted 7 lines · Frederic Weisbecker
kernel/rcu/tree_plugin.h +7 -0
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+	unsigned long flags;
+	struct rcu_node *rnp = rdp->mynode;
+
+	printk("De-offloading %d\n", rdp->cpu);
+	kthread_park(rdp->nocb_cb_kthread);

I am a bit concerned about this, because from this point until the end of this function, no RCU callbacks can be invoked for this CPU. This could be a problem if there is a callback flood in progress, and such callback floods are in fact one reason that the sysadm might want to be switching from offloaded to non-offloaded. Is it possible to move this kthread_park() to the end of this function? Yes, this can result in concurrent invocation of different callbacks for this CPU, but because we have excluded rcu_barrier(), that should be OK.

Quoted 2 lines

Or is there some failure mode that I am failing to see? (Wouldn't be the first time...)

Heh I actually worried about that. Ok the issue is that it leaves a window where nocb_cb and local caller of rcu_do_batch() can compete but the local caller doesn't lock the nocb_lock.

So there are two ways to solve that.:

1) - set cblist->offloaded = 0 locally
- From the kthread while calling rcu_do_batch():
check the value of cblist->offloaded everytime after we call rcu_nocb_lock() and stop messsing with the cblist and return when we see cblist->offloaded == 0
- Allow to handle cblist locally without taking the nocb_lock
- Park kthread

But there I'm worried about races. Imagine we have:

      Kthread                     Local
      --------                   -------
      rcu_do_batch() {
          rcu_nocb_lock()
          do stuff with cblist
          rcu_nocb_unlock()
                                 rcu_nocb_lock()
                                 set cblist->offloaded = 0
                                 rcu_nocb_unlock()
                                 ===> INT or preemption
                                 rcu_do_batch() {
                                     do stuff with cblist

Are we guaranteed that the Local CPU will see the updates from the kthread while calling rcu_do_batch()? I would tend to say yes but I'm not entirely sure...

Oh wait, that solution also implies that we can't re-enqueue extracted callbacks if we spent took much time in threaded rcu_do_batch(), as the cblist may have been offloaded while we executed the extracted callbacks.

That's a lot of corner cases to handle, which is why I prefer the other solution:

2) enum cblist_offloaded {
CBLIST_NOT_OFFLOADED,
CBLIST_(DE)OFFLOADING,
CBLIST_OFFLOADED
}

- Locally set cblist->offloaded = CBLIST_DEOFFLOADING
- From the kthread while calling rcu_do_batch(), do as usual.
- Local CPU can call rcu_do_batch() and if it sees CBLIST_DEOFFLOADING, rcu_nocb_lock() will take the lock.
- Park kthread
- Locally set cblist->offloaded = CBLIST_NOT_OFFLOADED
- Local calls to rcu_do_batch() won't take the lock anymore.

Quoted 4 lines
Quoted 4 lines
kernel/rcu/tree_plugin.h +4 -0
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
Quoted 8 lines

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2207

Quoted 5 lines
kernel/rcu/tree_plugin.h +5 -0
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_deoffload(rdp);
+
+	return 0;
+}

For example, is the problem caused by invocations of this rcu_nocb_rdp_deoffload() function?

How so?

Quoted 1 line

But if so, do we really need to acquire rcu_state.barrier_mutex?

Indeed it was probably not needed if we parked the kthread before anything, as we would have kept the callbacks ordering.

But now if we allow concurrent callbacks execution during the small window, we'll need it.

Quoted 5 lines

That aside, if it is possible to do the switch without interrupting callback invocation? Or is your idea that because we are always executing on the CPU being deoffloaded, that CPU has been prevented from posting callbacks in any case?

No in the tiny window between kthread_park() and the irqs being disabled, the workqueue can be preempted and thus call_rcu() can be called anytime.

Quoted 3 lines

If the latter, does that mean that it is not
possible to deoffload offlined CPUs? (Not sure whether this restriction is a real problem, but figured that I should ask.)

Ah in the case of offlined CPUs I simply call the function directly from the CPU that disables the nocb remotely. So we remotely park the kthread (that we always do anyway) and set offlined.

Thanks.

maintainer
#40 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 19 lines · Frederic Weisbecker
Quoted 18 lines · Paul E. McKenney

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2189

Quoted 7 lines · Frederic Weisbecker
kernel/rcu/tree_plugin.h +7 -0
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+	unsigned long flags;
+	struct rcu_node *rnp = rdp->mynode;
+
+	printk("De-offloading %d\n", rdp->cpu);
+	kthread_park(rdp->nocb_cb_kthread);

I am a bit concerned about this, because from this point until the end of this function, no RCU callbacks can be invoked for this CPU. This could be a problem if there is a callback flood in progress, and such callback floods are in fact one reason that the sysadm might want to be switching from offloaded to non-offloaded. Is it possible to move this kthread_park() to the end of this function? Yes, this can result in concurrent invocation of different callbacks for this CPU, but because we have excluded rcu_barrier(), that should be OK.

Quoted 6 lines
Quoted 2 lines

Or is there some failure mode that I am failing to see? (Wouldn't be the first time...)

Heh I actually worried about that. Ok the issue is that it leaves a window where nocb_cb and local caller of rcu_do_batch() can compete but the local caller doesn't lock the nocb_lock.

Indeed, my nightmare scenario involves some sort of preemption or similar long delay at that point. Callbacks pile up, and then OOM!

Quoted 44 lines

So there are two ways to solve that.:

1) - set cblist->offloaded = 0 locally
- From the kthread while calling rcu_do_batch():
check the value of cblist->offloaded everytime after we call rcu_nocb_lock() and stop messsing with the cblist and return when we see cblist->offloaded == 0
- Allow to handle cblist locally without taking the nocb_lock
- Park kthread

But there I'm worried about races. Imagine we have:

      Kthread                     Local
      --------                   -------
      rcu_do_batch() {
          rcu_nocb_lock()
          do stuff with cblist
          rcu_nocb_unlock()
                                 rcu_nocb_lock()
                                 set cblist->offloaded = 0
                                 rcu_nocb_unlock()
                                 ===> INT or preemption
                                 rcu_do_batch() {
                                     do stuff with cblist

Are we guaranteed that the Local CPU will see the updates from the kthread while calling rcu_do_batch()? I would tend to say yes but I'm not entirely sure...

Oh wait, that solution also implies that we can't re-enqueue extracted callbacks if we spent took much time in threaded rcu_do_batch(), as the cblist may have been offloaded while we executed the extracted callbacks.

That's a lot of corner cases to handle, which is why I prefer the other solution:

2) enum cblist_offloaded {
CBLIST_NOT_OFFLOADED,
CBLIST_(DE)OFFLOADING,
CBLIST_OFFLOADED
}

Quoted 8 lines

- Locally set cblist->offloaded = CBLIST_DEOFFLOADING
- From the kthread while calling rcu_do_batch(), do as usual.
- Local CPU can call rcu_do_batch() and if it sees CBLIST_DEOFFLOADING, rcu_nocb_lock() will take the lock.
- Park kthread
- Locally set cblist->offloaded = CBLIST_NOT_OFFLOADED
- Local calls to rcu_do_batch() won't take the lock anymore.

This last seems best to me. The transition from CBLIST_NOT_OFFLOADED to CBLIST_OFFLOADING of course needs to be on the CPU in question with at least bh disabled. Probably best to be holding rcu_nocb_lock(), but that might just be me being overly paranoid.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
kernel/rcu/tree_plugin.h +4 -0
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
Quoted 8 lines

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2207

Quoted 5 lines
kernel/rcu/tree_plugin.h +5 -0
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_deoffload(rdp);
+
+	return 0;
+}

For example, is the problem caused by invocations of this rcu_nocb_rdp_deoffload() function?

Quoted 1 line

How so?

It looked to me like it wasn't excluding either rcu_barrier() or CPU hotplug. It might also not have been pinning onto the CPU in question, but that might just be me misremembering. Then again, I didn't see a call to it, so maybe its callers set things up appropriately.

OK, I will bite... What is the purpose of rcu_nocb_rdp_deoffload()? ;-)

Quoted 7 lines
Quoted 1 line

But if so, do we really need to acquire rcu_state.barrier_mutex?

Indeed it was probably not needed if we parked the kthread before anything, as we would have kept the callbacks ordering.

But now if we allow concurrent callbacks execution during the small window, we'll need it.

Agreed! And I do believe that concurrent callback execution will prove better than a possibly indefinite gap in callback execution.

Quoted 7 lines
Quoted 4 lines

That aside, if it is possible to do the switch without interrupting callback invocation? Or is your idea that because we are always executing on the CPU being deoffloaded, that CPU has been prevented from posting callbacks in any case?

No in the tiny window between kthread_park() and the irqs being disabled, the workqueue can be preempted and thus call_rcu() can be called anytime.

Agreed! ;-)

Quoted 7 lines
Quoted 3 lines

If the latter, does that mean that it is not
possible to deoffload offlined CPUs? (Not sure whether this restriction is a real problem, but figured that I should ask.)

Ah in the case of offlined CPUs I simply call the function directly from the CPU that disables the nocb remotely. So we remotely park the kthread (that we always do anyway) and set offlined.

And the cpus_read_lock() in rcu_nocb_cpu_deoffload() prevents that CPU from coming back online, so looks good!

Thanx, Paul

author maintainer
#41 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 5 lines · Paul E. McKenney

On Thu, May 14, 2020 at 12:45:26AM +0200, Frederic Weisbecker wrote:
This last seems best to me. The transition from CBLIST_NOT_OFFLOADED to CBLIST_OFFLOADING of course needs to be on the CPU in question with at least bh disabled. Probably best to be holding rcu_nocb_lock(), but that might just be me being overly paranoid.

So that's in the case of offloading, right? Well, I don't think we'd need to even disable bh nor lock nocb. We just need the current CPU to see the local update of cblist->offloaded = CBLIST_OFFLOADING before the kthread is unparked:

    cblist->offloaded = CBLIST_OFFLOADING;
    /* Make sure subsequent softirq lock nocb */
    barrier();
    kthread_unpark(rdp->nocb_cb_thread);

Now, although that guarantees that nocb_cb will see CBLIST_OFFLOADING upon unparking, it's not guaranteed that the nocb_gp will see it on its next round. Ok so eventually you're right, I should indeed lock nocb...

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2207

Quoted 10 lines
kernel/rcu/tree_plugin.h +9 -0
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_deoffload(rdp);
+
+	return 0;
+}

For example, is the problem caused by invocations of this rcu_nocb_rdp_deoffload() function?

Quoted 8 lines
Quoted 1 line

How so?

It looked to me like it wasn't excluding either rcu_barrier() or CPU hotplug. It might also not have been pinning onto the CPU in question, but that might just be me misremembering. Then again, I didn't see a call to it, so maybe its callers set things up appropriately.

OK, I will bite... What is the purpose of rcu_nocb_rdp_deoffload()? ;-)

Ah it's called using work_on_cpu() which launch a workqueue on the target and waits for completion. And that whole thing is protected inside the barrier mutex and hotplug.

Quoted 2 lines

Agreed! And I do believe that concurrent callback execution will prove better than a possibly indefinite gap in callback execution.

Mutual agreement! :-)

Thanks.

maintainer
#42 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 12 lines · Frederic Weisbecker
Quoted 5 lines · Paul E. McKenney

On Thu, May 14, 2020 at 12:45:26AM +0200, Frederic Weisbecker wrote:
This last seems best to me. The transition from CBLIST_NOT_OFFLOADED to CBLIST_OFFLOADING of course needs to be on the CPU in question with at least bh disabled. Probably best to be holding rcu_nocb_lock(), but that might just be me being overly paranoid.

So that's in the case of offloading, right? Well, I don't think we'd need to even disable bh nor lock nocb. We just need the current CPU to see the local update of cblist->offloaded = CBLIST_OFFLOADING before the kthread is unparked:

Quoted 8 lines
    cblist->offloaded = CBLIST_OFFLOADING;
    /* Make sure subsequent softirq lock nocb */
    barrier();
    kthread_unpark(rdp->nocb_cb_thread);

Now, although that guarantees that nocb_cb will see CBLIST_OFFLOADING upon unparking, it's not guaranteed that the nocb_gp will see it on its next round. Ok so eventually you're right, I should indeed lock nocb...

I suspect that our future selves would hate us much less if we held that lock. ;-)

Quoted 20 lines
Quoted 20 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
kernel/rcu/tree_plugin.h +4 -0
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
Quoted 8 lines

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2207

Quoted 5 lines
kernel/rcu/tree_plugin.h +5 -0
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_deoffload(rdp);
+
+	return 0;
+}

For example, is the problem caused by invocations of this rcu_nocb_rdp_deoffload() function?

Quoted 1 line

How so?

It looked to me like it wasn't excluding either rcu_barrier() or CPU hotplug. It might also not have been pinning onto the CPU in question, but that might just be me misremembering. Then again, I didn't see a call to it, so maybe its callers set things up appropriately.

Quoted 5 lines
Quoted 1 line

OK, I will bite... What is the purpose of rcu_nocb_rdp_deoffload()? ;-)

Ah it's called using work_on_cpu() which launch a workqueue on the target and waits for completion. And that whole thing is protected inside the barrier mutex and hotplug.

Ah! Yet again, color me blind.

Thanx, Paul

Quoted 6 lines
Quoted 2 lines

Agreed! And I do believe that concurrent callback execution will prove better than a possibly indefinite gap in callback execution.

Mutual agreement! :-)

Thanks.

author maintainer
#43 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 18 lines · Paul E. McKenney
Quoted 17 lines · Frederic Weisbecker
Quoted 5 lines · Paul E. McKenney

On Thu, May 14, 2020 at 12:45:26AM +0200, Frederic Weisbecker wrote:
This last seems best to me. The transition from CBLIST_NOT_OFFLOADED to CBLIST_OFFLOADING of course needs to be on the CPU in question with at least bh disabled. Probably best to be holding rcu_nocb_lock(), but that might just be me being overly paranoid.

So that's in the case of offloading, right? Well, I don't think we'd need to even disable bh nor lock nocb. We just need the current CPU to see the local update of cblist->offloaded = CBLIST_OFFLOADING before the kthread is unparked:

    cblist->offloaded = CBLIST_OFFLOADING;
    /* Make sure subsequent softirq lock nocb */
    barrier();
    kthread_unpark(rdp->nocb_cb_thread);
Quoted 6 lines
Quoted 3 lines

Now, although that guarantees that nocb_cb will see CBLIST_OFFLOADING upon unparking, it's not guaranteed that the nocb_gp will see it on its next round. Ok so eventually you're right, I should indeed lock nocb...

I suspect that our future selves would hate us much less if we held that lock. ;-)

Also, taking the decision to hold that lock could teach a lesson to our past selves. Win-win! Let us become that most welcome time bridge!

#44 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

In reply to Frederic Weisbecker
Quoted 1 line · Frederic Weisbecker

Allow a CPU's rdp to quit the callback offlined mode.

nit: s/offlined/offloaded/ ?

Quoted 6 lines

The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.
nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

nit: you mean cblist is set to non-offloaded mode right?

Also, could you clarify better the rcu_barrier bits in the changelog. I know there's some issue if the cblist has both offloaded and non-offloaded callbacks, but it would be good to clarify this here better IMHO.

[...]

Quoted 45 lines
kernel/rcu/rcu_segcblist.c +2 -2
index d8ea2bef5574..4bed48da7702 100644
@@ -171,9 +171,9 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.
  */
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
 {
-	rsclp->offloaded = 1;
+	rsclp->offloaded = offload;
 }
 #endif
 
kernel/rcu/rcu_segcblist.h +1 -1
index 4c1503a82492..8f7c6c34cb1b 100644
@@ -83,7 +83,7 @@ void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
 void rcu_segcblist_init(struct rcu_segcblist *rsclp);
 void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
 #ifdef CONFIG_RCU_NOCB_CPU
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload);
 #endif
 bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
 bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
kernel/rcu/tree_plugin.h +4 -0
index f19e81e0c691..c74a4df8d5f2 100644
@@ -1943,6 +1943,10 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
 		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+		if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
+			continue;
+		}
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2188

Quoted 9 lines
kernel/rcu/tree_plugin.h +6 -0
@@ -2176,6 +2180,50 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)
 		do_nocb_deferred_wakeup_common(rdp);
 }
 
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+	unsigned long flags;
+	struct rcu_node *rnp = rdp->mynode;
+
+	printk("De-offloading %d\n", rdp->cpu);

nit: s/printk/pr_debug/ ?

thanks,

- Joel

Quoted 53 lines
kernel/rcu/tree_plugin.h +39 -1
+	kthread_park(rdp->nocb_cb_kthread);
+
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+	rcu_nocb_flush_bypass(rdp, NULL, jiffies);
+	raw_spin_lock_rcu_node(rnp);
+	rcu_segcblist_offload(&rdp->cblist, false);
+	raw_spin_unlock_rcu_node(rnp);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
+}
+
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_deoffload(rdp);
+
+	return 0;
+}
+
+void rcu_nocb_cpu_deoffload(int cpu)
+{
+	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
+
+	mutex_lock(&rcu_state.barrier_mutex);
+	cpus_read_lock();
+	if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
+		if (cpu_online(cpu)) {
+			work_on_cpu(cpu, rcu_nocb_rdp_deoffload, rdp);
+		} else {
+			__rcu_nocb_rdp_deoffload(rdp);
+		}
+		cpumask_clear_cpu(cpu, rcu_nocb_mask);
+	}
+	cpus_read_unlock();
+	mutex_unlock(&rcu_state.barrier_mutex);
+}
+
 void __init rcu_init_nohz(void)
 {
 	int cpu;
@@ -2218,7 +2266,7 @@ void __init rcu_init_nohz(void)
 		rdp = per_cpu_ptr(&rcu_data, cpu);
 		if (rcu_segcblist_empty(&rdp->cblist))
 			rcu_segcblist_init(&rdp->cblist);
-		rcu_segcblist_offload(&rdp->cblist);
+		rcu_segcblist_offload(&rdp->cblist, true);
 	}
 	rcu_organize_nocb_kthreads();
 }
Signature · 1 line
2.25.0
#45 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 7 lines · Joel Fernandes
Quoted 6 lines

The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.
nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

Quoted 5 lines

nit: you mean cblist is set to non-offloaded mode right?

Also, could you clarify better the rcu_barrier bits in the changelog. I know there's some issue if the cblist has both offloaded and non-offloaded callbacks, but it would be good to clarify this here better IMHO.

And for archival purposes: rcu_barrier needs excluding here because it is possible that for a brief period of time, the callback kthread has been parked to do the mode-switch, and it could be executing a bunch of callbacks when it was asked to park.

Meanwhile, more interrupts happen and more callbacks are queued which are now executing in softirq. This ruins the ordering of callbacks that rcu_barrier needs.

thanks,

- Joel

Quoted 116 lines

[...]

Quoted 45 lines
kernel/rcu/rcu_segcblist.c +2 -2
index d8ea2bef5574..4bed48da7702 100644
@@ -171,9 +171,9 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
  * Mark the specified rcu_segcblist structure as offloaded.  This
  * structure must be empty.
  */
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
 {
-	rsclp->offloaded = 1;
+	rsclp->offloaded = offload;
 }
 #endif
 
kernel/rcu/rcu_segcblist.h +1 -1
index 4c1503a82492..8f7c6c34cb1b 100644
@@ -83,7 +83,7 @@ void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
 void rcu_segcblist_init(struct rcu_segcblist *rsclp);
 void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
 #ifdef CONFIG_RCU_NOCB_CPU
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload);
 #endif
 bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
 bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
kernel/rcu/tree_plugin.h +4 -0
index f19e81e0c691..c74a4df8d5f2 100644
@@ -1943,6 +1943,10 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
 	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
 		raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+		if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
+			raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
+			continue;
+		}
 		bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
 		if (bypass_ncbs &&
 		    (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2188

Quoted 9 lines
kernel/rcu/tree_plugin.h +6 -0
@@ -2176,6 +2180,50 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)
 		do_nocb_deferred_wakeup_common(rdp);
 }
 
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+	unsigned long flags;
+	struct rcu_node *rnp = rdp->mynode;
+
+	printk("De-offloading %d\n", rdp->cpu);

nit: s/printk/pr_debug/ ?

thanks,

- Joel

Quoted 53 lines
kernel/rcu/tree_plugin.h +39 -1
+	kthread_park(rdp->nocb_cb_kthread);
+
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+	rcu_nocb_flush_bypass(rdp, NULL, jiffies);
+	raw_spin_lock_rcu_node(rnp);
+	rcu_segcblist_offload(&rdp->cblist, false);
+	raw_spin_unlock_rcu_node(rnp);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
+}
+
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_deoffload(rdp);
+
+	return 0;
+}
+
+void rcu_nocb_cpu_deoffload(int cpu)
+{
+	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
+
+	mutex_lock(&rcu_state.barrier_mutex);
+	cpus_read_lock();
+	if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
+		if (cpu_online(cpu)) {
+			work_on_cpu(cpu, rcu_nocb_rdp_deoffload, rdp);
+		} else {
+			__rcu_nocb_rdp_deoffload(rdp);
+		}
+		cpumask_clear_cpu(cpu, rcu_nocb_mask);
+	}
+	cpus_read_unlock();
+	mutex_unlock(&rcu_state.barrier_mutex);
+}
+
 void __init rcu_init_nohz(void)
 {
 	int cpu;
@@ -2218,7 +2266,7 @@ void __init rcu_init_nohz(void)
 		rdp = per_cpu_ptr(&rcu_data, cpu);
 		if (rcu_segcblist_empty(&rdp->cblist))
 			rcu_segcblist_init(&rdp->cblist);
-		rcu_segcblist_offload(&rdp->cblist);
+		rcu_segcblist_offload(&rdp->cblist, true);
 	}
 	rcu_organize_nocb_kthreads();
 }
Signature · 1 line
2.25.0
author maintainer
#46 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 15 lines · Joel Fernandes
Quoted 7 lines · Joel Fernandes
Quoted 6 lines

The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.
nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

Quoted 5 lines

nit: you mean cblist is set to non-offloaded mode right?

Also, could you clarify better the rcu_barrier bits in the changelog. I know there's some issue if the cblist has both offloaded and non-offloaded callbacks, but it would be good to clarify this here better IMHO.

Quoted 8 lines

And for archival purposes: rcu_barrier needs excluding here because it is possible that for a brief period of time, the callback kthread has been parked to do the mode-switch, and it could be executing a bunch of callbacks when it was asked to park.

Meanwhile, more interrupts happen and more callbacks are queued which are now executing in softirq. This ruins the ordering of callbacks that rcu_barrier needs.

I think in that case the callbacks would still be executed in order. We wait for the kthread to park before switching to softirq callback execution.

Initially it was to avoid callback ordering issues but I don't recall exactly which. Maybe it wasn't actually needed. But anyway I'll keep it for the next version where, for a brief period of time, nocb kthread will be able to compete with callback execution in softirq.

I'll clarify that in the changelog.

Thanks.

#47 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 21 lines · Frederic Weisbecker
Quoted 20 lines · Joel Fernandes
Quoted 7 lines · Joel Fernandes
Quoted 6 lines

The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.
nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

Quoted 5 lines

nit: you mean cblist is set to non-offloaded mode right?

Also, could you clarify better the rcu_barrier bits in the changelog. I know there's some issue if the cblist has both offloaded and non-offloaded callbacks, but it would be good to clarify this here better IMHO.

And for archival purposes: rcu_barrier needs excluding here because it is possible that for a brief period of time, the callback kthread has been parked to do the mode-switch, and it could be executing a bunch of callbacks when it was asked to park.

Quoted 6 lines
Quoted 3 lines

Meanwhile, more interrupts happen and more callbacks are queued which are now executing in softirq. This ruins the ordering of callbacks that rcu_barrier needs.

I think in that case the callbacks would still be executed in order. We wait for the kthread to park before switching to softirq callback execution.

Ah ok, you are parking the CB kthread after the no-cb CB's are already invoked (that's when parkme() is called -- i.e. after rcu_do_batch() in the CB kthread runs).

Yeah, I don't see the purpose of acquiring rcu_barrier mutex either now. Once you park, all CBs should have been invoked by the nocb CB thread right? kthread_park() waits for the thread to be parked before proceeding. And you don't de-offload before it is parked.

Quoted 4 lines

Initially it was to avoid callback ordering issues but I don't recall exactly which. Maybe it wasn't actually needed. But anyway I'll keep it for the next version where, for a brief period of time, nocb kthread will be able to compete with callback execution in softirq.

Which nocb kthread is competing? Do you mean GP or CB?

Either way, could you clarify how does softirqs compete? Until the thread is parked, you wouldn't de-offload. And once you de-offload, only then the softirq would be executing callbacks. So at any point of time, it is either the CB kthread executing CBs or the softirq executing CBs, not both. Or did I miss something?

thanks,

- Joel

Quoted 3 lines

I'll clarify that in the changelog.

Thanks.

maintainer
#48 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

Quoted 29 lines · Joel Fernandes
Quoted 21 lines · Frederic Weisbecker
Quoted 20 lines · Joel Fernandes
Quoted 7 lines · Joel Fernandes
Quoted 6 lines

The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.
nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

Quoted 5 lines

nit: you mean cblist is set to non-offloaded mode right?

Also, could you clarify better the rcu_barrier bits in the changelog. I know there's some issue if the cblist has both offloaded and non-offloaded callbacks, but it would be good to clarify this here better IMHO.

And for archival purposes: rcu_barrier needs excluding here because it is possible that for a brief period of time, the callback kthread has been parked to do the mode-switch, and it could be executing a bunch of callbacks when it was asked to park.

Quoted 6 lines
Quoted 3 lines

Meanwhile, more interrupts happen and more callbacks are queued which are now executing in softirq. This ruins the ordering of callbacks that rcu_barrier needs.

I think in that case the callbacks would still be executed in order. We wait for the kthread to park before switching to softirq callback execution.

Quoted 8 lines

Ah ok, you are parking the CB kthread after the no-cb CB's are already invoked (that's when parkme() is called -- i.e. after rcu_do_batch() in the CB kthread runs).

Yeah, I don't see the purpose of acquiring rcu_barrier mutex either now. Once you park, all CBs should have been invoked by the nocb CB thread right? kthread_park() waits for the thread to be parked before proceeding. And you don't de-offload before it is parked.

We absolutely must execute callbacks out of order in order to avoid OOM due to RCU callback floods. This is because if we don't execute callbacks out of order, there will be a time when we are not executing callbacks at all. If execution gets preempted at this point, it is quite possibly game over due to OOM.

Thanx, Paul

Quoted 21 lines
Quoted 4 lines

Initially it was to avoid callback ordering issues but I don't recall exactly which. Maybe it wasn't actually needed. But anyway I'll keep it for the next version where, for a brief period of time, nocb kthread will be able to compete with callback execution in softirq.

Which nocb kthread is competing? Do you mean GP or CB?

Either way, could you clarify how does softirqs compete? Until the thread is parked, you wouldn't de-offload. And once you de-offload, only then the softirq would be executing callbacks. So at any point of time, it is either the CB kthread executing CBs or the softirq executing CBs, not both. Or did I miss something?

thanks,

- Joel

Quoted 3 lines

I'll clarify that in the changelog.

Thanks.

author maintainer
#49 lore raw

Re: [PATCH 08/10] rcu: Allow to deactivate nocb on a CPU

In reply to Joel Fernandes
Quoted 4 lines · Joel Fernandes
Quoted 1 line · Frederic Weisbecker

Allow a CPU's rdp to quit the callback offlined mode.

nit: s/offlined/offloaded/ ?

Oh, looks like I did that everywhere :)

Quoted 9 lines
Quoted 7 lines

The switch happens on the target with IRQs disabled and rdp->nocb_lock held to avoid races between local callbacks handling and kthread offloaded callbacks handling.
nocb_cb kthread is first parked to avoid any future race with concurrent rcu_do_batch() executions. Then the cblist is set to offloaded so that the nocb_gp kthread ignores this rdp.

nit: you mean cblist is set to non-offloaded mode right?

Ah right!

Quoted 8 lines

kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2188

Quoted 6 lines
kernel/rcu/tree_plugin.h +6 -0
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+	unsigned long flags;
+	struct rcu_node *rnp = rdp->mynode;
+
+	printk("De-offloading %d\n", rdp->cpu);

nit: s/printk/pr_debug/ ?

Ok.

Thanks!

author maintainer
#50 lore raw

PATCH 9/10 rcu: Allow to re-offload a CPU that used to be nocb

In reply to Frederic Weisbecker

This is essentially the reverse operation of de-offloading. For now it's only supported on CPUs that used to be offloaded and therefore still have the relevant nocb_cb/nocb_gp kthreads around.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcupdate.h |  2 ++
 kernel/rcu/tree_plugin.h | 44 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
include/linux/rcupdate.h +2 -0
index 1d3a4c37c3c1..ee95e49d675f 100644
@@ -96,9 +96,11 @@ static inline void rcu_user_exit(void) { }
 
 #ifdef CONFIG_RCU_NOCB_CPU
 void rcu_init_nohz(void);
+void rcu_nocb_cpu_offload(int cpu);
 void rcu_nocb_cpu_deoffload(int cpu);
 #else /* #ifdef CONFIG_RCU_NOCB_CPU */
 static inline void rcu_init_nohz(void) { }
+static inline void rcu_nocb_cpu_offload(int cpu) { }
 static inline void rcu_nocb_cpu_deoffload(int cpu) { }
 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
 
kernel/rcu/tree_plugin.h +44 -0

50 lines not on this page · raw message

Signature · 1 line
2.25.0
maintainer
#51 lore raw

Re: [PATCH 09/10] rcu: Allow to re-offload a CPU that used to be nocb

Quoted 3 lines · Frederic Weisbecker

This is essentially the reverse operation of de-offloading. For now it's only supported on CPUs that used to be offloaded and therefore still have the relevant nocb_cb/nocb_gp kthreads around.

And I still believe that this is a good restriction to put in place, especially to start with. ;-)

Quoted 41 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 include/linux/rcupdate.h |  2 ++
 kernel/rcu/tree_plugin.h | 44 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
include/linux/rcupdate.h +2 -0
index 1d3a4c37c3c1..ee95e49d675f 100644
@@ -96,9 +96,11 @@ static inline void rcu_user_exit(void) { }
 
 #ifdef CONFIG_RCU_NOCB_CPU
 void rcu_init_nohz(void);
+void rcu_nocb_cpu_offload(int cpu);
 void rcu_nocb_cpu_deoffload(int cpu);
 #else /* #ifdef CONFIG_RCU_NOCB_CPU */
 static inline void rcu_init_nohz(void) { }
+static inline void rcu_nocb_cpu_offload(int cpu) { }
 static inline void rcu_nocb_cpu_deoffload(int cpu) { }
 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
 
kernel/rcu/tree_plugin.h +4 -0
index c74a4df8d5f2..ae4b5e9f2fc5 100644
@@ -2224,6 +2224,50 @@ void rcu_nocb_cpu_deoffload(int cpu)
 	mutex_unlock(&rcu_state.barrier_mutex);
 }
 
+static void __rcu_nocb_rdp_offload(struct rcu_data *rdp)
+{
+	unsigned long flags;
+	struct rcu_node *rnp = rdp->mynode;

kernel/rcu/tree_plugin.h void rcu_nocb_cpu_deoffload(int cpu) · line 2240

Quoted 10 lines
kernel/rcu/tree_plugin.h +10 -0
+
+	printk("Offloading %d\n", rdp->cpu);
+
+	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+	raw_spin_lock_rcu_node(rnp);
+	rcu_segcblist_offload(&rdp->cblist, true);
+	raw_spin_unlock_rcu_node(rnp);
+	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
+
+	kthread_unpark(rdp->nocb_cb_kthread);

And of course I have a similar set of questions here regarding interruption of callback invocation. ;-)

Thanx, Paul

Quoted 36 lines
kernel/rcu/tree_plugin.h +30 -0
+}
+
+static long rcu_nocb_rdp_offload(void *arg)
+{
+	struct rcu_data *rdp = arg;
+
+	WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+	__rcu_nocb_rdp_offload(rdp);
+
+	return 0;
+}
+
+void rcu_nocb_cpu_offload(int cpu)
+{
+	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
+
+	mutex_lock(&rcu_state.barrier_mutex);
+	cpus_read_lock();
+	if (!rcu_segcblist_is_offloaded(&rdp->cblist) && rdp->nocb_cb_kthread) {
+		if (cpu_online(cpu)) {
+			work_on_cpu(cpu, rcu_nocb_rdp_offload, rdp);
+		} else {
+			__rcu_nocb_rdp_offload(rdp);
+		}
+		cpumask_set_cpu(cpu, rcu_nocb_mask);
+	}
+	cpus_read_unlock();
+	mutex_unlock(&rcu_state.barrier_mutex);
+}
+
 void __init rcu_init_nohz(void)
 {
 	int cpu;
Signature · 1 line
2.25.0
author maintainer
#52 lore raw

PATCH 10/10 rcu: Nocb (de)activate through sysfs

In reply to Frederic Weisbecker

Not for merge.

Make nocb toggable for a given CPU using:
/sys/devices/system/cpu/cpu*/hotplug/nocb

This is only intended for those who want to test this patchset. The real interfaces will be cpuset/isolation and rcutorture.

Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/cpu.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
kernel/cpu.c +23 -0
index 2371292f30b0..ac6283dcb897 100644
@@ -2208,10 +2208,33 @@ static ssize_t show_cpuhp_fail(struct device *dev,
 
 static DEVICE_ATTR(fail, 0644, show_cpuhp_fail, write_cpuhp_fail);
 
+static ssize_t write_nocb(struct device *dev,
+			  struct device_attribute *attr,
+			  const char *buf, size_t count)
+{
+	int val, ret;
+
+	ret = kstrtoint(buf, 10, &val);
+	if (ret)
+		return ret;
+
+	if (val == 0)
+		rcu_nocb_cpu_deoffload(dev->id);
+	else if (val == 1)
+		rcu_nocb_cpu_offload(dev->id);
+	else
+		return -EINVAL;
+
+	return count;
+}
+
+static DEVICE_ATTR(nocb, 0644, NULL, write_nocb);
+
 static struct attribute *cpuhp_cpu_attrs[] = {
 	&dev_attr_state.attr,
 	&dev_attr_target.attr,
 	&dev_attr_fail.attr,
+	&dev_attr_nocb.attr,
 	NULL
 };
 
Signature · 1 line
2.25.0
maintainer
#53 lore raw

Re: [PATCH 10/10] rcu: Nocb (de)activate through sysfs

Quoted 7 lines · Frederic Weisbecker

Not for merge.

Make nocb toggable for a given CPU using:
/sys/devices/system/cpu/cpu*/hotplug/nocb

This is only intended for those who want to test this patchset. The real interfaces will be cpuset/isolation and rcutorture.

Makes sense!

Speaking of rcutorture, what level of testing has this series undergone?

Thanx, Paul

Quoted 52 lines
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Joel Fernandes <[email protected]>
---
 kernel/cpu.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
kernel/cpu.c +23 -0
index 2371292f30b0..ac6283dcb897 100644
@@ -2208,10 +2208,33 @@ static ssize_t show_cpuhp_fail(struct device *dev,
 
 static DEVICE_ATTR(fail, 0644, show_cpuhp_fail, write_cpuhp_fail);
 
+static ssize_t write_nocb(struct device *dev,
+			  struct device_attribute *attr,
+			  const char *buf, size_t count)
+{
+	int val, ret;
+
+	ret = kstrtoint(buf, 10, &val);
+	if (ret)
+		return ret;
+
+	if (val == 0)
+		rcu_nocb_cpu_deoffload(dev->id);
+	else if (val == 1)
+		rcu_nocb_cpu_offload(dev->id);
+	else
+		return -EINVAL;
+
+	return count;
+}
+
+static DEVICE_ATTR(nocb, 0644, NULL, write_nocb);
+
 static struct attribute *cpuhp_cpu_attrs[] = {
 	&dev_attr_state.attr,
 	&dev_attr_target.attr,
 	&dev_attr_fail.attr,
+	&dev_attr_nocb.attr,
 	NULL
 };
 
Signature · 1 line
2.25.0
author maintainer
#54 lore raw

Re: [PATCH 10/10] rcu: Nocb (de)activate through sysfs

Quoted 6 lines · Paul E. McKenney
Quoted 5 lines · Frederic Weisbecker

Not for merge.

Make nocb toggable for a given CPU using:
/sys/devices/system/cpu/cpu*/hotplug/nocb

Quoted 6 lines
Quoted 2 lines

This is only intended for those who want to test this patchset. The real interfaces will be cpuset/isolation and rcutorture.

Makes sense!

Speaking of rcutorture, what level of testing has this series undergone?

None yet, that first set was to shape up the design. But next iteration should see rcutorture coverage.

maintainer
#55 lore raw

Re: [PATCH 10/10] rcu: Nocb (de)activate through sysfs

Quoted 10 lines · Frederic Weisbecker
Quoted 9 lines · Paul E. McKenney
Quoted 7 lines · Frederic Weisbecker

Not for merge.

Make nocb toggable for a given CPU using:
/sys/devices/system/cpu/cpu*/hotplug/nocb

This is only intended for those who want to test this patchset. The real interfaces will be cpuset/isolation and rcutorture.

Quoted 6 lines
Quoted 3 lines

Makes sense!

Speaking of rcutorture, what level of testing has this series undergone?

None yet, that first set was to shape up the design. But next iteration should see rcutorture coverage.

Fair enough! I am sure that rcutorture will be ready when you are. Hey, it always has been ready for me! ;-)

Thanx, Paul

maintainer
#56 lore raw
In reply to Frederic Weisbecker
Quoted 11 lines · Frederic Weisbecker

This is a necessary step toward making nohz_full controllable through cpuset. Next step should be to allow a CPU to be nocb even if it wasn't part of the nocb set on boot.

The core design of this set is mostly based on suggestions from Paul of course.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
rcu/nohz

HEAD: 31cb4ee9da4e9cc6314498ff22d83f0d872b1a88

Very cool!!! A few comments on individual commits on a quick first scan, and more later.

Thanx, Paul

Quoted 26 lines

Thanks,
Frederic
---

Frederic Weisbecker (10):
rcu: Directly lock rdp->nocb_lock on nocb code entrypoints
rcu: Use direct rdp->nocb_lock operations on local calls
rcu: Make locking explicit in do_nocb_deferred_wakeup_common()
rcu: Implement rcu_segcblist_is_offloaded() config dependent
rcu: Remove useless conditional nocb unlock
rcu: Make nocb_cb kthread parkable
rcu: Temporarily assume that nohz full CPUs might not be NOCB
rcu: Allow to deactivate nocb on a CPU
rcu: Allow to re-offload a CPU that used to be nocb
rcu: Nocb (de)activate through sysfs

 include/linux/rcu_segcblist.h |   2 +
 include/linux/rcupdate.h      |   4 ++
 kernel/cpu.c                  |  23 +++++++
 kernel/rcu/rcu_segcblist.c    |   6 +-
 kernel/rcu/rcu_segcblist.h    |   8 ++-
 kernel/rcu/tree.c             |  24 +++----
 kernel/rcu/tree.h             |   2 +-
 kernel/rcu/tree_plugin.h      | 149 ++++++++++++++++++++++++++++++++++--------
 8 files changed, 172 insertions(+), 46 deletions(-)