rcu/nocb: Fixes and simplification
Frederic Weisbecker
maintainercover rcu/nocb: Fixes and simplification
Hi,
Some updates for the nocb side:
* 1st patch is a simplification on adding/deleting an rdp to a nocb group
* 2nd and 3rd are fixes for the kthread's creation failure path
Frederic Weisbecker (1):
rcu/nocb: Add/del rdp to iterate from rcuog itself
Zqiang (2):
rcu: Invert rcu_state.barrier_mutex VS hotplug lock locking order rcu/nocb: Fix NOCB kthreads spawn failure with
rcu_nocb_rdp_deoffload() direct call
kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 226 +++++++++++++++++++++++++---------------- 2 files changed, 139 insertions(+), 88 deletions(-)
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 1/3 rcu/nocb: Add/del rdp to iterate from rcuog itself
NOCB rdp's are part of a group whose list is iterated by the corresponding rdp leader.
This list is RCU traversed because an rdp can be either added or deleted concurrently. Upon addition, a new iteration to the list after a synchronization point (a pair of LOCK/UNLOCK ->nocb_gp_lock) is forced to make sure:
1) we didn't miss a new element added in the middle of an iteration
2) we didn't ignore a whole subset of the list due to an element being quickly deleted and then re-added.
3) we prevent from probably other surprises...
Although this layout is expected to be safe, it doesn't help anybody to sleep well.
Simplify instead the nocb state toggling with moving the list modification from the nocb (de-)offloading workqueue to the rcuog kthreads instead.
Whenever the rdp leader is expected to (re-)set the SEGCBLIST_KTHREAD_GP flag of a target rdp, the latter is queued so that the leader handles the flag flip along with adding or deleting the target rdp to the list to iterate. This way the list modification and iteration happen from the same kthread and those operations can't race altogether.
As a bonus, the flags for each rdp don't need to be checked locklessly before each iteration, which is one less opportunity to produce nightmares.
--- kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 138 +++++++++++++++++++++-------------------- 2 files changed, 71 insertions(+), 68 deletions(-)
kernel/rcu/tree.h +1 -0
@@ -235,6 +235,7 @@ struct rcu_data {* if rdp_gp. */ struct list_head nocb_entry_rdp;/* rcu_data node in wakeup chain. */ +struct rcu_data *nocb_toggling_rdp;/* rdp queued for (de-)offloading */ /* The following fields are used by CB kthread, hence new cacheline. */ struct rcu_data *nocb_gp_rdp____cacheline_internodealigned_in_smp ;
kernel/rcu/tree_nocb.h +70 -68
209 lines not on this page · raw message
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 2/3 rcu/nocb: Invert rcu_state.barrier_mutex VS hotplug lock locking order
In reply to Frederic WeisbeckerFrom: Zqiang <[email protected]>
In case of failure to spawn either rcuog or rcuo[p] kthreads for a given rdp, rcu_nocb_rdp_deoffload() needs to be called with the hotplug lock and the barrier_mutex held. However cpus write lock is already held while calling rcutree_prepare_cpu(). It's not possible to call rcu_nocb_rdp_deoffload() from there with just locking the barrier_mutex or this would result in a locking inversion against rcu_nocb_cpu_deoffload() which holds both locks in the reverse order.
Simply solve this with inverting the locking order inside rcu_nocb_cpu_[de]offload(). This will be a pre-requisite to toggle NOCB states toward cpusets anyway.
--- kernel/rcu/tree_nocb.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/rcu/tree_nocb.h +4 -4
@@ -1055,8 +1055,8 @@ int rcu_nocb_cpu_deoffload(int cpu)struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu);int ret =0 ; -mutex_lock (&rcu_state.barrier_mutex);cpus_read_lock (); +mutex_lock (&rcu_state.barrier_mutex);if (rcu_rdp_is_offloaded (rdp)) {if (cpu_online (cpu)) { ret =work_on_cpu (cpu, rcu_nocb_rdp_deoffload, rdp);
@@ -1067,8 +1067,8 @@ int rcu_nocb_cpu_deoffload(int cpu) ret = -EINVAL ; } } -cpus_read_unlock ();mutex_unlock (&rcu_state.barrier_mutex); +cpus_read_unlock ();return ret; }
@@ -1134,8 +1134,8 @@ int rcu_nocb_cpu_offload(int cpu)struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu);int ret =0 ; -mutex_lock (&rcu_state.barrier_mutex);cpus_read_lock (); +mutex_lock (&rcu_state.barrier_mutex);if (!rcu_rdp_is_offloaded (rdp)) {if (cpu_online (cpu)) { ret =work_on_cpu (cpu, rcu_nocb_rdp_offload, rdp);
@@ -1146,8 +1146,8 @@ int rcu_nocb_cpu_offload(int cpu) ret = -EINVAL ; } } -cpus_read_unlock ();mutex_unlock (&rcu_state.barrier_mutex); +cpus_read_unlock ();return ret; }
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 3/3 rcu/nocb: Fix NOCB kthreads spawn failure with rcu_nocb_rdp_deoffload() direct call
In reply to Frederic WeisbeckerFrom: Zqiang <[email protected]>
If the rcuog/o[p] kthreads spawn failed, the offloaded rdp needs to be explicitly deoffloaded, otherwise the target rdp is still considered offloaded even though nothing actually handles the callbacks.
--- kernel/rcu/tree_nocb.h | 80 +++++++++++++++++++++++++++++++++--------- 1 file changed, 64 insertions(+), 16 deletions(-)
kernel/rcu/tree_nocb.h +64 -16
@@ -986,10 +986,7 @@ static int rdp_offload_toggle(struct rcu_data *rdp, }raw_spin_unlock_irqrestore (&rdp_gp->nocb_gp_lock, flags); -if (wake_gp) -wake_up_process (rdp_gp->nocb_gp_kthread); - -return 0 ; +return wake_gp; }static long rcu_nocb_rdp_deoffload (void *arg)
@@ -997,9 +994,15 @@ static long rcu_nocb_rdp_deoffload(void *arg)struct rcu_data *rdp = arg;struct rcu_segcblist *cblist = &rdp->cblist;unsigned long flags; -int ret; +int wake_gp; +struct rcu_data *rdp_gp = rdp->nocb_gp_rdp; -WARN_ON_ONCE (rdp->cpu !=raw_smp_processor_id ()); +/* +* rcu_nocb_rdp_deoffload() may be called directly if +* rcuog/o[p] spawn failed, because at this time the rdp->cpu +* is not online yet. +*/ +WARN_ON_ONCE ((rdp->cpu !=raw_smp_processor_id ()) &&cpu_online (rdp->cpu));pr_info ("De-offloading %d\n" , rdp->cpu);
@@ -1023,10 +1026,41 @@ static long rcu_nocb_rdp_deoffload(void *arg)*/ rcu_segcblist_set_flags (cblist,SEGCBLIST_RCU_CORE );invoke_rcu_core (); - ret =rdp_offload_toggle (rdp, false, flags); -swait_event_exclusive (rdp->nocb_state_wq, - !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB | -SEGCBLIST_KTHREAD_GP )); + wake_gp =rdp_offload_toggle (rdp, false, flags); + +mutex_lock (&rdp_gp->nocb_gp_kthread_mutex); +if (rdp_gp->nocb_gp_kthread) { +if (wake_gp) +wake_up_process (rdp_gp->nocb_gp_kthread); + +/* +* If rcuo[p] kthread spawn failed, directly remove SEGCBLIST_KTHREAD_CB. +* Just wait SEGCBLIST_KTHREAD_GP to be cleared by rcuog. +*/ +if (!rdp->nocb_cb_kthread) { +rcu_nocb_lock_irqsave (rdp, flags); +rcu_segcblist_clear_flags (&rdp->cblist,SEGCBLIST_KTHREAD_CB ); +rcu_nocb_unlock_irqrestore (rdp, flags); + } + +swait_event_exclusive (rdp->nocb_state_wq, + !rcu_segcblist_test_flags (cblist, +SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP )); + }else { +/* +* No kthread to clear the flags for us or remove the rdp from the nocb list +* to iterate. Do it here instead. Locking doesn't look stricly necessary +* but we stick to paranoia in this rare path. +*/ +rcu_nocb_lock_irqsave (rdp, flags); +rcu_segcblist_clear_flags (&rdp->cblist, +SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP ); +rcu_nocb_unlock_irqrestore (rdp, flags); + +list_del (&rdp->nocb_entry_rdp); + } +mutex_unlock (&rdp_gp->nocb_gp_kthread_mutex); +/* * Lock one last time to acquire latest callback updates from kthreads * so we can later handle callbacks locally without locking.
@@ -1047,7 +1081,7 @@ static long rcu_nocb_rdp_deoffload(void *arg)WARN_ON_ONCE (rcu_cblist_n_cbs (&rdp->nocb_bypass)); -return ret; +return 0 ; }int rcu_nocb_cpu_deoffload (int cpu)
@@ -1079,7 +1113,8 @@ static long rcu_nocb_rdp_offload(void *arg)struct rcu_data *rdp = arg;struct rcu_segcblist *cblist = &rdp->cblist;unsigned long flags; -int ret; +int wake_gp; +struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;WARN_ON_ONCE (rdp->cpu !=raw_smp_processor_id ());/*
@@ -1089,6 +1124,9 @@ static long rcu_nocb_rdp_offload(void *arg)if (!rdp->nocb_gp_rdp)return -EINVAL ; +if (WARN_ON_ONCE (!rdp_gp->nocb_gp_kthread)) +return -EINVAL ; +pr_info ("Offloading %d\n" , rdp->cpu);/*
@@ -1113,7 +1151,9 @@ static long rcu_nocb_rdp_offload(void *arg)* WRITE flags READ callbacks * rcu_nocb_unlock() rcu_nocb_unlock() */ - ret =rdp_offload_toggle (rdp, true, flags); + wake_gp =rdp_offload_toggle (rdp, true, flags); +if (wake_gp) +wake_up_process (rdp_gp->nocb_gp_kthread);swait_event_exclusive (rdp->nocb_state_wq,rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB ) &&rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_GP ));
@@ -1126,7 +1166,7 @@ static long rcu_nocb_rdp_offload(void *arg)rcu_segcblist_clear_flags (cblist,SEGCBLIST_RCU_CORE );rcu_nocb_unlock_irqrestore (rdp, flags); -return ret; +return 0 ; }int rcu_nocb_cpu_offload (int cpu)
@@ -1248,7 +1288,7 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu) "rcuog/%d", rdp_gp->cpu);if (WARN_ONCE (IS_ERR (t),"%s: Could not start rcuo GP kthread, OOM is now expected behavior\n" , __func__)) {mutex_unlock (&rdp_gp->nocb_gp_kthread_mutex); -return ; + goto end; }WRITE_ONCE (rdp_gp->nocb_gp_kthread, t);if (kthread_prio)
@@ -1260,12 +1300,20 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu) t =kthread_run (rcu_nocb_cb_kthread, rdp,"rcuo%c/%d" , rcu_state.abbr, cpu);if (WARN_ONCE (IS_ERR (t),"%s: Could not start rcuo CB kthread, OOM is now expected behavior\n" , __func__)) -return ; + goto end;if (kthread_prio)sched_setscheduler_nocheck (t,SCHED_FIFO , &sp);WRITE_ONCE (rdp->nocb_cb_kthread, t);WRITE_ONCE (rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread); +return ; +end: +mutex_lock (&rcu_state.barrier_mutex); +if (rcu_rdp_is_offloaded (rdp)) { +rcu_nocb_rdp_deoffload (rdp); +cpumask_clear_cpu (cpu, rcu_nocb_mask); + } +mutex_unlock (&rcu_state.barrier_mutex); }/* How many CB CPU IDs per GP kthread? Default of -1 for sqrt(nr_cpu_ids). */
Signature · 1 line
2.25.1
Paul E. McKenney
maintainerQuoted 10 lines · Frederic Weisbecker
Hi,
Some updates for the nocb side:
* 1st patch is a simplification on adding/deleting an rdp to a nocb group
* 2nd and 3rd are fixes for the kthread's creation failure path
Frederic Weisbecker (1):
rcu/nocb: Add/del rdp to iterate from rcuog itself
A very welcome simplification!
Quoted 4 lines
Zqiang (2):
rcu: Invert rcu_state.barrier_mutex VS hotplug lock locking order rcu/nocb: Fix NOCB kthreads spawn failure with
rcu_nocb_rdp_deoffload()direct call
And here is to added robustness!
I queued these for review and testing, thank you! By default, these would go into the v5.20 (or will it be v6.0?) pile. Please let me know if they are more urgent and need to go into v5.19 instead.
Thanx, Paul
Quoted 7 lines
kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 226 +++++++++++++++++++++++++---------------- 2 files changed, 139 insertions(+), 88 deletions(-)Signature · 1 line
2.25.1