CPU-hotplug updates
Subsystems (3)
cover CPU-hotplug updates
Hello!
This series updates RCU's interaction with CPU hotplug, perhaps most notably making callback migration happen very shortly after the outgoing CPU is gone.
1. Make expedited GPs correctly handle hardware CPU insertion.
2. Migrate callbacks earlier in the CPU-offline timeline.
3. Fix typo suppressing CPU-hotplug statistics.
4. Remove orphan/adopt event-tracing fields.
5. Check for NOCB CPUs and empty lists earlier in CB migration.
6. Make NOCB CPUs migrate CBs directly from outgoing CPU.
7. Advance outgoing CPU's callbacks before migrating them.
8. Eliminate rcu_state ->orphan_lock.
9. Advance callbacks after migration.
10. Localize rcu_state ->orphan_pend and ->orphan_done.
11. Remove unused RCU list functions.
12. Move callback-list warning to irq-disable region, thus suppressing
rare but real false-positive dmesg splats.
Thanx, Paul
------------------------------------------------------------------------
include/linux/rcupdate.h | 1 kernel/cpu.c | 1 kernel/rcu/rcu_segcblist.c | 108 +++----------- kernel/rcu/rcu_segcblist.h | 28 --- kernel/rcu/tree.c | 342 +++++++++++++++++++-------------------------- kernel/rcu/tree.h | 13 - kernel/rcu/tree_exp.h | 2 kernel/rcu/tree_plugin.h | 31 +--- kernel/torture.c | 2 9 files changed, 189 insertions(+), 339 deletions(-)
PATCH 1/12 rcu: Make expedited GPs correctly handle hardware CPU insertion
The update of the ->expmaskinitnext and of ->ncpus are unsynchronized, with the value of ->ncpus being incremented long before the corresponding ->expmaskinitnext mask is updated. If an RCU expedited grace period sees ->ncpus change, it will update the ->expmaskinit masks from the new ->expmaskinitnext masks. But it is possible that ->ncpus has already been updated, but the ->expmaskinitnext masks still have their old values. For the current expedited grace period, no harm done. The CPU could not have been online before the grace period started, so there is no need to wait for its non-existent pre-existing readers.
But the next RCU expedited grace period is in a world of hurt. The value of ->ncpus has already been updated, so this grace period will assume that the ->expmaskinitnext masks have not changed. But they have, and they won't be taken into account until the next never-been-online CPU comes online. This means that RCU will be ignoring some CPUs that it should be paying attention to.
The solution is to update ->ncpus and ->expmaskinitnext while holding the ->lock for the rcu_node structure containing the ->expmaskinitnext mask. Because smp_store_release() is now used to update ->ncpus and smp_load_acquire() is now used to locklessly read it, if the expedited grace period sees ->ncpus change, then the updating CPU has to already be holding the corresponding ->lock. Therefore, when the expedited grace period later acquires that ->lock, it is guaranteed to see the new value of ->expmaskinitnext.
On the other hand, if the expedited grace period loads ->ncpus just before an update, earlier full memory barriers guarantee that the incoming CPU isn't far enough along to be running any RCU readers.
This commit therefore makes the required change.
--- kernel/rcu/tree.c | 10 ++++++++-- kernel/rcu/tree_exp.h | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-)
kernel/rcu/tree.c +8 -2
@@ -3777,8 +3777,6 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp)*/ rnp = rdp->mynode;raw_spin_lock_rcu_node (rnp);/* irqs already disabled. */ -if (!rdp->beenonline) -WRITE_ONCE (rsp->ncpus,READ_ONCE (rsp->ncpus) +1 ); rdp->beenonline = true;/* We have now been online. */ rdp->gpnum = rnp->completed;/* Make CPU later note any new GP. */ rdp->completed = rnp->completed;
@@ -3882,6 +3880,8 @@ void rcu_cpu_starting(unsigned int cpu) {unsigned long flags;unsigned long mask; +int nbits; +unsigned long oldmask;struct rcu_data *rdp;struct rcu_node *rnp;struct rcu_state *rsp;
@@ -3892,9 +3892,15 @@ void rcu_cpu_starting(unsigned int cpu) mask = rdp->grpmask;raw_spin_lock_irqsave_rcu_node (rnp, flags); rnp->qsmaskinitnext |= mask; + oldmask = rnp->expmaskinitnext; rnp->expmaskinitnext |= mask; + oldmask ^= rnp->expmaskinitnext; + nbits =bitmap_weight (&oldmask,BITS_PER_LONG ); +/* Allow lockless access for expedited grace periods. */ +smp_store_release (&rsp->ncpus, rsp->ncpus + nbits);/* ^^^ */ raw_spin_unlock_irqrestore_rcu_node (rnp, flags); } +smp_mb ();/* Ensure RCU read-side usage follows above initialization. */ }#ifdef CONFIG_HOTPLUG_CPU
kernel/rcu/tree_exp.h +1 -1
@@ -73,7 +73,7 @@ static void sync_exp_reset_tree_hotplug(struct rcu_state *rsp)unsigned long flags;unsigned long mask;unsigned long oldmask; -int ncpus =READ_ONCE (rsp->ncpus); +int ncpus =smp_load_acquire (&rsp->ncpus);/* Order against locking. */ struct rcu_node *rnp;struct rcu_node *rnp_up;
Signature · 1 line
2.5.2
PATCH 2/12 rcu: Migrate callbacks earlier in the CPU-offline timeline
In reply to Paul E. McKenneyRCU callbacks must be migrated away from an outgoing CPU, and this is done near the end of the CPU-hotplug operation, after the outgoing CPU is long gone. Unfortunately, this means that other CPU-hotplug callbacks can execute while the outgoing CPU's callbacks are still immobilized on the long-gone CPU's callback lists. If any of these CPU-hotplug callbacks must wait, either directly or indirectly, for the invocation of any of the immobilized RCU callbacks, the system will hang.
This commit avoids such hangs by migrating the callbacks away from the outgoing CPU immediately upon its departure, shortly after the return from __cpu_die() in takedown_cpu(). Thus, RCU is able to advance these callbacks and invoke them, which allows all the after-the-fact CPU-hotplug callbacks to wait on these RCU callbacks without risk of a hang.
While in the neighborhood, this commit also moves rcu_send_cbs_to_orphanage() and rcu_adopt_orphan_cbs() under a pre-existing #ifdef to avoid including dead code on the one hand and to avoid define-without-use warnings on the other hand.
--- include/linux/rcupdate.h | 1 + kernel/cpu.c | 1 + kernel/rcu/tree.c | 209 +++++++++++++++++++++++++---------------------- 3 files changed, 115 insertions(+), 96 deletions(-)
include/linux/rcupdate.h +1 -0
@@ -110,6 +110,7 @@ void rcu_bh_qs(void);void rcu_check_callbacks (int user);void rcu_report_dead (unsigned int cpu);void rcu_cpu_starting (unsigned int cpu); +void rcutree_migrate_callbacks (int cpu);#ifdef CONFIG_RCU_STALL_COMMON void rcu_sysrq_start (void );
kernel/cpu.c +1 -0
@@ -649,6 +649,7 @@ static int takedown_cpu(unsigned int cpu)__cpu_die (cpu);tick_cleanup_dead_cpu (cpu); +rcutree_migrate_callbacks (cpu);return 0 ; }
kernel/rcu/tree.c +113 -96
236 lines not on this page · raw message
Signature · 1 line
2.5.2
PATCH 3/12 torture: Fix typo suppressing CPU-hotplug statistics
In reply to Paul E. McKenneyThe torture status line contains a series of values preceded by "onoff:". The last value in that line, the one preceding the "HZ=" string, is always zero. The reason that it is always zero is that torture_offline() was incrementing the sum_offl pointer instead of the value that this pointer referenced. This commit therefore makes this increment operate on the statistic rather than the pointer to the statistic.
--- kernel/torture.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/torture.c +1 -1
@@ -117,7 +117,7 @@ bool torture_offline(int cpu, long *n_offl_attempts, long *n_offl_successes, torture_type, cpu); (*n_offl_successes)++; delta = jiffies - starttime; - sum_offl += delta; + *sum_offl += delta;if (*min_offl <0 ) { *min_offl = delta; *max_offl = delta;
Signature · 1 line
2.5.2
PATCH 4/12 rcu: Remove orphan/adopt event-tracing fields
In reply to Paul E. McKenneyThe rcu_node structure's ->n_cbs_orphaned and ->n_cbs_adopted fields are updated, but never read. This commit therefore removes them.
--- kernel/rcu/tree.c | 2 -- kernel/rcu/tree.h | 2 -- 2 files changed, 4 deletions(-)
kernel/rcu/tree.c +0 -2
@@ -3871,7 +3871,6 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,* because _rcu_barrier() excludes CPU-hotplug operations, so it * cannot be running now. Thus no memory barrier is required. */ - rdp->n_cbs_orphaned +=rcu_segcblist_n_cbs (&rdp->cblist);rcu_segcblist_extract_count (&rdp->cblist, &rsp->orphan_done);/*
@@ -3910,7 +3909,6 @@ static void rcu_adopt_orphan_cbs(struct rcu_state *rsp, unsigned long flags)return ;/* Do the accounting first. */ - rdp->n_cbs_adopted += rsp->orphan_done.len;if (rsp->orphan_done.len_lazy != rsp->orphan_done.len)rcu_idle_count_callbacks_posted ();rcu_segcblist_insert_count (&rdp->cblist, &rsp->orphan_done);
kernel/rcu/tree.h +0 -2
@@ -219,8 +219,6 @@ struct rcu_data {/* qlen at last check for QS forcing */ unsigned long n_cbs_invoked;/* count of RCU cbs invoked. */ unsigned long n_nocbs_invoked;/* count of no-CBs RCU cbs invoked. */ -unsigned long n_cbs_orphaned;/* RCU cbs orphaned by dying CPU */ -unsigned long n_cbs_adopted;/* RCU cbs adopted from dying CPU */ unsigned long n_force_qs_snap;/* did other CPU force QS recently? */ long blimit;/* Upper limit on a processed batch */
Signature · 1 line
2.5.2
PATCH 5/12 rcu: Check for NOCB CPUs and empty lists earlier in CB migration
In reply to Paul E. McKenneyThe current CPU-hotplug RCU-callback-migration code checks for the source (newly offlined) CPU being a NOCBs CPU down in rcu_send_cbs_to_orphanage(). This commit simplifies callback migration a bit by moving this check up to rcu_migrate_callbacks(). This commit also adds a check for the source CPU having no callbacks, which eases analysis of the rcu_send_cbs_to_orphanage() and rcu_adopt_orphan_cbs() functions.
--- kernel/rcu/tree.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +3 -4
@@ -3862,10 +3862,6 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp, {lockdep_assert_held (&rsp->orphan_lock); -/* No-CBs CPUs do not have orphanable callbacks. */ -if (!IS_ENABLED (CONFIG_HOTPLUG_CPU ) ||rcu_is_nocb_cpu (rdp->cpu)) -return ; -/* * Orphan the callbacks. First adjust the counts. This is safe * because _rcu_barrier() excludes CPU-hotplug operations, so it
@@ -3935,6 +3931,9 @@ static void rcu_migrate_callbacks(int cpu, struct rcu_state *rsp)struct rcu_data *rdp =per_cpu_ptr (rsp->rda, cpu);struct rcu_node *rnp = rdp->mynode;/* Outgoing CPU's rdp & rnp. */ +if (rcu_is_nocb_cpu (cpu) ||rcu_segcblist_empty (&rdp->cblist)) +return ;/* No callbacks to migrate. */ +raw_spin_lock_irqsave (&rsp->orphan_lock, flags);rcu_send_cbs_to_orphanage (cpu, rsp, rnp, rdp);rcu_adopt_orphan_cbs (rsp, flags);
Signature · 1 line
2.5.2
PATCH 6/12 rcu: Make NOCB CPUs migrate CBs directly from outgoing CPU
In reply to Paul E. McKenneyRCU's CPU-hotplug callback-migration code first moves the outgoing CPU's callbacks to ->orphan_done and ->orphan_pend, and only then moves them to the NOCB callback list. This commit avoids the extra step (and simplifies the code) by moving the callbacks directly from the outgoing CPU's callback list to the NOCB callback list.
--- kernel/rcu/tree.c | 14 ++++++++------ kernel/rcu/tree.h | 2 +- kernel/rcu/tree_plugin.h | 31 ++++++++++--------------------- 3 files changed, 19 insertions(+), 28 deletions(-)
kernel/rcu/tree.c +8 -6
@@ -3899,11 +3899,6 @@ static void rcu_adopt_orphan_cbs(struct rcu_state *rsp, unsigned long flags)lockdep_assert_held (&rsp->orphan_lock); -/* No-CBs CPUs are handled specially. */ -if (!IS_ENABLED (CONFIG_HOTPLUG_CPU ) || -rcu_nocb_adopt_orphan_cbs (rsp, rdp, flags)) -return ; -/* Do the accounting first. */ if (rsp->orphan_done.len_lazy != rsp->orphan_done.len)rcu_idle_count_callbacks_posted ();
@@ -3928,13 +3923,20 @@ static void rcu_adopt_orphan_cbs(struct rcu_state *rsp, unsigned long flags)static void rcu_migrate_callbacks (int cpu,struct rcu_state *rsp) {unsigned long flags; +struct rcu_data *my_rdp;struct rcu_data *rdp =per_cpu_ptr (rsp->rda, cpu);struct rcu_node *rnp = rdp->mynode;/* Outgoing CPU's rdp & rnp. */ if (rcu_is_nocb_cpu (cpu) ||rcu_segcblist_empty (&rdp->cblist))return ;/* No callbacks to migrate. */ -raw_spin_lock_irqsave (&rsp->orphan_lock, flags); +local_irq_save (flags); + my_rdp =this_cpu_ptr (rsp->rda); +if (rcu_nocb_adopt_orphan_cbs (my_rdp, rdp, flags)) { +local_irq_restore (flags); +return ; + } +raw_spin_lock (&rsp->orphan_lock);/* irqs already disabled. */ rcu_send_cbs_to_orphanage (cpu, rsp, rnp, rdp);rcu_adopt_orphan_cbs (rsp, flags);raw_spin_unlock_irqrestore (&rsp->orphan_lock, flags);
kernel/rcu/tree.h +1 -1
@@ -493,7 +493,7 @@ static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq);static void rcu_init_one_nocb (struct rcu_node *rnp);static bool __call_rcu_nocb (struct rcu_data *rdp,struct rcu_head *rhp,bool lazy,unsigned long flags); -static bool rcu_nocb_adopt_orphan_cbs (struct rcu_state *rsp, +static bool rcu_nocb_adopt_orphan_cbs (struct rcu_data *my_rdp,struct rcu_data *rdp,unsigned long flags);static int rcu_nocb_need_deferred_wakeup (struct rcu_data *rdp);
kernel/rcu/tree_plugin.h +10 -21
@@ -1961,30 +1961,19 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,* Adopt orphaned callbacks on a no-CBs CPU, or return 0 if this is * not a no-CBs CPU. */ -static bool __maybe_unused rcu_nocb_adopt_orphan_cbs (struct rcu_state *rsp, +static bool __maybe_unused rcu_nocb_adopt_orphan_cbs (struct rcu_data *my_rdp,struct rcu_data *rdp,unsigned long flags) { -long ql = rsp->orphan_done.len; -long qll = rsp->orphan_done.len_lazy; - -/* If this is not a no-CBs CPU, tell the caller to do it the old way. */ +RCU_LOCKDEP_WARN (!irqs_disabled (),"rcu_nocb_adopt_orphan_cbs() invoked with irqs enabled!!!" );if (!rcu_is_nocb_cpu (smp_processor_id ())) -return false; - -/* First, enqueue the donelist, if any. This preserves CB ordering. */ -if (rsp->orphan_done.head) { -__call_rcu_nocb_enqueue (rdp,rcu_cblist_head (&rsp->orphan_done), -rcu_cblist_tail (&rsp->orphan_done), - ql, qll, flags); - } -if (rsp->orphan_pend.head) { -__call_rcu_nocb_enqueue (rdp,rcu_cblist_head (&rsp->orphan_pend), -rcu_cblist_tail (&rsp->orphan_pend), - ql, qll, flags); - } -rcu_cblist_init (&rsp->orphan_done); -rcu_cblist_init (&rsp->orphan_pend); +return false;/* Not NOCBs CPU, caller must migrate CBs. */ +__call_rcu_nocb_enqueue (my_rdp,rcu_segcblist_head (&rdp->cblist), +rcu_segcblist_tail (&rdp->cblist), +rcu_segcblist_n_cbs (&rdp->cblist), +rcu_segcblist_n_lazy_cbs (&rdp->cblist), flags); +rcu_segcblist_init (&rdp->cblist); +rcu_segcblist_disable (&rdp->cblist);return true; }
@@ -2459,7 +2448,7 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,return false; } -static bool __maybe_unused rcu_nocb_adopt_orphan_cbs (struct rcu_state *rsp, +static bool __maybe_unused rcu_nocb_adopt_orphan_cbs (struct rcu_data *my_rdp,struct rcu_data *rdp,unsigned long flags) {
Signature · 1 line
2.5.2
PATCH 7/12 rcu: Advance outgoing CPU's callbacks before migrating them
In reply to Paul E. McKenneyIt is possible that the outgoing CPU is unaware of recent grace periods, and so it is also possible that some of its pending callbacks are actually ready to be invoked. The current callback-migration code would needlessly force these callbacks to pass through another grace period. This commit therefore invokes rcu_advance_cbs() on the outgoing CPU's callbacks in order to give them full credit for having passed through any recent grace periods.
This also fixes an odd theoretical bug where there are no callbacks in the system except for those on the outgoing CPU, none of those callbacks have yet been associated with a grace-period number, there is never again another callback registered, and the surviving CPU never again takes a scheduling-clock interrupt, never goes idle, and never enters nohz_full userspace execution. Yes, this is (just barely) possible. It requires that the surviving CPU be a nohz_full CPU, that its scheduler-clock interrupt be shut off, and that it loop forever in the kernel. You get bonus points if you can make this one happen! ;-)
--- kernel/rcu/tree.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +6 -1
@@ -3926,6 +3926,7 @@ static void rcu_migrate_callbacks(int cpu, struct rcu_state *rsp)struct rcu_data *my_rdp;struct rcu_data *rdp =per_cpu_ptr (rsp->rda, cpu);struct rcu_node *rnp = rdp->mynode;/* Outgoing CPU's rdp & rnp. */ +struct rcu_node *rnp_root =rcu_get_root (rdp->rsp);if (rcu_is_nocb_cpu (cpu) ||rcu_segcblist_empty (&rdp->cblist))return ;/* No callbacks to migrate. */
@@ -3936,7 +3937,11 @@ static void rcu_migrate_callbacks(int cpu, struct rcu_state *rsp)local_irq_restore (flags);return ; } -raw_spin_lock (&rsp->orphan_lock);/* irqs already disabled. */ +raw_spin_lock_rcu_node (rnp_root);/* irqs already disabled. */ +rcu_advance_cbs (rsp, rnp_root, rdp);/* Leverage recent GPs. */ +raw_spin_unlock_rcu_node (rnp_root); + +raw_spin_lock (&rsp->orphan_lock);rcu_send_cbs_to_orphanage (cpu, rsp, rnp, rdp);rcu_adopt_orphan_cbs (rsp, flags);raw_spin_unlock_irqrestore (&rsp->orphan_lock, flags);
Signature · 1 line
2.5.2
PATCH 8/12 rcu: Eliminate rcu_state ->orphan_lock
In reply to Paul E. McKenneyThe ->orphan_lock is acquired and released only within the rcu_migrate_callbacks() function, which now acquires the root rcu_node structure's ->lock. This commit therefore eliminates the ->orphan_lock in favor of the root rcu_node structure's ->lock.
--- kernel/rcu/tree.c | 15 +++------------ kernel/rcu/tree.h | 3 --- 2 files changed, 3 insertions(+), 15 deletions(-)
kernel/rcu/tree.c +3 -12
@@ -97,7 +97,6 @@ struct rcu_state sname##_state = { \ .gp_state = RCU_GP_IDLE, \ .gpnum = 0UL - 300UL, \ .completed = 0UL - 300UL, \ - .orphan_lock = __RAW_SPIN_LOCK_UNLOCKED(&sname##_state.orphan_lock), \ .orphan_pend = RCU_CBLIST_INITIALIZER(sname##_state.orphan_pend), \ .orphan_done = RCU_CBLIST_INITIALIZER(sname##_state.orphan_done), \ .barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \
@@ -3853,15 +3852,12 @@ void rcu_report_dead(unsigned int cpu)/* * Send the specified CPU's RCU callbacks to the orphanage. The -* specified CPU must be offline, and the caller must hold the -* ->orphan_lock. +* specified CPU must be offline. */ static void rcu_send_cbs_to_orphanage (int cpu,struct rcu_state *rsp,struct rcu_node *rnp,struct rcu_data *rdp) { -lockdep_assert_held (&rsp->orphan_lock); -/* * Orphan the callbacks. First adjust the counts. This is safe * because _rcu_barrier() excludes CPU-hotplug operations, so it
@@ -3891,14 +3887,12 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,/* * Adopt the RCU callbacks from the specified rcu_state structure's -* orphanage. The caller must hold the ->orphan_lock. +* orphanage. */ static void rcu_adopt_orphan_cbs (struct rcu_state *rsp,unsigned long flags) {struct rcu_data *rdp =raw_cpu_ptr (rsp->rda); -lockdep_assert_held (&rsp->orphan_lock); -/* Do the accounting first. */ if (rsp->orphan_done.len_lazy != rsp->orphan_done.len)rcu_idle_count_callbacks_posted ();
@@ -3939,12 +3933,9 @@ static void rcu_migrate_callbacks(int cpu, struct rcu_state *rsp) }raw_spin_lock_rcu_node (rnp_root);/* irqs already disabled. */ rcu_advance_cbs (rsp, rnp_root, rdp);/* Leverage recent GPs. */ -raw_spin_unlock_rcu_node (rnp_root); - -raw_spin_lock (&rsp->orphan_lock);rcu_send_cbs_to_orphanage (cpu, rsp, rnp, rdp);rcu_adopt_orphan_cbs (rsp, flags); -raw_spin_unlock_irqrestore (&rsp->orphan_lock, flags); +raw_spin_unlock_irqrestore_rcu_node (rnp_root, flags); WARN_ONCE(rcu_segcblist_n_cbs(&rdp->cblist) != 0 || !rcu_segcblist_empty(&rdp->cblist), "rcu_cleanup_dead_cpu: Callbacks on offline CPU %d: qlen=%lu, 1stCB=%p\n",
kernel/rcu/tree.h +0 -3
@@ -348,14 +348,11 @@ struct rcu_state {/* End of fields guarded by root rcu_node's lock. */ -raw_spinlock_t orphan_lock____cacheline_internodealigned_in_smp ; -/* Protect following fields. */ struct rcu_cblist orphan_pend;/* Orphaned callbacks that */ /* need a grace period. */ struct rcu_cblist orphan_done;/* Orphaned callbacks that */ /* are ready to invoke. */ /* (Contains counts.) */ -/* End of fields guarded by orphan_lock. */ struct mutex barrier_mutex;/* Guards barrier fields. */ atomic_t barrier_cpu_count;/* # CPUs waiting on. */
Signature · 1 line
2.5.2
PATCH 9/12 rcu: Advance callbacks after migration
In reply to Paul E. McKenneyWhen migrating callbacks from a newly offlined CPU, we are already holding the root rcu_node structure's lock, so it costs almost nothing to advance and accelerate the newly migrated callbacks. This patch therefore makes this advancing and acceleration happen.
--- kernel/rcu/tree.c | 1 + 1 file changed, 1 insertion(+)
kernel/rcu/tree.c +1 -0
@@ -3935,6 +3935,7 @@ static void rcu_migrate_callbacks(int cpu, struct rcu_state *rsp)rcu_advance_cbs (rsp, rnp_root, rdp);/* Leverage recent GPs. */ rcu_send_cbs_to_orphanage (cpu, rsp, rnp, rdp);rcu_adopt_orphan_cbs (rsp, flags); +rcu_advance_cbs (rsp, rnp_root, my_rdp);/* Assign GP to pending CBs. */ raw_spin_unlock_irqrestore_rcu_node (rnp_root, flags); WARN_ONCE(rcu_segcblist_n_cbs(&rdp->cblist) != 0 || !rcu_segcblist_empty(&rdp->cblist),
Signature · 1 line
2.5.2
PATCH 10/12 rcu: Localize rcu_state ->orphan_pend and ->orphan_done
In reply to Paul E. McKenneyGiven that the rcu_state structure's >orphan_pend and ->orphan_done fields are used only during migration of callbacks from the recently offlined CPU to a surviving CPU, if rcu_send_cbs_to_orphanage() and rcu_adopt_orphan_cbs() are combined, these fields can become local variables in the combined function. This commit therefore combines rcu_send_cbs_to_orphanage() and rcu_adopt_orphan_cbs() into a new rcu_segcblist_merge() function and removes the ->orphan_pend and ->orphan_done fields.
--- kernel/rcu/rcu_segcblist.c | 24 +++++++++++++++ kernel/rcu/rcu_segcblist.h | 2 ++ kernel/rcu/tree.c | 73 +++------------------------------------------- kernel/rcu/tree.h | 6 ---- 4 files changed, 30 insertions(+), 75 deletions(-)
kernel/rcu/rcu_segcblist.c +24 -0
@@ -503,3 +503,27 @@ bool rcu_segcblist_future_gp_needed(struct rcu_segcblist *rsclp,return true;return false; } + +/* +* Merge the source rcu_segcblist structure into the destination +* rcu_segcblist structure, then initialize the source. Any pending +* callbacks from the source get to start over. It is best to +* advance and accelerate both the destination and the source +* before merging. +*/ +void rcu_segcblist_merge (struct rcu_segcblist *dst_rsclp, +struct rcu_segcblist *src_rsclp) +{ +struct rcu_cblist donecbs; +struct rcu_cblist pendcbs; + +rcu_cblist_init (&donecbs); +rcu_cblist_init (&pendcbs); +rcu_segcblist_extract_count (src_rsclp, &donecbs); +rcu_segcblist_extract_done_cbs (src_rsclp, &donecbs); +rcu_segcblist_extract_pend_cbs (src_rsclp, &pendcbs); +rcu_segcblist_insert_count (dst_rsclp, &donecbs); +rcu_segcblist_insert_done_cbs (dst_rsclp, &donecbs); +rcu_segcblist_insert_pend_cbs (dst_rsclp, &pendcbs); +rcu_segcblist_init (src_rsclp); +}
kernel/rcu/rcu_segcblist.h +2 -0
@@ -162,3 +162,5 @@ void rcu_segcblist_advance(struct rcu_segcblist *rsclp, unsigned long seq);bool rcu_segcblist_accelerate (struct rcu_segcblist *rsclp,unsigned long seq);bool rcu_segcblist_future_gp_needed (struct rcu_segcblist *rsclp,unsigned long seq); +void rcu_segcblist_merge (struct rcu_segcblist *dst_rsclp, +struct rcu_segcblist *src_rsclp);
kernel/rcu/tree.c +4 -69
103 lines not on this page · raw message
kernel/rcu/tree.h +0 -6
@@ -348,12 +348,6 @@ struct rcu_state {/* End of fields guarded by root rcu_node's lock. */ -struct rcu_cblist orphan_pend;/* Orphaned callbacks that */ -/* need a grace period. */ -struct rcu_cblist orphan_done;/* Orphaned callbacks that */ -/* are ready to invoke. */ -/* (Contains counts.) */ -struct mutex barrier_mutex;/* Guards barrier fields. */ atomic_t barrier_cpu_count;/* # CPUs waiting on. */ struct completion barrier_completion;/* Wake at barrier end. */
Signature · 1 line
2.5.2
PATCH 11/12 rcu: Remove unused RCU list functions
In reply to Paul E. McKenneyGiven changes to callback migration, rcu_cblist_head(), rcu_cblist_tail(), rcu_cblist_count_cbs(), rcu_segcblist_segempty(), rcu_segcblist_dequeued_lazy(), and rcu_segcblist_new_cbs() are no longer used. This commit therefore removes them.
--- kernel/rcu/rcu_segcblist.c | 84 ---------------------------------------------- kernel/rcu/rcu_segcblist.h | 26 -------------- 2 files changed, 110 deletions(-)
kernel/rcu/rcu_segcblist.c +0 -84
@@ -36,24 +36,6 @@ void rcu_cblist_init(struct rcu_cblist *rclp) }/* -* Debug function to actually count the number of callbacks. -* If the number exceeds the limit specified, return -1. -*/ -long rcu_cblist_count_cbs (struct rcu_cblist *rclp,long lim) -{ -int cnt =0 ; -struct rcu_head **rhpp = &rclp->head; - -for (;;) { -if (!*rhpp) -return cnt; -if (++cnt > lim) -return -1 ; - rhpp = &(*rhpp)->next; - } -} - -/* * Dequeue the oldest rcu_head structure from the specified callback * list. This function assumes that the callback is non-lazy, but * the caller can later invoke rcu_cblist_dequeued_lazy() if it
@@ -103,17 +85,6 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp) }/* -* Is the specified segment of the specified rcu_segcblist structure -* empty of callbacks? -*/ -bool rcu_segcblist_segempty (struct rcu_segcblist *rsclp,int seg) -{ -if (seg ==RCU_DONE_TAIL ) -return &rsclp->head == rsclp->tails[RCU_DONE_TAIL ]; -return rsclp->tails[seg -1 ] == rsclp->tails[seg]; -} - -/* * Does the specified rcu_segcblist structure contain callbacks that * are ready to be invoked? */
@@ -134,50 +105,6 @@ bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp) }/* -* Dequeue and return the first ready-to-invoke callback. If there -* are no ready-to-invoke callbacks, return NULL. Disables interrupts -* to avoid interference. Does not protect from interference from other -* CPUs or tasks. -*/ -struct rcu_head *rcu_segcblist_dequeue (struct rcu_segcblist *rsclp) -{ -unsigned long flags; -int i; -struct rcu_head *rhp; - -local_irq_save (flags); -if (!rcu_segcblist_ready_cbs (rsclp)) { -local_irq_restore (flags); -return NULL ; - } - rhp = rsclp->head; -BUG_ON (!rhp); - rsclp->head = rhp->next; -for (i =RCU_DONE_TAIL ; i <RCU_CBLIST_NSEGS ; i++) { -if (rsclp->tails[i] != &rhp->next) -break ; - rsclp->tails[i] = &rsclp->head; - } -smp_mb ();/* Dequeue before decrement for rcu_barrier(). */ -WRITE_ONCE (rsclp->len, rsclp->len -1 ); -local_irq_restore (flags); -return rhp; -} - -/* -* Account for the fact that a previously dequeued callback turned out -* to be marked as lazy. -*/ -void rcu_segcblist_dequeued_lazy (struct rcu_segcblist *rsclp) -{ -unsigned long flags; - -local_irq_save (flags); - rsclp->len_lazy--; -local_irq_restore (flags); -} - -/* * Return a pointer to the first callback in the specified rcu_segcblist * structure. This is useful for diagnostics. */
@@ -203,17 +130,6 @@ struct rcu_head *rcu_segcblist_first_pend_cb(struct rcu_segcblist *rsclp) }/* -* Does the specified rcu_segcblist structure contain callbacks that -* have not yet been processed beyond having been posted, that is, -* does it contain callbacks in its last segment? -*/ -bool rcu_segcblist_new_cbs (struct rcu_segcblist *rsclp) -{ -return rcu_segcblist_is_enabled (rsclp) && - !rcu_segcblist_restempty (rsclp,RCU_NEXT_READY_TAIL ); -} - -/* * Enqueue the specified callback onto the specified rcu_segcblist * structure, updating accounting as needed. Note that the ->len * field may be accessed locklessly, hence the WRITE_ONCE().
kernel/rcu/rcu_segcblist.h +0 -26
@@ -31,29 +31,7 @@ static inline void rcu_cblist_dequeued_lazy(struct rcu_cblist *rclp) rclp->len_lazy--; } -/* -* Interim function to return rcu_cblist head pointer. Longer term, the -* rcu_cblist will be used more pervasively, removing the need for this -* function. -*/ -static inline struct rcu_head *rcu_cblist_head (struct rcu_cblist *rclp) -{ -return rclp->head; -} - -/* -* Interim function to return rcu_cblist head pointer. Longer term, the -* rcu_cblist will be used more pervasively, removing the need for this -* function. -*/ -static inline struct rcu_head **rcu_cblist_tail (struct rcu_cblist *rclp) -{ -WARN_ON_ONCE (!rclp->head); -return rclp->tail; -} -void rcu_cblist_init (struct rcu_cblist *rclp); -long rcu_cblist_count_cbs (struct rcu_cblist *rclp,long lim);struct rcu_head *rcu_cblist_dequeue (struct rcu_cblist *rclp);/*
@@ -134,14 +112,10 @@ static inline struct rcu_head **rcu_segcblist_tail(struct rcu_segcblist *rsclp)void rcu_segcblist_init (struct rcu_segcblist *rsclp);void rcu_segcblist_disable (struct rcu_segcblist *rsclp); -bool rcu_segcblist_segempty (struct rcu_segcblist *rsclp,int seg);bool rcu_segcblist_ready_cbs (struct rcu_segcblist *rsclp);bool rcu_segcblist_pend_cbs (struct rcu_segcblist *rsclp); -struct rcu_head *rcu_segcblist_dequeue (struct rcu_segcblist *rsclp); -void rcu_segcblist_dequeued_lazy (struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_cb (struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_pend_cb (struct rcu_segcblist *rsclp); -bool rcu_segcblist_new_cbs (struct rcu_segcblist *rsclp);void rcu_segcblist_enqueue (struct rcu_segcblist *rsclp,struct rcu_head *rhp,bool lazy); bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,
Signature · 1 line
2.5.2
PATCH 12/12 rcu: Move callback-list warning to irq-disable region
In reply to Paul E. McKenneyAfter adopting callbacks from a newly offlined CPU, the adopting CPU checks to make sure that its callback list's count is zero only if the list has no callbacks and vice versa. Unfortunately, it does so after enabling interrupts, which means that false positives are possible due to interrupt handlers invoking call_rcu(). Although these false positives are improbable, rcutorture did make it happen once.
This commit therefore moves this check to an irq-disabled region of code, thus suppressing the false positive.
--- kernel/rcu/tree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree.c +2 -2
@@ -3869,14 +3869,14 @@ static void rcu_migrate_callbacks(int cpu, struct rcu_state *rsp)rcu_advance_cbs (rsp, rnp_root, rdp);/* Leverage recent GPs. */ rcu_advance_cbs (rsp, rnp_root, my_rdp);/* Assign GP to pending CBs. */ rcu_segcblist_merge (&my_rdp->cblist, &rdp->cblist); +WARN_ON_ONCE (rcu_segcblist_empty (&my_rdp->cblist) != + !rcu_segcblist_n_cbs (&my_rdp->cblist));raw_spin_unlock_irqrestore_rcu_node (rnp_root, flags);WARN_ONCE (rcu_segcblist_n_cbs (&rdp->cblist) !=0 || !rcu_segcblist_empty (&rdp->cblist),"rcu_cleanup_dead_cpu: Callbacks on offline CPU %d: qlen=%lu, 1stCB=%p\n" , cpu,rcu_segcblist_n_cbs (&rdp->cblist),rcu_segcblist_first_cb (&rdp->cblist)); -WARN_ON_ONCE (rcu_segcblist_empty (&my_rdp->cblist) != - !rcu_segcblist_n_cbs (&my_rdp->cblist)); }/*
Signature · 1 line
2.5.2