rcu: further grace-period processing cleanups
cover rcu: further grace-period processing cleanups
This patch set provides a couple of additional grace-period-processing cleanups:
1. Propagate ->completed through the rcu_node hierarchy
when RCU goes idle to prevent an additional needless grace period from being started.
2. Make force_quiescent_state() take a snapshot of ->gpnum-1
rather than ->completed for consistency with rcu_sched_qs() and friends.
PATCH 1/2 rcu: accelerate callback processing on CPUs not detecting GP end
An earlier fix for a race resulted in a situation where the CPUs other than the CPU that detected the end of the grace period would not process their callbacks until the next grace period started. This means that these other CPUs would unnecessarily demand that an extra grace period be started. This patch eliminates this extra grace period and speeds callback processing by propagating rsp->completed to the rcu_node structures in the case where the CPU detecting the end of the grace period sees no reason to start a new grace period.
--- kernel/rcutree.c | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-)
kernel/rcutree.c +17 -1
@@ -676,7 +676,23 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)struct rcu_node *rnp =rcu_get_root (rsp);if (!cpu_needs_another_gp (rsp, rdp)) { -spin_unlock_irqrestore (&rnp->lock, flags); +if (rnp->completed == rsp->completed) { +spin_unlock_irqrestore (&rnp->lock, flags); +return ; + } +spin_unlock (&rnp->lock);/* irqs remain disabled. */ + +/* +* Propagate new ->completed value to rcu_node structures +* so that other CPUs don't have to wait until the start +* of the next grace period to process their callbacks. +*/ +rcu_for_each_node_breadth_first (rsp, rnp) { +spin_lock (&rnp->lock);/* irqs already disabled. */ + rnp->completed = rsp->completed; +spin_unlock (&rnp->lock);/* irqs remain disabled. */ + } +local_irq_restore (flags);return ; }
Signature · 1 line
1.5.2.5
PATCH 2/2 rcu: simplify association of forced quiescent states with grace periods
In reply to Paul E. McKenneyFrom: Paul E. McKenney <[email protected]>
The force_quiescent_state() function also took a snapshot
of the ->completed field, which was as obnoxious as it was in rcu_sched_qs() and friends. So snapshot ->gpnum-1. Also, since the dyntick_record_completed() and dyntick_recall_completed() functions are now simple assignments that are independent of CONFIG_NO_HZ, and since their names are now misleading, get rid of them.
--- kernel/rcutree.c | 27 +++------------------------ 1 files changed, 3 insertions(+), 24 deletions(-)
kernel/rcutree.c +3 -24
@@ -178,29 +178,9 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp)return &rsp->node[0 ]; } -/* -* Record the specified "completed" value, which is later used to validate -* dynticks counter manipulations and CPU-offline checks. Specify -* "rsp->completed - 1" to unconditionally invalidate any future dynticks -* manipulations and CPU-offline checks. Such invalidation is useful at -* the beginning of a grace period. -*/ -static void dyntick_record_completed (struct rcu_state *rsp,long comp) -{ - rsp->completed_fqs = comp; -} -#ifdef CONFIG_SMP /* -* Recall the previously recorded value of the completion for dynticks. -*/ -static long dyntick_recall_completed (struct rcu_state *rsp) -{ -return rsp->completed_fqs; -} - -/* * If the specified CPU is offline, tell the caller that it is in * a quiescent state. Otherwise, whack it with a reschedule IPI. * Grace periods can end up waiting on an offline CPU when that
@@ -702,7 +682,6 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags) rsp->signaled =RCU_GP_INIT ;/* Hold off force_quiescent_state. */ rsp->jiffies_force_qs = jiffies +RCU_JIFFIES_TILL_FORCE_QS ;record_gp_stall_check_time (rsp); -dyntick_record_completed (rsp, rsp->completed -1 );/* Special-case the common single-level case. */ if (NUM_RCU_NODES ==1 ) {
@@ -1201,7 +1180,7 @@ static void force_quiescent_state(struct rcu_state *rsp, int relaxed) goto unlock_ret;/* no emergency and done recently. */ rsp->n_force_qs++;spin_lock (&rnp->lock); - lastcomp = rsp->completed; + lastcomp = rsp->gpnum -1 ; signaled = rsp->signaled; rsp->jiffies_force_qs = jiffies +RCU_JIFFIES_TILL_FORCE_QS ;if (lastcomp == rsp->gpnum) {
@@ -1235,7 +1214,7 @@ static void force_quiescent_state(struct rcu_state *rsp, int relaxed)if (lastcomp == rsp->completed && rsp->signaled == signaled) { rsp->signaled =RCU_FORCE_QS ; -dyntick_record_completed (rsp, lastcomp); + rsp->completed_fqs = lastcomp; forcenow = signaled ==RCU_SAVE_COMPLETED ; }spin_unlock (&rnp->lock);
@@ -1246,7 +1225,7 @@ static void force_quiescent_state(struct rcu_state *rsp, int relaxed)case RCU_FORCE_QS :/* Check dyntick-idle state, send IPI to laggarts. */ -if (rcu_process_dyntick (rsp,dyntick_recall_completed (rsp), +if (rcu_process_dyntick (rsp, rsp->completed_fqs, rcu_implicit_dynticks_qs)) goto unlock_ret;
Signature · 1 line
1.5.2.5
[tip:core/rcu] rcu: Simplify association of forced quiescent states with grace periods
Commit-ID: 8e9aa8f067d2dcd9457980ced618e1cffbcfba46 Gitweb: http://git.kernel.org/tip/8e9aa8f067d2dcd9457980ced618e1cffbcfba46 Author: Paul E. McKenney <[email protected]> AuthorDate: Thu, 12 Nov 2009 22:35:04 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 13 Nov 2009 10:18:36 +0100
rcu: Simplify association of forced quiescent states with grace periods
The force_quiescent_state() function also took a snapshot of the ->completed field, which was as obnoxious as it was in rcu_sched_qs() and friends. So snapshot ->gpnum-1.
Also, since the dyntick_record_completed() and dyntick_recall_completed() functions are now simple assignments that are independent of CONFIG_NO_HZ, and since their names are now misleading, get rid of them.
LKML-Reference: <12580941042308-git-send-email->
--- kernel/rcutree.c | 27 +++------------------------ 1 files changed, 3 insertions(+), 24 deletions(-)
kernel/rcutree.c +3 -24
@@ -178,29 +178,9 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp)return &rsp->node[0 ]; } -/* -* Record the specified "completed" value, which is later used to validate -* dynticks counter manipulations and CPU-offline checks. Specify -* "rsp->completed - 1" to unconditionally invalidate any future dynticks -* manipulations and CPU-offline checks. Such invalidation is useful at -* the beginning of a grace period. -*/ -static void dyntick_record_completed (struct rcu_state *rsp,long comp) -{ - rsp->completed_fqs = comp; -} -#ifdef CONFIG_SMP /* -* Recall the previously recorded value of the completion for dynticks. -*/ -static long dyntick_recall_completed (struct rcu_state *rsp) -{ -return rsp->completed_fqs; -} - -/* * If the specified CPU is offline, tell the caller that it is in * a quiescent state. Otherwise, whack it with a reschedule IPI. * Grace periods can end up waiting on an offline CPU when that
@@ -702,7 +682,6 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags) rsp->signaled =RCU_GP_INIT ;/* Hold off force_quiescent_state. */ rsp->jiffies_force_qs = jiffies +RCU_JIFFIES_TILL_FORCE_QS ;record_gp_stall_check_time (rsp); -dyntick_record_completed (rsp, rsp->completed -1 );/* Special-case the common single-level case. */ if (NUM_RCU_NODES ==1 ) {
@@ -1214,7 +1193,7 @@ static void force_quiescent_state(struct rcu_state *rsp, int relaxed) goto unlock_ret;/* no emergency and done recently. */ rsp->n_force_qs++;spin_lock (&rnp->lock); - lastcomp = rsp->completed; + lastcomp = rsp->gpnum -1 ; signaled = rsp->signaled; rsp->jiffies_force_qs = jiffies +RCU_JIFFIES_TILL_FORCE_QS ;if (lastcomp == rsp->gpnum) {
@@ -1248,7 +1227,7 @@ static void force_quiescent_state(struct rcu_state *rsp, int relaxed)if (lastcomp == rsp->completed && rsp->signaled == signaled) { rsp->signaled =RCU_FORCE_QS ; -dyntick_record_completed (rsp, lastcomp); + rsp->completed_fqs = lastcomp; forcenow = signaled ==RCU_SAVE_COMPLETED ; }spin_unlock (&rnp->lock);
@@ -1259,7 +1238,7 @@ static void force_quiescent_state(struct rcu_state *rsp, int relaxed)case RCU_FORCE_QS :/* Check dyntick-idle state, send IPI to laggarts. */ -if (rcu_process_dyntick (rsp,dyntick_recall_completed (rsp), +if (rcu_process_dyntick (rsp, rsp->completed_fqs, rcu_implicit_dynticks_qs)) goto unlock_ret;
[tip:core/rcu] rcu: Accelerate callback processing on CPUs not detecting GP end
In reply to Paul E. McKenneyCommit-ID: b32e9eb6ad29572b4451847d0e8227c9be2b6d69 Gitweb: http://git.kernel.org/tip/b32e9eb6ad29572b4451847d0e8227c9be2b6d69 Author: Paul E. McKenney <[email protected]> AuthorDate: Thu, 12 Nov 2009 22:35:03 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 13 Nov 2009 10:18:36 +0100
rcu: Accelerate callback processing on CPUs not detecting GP end
An earlier fix for a race resulted in a situation where the CPUs other than the CPU that detected the end of the grace period would not process their callbacks until the next grace period started.
This means that these other CPUs would unnecessarily demand that an extra grace period be started.
This patch eliminates this extra grace period and speeds callback processing by propagating rsp->completed to the rcu_node structures in the case where the CPU detecting the end of the grace period sees no reason to start a new grace period.
LKML-Reference: <1258094104417-git-send-email->
--- kernel/rcutree.c | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-)
kernel/rcutree.c +17 -1
@@ -676,7 +676,23 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)struct rcu_node *rnp =rcu_get_root (rsp);if (!cpu_needs_another_gp (rsp, rdp)) { -spin_unlock_irqrestore (&rnp->lock, flags); +if (rnp->completed == rsp->completed) { +spin_unlock_irqrestore (&rnp->lock, flags); +return ; + } +spin_unlock (&rnp->lock);/* irqs remain disabled. */ + +/* +* Propagate new ->completed value to rcu_node structures +* so that other CPUs don't have to wait until the start +* of the next grace period to process their callbacks. +*/ +rcu_for_each_node_breadth_first (rsp, rnp) { +spin_lock (&rnp->lock);/* irqs already disabled. */ + rnp->completed = rsp->completed; +spin_unlock (&rnp->lock);/* irqs remain disabled. */ + } +local_irq_restore (flags);return ; }