rcu: Use rcu_gp_kthread_wake() to wake up grace period kthreads
PATCH v2 1/1 rcu: Use rcu_gp_kthread_wake() to wake up grace period kthreads
Updated changelog regarding the removed comment about the implied barrier provided by wake_up() logic.
--
Pranith
The rcu_gp_kthread_wake() function checks for three conditions before waking up grace period kthreads:
* Is the thread we are trying to wake up the current thread? * Are the gp_flags zero? (all threads wait on non-zero gp_flags condition) * Is there no thread created for this flavour, hence nothing to wake up?
If any one of these condition is true, we do not call wake_up(). It was found that there are quite a few avoidable wake ups both during idle time and under stress induced by rcutorture.
Idle:
Total:66000, unnecessary:66000, case1:61827, case2:66000, case3:0 Total:68000, unnecessary:68000, case1:63696, case2:68000, case3:0
rcutorture:
Total:254000, unnecessary:254000, case1:199913, case2:254000, case3:0 Total:256000, unnecessary:256000, case1:201784, case2:256000, case3:0
Here case{1-3} are the cases listed above. We can avoid these wake ups by using rcu_gp_kthread_wake() to conditionally wake up the grace period kthreads.
There is a comment about an implied barrier supplied by the wake_up() logic. This barrier is necessary for the awakened thread to see the updated ->gp_flags. This flag is always being updated with the root node lock held. Also, the awakened thread tries to acquire the root node lock before reading ->gp_flags because of which there is proper ordering.
Hence this commit tries to avoid calling wake_up() whenever we can by using rcu_gp_kthread_wake() function.
--- kernel/rcu/tree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree.c +2 -2
@@ -1938,7 +1938,7 @@ static void rcu_report_qs_rsp(struct rcu_state *rsp, unsigned long flags) {WARN_ON_ONCE (!rcu_gp_in_progress (rsp));raw_spin_unlock_irqrestore (&rcu_get_root (rsp)->lock, flags); -wake_up (&rsp->gp_wq);/* Memory barrier implied by wake_up() path. */ +rcu_gp_kthread_wake (rsp); }/*
@@ -2516,7 +2516,7 @@ static void force_quiescent_state(struct rcu_state *rsp)ACCESS_ONCE (rsp->gp_flags) =ACCESS_ONCE (rsp->gp_flags) |RCU_GP_FLAG_FQS ;raw_spin_unlock_irqrestore (&rnp_old->lock, flags); -wake_up (&rsp->gp_wq);/* Memory barrier implied by wake_up() path. */ +rcu_gp_kthread_wake (rsp); }/*
Signature · 1 line
1.9.1
Paul E. McKenney
AppliedQuoted 5 lines · Pranith Kumar
Updated changelog regarding the removed comment about the implied barrier provided by
wake_up()logic.--
Pranith
Queued for 3.18, thank you!
Thanx, Paul
Quoted 64 lines
The
rcu_gp_kthread_wake()function checks for three conditions before waking up grace period kthreads:* Is the thread we are trying to wake up the current thread? * Are the gp_flags zero? (all threads wait on non-zero gp_flags condition) * Is there no thread created for this flavour, hence nothing to wake up?If any one of these condition is true, we do not call
wake_up(). It was found that there are quite a few avoidable wake ups both during idle time and under stress induced by rcutorture.Idle:
Total:66000, unnecessary:66000, case1:61827, case2:66000, case3:0 Total:68000, unnecessary:68000, case1:63696, case2:68000, case3:0
rcutorture:
Total:254000, unnecessary:254000, case1:199913, case2:254000, case3:0 Total:256000, unnecessary:256000, case1:201784, case2:256000, case3:0
Here case{1-3} are the cases listed above. We can avoid these wake ups by using
rcu_gp_kthread_wake()to conditionally wake up the grace period kthreads.There is a comment about an implied barrier supplied by the
wake_up()logic. This barrier is necessary for the awakened thread to see the updated ->gp_flags. This flag is always being updated with the root node lock held. Also, the awakened thread tries to acquire the root node lock before reading ->gp_flags because of which there is proper ordering.Hence this commit tries to avoid calling
wake_up()whenever we can by usingrcu_gp_kthread_wake()function.Signed-off-by Pranith Kumar <[email protected]>CC Mathieu Desnoyers <[email protected]>--- kernel/rcu/tree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/rcu/tree.c +2
-2@@ -1938,7 +1938,7 @@ static void rcu_report_qs_rsp(struct rcu_state *rsp, unsigned long flags) {WARN_ON_ONCE (!rcu_gp_in_progress (rsp));raw_spin_unlock_irqrestore (&rcu_get_root (rsp)->lock, flags); -wake_up (&rsp->gp_wq);/* Memory barrier implied by wake_up() path. */ +rcu_gp_kthread_wake (rsp); }/* @@ -2516,7 +2516,7 @@ static void force_quiescent_state(struct rcu_state *rsp)ACCESS_ONCE (rsp->gp_flags) =ACCESS_ONCE (rsp->gp_flags) |RCU_GP_FLAG_FQS ;raw_spin_unlock_irqrestore (&rnp_old->lock, flags); -wake_up (&rsp->gp_wq);/* Memory barrier implied by wake_up() path. */ +rcu_gp_kthread_wake (rsp); }/* Signature · 1 line
1.9.1