Forward-progress changes for v5.1
Subsystems (2)
cover Forward-progress changes for v5.1
Hello!
This series contains commits that improve RCU's ability to make forward progress on grace periods:
1. Accommodate zero jiffies_till_first_fqs and kthread kicking.
2-6. Move per-CPU variables to rcu_data structure, now that rcu_data
is strictly per-CPU itself due to there only being one flavor. Also remove rcu_cpu_kthread_loops and rcu_cpu_kthread_cpu, which are unused.
7. Update outdated NOCB comments.
8. Improve diagnostics for failed RCU grace-period start.
9. Protect rcu_check_gp_kthread_starvation() access to ->gp_flags.
10. Add sysrq rcu_node-dump capability to assist with debugging of
forward-progress issues.
11. Do RCU GP kthread self-wakeup from softirq and interrupt, thus
fixing a grace-period hang, courtesy of Jun Zhang.
12. Prevent needless ->gp_seq_needed update in __note_gp_changes(),
courtesy of Jun Zhang.
Thanx, Paul
------------------------------------------------------------------------
Documentation/admin-guide/kernel-parameters.txt | 5 + kernel/rcu/tree.c | 106 ++++++++++++++++-------- kernel/rcu/tree.h | 19 ++-- kernel/rcu/tree_plugin.h | 68 ++++++--------- 4 files changed, 117 insertions(+), 81 deletions(-)
PATCH 1/12 rcu: Accommodate zero jiffies_till_first_fqs and kthread kicking
It is perfectly fine to set the rcutree.jiffies_till_first_fqs boot parameter to zero, in fact, this can be useful on specialty systems that usually have at least one idle CPU and that need fast grace periods. This is because this setting causes the RCU grace-period kthread to scan for idle threads immediately after grace-period initialization, as opposed to waiting several jiffies to do so.
It is also perfectly fine to set the rcutree.rcu_kick_kthreads kernel parameter, which gives the RCU grace-period kthread an extra wakeup if it doesn't make progress for a period of three times the setting of the rcutree.jiffies_till_first_fqs boot parameter. This is of course problematic when the value of this parameter is zero, as it can result in unnecessary wakeup IPIs along with unnecessary WARN_ONCE() invocations.
This commit therefore defers kthread kicking for at least two jiffies, regardless of the setting of rcutree.jiffies_till_first_fqs.
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -1939,7 +1939,7 @@ static void rcu_gp_fqs_loop(void)if (!ret) { rcu_state.jiffies_force_qs = jiffies + j;WRITE_ONCE (rcu_state.jiffies_kick_kthreads, - jiffies +3 * j); + jiffies + (j ?3 * j :2 )); } trace_rcu_grace_period(rcu_state.name, READ_ONCE(rcu_state.gp_seq),
Signature · 1 line
2.17.1
PATCH 2/12 rcu: Move rcu_cpu_kthread_task to rcu_data structure
In reply to Paul E. McKenneyGiven that RCU has a perfectly good per-CPU rcu_data structure, most per-CPU quantities should be stored there.
This commit therefore moves the rcu_cpu_kthread_task per-CPU variable to the rcu_data structure. This also makes this variable unconditionally present, which should be acceptable given the memory reduction due to the RCU flavor consolidation and also due to simplifications this will enable.
--- kernel/rcu/tree.h | 6 +++++- kernel/rcu/tree_plugin.h | 11 +++++------ 2 files changed, 10 insertions(+), 7 deletions(-)
kernel/rcu/tree.h +5 -1
@@ -234,7 +234,11 @@ struct rcu_data {/* Leader CPU takes GP-end wakeups. */ #endif /* #ifdef CONFIG_RCU_NOCB_CPU */ -/* 6) Diagnostic data, including RCU CPU stall warnings. */ +/* 6) RCU priority boosting. */ +struct task_struct *rcu_cpu_kthread_task; +/* rcuc per-CPU kthread or NULL. */ + +/* 7) Diagnostic data, including RCU CPU stall warnings. */ unsigned int softirq_snap;/* Snapshot of softirq activity. */ /* ->rcu_iw* fields protected by leaf rcu_node ->lock. */ struct irq_work rcu_iw;/* Check for non-irq activity. */
kernel/rcu/tree_plugin.h +5 -6
@@ -40,7 +40,6 @@/* * Control variables for per-CPU and per-rcu_node kthreads. */ -static DEFINE_PER_CPU(struct task_struct *, rcu_cpu_kthread_task); DEFINE_PER_CPU(unsigned int, rcu_cpu_kthread_status); DEFINE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);DEFINE_PER_CPU (char, rcu_cpu_has_work);
@@ -1308,9 +1307,9 @@ static void invoke_rcu_callbacks_kthread(void)local_irq_save (flags);__this_cpu_write (rcu_cpu_has_work,1 ); -if (__this_cpu_read (rcu_cpu_kthread_task) !=NULL && - current !=__this_cpu_read (rcu_cpu_kthread_task)) { -rcu_wake_cond (__this_cpu_read (rcu_cpu_kthread_task), +if (__this_cpu_read (rcu_data.rcu_cpu_kthread_task) !=NULL && + current !=__this_cpu_read (rcu_data.rcu_cpu_kthread_task)) { +rcu_wake_cond (__this_cpu_read (rcu_data.rcu_cpu_kthread_task),__this_cpu_read (rcu_cpu_kthread_status)); }local_irq_restore (flags);
@@ -1322,7 +1321,7 @@ static void invoke_rcu_callbacks_kthread(void)*/ static bool rcu_is_callbacks_kthread (void ) { -return __this_cpu_read (rcu_cpu_kthread_task) == current; +return __this_cpu_read (rcu_data.rcu_cpu_kthread_task) == current; }#define RCU_BOOST_DELAY_JIFFIES DIV_ROUND_UP(CONFIG_RCU_BOOST_DELAY * HZ, 1000)
@@ -1459,7 +1458,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) }static struct smp_hotplug_thread rcu_cpu_thread_spec = { - .store = &rcu_cpu_kthread_task, + .store = &rcu_data.rcu_cpu_kthread_task, .thread_should_run = rcu_cpu_kthread_should_run, .thread_fn = rcu_cpu_kthread, .thread_comm ="rcuc/%u" ,
Signature · 1 line
2.17.1
PATCH 3/12 rcu: Move rcu_cpu_kthread_status to rcu_data structure
In reply to Paul E. McKenneyGiven that RCU has a perfectly good per-CPU rcu_data structure, most per-CPU quantities should be stored there.
This commit therefore moves the rcu_cpu_kthread_status per-CPU variable to the rcu_data structure. This also makes this variable unconditionally present, which should be acceptable given the memory reduction due to the RCU flavor consolidation and also due to simplifications this will enable.
--- kernel/rcu/tree.h | 3 ++- kernel/rcu/tree_plugin.h | 7 +++---- 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/rcu/tree.h +2 -1
@@ -237,6 +237,8 @@ struct rcu_data {/* 6) RCU priority boosting. */ struct task_struct *rcu_cpu_kthread_task;/* rcuc per-CPU kthread or NULL. */ +unsigned int rcu_cpu_kthread_status; +/* Running status for rcuc. */ /* 7) Diagnostic data, including RCU CPU stall warnings. */ unsigned int softirq_snap;/* Snapshot of softirq activity. */
@@ -407,7 +409,6 @@ static const char *tp_rcu_varname __used __tracepoint_string = rcu_name;int rcu_dynticks_snap (struct rcu_data *rdp);#ifdef CONFIG_RCU_BOOST -DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_status);DECLARE_PER_CPU (int, rcu_cpu_kthread_cpu); DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);DECLARE_PER_CPU (char, rcu_cpu_has_work);
kernel/rcu/tree_plugin.h +3 -4
@@ -40,7 +40,6 @@/* * Control variables for per-CPU and per-rcu_node kthreads. */ -DEFINE_PER_CPU(unsigned int, rcu_cpu_kthread_status); DEFINE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);DEFINE_PER_CPU (char, rcu_cpu_has_work);
@@ -1310,7 +1309,7 @@ static void invoke_rcu_callbacks_kthread(void)if (__this_cpu_read (rcu_data.rcu_cpu_kthread_task) !=NULL && current !=__this_cpu_read (rcu_data.rcu_cpu_kthread_task)) {rcu_wake_cond (__this_cpu_read (rcu_data.rcu_cpu_kthread_task), -__this_cpu_read (rcu_cpu_kthread_status)); +__this_cpu_read (rcu_data.rcu_cpu_kthread_status)); }local_irq_restore (flags); }
@@ -1383,7 +1382,7 @@ static void rcu_cpu_kthread_setup(unsigned int cpu)static void rcu_cpu_kthread_park (unsigned int cpu) { -per_cpu (rcu_cpu_kthread_status, cpu) =RCU_KTHREAD_OFFCPU ; +per_cpu (rcu_data.rcu_cpu_kthread_status, cpu) =RCU_KTHREAD_OFFCPU ; }static int rcu_cpu_kthread_should_run (unsigned int cpu)
@@ -1398,7 +1397,7 @@ static int rcu_cpu_kthread_should_run(unsigned int cpu)*/ static void rcu_cpu_kthread (unsigned int cpu) { -unsigned int *statusp =this_cpu_ptr (&rcu_cpu_kthread_status); +unsigned int *statusp =this_cpu_ptr (&rcu_data.rcu_cpu_kthread_status);char work, *workp =this_cpu_ptr (&rcu_cpu_has_work);int spincnt;
Signature · 1 line
2.17.1
PATCH 4/12 rcu: Remove unused rcu_cpu_kthread_loops per-CPU variable
In reply to Paul E. McKenneyThe rcu_cpu_kthread_loops variable used to provide debugfs information, but is no longer used. This commit therefore removes it.
--- kernel/rcu/tree.h | 1 - kernel/rcu/tree_plugin.h | 2 -- 2 files changed, 3 deletions(-)
kernel/rcu/tree.h +0 -1
@@ -410,7 +410,6 @@ int rcu_dynticks_snap(struct rcu_data *rdp);#ifdef CONFIG_RCU_BOOST DECLARE_PER_CPU (int, rcu_cpu_kthread_cpu); -DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);DECLARE_PER_CPU (char, rcu_cpu_has_work);#endif /* #ifdef CONFIG_RCU_BOOST */
kernel/rcu/tree_plugin.h +0 -2
@@ -40,7 +40,6 @@/* * Control variables for per-CPU and per-rcu_node kthreads. */ -DEFINE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);DEFINE_PER_CPU (char, rcu_cpu_has_work);#else /* #ifdef CONFIG_RCU_BOOST */
@@ -1405,7 +1404,6 @@ static void rcu_cpu_kthread(unsigned int cpu)trace_rcu_utilization (TPS ("Start CPU kthread@rcu_wait" ));local_bh_disable (); *statusp =RCU_KTHREAD_RUNNING ; -this_cpu_inc (rcu_cpu_kthread_loops);local_irq_disable (); work = *workp; *workp =0 ;
Signature · 1 line
2.17.1
PATCH 5/12 rcu: Move rcu_cpu_has_work to rcu_data structure
In reply to Paul E. McKenneyGiven that RCU has a perfectly good per-CPU rcu_data structure, most per-CPU quantities should be stored there.
This commit therefore moves the rcu_cpu_has_work per-CPU variable to the rcu_data structure. This also makes this variable unconditionally present, which should be acceptable given the memory reduction due to the RCU flavor consolidation and also due to simplifications this will enable.
--- kernel/rcu/tree.h | 3 +-- kernel/rcu/tree_plugin.h | 15 ++++----------- 2 files changed, 5 insertions(+), 13 deletions(-)
kernel/rcu/tree.h +1 -2
@@ -238,7 +238,7 @@ struct rcu_data {struct task_struct *rcu_cpu_kthread_task;/* rcuc per-CPU kthread or NULL. */ unsigned int rcu_cpu_kthread_status; -/* Running status for rcuc. */ +char rcu_cpu_has_work;/* 7) Diagnostic data, including RCU CPU stall warnings. */ unsigned int softirq_snap;/* Snapshot of softirq activity. */
@@ -410,7 +410,6 @@ int rcu_dynticks_snap(struct rcu_data *rdp);#ifdef CONFIG_RCU_BOOST DECLARE_PER_CPU (int, rcu_cpu_kthread_cpu); -DECLARE_PER_CPU (char, rcu_cpu_has_work);#endif /* #ifdef CONFIG_RCU_BOOST */ /* Forward declarations for rcutree_plugin.h */
kernel/rcu/tree_plugin.h +4 -11
@@ -34,14 +34,7 @@#include "../time/tick-internal.h" #ifdef CONFIG_RCU_BOOST -#include "../locking/rtmutex_common.h" - -/* -* Control variables for per-CPU and per-rcu_node kthreads. -*/ -DEFINE_PER_CPU (char, rcu_cpu_has_work); -#else /* #ifdef CONFIG_RCU_BOOST */ /*
@@ -1304,7 +1297,7 @@ static void invoke_rcu_callbacks_kthread(void)unsigned long flags;local_irq_save (flags); -__this_cpu_write (rcu_cpu_has_work,1 ); +__this_cpu_write (rcu_data.rcu_cpu_has_work,1 );if (__this_cpu_read (rcu_data.rcu_cpu_kthread_task) !=NULL && current !=__this_cpu_read (rcu_data.rcu_cpu_kthread_task)) { rcu_wake_cond(__this_cpu_read(rcu_data.rcu_cpu_kthread_task),
@@ -1386,7 +1379,7 @@ static void rcu_cpu_kthread_park(unsigned int cpu)static int rcu_cpu_kthread_should_run (unsigned int cpu) { -return __this_cpu_read (rcu_cpu_has_work); +return __this_cpu_read (rcu_data.rcu_cpu_has_work); }/*
@@ -1397,7 +1390,7 @@ static int rcu_cpu_kthread_should_run(unsigned int cpu)static void rcu_cpu_kthread (unsigned int cpu) {unsigned int *statusp =this_cpu_ptr (&rcu_data.rcu_cpu_kthread_status); -char work, *workp =this_cpu_ptr (&rcu_cpu_has_work); +char work, *workp =this_cpu_ptr (&rcu_data.rcu_cpu_has_work);int spincnt;for (spincnt =0 ; spincnt <10 ; spincnt++) {
@@ -1472,7 +1465,7 @@ static void __init rcu_spawn_boost_kthreads(void)int cpu;for_each_possible_cpu (cpu) -per_cpu (rcu_cpu_has_work, cpu) =0 ; +per_cpu (rcu_data.rcu_cpu_has_work, cpu) =0 ;if (WARN_ONCE (smpboot_register_percpu_thread (&rcu_cpu_thread_spec),"%s: Could not start rcub kthread, OOM is now expected behavior\n" , __func__))return ;rcu_for_each_leaf_node (rnp)
Signature · 1 line
2.17.1
PATCH 6/12 rcu: Remove unused rcu_cpu_kthread_cpu per-CPU variable
In reply to Paul E. McKenneyThe rcu_cpu_kthread_cpu used to provide debugfs information, but is no longer used. This commit therefore removes it.
--- kernel/rcu/tree.h | 4 ---- 1 file changed, 4 deletions(-)
kernel/rcu/tree.h +0 -4
@@ -408,10 +408,6 @@ static const char *tp_rcu_varname __used __tracepoint_string = rcu_name;int rcu_dynticks_snap (struct rcu_data *rdp); -#ifdef CONFIG_RCU_BOOST -DECLARE_PER_CPU (int, rcu_cpu_kthread_cpu); -#endif /* #ifdef CONFIG_RCU_BOOST */ -/* Forward declarations for rcutree_plugin.h */ static void rcu_bootup_announce (void );static void rcu_qs (void );
Signature · 1 line
2.17.1
PATCH 7/12 rcu: Update NOCB comments
In reply to Paul E. McKenneyThis commit updates a few obsolete comments in the RCU callback-offload code.
--- kernel/rcu/tree_plugin.h | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-)
kernel/rcu/tree_plugin.h +16 -17
@@ -1857,22 +1857,24 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp)/* * Offload callback processing from the boot-time-specified set of CPUs -* specified by rcu_nocb_mask. For each CPU in the set, there is a -* kthread created that pulls the callbacks from the corresponding CPU, -* waits for a grace period to elapse, and invokes the callbacks. -* The no-CBs CPUs do a wake_up() on their kthread when they insert -* a callback into any empty list, unless the rcu_nocb_poll boot parameter -* has been specified, in which case each kthread actively polls its -* CPU. (Which isn't so great for energy efficiency, but which does -* reduce RCU's overhead on that CPU.) +* specified by rcu_nocb_mask. For the CPUs in the set, there are kthreads +* created that pull the callbacks from the corresponding CPU, wait for +* a grace period to elapse, and invoke the callbacks. These kthreads +* are organized into leaders, which manage incoming callbacks, wait for +* grace periods, and awaken followers, and the followers, which only +* invoke callbacks. Each leader is its own follower. The no-CBs CPUs +* do a wake_up() on their kthread when they insert a callback into any +* empty list, unless the rcu_nocb_poll boot parameter has been specified, +* in which case each kthread actively polls its CPU. (Which isn't so great +* for energy efficiency, but which does reduce RCU's overhead on that CPU.) * * This is intended to be used in conjunction with Frederic Weisbecker's * adaptive-idle work, which would seriously reduce OS jitter on CPUs * running CPU-bound user-mode computations. * -* Offloading of callback processing could also in theory be used as -* an energy-efficiency measure because CPUs with no RCU callbacks -* queued are more aggressive about entering dyntick-idle mode. +* Offloading of callbacks can also be used as an energy-efficiency +* measure because CPUs with no RCU callbacks queued are more aggressive +* about entering dyntick-idle mode. */
@@ -1976,10 +1978,7 @@ static void wake_nocb_leader_defer(struct rcu_data *rdp, int waketype,raw_spin_unlock_irqrestore (&rdp->nocb_lock, flags); } -/* -* Does the specified CPU need an RCU callback for this invocation -* of rcu_barrier()? -*/ +/* Does rcu_barrier need to queue an RCU callback on the specified CPU? */ static bool rcu_nocb_cpu_needs_barrier (int cpu) {struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu);
@@ -1995,8 +1994,8 @@ static bool rcu_nocb_cpu_needs_barrier(int cpu)* callbacks would be posted. In the worst case, the first * barrier in rcu_barrier() suffices (but the caller cannot * necessarily rely on this, not a substitute for the caller -* getting the concurrency design right!). There must also be -* a barrier between the following load an posting of a callback +* getting the concurrency design right!). There must also be a +* barrier between the following load and posting of a callback * (if a callback is in fact needed). This is associated with an * atomic_inc() in the caller. */
Signature · 1 line
2.17.1
PATCH 8/12 rcu: Improve diagnostics for failed RCU grace-period start
In reply to Paul E. McKenneyIf a grace period fails to start (for example, because you commented out the last two lines of rcu_accelerate_cbs_unlocked()), rcu_core() will invoke rcu_check_gp_start_stall(), which will notice and complain. However, this complaint is lacking crucial debugging information such as when the last wakeup executed and what the value of ->gp_seq was at that time. This commit therefore removes the current pr_alert() from rcu_check_gp_start_stall(), instead invoking show_rcu_gp_kthreads(), which has been updated to print the needed information, which is collected by rcu_gp_kthread_wake().
--- kernel/rcu/tree.c | 55 +++++++++++++++++++++++++++-------------------- kernel/rcu/tree.h | 2 ++ 2 files changed, 34 insertions(+), 23 deletions(-)
kernel/rcu/tree.c +32 -23
@@ -512,6 +512,14 @@ void rcu_force_quiescent_state(void) }EXPORT_SYMBOL_GPL (rcu_force_quiescent_state); +/* +* Return the root node of the rcu_state structure. +*/ +static struct rcu_node *rcu_get_root (void ) +{ +return &rcu_state.node[0 ]; +} +/* * Convert a ->gp_state value to a character string. */
@@ -529,19 +537,30 @@ void show_rcu_gp_kthreads(void) {int cpu;unsigned long j; +unsigned long ja; +unsigned long jr; +unsigned long jw;struct rcu_data *rdp;struct rcu_node *rnp; - j = jiffies -READ_ONCE (rcu_state.gp_activity); -pr_info ("%s: wait state: %s(%d) ->state: %#lx delta ->gp_activity %ld\n" , + j = jiffies; + ja = j -READ_ONCE (rcu_state.gp_activity); + jr = j -READ_ONCE (rcu_state.gp_req_activity); + jw = j -READ_ONCE (rcu_state.gp_wake_time); +pr_info ("%s: wait state: %s(%d) ->state: %#lx delta ->gp_activity %lu ->gp_req_activity %lu ->gp_wake_time %lu ->gp_wake_seq %ld ->gp_seq %ld ->gp_seq_needed %ld ->gp_flags %#x\n" , rcu_state.name,gp_state_getname (rcu_state.gp_state), - rcu_state.gp_state, rcu_state.gp_kthread->state, j); + rcu_state.gp_state, + rcu_state.gp_kthread ? rcu_state.gp_kthread->state :0x1ffffL , + ja, jr, jw, (long )READ_ONCE (rcu_state.gp_wake_seq), + (long )READ_ONCE (rcu_state.gp_seq), + (long )READ_ONCE (rcu_get_root ()->gp_seq_needed), +READ_ONCE (rcu_state.gp_flags));rcu_for_each_node_breadth_first (rnp) {if (ULONG_CMP_GE (rcu_state.gp_seq, rnp->gp_seq_needed))continue ; -pr_info ("\trcu_node %d:%d ->gp_seq %lu ->gp_seq_needed %lu\n" , - rnp->grplo, rnp->grphi, rnp->gp_seq, - rnp->gp_seq_needed); +pr_info ("\trcu_node %d:%d ->gp_seq %ld ->gp_seq_needed %ld\n" , + rnp->grplo, rnp->grphi, (long )rnp->gp_seq, + (long )rnp->gp_seq_needed);if (!rcu_is_leaf_node (rnp))continue ;for_each_leaf_node_possible_cpu (rnp, cpu) {
@@ -550,8 +569,8 @@ void show_rcu_gp_kthreads(void)ULONG_CMP_GE (rcu_state.gp_seq, rdp->gp_seq_needed)) continue; -pr_info ("\tcpu %d ->gp_seq_needed %lu\n" , - cpu, rdp->gp_seq_needed); +pr_info ("\tcpu %d ->gp_seq_needed %ld\n" , + cpu, (long )rdp->gp_seq_needed); } }/* sched_show_task(rcu_state.gp_kthread); */
@@ -577,14 +596,6 @@ void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, }EXPORT_SYMBOL_GPL (rcutorture_get_gp_data); -/* -* Return the root node of the rcu_state structure. -*/ -static struct rcu_node *rcu_get_root (void ) -{ -return &rcu_state.node[0 ]; -} -/* * Enter an RCU extended quiescent state, which can be either the * idle loop or adaptive-tickless usermode execution.
@@ -1560,7 +1571,8 @@ static bool rcu_future_gp_cleanup(struct rcu_node *rnp)* Awaken the grace-period kthread. Don't do a self-awaken, and don't * bother awakening when there is nothing for the grace-period kthread * to do (as in several CPUs raced to awaken, and we lost), and finally -* don't try to awaken a kthread that has not yet been created. +* don't try to awaken a kthread that has not yet been created. If +* all those checks are passed, track some debug information and awaken. */ static void rcu_gp_kthread_wake (void ) {
@@ -1568,6 +1580,8 @@ static void rcu_gp_kthread_wake(void) !READ_ONCE (rcu_state.gp_flags) || !rcu_state.gp_kthread) return; +WRITE_ONCE (rcu_state.gp_wake_time, jiffies); +WRITE_ONCE (rcu_state.gp_wake_seq,READ_ONCE (rcu_state.gp_seq));swake_up_one (&rcu_state.gp_wq); }
@@ -2657,16 +2671,11 @@ rcu_check_gp_start_stall(struct rcu_node *rnp, struct rcu_data *rdp,raw_spin_unlock_irqrestore_rcu_node (rnp, flags);return ; } -pr_alert ("%s: g%ld->%ld gar:%lu ga:%lu f%#x gs:%d %s->state:%#lx\n" , - __func__, (long )READ_ONCE (rcu_state.gp_seq), - (long )READ_ONCE (rnp_root->gp_seq_needed), - j - rcu_state.gp_req_activity, j - rcu_state.gp_activity, - rcu_state.gp_flags, rcu_state.gp_state, rcu_state.name, - rcu_state.gp_kthread ? rcu_state.gp_kthread->state :0x1ffffL );WARN_ON (1 );if (rnp_root != rnp)raw_spin_unlock_rcu_node (rnp_root);raw_spin_unlock_irqrestore_rcu_node (rnp, flags); +show_rcu_gp_kthreads (); }/*
kernel/rcu/tree.h +2 -0
@@ -309,6 +309,8 @@ struct rcu_state {struct swait_queue_head gp_wq;/* Where GP task waits. */ short gp_flags;/* Commands for GP task. */ short gp_state;/* GP kthread sleep state. */ +unsigned long gp_wake_time;/* Last GP kthread wake. */ +unsigned long gp_wake_seq;/* ->gp_seq at ^^^. */ /* End of fields guarded by root rcu_node's lock. */
Signature · 1 line
2.17.1
PATCH 9/12 rcu: Protect rcu_check_gp_kthread_starvation() access to ->gp_flags
In reply to Paul E. McKenneyThe rcu_check_gp_kthread_starvation() function can be invoked without holding locks, so the access to the rcu_state structure's ->gp_flags field must be protected with READ_ONCE(). This commit therefore adds this protection.
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -1192,7 +1192,7 @@ static void rcu_check_gp_kthread_starvation(void)pr_err ("%s kthread starved for %ld jiffies! g%ld f%#x %s(%d) ->state=%#lx ->cpu=%d\n" , rcu_state.name, j, (long )rcu_seq_current (&rcu_state.gp_seq), - rcu_state.gp_flags, +READ_ONCE (rcu_state.gp_flags),gp_state_getname (rcu_state.gp_state), rcu_state.gp_state, gpk ? gpk->state : ~0 , gpk ?task_cpu (gpk) :-1 );if (gpk) {
Signature · 1 line
2.17.1
PATCH 10/12 rcu: Add sysrq rcu_node-dump capability
In reply to Paul E. McKenneyLife is hard if RCU manages to get stuck without triggering RCU CPU stall warnings or triggering the rcu_check_gp_start_stall() checks for failing to start a grace period. This commit therefore adds a boot-time-selectable sysrq key (commandeering "y") that allows manually dumping Tree RCU state. The new rcutree.sysrq_rcu kernel boot parameter must be set for this sysrq to be available.
--- .../admin-guide/kernel-parameters.txt | 5 ++++ kernel/rcu/tree.c | 25 +++++++++++++++++++ 2 files changed, 30 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +5 -0
@@ -3721,6 +3721,11 @@ This wake_up() will be accompanied by a WARN_ONCE() splat and an ftrace_dump(). + rcutree.sysrq_rcu= [KNL] + Commandeer a sysrq key to dump out Tree RCU's + rcu_node tree with an eye towards determining + why a new grace period has not yet started. + rcuperf.gp_async= [KNL] Measure performance of asynchronous grace-period primitives such as call_rcu().
kernel/rcu/tree.c +25 -0
@@ -62,6 +62,7 @@#include <linux/suspend.h> #include <linux/ftrace.h> #include <linux/tick.h> +#include <linux/sysrq.h> #include "tree.h" #include "rcu.h"
@@ -115,6 +116,9 @@ int num_rcu_lvl[] = NUM_RCU_LVL_INIT;int rcu_num_nodes__read_mostly =NUM_RCU_NODES ;/* Total # rcu_nodes in use. */ /* panic() on RCU Stall sysctl. */ int sysctl_panic_on_rcu_stall__read_mostly ; +/* Commandeer a sysrq key to dump RCU's tree. */ +static bool sysrq_rcu; +module_param (sysrq_rcu, bool,0444 );/* * The rcu_scheduler_active variable is initialized to the value
@@ -577,6 +581,27 @@ void show_rcu_gp_kthreads(void) }EXPORT_SYMBOL_GPL (show_rcu_gp_kthreads); +/* Dump grace-period-request information due to commandeered sysrq. */ +static void sysrq_show_rcu (int key) +{ +show_rcu_gp_kthreads (); +} + +static struct sysrq_key_op sysrq_rcudump_op = { + .handler = sysrq_show_rcu, + .help_msg ="show-rcu(y)" , + .action_msg ="Show RCU tree" , + .enable_mask =SYSRQ_ENABLE_DUMP , +}; + +static int __init rcu_sysrq_init (void ) +{ +if (sysrq_rcu) +return register_sysrq_key ('y' , &sysrq_rcudump_op); +return 0 ; +} +early_initcall (rcu_sysrq_init); +/* * Send along grace-period-related data for rcutorture diagnostics. */
Signature · 1 line
2.17.1
PATCH 11/12 rcu: Do RCU GP kthread self-wakeup from softirq and interrupt
In reply to Paul E. McKenneyFrom: "Zhang, Jun" <[email protected]>
The rcu_gp_kthread_wake() function is invoked when it might be necessary to wake the RCU grace-period kthread. Because self-wakeups are normally a useless waste of CPU cycles, if rcu_gp_kthread_wake() is invoked from this kthread, it naturally refuses to do the wakeup.
Unfortunately, natural though it might be, this heuristic fails when rcu_gp_kthread_wake() is invoked from an interrupt or softirq handler that interrupted the grace-period kthread just after the final check of the wait-event condition but just before the schedule() call. In this case, a wakeup is required, even though the call to rcu_gp_kthread_wake() is within the RCU grace-period kthread's context. Failing to provide this wakeup can result in grace periods failing to start, which in turn results in out-of-memory conditions.
This race window is quite narrow, but it actually did happen during real testing. It would of course need to be fixed even if it was strictly theoretical in nature.
This patch does not Cc stable because it does not apply cleanly to earlier kernel versions.
Signed-off: "Zhang, Jun" <[email protected]>
Signed-off: "He, Bo" <[email protected]>
Signed-off: "xiao, jin" <[email protected]>
Signed-off: Bai, Jie A <[email protected]>
[ paulmck: Switch from !in_softirq() to "!in_interrupt() &&
!in_serving_softirq() to avoid redundant wakeups and to also handle the interrupt-handler scenario as well as the softirq-handler scenario that actually occurred in testing. ]
--- kernel/rcu/tree.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +14 -6
@@ -1593,15 +1593,23 @@ static bool rcu_future_gp_cleanup(struct rcu_node *rnp) }/* -* Awaken the grace-period kthread. Don't do a self-awaken, and don't -* bother awakening when there is nothing for the grace-period kthread -* to do (as in several CPUs raced to awaken, and we lost), and finally -* don't try to awaken a kthread that has not yet been created. If -* all those checks are passed, track some debug information and awaken. +* Awaken the grace-period kthread. Don't do a self-awaken (unless in +* an interrupt or softirq handler), and don't bother awakening when there +* is nothing for the grace-period kthread to do (as in several CPUs raced +* to awaken, and we lost), and finally don't try to awaken a kthread that +* has not yet been created. If all those checks are passed, track some +* debug information and awaken. +* +* So why do the self-wakeup when in an interrupt or softirq handler +* in the grace-period kthread's context? Because the kthread might have +* been interrupted just as it was going to sleep, and just after the final +* pre-sleep check of the awaken condition. In this case, a wakeup really +* is required, and is therefore supplied. */ static void rcu_gp_kthread_wake (void ) { -if (current == rcu_state.gp_kthread || +if ((current == rcu_state.gp_kthread && + !in_interrupt () && !in_serving_softirq ()) || !READ_ONCE (rcu_state.gp_flags) || !rcu_state.gp_kthread)return ;
Signature · 1 line
2.17.1
PATCH 12/12 rcu: Prevent needless ->gp_seq_needed update in __note_gp_changes()
In reply to Paul E. McKenneyFrom: "Zhang, Jun" <[email protected]>
Currently, __note_gp_changes() checks to see if the rcu_node structure's ->gp_seq_needed is greater than or equal to that of the rcu_data structure, and if so, updates the rcu_data structure's ->gp_seq_needed field. This results in a useless store in the case where the two fields are equal.
This commit therefore carries out this store only in the case where the rcu_node structure's ->gp_seq_needed is strictly greater than that of the rcu_data structure.
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -1758,7 +1758,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)zero_cpu_stall_ticks (rdp); } rdp->gp_seq = rnp->gp_seq;/* Remember new grace-period state. */ -if (ULONG_CMP_GE (rnp->gp_seq_needed, rdp->gp_seq_needed) || rdp->gpwrap) +if (ULONG_CMP_LT (rdp->gp_seq_needed, rnp->gp_seq_needed) || rdp->gpwrap) rdp->gp_seq_needed = rnp->gp_seq_needed;WRITE_ONCE (rdp->gpwrap, false);rcu_gpnum_ovf (rnp, rdp);
Signature · 1 line
2.17.1