Miscellaneous fixes
Subsystems (8)
cover Miscellaneous fixes
Hello!
This series provides miscellaneous fixes:
1. Remove unnecessary spinlock in rcu_boot_init_percpu_data(),
courtesy of Lihao Liang.
2. Eliminate NO_HZ_FULL_ALL, instead relying on the nohz_full
kernel boot parameter.
3. Fix CPU offload boot message when no CPUs are offloaded.
4. Fix typo in rcutorture documentation, courtesy of Lihao Liang.
5. Fix typo in the rcu_head structure's comments,courtesy of Lihao
Liang.
6. Call touch_nmi_watchdog() while printing stall warnings,
courtesy of Tejun Heo.
7. Remove obsolete boost statistics for debugfs.
8-10. Remove obsolete statistics for debugfs.
11. More clearly identify grace-period kthread stack dump.
12. Consolidate rcu.h #ifdefs.
13. Remove redundant nxttail index macro define, courtesy of
Liu Changcheng.
14. Use wrapper for lockdep asserts, courtesy of Matthew Wilcox.
15. Fix init_rcu_head() comment.
16. Fix misprint in srcu_funnel_exp_start, courtesy of Ildar Ismagilov.
17. Add more tracing of expedited grace periods.
18. Trace expedited GP delays due to transitioning CPUs.
19. Make expedited RCU CPU selection avoid unnecessary stores.
20. Create RCU-specific workqueues with rescuers.
Thanx, Paul
------------------------------------------------------------------------
Documentation/timers/NO_HZ.txt | 7 - include/linux/rcupdate.h | 10 + include/linux/types.h | 2 include/trace/events/rcu.h | 4 kernel/rcu/rcu.h | 38 ++++-- kernel/rcu/srcutree.c | 12 -- kernel/rcu/tree.c | 72 ++++-------- kernel/rcu/tree.h | 36 ------ kernel/rcu/tree_exp.h | 38 ++++-- kernel/rcu/tree_plugin.h | 34 +++-- kernel/time/Kconfig | 10 - kernel/time/tick-sched.c | 22 --- tools/testing/selftests/rcutorture/configs/rcu/TASKS03 | 1 tools/testing/selftests/rcutorture/configs/rcu/TASKS03.boot | 2 tools/testing/selftests/rcutorture/configs/rcu/TREE04 | 1 tools/testing/selftests/rcutorture/configs/rcu/TREE04.boot | 2 tools/testing/selftests/rcutorture/configs/rcu/TREE07 | 1 tools/testing/selftests/rcutorture/doc/rcu-test-image.txt | 2 18 files changed, 128 insertions(+), 166 deletions(-)
PATCH 4/20 doc: Fix typo in rcutorture documentation
From: Lihao Liang <[email protected]>
--- tools/testing/selftests/rcutorture/doc/rcu-test-image.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/rcutorture/doc/rcu-test-image.txt +1 -1
@@ -1,4 +1,4 @@ -This document describes one way to created the rcu-test-image file +This document describes one way to create the rcu-test-image file that contains the filesystem used by the guest-OS kernel. There are probably much better ways of doing this, and this filesystem could no doubt be smaller. It is probably also possible to simply download
Signature · 1 line
2.5.2
PATCH 5/20 doc: Fix typo in rcu_head comments
In reply to Paul E. McKenneyFrom: Lihao Liang <[email protected]>
--- include/linux/types.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/types.h +1 -1
@@ -217,7 +217,7 @@ struct ustat {* * This guarantee is important for few reasons: * - future call_rcu_lazy() will make use of lower bits in the pointer; -* - the structure shares storage spacer in struct page with @compound_head, +* - the structure shares storage space in struct page with @compound_head, * which encode PageTail() in bit 0. The guarantee is needed to avoid * false-positive PageTail(). */
Signature · 1 line
2.5.2
PATCH 1/20 rcu: Remove unnecessary spinlock in rcu_boot_init_percpu_data()
In reply to Paul E. McKenneyFrom: Lihao Liang <[email protected]>
Since rcu_boot_init_percpu_data() is only called at boot time, there is no data race and spinlock is not needed.
--- kernel/rcu/tree.c | 4 ---- 1 file changed, 4 deletions(-)
kernel/rcu/tree.c +0 -4
@@ -3636,12 +3636,9 @@ static void rcu_init_new_rnp(struct rcu_node *rnp_leaf)static void __init rcu_boot_init_percpu_data (int cpu,struct rcu_state *rsp) { -unsigned long flags;struct rcu_data *rdp =per_cpu_ptr (rsp->rda, cpu); -struct rcu_node *rnp =rcu_get_root (rsp);/* Set up local state, ensuring consistent view of global state. */ -raw_spin_lock_irqsave_rcu_node (rnp, flags); rdp->grpmask =leaf_node_cpu_bit (rdp->mynode, cpu); rdp->dynticks = &per_cpu (rcu_dynticks, cpu);WARN_ON_ONCE (rdp->dynticks->dynticks_nesting !=1 );
@@ -3649,7 +3646,6 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp) rdp->cpu = cpu; rdp->rsp = rsp;rcu_boot_init_nocb_percpu_data (rdp); -raw_spin_unlock_irqrestore_rcu_node (rnp, flags); }/*
Signature · 1 line
2.5.2
PATCH 11/20 rcu: More clearly identify grace-period kthread stack dump
In reply to Paul E. McKenneyIt is not always obvious that the stack dump from a starved grace-period kthread isn't instead that of a CPU stalling the current grace period. This commit therefore adds a pr_err() flagging these dumps.
--- kernel/rcu/tree.c | 1 + 1 file changed, 1 insertion(+)
kernel/rcu/tree.c +1 -0
@@ -1350,6 +1350,7 @@ static void rcu_check_gp_kthread_starvation(struct rcu_state *rsp) rsp->gp_kthread ? rsp->gp_kthread->state : ~0 , rsp->gp_kthread ? task_cpu(rsp->gp_kthread) : -1);if (rsp->gp_kthread) { +pr_err ("RCU grace-period kthread stack dump:\n" );sched_show_task (rsp->gp_kthread);wake_up_process (rsp->gp_kthread); }
Signature · 1 line
2.5.2
PATCH 8/20 rcu: Remove obsolete callback-invocation statistics for debugfs
In reply to Paul E. McKenneyThe debugfs interface displayed statistics on RCU callback invocation but this interface has since been removed. This commit therefore removes the no-longer-used rcu_data structure's ->n_cbs_invoked and ->n_nocbs_invoked fields along with their updates.
If this information proves necessary in the future, the corresponding event traces will be added.
--- kernel/rcu/tree.c | 1 - kernel/rcu/tree.h | 2 -- kernel/rcu/tree_plugin.h | 1 - 3 files changed, 4 deletions(-)
kernel/rcu/tree.c +0 -1
@@ -2691,7 +2691,6 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)/* Update counts and requeue any remaining callbacks. */ rcu_segcblist_insert_done_cbs (&rdp->cblist, &rcl);smp_mb ();/* List handling before counting for rcu_barrier(). */ - rdp->n_cbs_invoked += count;rcu_segcblist_insert_count (&rdp->cblist, &rcl);/* Reinstate batch limit if we have worked down the excess. */
kernel/rcu/tree.h +0 -2
@@ -211,8 +211,6 @@ struct rcu_data {/* different grace periods. */ long qlen_last_fqs_check;/* qlen at last check for QS forcing */ -unsigned long n_cbs_invoked;/* count of RCU cbs invoked. */ -unsigned long n_nocbs_invoked;/* count of no-CBs RCU cbs invoked. */ unsigned long n_force_qs_snap;/* did other CPU force QS recently? */ long blimit;/* Upper limit on a processed batch */
kernel/rcu/tree_plugin.h +0 -1
@@ -2243,7 +2243,6 @@ static int rcu_nocb_kthread(void *arg)smp_mb__before_atomic ();/* _add after CB invocation. */ atomic_long_add (-c, &rdp->nocb_q_count);atomic_long_add (-cl, &rdp->nocb_q_count_lazy); - rdp->n_nocbs_invoked += c; }return 0 ; }
Signature · 1 line
2.5.2
PATCH 13/20 rcu: Remove redundant nxttail index macro define
In reply to Paul E. McKenneyFrom: "Liu, Changcheng" <[email protected]>
RCU's nxttail has been optimized to be a rcu_segcblist, which is a multi-tailed linked list with macros defined for the indexes for each tail. The indexes have been defined in linux/rcu_segcblist.h, so this commit removes the redundant definitions in kernel/rcu/tree.h.
--- kernel/rcu/tree.h | 7 ------- 1 file changed, 7 deletions(-)
kernel/rcu/tree.h +0 -7
@@ -178,13 +178,6 @@ union rcu_noqs {u16 s;/* Set of bits, aggregate OR here. */ }; -/* Index values for nxttail array in struct rcu_data. */ -#define RCU_DONE_TAIL 0/* Also RCU_WAIT head. */ -#define RCU_WAIT_TAIL 1/* Also RCU_NEXT_READY head. */ -#define RCU_NEXT_READY_TAIL 2/* Also RCU_NEXT head. */ -#define RCU_NEXT_TAIL 3 -#define RCU_NEXT_SIZE 4 -/* Per-CPU data for read-copy update. */ struct rcu_data {/* 1) quiescent-state and grace-period handling : */
Signature · 1 line
2.5.2
PATCH 20/20 rcu: Create RCU-specific workqueues with rescuers
In reply to Paul E. McKenneyRCU's expedited grace periods can participate in out-of-memory deadlocks due to all available system_wq kthreads being blocked and there not being memory available to create more. This commit prevents such deadlocks by allocating an RCU-specific workqueue_struct at early boot time, and providing it with a rescuer to ensure forward progress. This uses the shiny new init_rescuer() function provided by Tejun (but indirectly).
This commit also causes SRCU to use this new RCU-specific workqueue_struct. Note that SRCU's use of workqueues never blocks them waiting for readers, so this should be safe from a forward-progress viewpoint. Note that this moves SRCU from system_power_efficient_wq to a normal workqueue. In the unlikely event that this results in measurable degradation, a separate power-efficient workqueue will be creates for SRCU.
--- kernel/rcu/rcu.h | 1 + kernel/rcu/srcutree.c | 8 +++----- kernel/rcu/tree.c | 6 ++++++ kernel/rcu/tree_exp.h | 2 +- 4 files changed, 11 insertions(+), 6 deletions(-)
kernel/rcu/rcu.h +1 -0
@@ -485,6 +485,7 @@ void show_rcu_gp_kthreads(void);void rcu_force_quiescent_state (void );void rcu_bh_force_quiescent_state (void );void rcu_sched_force_quiescent_state (void ); +extern struct workqueue_struct *rcu_gp_wq;#endif /* #else #ifdef CONFIG_TINY_RCU */ #ifdef CONFIG_RCU_NOCB_CPU
kernel/rcu/srcutree.c +3 -5
@@ -492,8 +492,7 @@ static bool srcu_queue_delayed_work_on(int cpu, struct workqueue_struct *wq,*/ static void srcu_schedule_cbs_sdp (struct srcu_data *sdp,unsigned long delay) { -srcu_queue_delayed_work_on (sdp->cpu, system_power_efficient_wq, - &sdp->work, delay); +srcu_queue_delayed_work_on (sdp->cpu, rcu_gp_wq, &sdp->work, delay); }/*
@@ -691,8 +690,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *sp, struct srcu_data *sdp, rcu_seq_state(sp->srcu_gp_seq) == SRCU_STATE_IDLE) {WARN_ON_ONCE (ULONG_CMP_GE (sp->srcu_gp_seq, sp->srcu_gp_seq_needed));srcu_gp_start (sp); -queue_delayed_work (system_power_efficient_wq, &sp->work, -srcu_get_delay (sp)); +queue_delayed_work (rcu_gp_wq, &sp->work,srcu_get_delay (sp)); }spin_unlock_irqrestore_rcu_node (sp, flags); }
@@ -1225,7 +1223,7 @@ static void srcu_reschedule(struct srcu_struct *sp, unsigned long delay)spin_unlock_irq_rcu_node (sp);if (pushgp) -queue_delayed_work (system_power_efficient_wq, &sp->work, delay); +queue_delayed_work (rcu_gp_wq, &sp->work, delay); }/*
kernel/rcu/tree.c +6 -0
@@ -4167,6 +4167,8 @@ static void __init rcu_dump_rcu_node_tree(struct rcu_state *rsp)pr_cont ("\n" ); } +struct workqueue_struct *rcu_gp_wq; +void __init rcu_init (void ) {int cpu;
@@ -4193,6 +4195,10 @@ void __init rcu_init(void)rcu_cpu_starting (cpu);rcutree_online_cpu (cpu); } + +/* Create workqueue for expedited GPs and for Tree SRCU. */ + rcu_gp_wq =alloc_workqueue ("rcu_gp" ,WQ_MEM_RECLAIM ,0 ); +WARN_ON (!rcu_gp_wq); }#include "tree_exp.h"
kernel/rcu/tree_exp.h +1 -1
@@ -626,7 +626,7 @@ static void _synchronize_rcu_expedited(struct rcu_state *rsp, rew.rew_rsp = rsp; rew.rew_s = s;INIT_WORK_ONSTACK (&rew.rew_work, wait_rcu_exp_gp); -schedule_work (&rew.rew_work); +queue_work (rcu_gp_wq, &rew.rew_work); }/* Wait for expedited grace period to complete. */
Signature · 1 line
2.5.2
PATCH 10/20 rcu: Remove obsolete force-quiescent-state statistics for debugfs
In reply to Paul E. McKenneyThe debugfs interface displayed statistics on RCU-pending checks but this interface has since been removed. This commit therefore removes the no-longer-used rcu_state structure's ->n_force_qs_lh and ->n_force_qs_ngp fields along with their updates. (Though the ->n_force_qs_ngp field was actually not used at all, embarrassingly enough.)
If this information proves necessary in the future, the corresponding event traces will be added.
--- kernel/rcu/tree.c | 5 +---- kernel/rcu/tree.h | 4 ---- 2 files changed, 1 insertion(+), 8 deletions(-)
kernel/rcu/tree.c +1 -4
@@ -2844,10 +2844,8 @@ static void force_quiescent_state(struct rcu_state *rsp) !raw_spin_trylock (&rnp->fqslock);if (rnp_old !=NULL )raw_spin_unlock (&rnp_old->fqslock); -if (ret) { - rsp->n_force_qs_lh++; +if (ret)return ; - } rnp_old = rnp; }/* rnp_old == rcu_get_root(rsp), rnp == NULL. */
@@ -2856,7 +2854,6 @@ static void force_quiescent_state(struct rcu_state *rsp)raw_spin_lock_irqsave_rcu_node (rnp_old, flags);raw_spin_unlock (&rnp_old->fqslock);if (READ_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) { - rsp->n_force_qs_lh++;raw_spin_unlock_irqrestore_rcu_node (rnp_old, flags);return ;/* Someone beat us to it. */ }
kernel/rcu/tree.h +0 -4
@@ -355,10 +355,6 @@ struct rcu_state {/* kthreads, if configured. */ unsigned long n_force_qs;/* Number of calls to */ /* force_quiescent_state(). */ -unsigned long n_force_qs_lh;/* ~Number of calls leaving */ -/* due to lock unavailable. */ -unsigned long n_force_qs_ngp;/* Number of calls leaving */ -/* due to no GP active. */ unsigned long gp_start;/* Time at which GP started, */ /* but in jiffies. */ unsigned long gp_activity;/* Time of last GP kthread */
Signature · 1 line
2.5.2
PATCH 16/20 rcu: Fix misprint in srcu_funnel_exp_start
In reply to Paul E. McKenneyFrom: Ildar Ismagilov <[email protected]>
The srcu_funnel_exp_start() function checks to see if the srcu_struct structure's expedited grace period counter needs updating to reflect a newly arrived request for an expedited SRCU grace period. Unfortunately, the check is backwards, so this commit reverses the sense of the test.
--- kernel/rcu/srcutree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/srcutree.c +1 -1
@@ -626,7 +626,7 @@ static void srcu_funnel_exp_start(struct srcu_struct *sp, struct srcu_node *snp,spin_unlock_irqrestore_rcu_node (snp, flags); }spin_lock_irqsave_rcu_node (sp, flags); -if (!ULONG_CMP_LT (sp->srcu_gp_seq_needed_exp, s)) +if (ULONG_CMP_LT (sp->srcu_gp_seq_needed_exp, s)) sp->srcu_gp_seq_needed_exp = s;spin_unlock_irqrestore_rcu_node (sp, flags); }
Signature · 1 line
2.5.2
PATCH 9/20 rcu: Remove obsolete __rcu_pending() statistics for debugfs
In reply to Paul E. McKenneyThe debugfs interface displayed statistics on RCU-pending checks but this interface has since been removed. This commit therefore removes the no-longer-used rcu_data structure's ->n_rcu_pending, ->n_rp_core_needs_qs, ->n_rp_report_qs, ->n_rp_cb_ready, ->n_rp_cpu_needs_gp, ->n_rp_gp_completed, ->n_rp_gp_started, ->n_rp_nocb_defer_wakeup, and ->n_rp_need_nothing fields along with their updates.
If this information proves necessary in the future, the corresponding event traces will be added.
--- kernel/rcu/tree.c | 31 ++++++------------------------- kernel/rcu/tree.h | 17 +++-------------- 2 files changed, 9 insertions(+), 39 deletions(-)
kernel/rcu/tree.c +6 -25
@@ -3354,8 +3354,6 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp) {struct rcu_node *rnp = rdp->mynode; - rdp->n_rcu_pending++; -/* Check for CPU stalls, if enabled. */ check_cpu_stall (rsp, rdp);
@@ -3364,48 +3362,31 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)return 0 ;/* Is the RCU core waiting for a quiescent state from this CPU? */ -if (rcu_scheduler_fully_active && - rdp->core_needs_qs && rdp->cpu_no_qs.b.norm && - rdp->rcu_qs_ctr_snap ==__this_cpu_read (rcu_dynticks.rcu_qs_ctr)) { - rdp->n_rp_core_needs_qs++; - }else if (rdp->core_needs_qs && !rdp->cpu_no_qs.b.norm) { - rdp->n_rp_report_qs++; +if (rdp->core_needs_qs && !rdp->cpu_no_qs.b.norm)return 1 ; - }/* Does this CPU have callbacks ready to invoke? */ -if (rcu_segcblist_ready_cbs (&rdp->cblist)) { - rdp->n_rp_cb_ready++; +if (rcu_segcblist_ready_cbs (&rdp->cblist))return 1 ; - }/* Has RCU gone idle with this CPU needing another grace period? */ -if (cpu_needs_another_gp (rsp, rdp)) { - rdp->n_rp_cpu_needs_gp++; +if (cpu_needs_another_gp (rsp, rdp))return 1 ; - }/* Has another RCU grace period completed? */ -if (READ_ONCE (rnp->completed) != rdp->completed) {/* outside lock */ - rdp->n_rp_gp_completed++; +if (READ_ONCE (rnp->completed) != rdp->completed)/* outside lock */ return 1 ; - }/* Has a new RCU grace period started? */ if (READ_ONCE (rnp->gpnum) != rdp->gpnum || -unlikely (READ_ONCE (rdp->gpwrap))) {/* outside lock */ - rdp->n_rp_gp_started++; +unlikely (READ_ONCE (rdp->gpwrap)))/* outside lock */ return 1 ; - }/* Does this CPU need a deferred NOCB wakeup? */ -if (rcu_nocb_need_deferred_wakeup (rdp)) { - rdp->n_rp_nocb_defer_wakeup++; +if (rcu_nocb_need_deferred_wakeup (rdp))return 1 ; - }/* nothing to do */ - rdp->n_rp_need_nothing++;return 0 ; }
kernel/rcu/tree.h +3 -14
@@ -226,18 +226,7 @@ struct rcu_data {/* Grace period that needs help */ /* from cond_resched(). */ -/* 5) __rcu_pending() statistics. */ -unsigned long n_rcu_pending;/* rcu_pending() calls since boot. */ -unsigned long n_rp_core_needs_qs; -unsigned long n_rp_report_qs; -unsigned long n_rp_cb_ready; -unsigned long n_rp_cpu_needs_gp; -unsigned long n_rp_gp_completed; -unsigned long n_rp_gp_started; -unsigned long n_rp_nocb_defer_wakeup; -unsigned long n_rp_need_nothing; - -/* 6) _rcu_barrier(), OOM callbacks, and expediting. */ +/* 5) _rcu_barrier(), OOM callbacks, and expediting. */ struct rcu_head barrier_head;#ifdef CONFIG_RCU_FAST_NO_HZ struct rcu_head oom_head;
@@ -248,7 +237,7 @@ struct rcu_data {atomic_long_t exp_workdone3;/* # done by others #3. */ int exp_dynticks_snap;/* Double-check need for IPI. */ -/* 7) Callback offloading. */ +/* 6) Callback offloading. */ #ifdef CONFIG_RCU_NOCB_CPU struct rcu_head *nocb_head;/* CBs waiting for kthread. */ struct rcu_head **nocb_tail;
@@ -275,7 +264,7 @@ struct rcu_data {/* Leader CPU takes GP-end wakeups. */ #endif /* #ifdef CONFIG_RCU_NOCB_CPU */ -/* 8) RCU CPU stall data. */ +/* 7) RCU CPU stall data. */ unsigned int softirq_snap;/* Snapshot of softirq activity. */ /* ->rcu_iw* fields protected by leaf rcu_node ->lock. */ struct irq_work rcu_iw;/* Check for non-irq activity. */
Signature · 1 line
2.5.2
PATCH 7/20 rcu: Remove obsolete boost statistics for debugfs
In reply to Paul E. McKenneyThe debugfs interface displayed statistics on RCU priority boosting, but this interface has since been removed. This commit therefore removes the no-longer-used rcu_data structure's ->n_tasks_boosted, ->n_exp_boosts, and ->n_exp_boosts and their updates.
If this information proves necessary in the future, the corresponding event traces will be added.
--- kernel/rcu/tree.h | 6 ------ kernel/rcu/tree_plugin.h | 8 ++------ 2 files changed, 2 insertions(+), 12 deletions(-)
kernel/rcu/tree.h +0 -6
@@ -146,12 +146,6 @@ struct rcu_node {/* boosting for this rcu_node structure. */ unsigned int boost_kthread_status;/* State of boost_kthread_task for tracing. */ -unsigned long n_tasks_boosted; -/* Total number of tasks boosted. */ -unsigned long n_exp_boosts; -/* Number of tasks boosted for expedited GP. */ -unsigned long n_normal_boosts; -/* Number of tasks boosted for normal GP. */ #ifdef CONFIG_RCU_NOCB_CPU struct swait_queue_head nocb_gp_wq[2 ];/* Place for rcu_nocb_kthread() to wait GP. */
kernel/rcu/tree_plugin.h +2 -6
@@ -963,14 +963,10 @@ static int rcu_boost(struct rcu_node *rnp)* expedited grace period must boost all blocked tasks, including * those blocking the pre-existing normal grace period. */ -if (rnp->exp_tasks !=NULL ) { +if (rnp->exp_tasks !=NULL ) tb = rnp->exp_tasks; - rnp->n_exp_boosts++; - }else { +else tb = rnp->boost_tasks; - rnp->n_normal_boosts++; - } - rnp->n_tasks_boosted++;/* * We boost task t by manufacturing an rt_mutex that appears to
Signature · 1 line
2.5.2
PATCH 2/20 sched/isolation: Eliminate NO_HZ_FULL_ALL
In reply to Paul E. McKenneyCommit 6f1982fedd59 ("sched/isolation: Handle the nohz_full= parameter") broke CONFIG_NO_HZ_FULL_ALL=y kernels. This breakage is due to the code under CONFIG_NO_HZ_FULL_ALL failing to invoke the shiny new housekeeping functions. This means that rcutorture scenario TREE04 now emits RCU CPU stall warnings due to the RCU grace-period kthreads not being awakened at a time of their choosing, or perhaps even not at all:
[ 27.731422] rcu_bh kthread starved for 21001 jiffies! g18446744073709551369 c18446744073709551368 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=3 [ 27.731423] rcu_bh I14936 9 2 0x80080000 [ 27.731435] Call Trace: [ 27.731440] __schedule+0x31a/0x6d0 [ 27.731442] schedule+0x31/0x80 [ 27.731446] schedule_timeout+0x15a/0x320 [ 27.731453] ? call_timer_fn+0x130/0x130 [ 27.731457] rcu_gp_kthread+0x66c/0xea0 [ 27.731458] ? rcu_gp_kthread+0x66c/0xea0
Because no one has complained about CONFIG_NO_HZ_FULL_ALL=y being broken, I hypothesize that no one is in fact using it, other than rcutorture. This commit therefore eliminates CONFIG_NO_HZ_FULL_ALL and updates rcutorture's config files to instead use the nohz_full= kernel parameter to put the desired CPUs into nohz_full mode.
--- Documentation/timers/NO_HZ.txt | 7 ------- kernel/time/Kconfig | 10 ---------- kernel/time/tick-sched.c | 22 ++-------------------- .../selftests/rcutorture/configs/rcu/TASKS03 | 1 - .../selftests/rcutorture/configs/rcu/TASKS03.boot | 2 +- .../selftests/rcutorture/configs/rcu/TREE04 | 1 - .../selftests/rcutorture/configs/rcu/TREE04.boot | 2 +- .../selftests/rcutorture/configs/rcu/TREE07 | 1 - 8 files changed, 4 insertions(+), 42 deletions(-)
Documentation/timers/NO_HZ.txt +0 -7
@@ -131,13 +131,6 @@ error message, and the boot CPU will be removed from the mask. Note that this means that your system must have at least two CPUs in order for CONFIG_NO_HZ_FULL=y to do anything for you. -Alternatively, the CONFIG_NO_HZ_FULL_ALL=y Kconfig parameter specifies -that all CPUs other than the boot CPU are adaptive-ticks CPUs. This -Kconfig parameter will be overridden by the "nohz_full=" boot parameter, -so that if both the CONFIG_NO_HZ_FULL_ALL=y Kconfig parameter and -the "nohz_full=1" boot parameter is specified, the boot parameter will -prevail so that only CPU 1 will be an adaptive-ticks CPU. - Finally, adaptive-ticks CPUs must have their RCU callbacks offloaded. This is covered in the "RCU IMPLICATIONS" section below.
kernel/time/Kconfig +0 -10
@@ -113,16 +113,6 @@ config NO_HZ_FULL endchoice -config NO_HZ_FULL_ALL - bool "Full dynticks system on all CPUs by default (except CPU 0)" - depends on NO_HZ_FULL - help - If the user doesn't pass the nohz_full boot option to - define the range of full dynticks CPUs, consider that all - CPUs in the system are full dynticks by default. - Note the boot CPU will still be kept outside the range to - handle the timekeeping duty. - config NO_HZ bool "Old Idle dynticks config" depends on !ARCH_USES_GETTIMEOFFSET && GENERIC_CLOCKEVENTS
kernel/time/tick-sched.c +2 -20
@@ -405,30 +405,12 @@ static int tick_nohz_cpu_down(unsigned int cpu)return 0 ; } -static int tick_nohz_init_all (void ) -{ -int err =-1 ; - -#ifdef CONFIG_NO_HZ_FULL_ALL -if (!alloc_cpumask_var (&tick_nohz_full_mask,GFP_KERNEL )) { -WARN (1 ,"NO_HZ: Can't allocate full dynticks cpumask\n" ); -return err; - } - err =0 ; -cpumask_setall (tick_nohz_full_mask); - tick_nohz_full_running = true; -#endif -return err; -} -void __init tick_nohz_init (void ) {int cpu, ret; -if (!tick_nohz_full_running) { -if (tick_nohz_init_all () <0 ) -return ; - } +if (!tick_nohz_full_running) +return ;/* * Full dynticks uses irq work to drive the tick rescheduling on safe
tools/testing/selftests/rcutorture/configs/rcu/TASKS03 +0 -1
@@ -9,5 +9,4 @@ CONFIG_PREEMPT=y CONFIG_HZ_PERIODIC=n CONFIG_NO_HZ_IDLE=n CONFIG_NO_HZ_FULL=y -CONFIG_NO_HZ_FULL_ALL=y #CHECK#CONFIG_RCU_EXPERT=n
tools/testing/selftests/rcutorture/configs/rcu/TASKS03.boot +1 -1
@@ -1 +1 @@ -rcutorture.torture_type=tasks +rcutorture.torture_type=tasks nohz_full=1
tools/testing/selftests/rcutorture/configs/rcu/TREE04 +0 -1
@@ -7,7 +7,6 @@ CONFIG_PREEMPT=n CONFIG_HZ_PERIODIC=n CONFIG_NO_HZ_IDLE=n CONFIG_NO_HZ_FULL=y -CONFIG_NO_HZ_FULL_ALL=y CONFIG_RCU_FAST_NO_HZ=y CONFIG_RCU_TRACE=y CONFIG_HOTPLUG_CPU=n
tools/testing/selftests/rcutorture/configs/rcu/TREE04.boot +1 -1
@@ -1 +1 @@ -rcutorture.torture_type=rcu_bh rcutree.rcu_fanout_leaf=4 +rcutorture.torture_type=rcu_bh rcutree.rcu_fanout_leaf=4 nohz_full=1-7
tools/testing/selftests/rcutorture/configs/rcu/TREE07 +0 -1
@@ -7,7 +7,6 @@ CONFIG_PREEMPT=n CONFIG_HZ_PERIODIC=n CONFIG_NO_HZ_IDLE=n CONFIG_NO_HZ_FULL=y -CONFIG_NO_HZ_FULL_ALL=n CONFIG_RCU_FAST_NO_HZ=n CONFIG_RCU_TRACE=y CONFIG_HOTPLUG_CPU=y
Signature · 1 line
2.5.2
PATCH 18/20 rcu: Trace expedited GP delays due to transitioning CPUs
In reply to Paul E. McKenneyIf a CPU is transitioning to or from offline state, an expedited grace period may undergo a timed wait. This timed wait can unduly delay grace periods, so this commit adds a trace statement to make it visible.
--- include/trace/events/rcu.h | 1 + kernel/rcu/tree_exp.h | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)
include/trace/events/rcu.h +1 -0
@@ -181,6 +181,7 @@ TRACE_EVENT(rcu_grace_period_init,* "start": Started a real expedited grace period. * "reset": Started resetting the tree * "select": Started selecting the CPUs to wait on. +* "selectofl": Selected CPU partially offline. * "startwait": Started waiting on selected CPUs. * "end": Ended a real expedited grace period. * "endwake": Woke piggybackers up.
kernel/rcu/tree_exp.h +2 -1
@@ -32,7 +32,7 @@ static void rcu_exp_gp_seq_start(struct rcu_state *rsp)* Return then value that expedited-grace-period counter will have * at the end of the current grace period. */ -static unsigned long rcu_exp_gp_seq_endval (struct rcu_state *rsp) +static __maybe_unused unsigned long rcu_exp_gp_seq_endval (struct rcu_state *rsp) {return rcu_seq_endval (&rsp->expedited_sequence); }
@@ -428,6 +428,7 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp, (rnp->expmask & mask)) {/* Online, so delay for a bit and try again. */ raw_spin_unlock_irqrestore_rcu_node (rnp, flags); +trace_rcu_exp_grace_period (rsp->name,rcu_exp_gp_seq_endval (rsp),TPS ("selectofl" ));schedule_timeout_uninterruptible (1 ); goto retry_ipi; }
Signature · 1 line
2.5.2
PATCH 19/20 rcu: Make expedited RCU CPU selection avoid unnecessary stores
In reply to Paul E. McKenneyThis commit reworks the first loop in sync_rcu_exp_select_cpus() to avoid doing unnecssary stores to other CPUs' rcu_data structures. This speeds up that first loop by roughly a factor of two on an old x86 system. In the case where the system is mostly idle, this loop incurs a large fraction of the overhead of the synchronize_rcu_expedited(). There is less benefit on busy systems because the overhead of the smp_call_function_single() in the second loop dominates in that case.
However, it is not unusual to do configuration chances involving RCU grace periods (both expedited and normal) while the system is mostly idle, so this optimization is worth doing.
While we are in the area, this commit also adds parentheses to arguments used by the for_each_leaf_node_possible_cpu() macro.
--- kernel/rcu/rcu.h | 16 +++++++++++++--- kernel/rcu/tree_exp.h | 21 ++++++++++++++------- 2 files changed, 27 insertions(+), 10 deletions(-)
kernel/rcu/rcu.h +13 -3
@@ -301,9 +301,19 @@ static inline void rcu_init_levelspread(int *levelspread, const int *levelcnt)* Iterate over all possible CPUs in a leaf RCU node. */ #define for_each_leaf_node_possible_cpu (rnp, cpu) \ - for ((cpu) = cpumask_next(rnp->grplo - 1, cpu_possible_mask); \ - cpu <= rnp->grphi; \ - cpu = cpumask_next((cpu), cpu_possible_mask)) + for ((cpu) = cpumask_next((rnp)->grplo - 1, cpu_possible_mask); \ + (cpu) <= rnp->grphi; \ + (cpu) = cpumask_next((cpu), cpu_possible_mask)) + +/* +* Iterate over all CPUs in a leaf RCU node's specified mask. +*/ +#define rcu_find_next_bit (rnp, cpu, mask) \ + ((rnp)->grplo + find_next_bit(&(mask), BITS_PER_LONG, (cpu))) +#define for_each_leaf_node_cpu_mask (rnp, cpu, mask) \ + for ((cpu) = rcu_find_next_bit((rnp), 0, (mask)); \ + (cpu) <= rnp->grphi; \ + (cpu) = rcu_find_next_bit((rnp), (cpu) + 1 - (rnp->grplo), (mask)))/* * Wrappers for the rcu_node::lock acquire and release.
kernel/rcu/tree_exp.h +14 -7
@@ -383,15 +383,22 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp,/* Each pass checks a CPU for identity, offline, and idle. */ mask_ofl_test =0 ; -for_each_leaf_node_possible_cpu (rnp, cpu) { +for_each_leaf_node_cpu_mask (rnp, cpu, rnp->expmask) { +unsigned long mask =leaf_node_cpu_bit (rnp, cpu);struct rcu_data *rdp =per_cpu_ptr (rsp->rda, cpu); +struct rcu_dynticks *rdtp =per_cpu_ptr (&rcu_dynticks, cpu); +int snap; - rdp->exp_dynticks_snap = -rcu_dynticks_snap (rdp->dynticks);if (raw_smp_processor_id () == cpu || -rcu_dynticks_in_eqs (rdp->exp_dynticks_snap) || - !(rnp->qsmaskinitnext & rdp->grpmask)) - mask_ofl_test |= rdp->grpmask; + !(rnp->qsmaskinitnext & mask)) { + mask_ofl_test |= mask; + }else { + snap =rcu_dynticks_snap (rdtp); +if (rcu_dynticks_in_eqs (snap)) + mask_ofl_test |= mask; +else + rdp->exp_dynticks_snap = snap; + } } mask_ofl_ipi = rnp->expmask & ~mask_ofl_test;
@@ -405,7 +412,7 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp,raw_spin_unlock_irqrestore_rcu_node (rnp, flags);/* IPI the remaining CPUs for expedited quiescent state. */ -for_each_leaf_node_possible_cpu (rnp, cpu) { +for_each_leaf_node_cpu_mask (rnp, cpu, rnp->expmask) {unsigned long mask =leaf_node_cpu_bit (rnp, cpu);struct rcu_data *rdp =per_cpu_ptr (rsp->rda, cpu);
Signature · 1 line
2.5.2
PATCH 14/20 rcu: Use wrapper for lockdep asserts
In reply to Paul E. McKenneyFrom: Matthew Wilcox <[email protected]>
Commits c0b334c5bfa9 and ea9b0c8a26a2 introduced new sparse warnings by accessing rcu_node->lock directly and ignoring the __private marker. Introduce a new wrapper and use it. Also fix a similar problem in srcutree.c introduced by a3883df3935e.
--- kernel/rcu/rcu.h | 5 ++++- kernel/rcu/srcutree.c | 2 +- kernel/rcu/tree.c | 24 ++++++++++++------------ kernel/rcu/tree_plugin.h | 4 ++-- 4 files changed, 19 insertions(+), 16 deletions(-)
kernel/rcu/rcu.h +4 -1
@@ -337,7 +337,7 @@ do { \ } while (0)#define raw_spin_unlock_irqrestore_rcu_node (p, flags) \ - raw_spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \ + raw_spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags)#define raw_spin_trylock_rcu_node (p) \ ({ \
@@ -348,6 +348,9 @@ do { \ ___locked; \ }) +#define raw_lockdep_assert_held_rcu_node (p) \ + lockdep_assert_held(&ACCESS_PRIVATE(p, lock)) +#endif /* #if defined(SRCU) || !defined(TINY_RCU) */ #ifdef CONFIG_TINY_RCU
kernel/rcu/srcutree.c +1 -1
@@ -439,7 +439,7 @@ static void srcu_gp_start(struct srcu_struct *sp)struct srcu_data *sdp =this_cpu_ptr (sp->sda);int state; -lockdep_assert_held (&sp->lock); +lockdep_assert_held (&ACCESS_PRIVATE (sp, lock));WARN_ON_ONCE (ULONG_CMP_GE (sp->srcu_gp_seq, sp->srcu_gp_seq_needed));rcu_segcblist_advance (&sdp->srcu_cblist,rcu_seq_current (&sp->srcu_gp_seq));
kernel/rcu/tree.c +12 -12
@@ -1161,7 +1161,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)*/ static void rcu_gpnum_ovf (struct rcu_node *rnp,struct rcu_data *rdp) { -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);if (ULONG_CMP_LT (READ_ONCE (rdp->gpnum) +ULONG_MAX /4 , rnp->gpnum))WRITE_ONCE (rdp->gpwrap, true);if (ULONG_CMP_LT (rdp->rcu_iw_gpnum +ULONG_MAX /4 , rnp->gpnum))
@@ -1629,7 +1629,7 @@ void rcu_cpu_stall_reset(void)static unsigned long rcu_cbs_completed (struct rcu_state *rsp,struct rcu_node *rnp) { -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);/* * If RCU is idle, we just wait for the next grace period.
@@ -1676,7 +1676,7 @@ rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp,bool ret = false;struct rcu_node *rnp_root =rcu_get_root (rdp->rsp); -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);/* * Pick up grace-period number for new callbacks. If this
@@ -1804,7 +1804,7 @@ static bool rcu_accelerate_cbs(struct rcu_state *rsp, struct rcu_node *rnp, {bool ret = false; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);/* If no pending (not yet ready to invoke) callbacks, nothing to do. */ if (!rcu_segcblist_pend_cbs (&rdp->cblist))
@@ -1844,7 +1844,7 @@ static bool rcu_accelerate_cbs(struct rcu_state *rsp, struct rcu_node *rnp,static bool rcu_advance_cbs (struct rcu_state *rsp,struct rcu_node *rnp,struct rcu_data *rdp) { -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);/* If no pending (not yet ready to invoke) callbacks, nothing to do. */ if (!rcu_segcblist_pend_cbs (&rdp->cblist))
@@ -1872,7 +1872,7 @@ static bool __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp,bool ret;bool need_gp; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);/* Handle the ends of any preceding grace periods first. */ if (rdp->completed == rnp->completed &&
@@ -2297,7 +2297,7 @@ static bool rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp) { - lockdep_assert_held(&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);if (!rsp->gp_kthread || !cpu_needs_another_gp (rsp, rdp)) {/* * Either we have not yet spawned the grace-period
@@ -2359,7 +2359,7 @@ static bool rcu_start_gp(struct rcu_state *rsp)static void rcu_report_qs_rsp (struct rcu_state *rsp,unsigned long flags) __releases(rcu_get_root(rsp)->lock) { -lockdep_assert_held (&rcu_get_root (rsp)->lock); +raw_lockdep_assert_held_rcu_node (rcu_get_root (rsp));WARN_ON_ONCE (!rcu_gp_in_progress (rsp));WRITE_ONCE (rsp->gp_flags,READ_ONCE (rsp->gp_flags) |RCU_GP_FLAG_FQS );raw_spin_unlock_irqrestore_rcu_node (rcu_get_root (rsp), flags);
@@ -2384,7 +2384,7 @@ rcu_report_qs_rnp(unsigned long mask, struct rcu_state *rsp,unsigned long oldmask =0 ;struct rcu_node *rnp_c; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);/* Walk up the rcu_node hierarchy. */ for (;;) {
@@ -2448,7 +2448,7 @@ static void rcu_report_unblock_qs_rnp(struct rcu_state *rsp,unsigned long mask;struct rcu_node *rnp_p; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);if (rcu_state_p == &rcu_sched_state || rsp != rcu_state_p || rnp->qsmask !=0 ||rcu_preempt_blocked_readers_cgp (rnp)) {raw_spin_unlock_irqrestore_rcu_node (rnp, flags);
@@ -2593,7 +2593,7 @@ static void rcu_cleanup_dead_rnp(struct rcu_node *rnp_leaf)long mask;struct rcu_node *rnp = rnp_leaf; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);if (!IS_ENABLED (CONFIG_HOTPLUG_CPU ) || rnp->qsmaskinit ||rcu_preempt_has_tasks (rnp))return ;
@@ -3596,7 +3596,7 @@ static void rcu_init_new_rnp(struct rcu_node *rnp_leaf)long mask;struct rcu_node *rnp = rnp_leaf; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);for (;;) { mask = rnp->grpmask; rnp = rnp->parent;
kernel/rcu/tree_plugin.h +2 -2
@@ -180,7 +180,7 @@ static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp) (rnp->expmask & rdp->grpmask ?RCU_EXP_BLKD :0 );struct task_struct *t = current; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);WARN_ON_ONCE (rdp->mynode != rnp);WARN_ON_ONCE (rnp->level != rcu_num_lvls -1 );
@@ -1044,7 +1044,7 @@ static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags) {struct task_struct *t; -lockdep_assert_held (&rnp->lock); +raw_lockdep_assert_held_rcu_node (rnp);if (!rcu_preempt_blocked_readers_cgp (rnp) && rnp->exp_tasks ==NULL ) {raw_spin_unlock_irqrestore_rcu_node (rnp, flags);return ;
Signature · 1 line
2.5.2
PATCH 12/20 rcu: Consolidate rcu.h #ifdefs
In reply to Paul E. McKenneyThe kernel/rcu/rcu.h file has a pair of consecutive #ifdefs on CONFIG_TINY_RCU, so this commit consolidates them, thus saving a few lines of code.
--- kernel/rcu/rcu.h | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
kernel/rcu/rcu.h +2 -6
@@ -356,24 +356,20 @@ static inline bool rcu_gp_is_normal(void) { return true; }static inline bool rcu_gp_is_expedited (void ) {return false; }static inline void rcu_expedite_gp (void ) { }static inline void rcu_unexpedite_gp (void ) { } +static inline void rcu_request_urgent_qs_task (struct task_struct *t) { }#else /* #ifdef CONFIG_TINY_RCU */ bool rcu_gp_is_normal (void );/* Internal RCU use. */ bool rcu_gp_is_expedited (void );/* Internal RCU use. */ void rcu_expedite_gp (void );void rcu_unexpedite_gp (void );void rcupdate_announce_bootup_oddness (void ); +void rcu_request_urgent_qs_task (struct task_struct *t);#endif /* #else #ifdef CONFIG_TINY_RCU */ #define RCU_SCHEDULER_INACTIVE 0#define RCU_SCHEDULER_INIT 1#define RCU_SCHEDULER_RUNNING 2 -#ifdef CONFIG_TINY_RCU -static inline void rcu_request_urgent_qs_task (struct task_struct *t) { } -#else /* #ifdef CONFIG_TINY_RCU */ -void rcu_request_urgent_qs_task (struct task_struct *t); -#endif /* #else #ifdef CONFIG_TINY_RCU */ -enum rcutorture_type {RCU_FLAVOR ,RCU_BH_FLAVOR ,
Signature · 1 line
2.5.2
PATCH 15/20 rcu: Fix init_rcu_head() comment.
In reply to Paul E. McKenneyThe current (and implicit) comment header for init_rcu_head() and destroy_rcu_head() incorrectly says that they are not needed for statically allocated rcu_head structures. This commit therefore fixes this comment.
--- include/linux/rcupdate.h | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
include/linux/rcupdate.h +6 -4
@@ -214,10 +214,12 @@ do { \#endif /* -* init_rcu_head_on_stack()/destroy_rcu_head_on_stack() are needed for dynamic -* initialization and destruction of rcu_head on the stack. rcu_head structures -* allocated dynamically in the heap or defined statically don't need any -* initialization. +* The init_rcu_head_on_stack() and destroy_rcu_head_on_stack() calls +* are needed for dynamic initialization and destruction of rcu_head +* on the stack, and init_rcu_head()/destroy_rcu_head() are needed for +* dynamic initialization and destruction of statically allocated rcu_head +* structures. However, rcu_head structures allocated dynamically in the +* heap don't need any initialization. */ #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD void init_rcu_head (struct rcu_head *head);
Signature · 1 line
2.5.2
PATCH 17/20 rcu: Add more tracing of expedited grace periods
In reply to Paul E. McKenneyThis commit adds more tracing of expedited grace periods to enable improved debugging of slowdowns.
--- include/trace/events/rcu.h | 3 +++ kernel/rcu/rcu.h | 8 +++++++- kernel/rcu/tree_exp.h | 12 ++++++++++++ 3 files changed, 22 insertions(+), 1 deletion(-)
include/trace/events/rcu.h +3 -0
@@ -179,6 +179,9 @@ TRACE_EVENT(rcu_grace_period_init,* * "snap": Captured snapshot of expedited grace period sequence number. * "start": Started a real expedited grace period. +* "reset": Started resetting the tree +* "select": Started selecting the CPUs to wait on. +* "startwait": Started waiting on selected CPUs. * "end": Ended a real expedited grace period. * "endwake": Woke piggybackers up. * "done": Someone else did the expedited grace period for us.
kernel/rcu/rcu.h +7 -1
@@ -77,12 +77,18 @@ static inline void rcu_seq_start(unsigned long *sp)WARN_ON_ONCE (rcu_seq_state (*sp) !=1 ); } +/* Compute the end-of-grace-period value for the specified sequence number. */ +static inline unsigned long rcu_seq_endval (unsigned long *sp) +{ +return (*sp |RCU_SEQ_STATE_MASK ) +1 ; +} +/* Adjust sequence number for end of update-side operation. */ static inline void rcu_seq_end (unsigned long *sp) {smp_mb ();/* Ensure update-side operation before counter increment. */ WARN_ON_ONCE (!rcu_seq_state (*sp)); -WRITE_ONCE (*sp, (*sp |RCU_SEQ_STATE_MASK ) +1 ); +WRITE_ONCE (*sp,rcu_seq_endval (sp)); }/* Take a snapshot of the update side's sequence number. */
kernel/rcu/tree_exp.h +12 -0
@@ -29,6 +29,15 @@ static void rcu_exp_gp_seq_start(struct rcu_state *rsp) }/* +* Return then value that expedited-grace-period counter will have +* at the end of the current grace period. +*/ +static unsigned long rcu_exp_gp_seq_endval (struct rcu_state *rsp) +{ +return rcu_seq_endval (&rsp->expedited_sequence); +} + +/* * Record the end of an expedited grace period. */ static void rcu_exp_gp_seq_end (struct rcu_state *rsp)
@@ -366,7 +375,9 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp,int ret;struct rcu_node *rnp; +trace_rcu_exp_grace_period (rsp->name,rcu_exp_gp_seq_endval (rsp),TPS ("reset" ));sync_exp_reset_tree (rsp); +trace_rcu_exp_grace_period (rsp->name,rcu_exp_gp_seq_endval (rsp),TPS ("select" ));rcu_for_each_leaf_node (rsp, rnp) {raw_spin_lock_irqsave_rcu_node (rnp, flags);
@@ -443,6 +454,7 @@ static void synchronize_sched_expedited_wait(struct rcu_state *rsp)struct rcu_node *rnp_root =rcu_get_root (rsp);int ret; +trace_rcu_exp_grace_period (rsp->name,rcu_exp_gp_seq_endval (rsp),TPS ("startwait" )); jiffies_stall =rcu_jiffies_till_stall_check (); jiffies_start = jiffies;
Signature · 1 line
2.5.2
PATCH 3/20 rcu: Fix CPU offload boot message when no CPUs are offloaded
In reply to Paul E. McKenneyIn CONFIG_RCU_NOCB_CPU=y kernels, if the boot parameters indicate that none of the CPUs should in fact be offloaded, the following somewhat obtuse message appears:
Offload RCU callbacks from CPUs: .
This commit therefore makes the message at least grammatically correct in this case:
Offload RCU callbacks from CPUs: (none)
--- kernel/rcu/tree_plugin.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +5 -2
@@ -2312,8 +2312,11 @@ void __init rcu_init_nohz(void)cpumask_and (rcu_nocb_mask, cpu_possible_mask, rcu_nocb_mask); } -pr_info ("\tOffload RCU callbacks from CPUs: %*pbl.\n" , -cpumask_pr_args (rcu_nocb_mask)); +if (cpumask_empty (rcu_nocb_mask)) +pr_info ("\tOffload RCU callbacks from CPUs: (none).\n" ); +else +pr_info ("\tOffload RCU callbacks from CPUs: %*pbl.\n" , +cpumask_pr_args (rcu_nocb_mask));if (rcu_nocb_poll)pr_info ("\tPoll for callbacks from no-CBs CPUs.\n" );
Signature · 1 line
2.5.2
PATCH 6/20 rcu: Call touch_nmi_watchdog() while printing stall warnings
In reply to Paul E. McKenneyFrom: Tejun Heo <[email protected]>
When RCU stall warning triggers, it can print out a lot of messages while holding spinlocks. If the console device is slow (e.g. an actual or IPMI serial console), it may end up triggering NMI hard lockup watchdog like the following.
*** CPU printking while holding RCU spinlock
PID: 4149739 TASK: ffff881a46baa880 CPU: 13 COMMAND: "CPUThreadPool8"
#0 [ffff881fff945e48] crash_nmi_callback at ffffffff8103f7d0 #1 [ffff881fff945e58] nmi_handle at ffffffff81020653
#2 [ffff881fff945eb0] default_do_nmi at ffffffff81020c36
#3 [ffff881fff945ed0] do_nmi at ffffffff81020d32
#4 [ffff881fff945ef0] end_repeat_nmi at ffffffff81956a7e
[exception RIP: io_serial_in+21]
RIP: ffffffff81630e55 RSP: ffff881fff943b88 RFLAGS: 00000002
RAX: 000000000000ca00 RBX: ffffffff8230e188 RCX: 0000000000000000
RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff8230e188
RBP: ffff881fff943bb0 R8: 0000000000000000 R9: ffffffff820cb3c4
R10: 0000000000000019 R11: 0000000000002000 R12: 00000000000026e1
R13: 0000000000000020 R14: ffffffff820cd398 R15: 0000000000000035
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000 --- <NMI exception stack> ---
#5 [ffff881fff943b88] io_serial_in at ffffffff81630e55
#6 [ffff881fff943b90] wait_for_xmitr at ffffffff8163175c #7 [ffff881fff943bb8] serial8250_console_putchar at ffffffff816317dc #8 [ffff881fff943bd8] uart_console_write at ffffffff8162ac00 #9 [ffff881fff943c08] serial8250_console_write at ffffffff81634691
#10 [ffff881fff943c80] univ8250_console_write at ffffffff8162f7c2 #11 [ffff881fff943c90] console_unlock at ffffffff810dfc55 #12 [ffff881fff943cf0] vprintk_emit at ffffffff810dffb5 #13 [ffff881fff943d50] vprintk_default at ffffffff810e01bf #14 [ffff881fff943d60] vprintk_func at ffffffff810e1127 #15 [ffff881fff943d70] printk at ffffffff8119a8a4
#16 [ffff881fff943dd0] print_cpu_stall_info at ffffffff810eb78c #17 [ffff881fff943e88] rcu_check_callbacks at ffffffff810ef133 #18 [ffff881fff943ee8] update_process_times at ffffffff810f3497 #19 [ffff881fff943f10] tick_sched_timer at ffffffff81103037 #20 [ffff881fff943f38] __hrtimer_run_queues at ffffffff810f3f38 #21 [ffff881fff943f88] hrtimer_interrupt at ffffffff810f442b
*** CPU triggering the hardlockup watchdog
PID: 4149709 TASK: ffff88010f88c380 CPU: 26 COMMAND: "CPUThreadPool35"
#0 [ffff883fff1059d0] machine_kexec at ffffffff8104a874
#1 [ffff883fff105a30] __crash_kexec at ffffffff811116cc
#2 [ffff883fff105af0] __crash_kexec at ffffffff81111795
#3 [ffff883fff105b08] panic at ffffffff8119a6ae
#4 [ffff883fff105b98] watchdog_overflow_callback at ffffffff81135dbd #5 [ffff883fff105bb0] __perf_event_overflow at ffffffff81186866 #6 [ffff883fff105be8] perf_event_overflow at ffffffff81192bc4 #7 [ffff883fff105bf8] intel_pmu_handle_irq at ffffffff8100b265 #8 [ffff883fff105df8] perf_event_nmi_handler at ffffffff8100489f #9 [ffff883fff105e58] nmi_handle at ffffffff81020653
#10 [ffff883fff105eb0] default_do_nmi at ffffffff81020b94
#11 [ffff883fff105ed0] do_nmi at ffffffff81020d32
#12 [ffff883fff105ef0] end_repeat_nmi at ffffffff81956a7e
[exception RIP: queued_spin_lock_slowpath+248]
RIP: ffffffff810da958 RSP: ffff883fff103e68 RFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000046 RCX: 00000000006d0000
RDX: ffff883fff49a950 RSI: 0000000000d10101 RDI: ffffffff81e54300
RBP: ffff883fff103e80 R8: ffff883fff11a950 R9: 0000000000000000
R10: 000000000e5873ba R11: 000000000000010f R12: ffffffff81e54300
R13: 0000000000000000 R14: ffff88010f88c380 R15: ffffffff81e54300
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> ---
#13 [ffff883fff103e68] queued_spin_lock_slowpath at ffffffff810da958 #14 [ffff883fff103e70] _raw_spin_lock_irqsave at ffffffff8195550b #15 [ffff883fff103e88] rcu_check_callbacks at ffffffff810eed18 #16 [ffff883fff103ee8] update_process_times at ffffffff810f3497 #17 [ffff883fff103f10] tick_sched_timer at ffffffff81103037 #18 [ffff883fff103f38] __hrtimer_run_queues at ffffffff810f3f38 #19 [ffff883fff103f88] hrtimer_interrupt at ffffffff810f442b --- <IRQ stack> ---
Avoid spuriously triggering NMI hardlockup watchdog by touching it from the print functions. show_state_filter() shares the same problem and solution.
v2: Relocate the comment to where it belongs.
--- kernel/rcu/tree_plugin.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +13 -1
@@ -560,8 +560,14 @@ static void rcu_print_detail_task_stall_rnp(struct rcu_node *rnp) } t =list_entry (rnp->gp_tasks->prev, struct task_struct, rcu_node_entry); -list_for_each_entry_continue (t, &rnp->blkd_tasks, rcu_node_entry) +list_for_each_entry_continue (t, &rnp->blkd_tasks, rcu_node_entry) { +/* +* We could be printing a lot while holding a spinlock. +* Avoid triggering hard lockup. +*/ +touch_nmi_watchdog ();sched_show_task (t); + }raw_spin_unlock_irqrestore_rcu_node (rnp, flags); }
@@ -1677,6 +1683,12 @@ static void print_cpu_stall_info(struct rcu_state *rsp, int cpu)char *ticks_title;unsigned long ticks_value; +/* +* We could be printing a lot while holding a spinlock. Avoid +* triggering hard lockup. +*/ +touch_nmi_watchdog (); +if (rsp->gpnum == rdp->gpnum) { ticks_title ="ticks this GP" ; ticks_value = rdp->ticks_this_gp;
Signature · 1 line
2.5.2