Miscellaneous fixes for 3.16
cover Miscellaneous fixes for 3.16
Hello!
This series provides miscellaneous fixes:
1. Apply ACCESS_ONCE() to unprotected ->gp_flags accesses.
2. Fix typo in comment, courtesy of Liu Ping Fan.
3. Make RCU CPU stall warnings print grace-period numbers in
signed format to improve readability of stall-warning output.
4. Make cpu_needs_another_gp() take future grace-period needs
into account.
5. Remove unused ->preemptible field from the rcu_data structure,
courtesty of Iulia Manda.
6. Apply ACCESS_ONCE() to unprotected ->jiffies_stall accesses,
courtesty of Iulia Manda.
7. Make callers responsible for grace-period kthread wakeup in
order to avoid potential silent grace-period stalls.
8. Remove "extern" from RCU function declarations, courtesy of
Iulia Manda.
9. Apply ACCESS_ONCE() to additional ->jiffies_stall accesses,
courtesy of Himangi Saraogi.
10. Add event tracing to dyntick_save_progress_counter(), courtesy
of Andreea-Cristina Bernat.
11. Make rcu_init_one() use nr_cpu_ids instead of NR_CPUS for
data-structure setup limit check, courtesy of Himangi Saraogi.
12. Remove redundant kfree_call_rcu() definition by using the
rcu_state pointer, courtesy of Andreea-Cristina Bernat.
13. Merge rcu_sched_force_quiescent_state() definition with
rcu_force_quiescent_state() by using the rcu_state pointer, courtesy of Andreea-Cristina Bernat.
14. Document RCU_INIT_POINTER()'s lack of ordering guarantees.
15. Automatically bind RCU's grace-period kthreads to timekeeping
CPU for NO_HZ_FULL builds.
16. Make large and small sysidle systems use equivalent state machine.
17. Remove duplicate resched_cpu() declaration, courtesy of
Pranith Kumar.
18. Replace deprecated __this_cpu_ptr() uses with raw_cpu_ptr(),
courtesy of Christoph Lameter.
19. Make softirq processing provide a quiescent state only once
per full pass over all softirqs rather than once per action, courtesy of Eric Dumazet.
Thanx, Paul
------------------------------------------------------------------------
b/include/linux/rcupdate.h | 5 b/kernel/rcu/tiny_plugin.h | 8 - b/kernel/rcu/tree.c | 261 ++++++++++++++++++++++++++++----------------- b/kernel/rcu/tree.h | 3 b/kernel/rcu/tree_plugin.h | 120 +++++--------------- b/kernel/softirq.c | 4 6 files changed, 206 insertions(+), 195 deletions(-)
PATCH 1/19 rcu: Protect ->gp_flags accesses with ACCESS_ONCE()
From: "Paul E. McKenney" <[email protected]>
A number of ->gp_flags accesses don't have ACCESS_ONCE(), but all of the can race against other loads or stores. This commit therefore applies ACCESS_ONCE() to the unprotected ->gp_flags accesses.
--- kernel/rcu/tree.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +6 -6
@@ -1403,12 +1403,12 @@ static int rcu_gp_init(struct rcu_state *rsp)rcu_bind_gp_kthread ();raw_spin_lock_irq (&rnp->lock);smp_mb__after_unlock_lock (); -if (rsp->gp_flags ==0 ) { +if (!ACCESS_ONCE (rsp->gp_flags)) {/* Spurious wakeup, tell caller to go back to sleep. */ raw_spin_unlock_irq (&rnp->lock);return 0 ; } - rsp->gp_flags =0 ;/* Clear all flags: New grace period. */ +ACCESS_ONCE (rsp->gp_flags) =0 ;/* Clear all flags: New grace period. */ if (WARN_ON_ONCE (rcu_gp_in_progress (rsp))) {/*
@@ -1501,7 +1501,7 @@ static int rcu_gp_fqs(struct rcu_state *rsp, int fqs_state_in)if (ACCESS_ONCE (rsp->gp_flags) &RCU_GP_FLAG_FQS ) {raw_spin_lock_irq (&rnp->lock);smp_mb__after_unlock_lock (); - rsp->gp_flags &= ~RCU_GP_FLAG_FQS ; +ACCESS_ONCE (rsp->gp_flags) &= ~RCU_GP_FLAG_FQS ;raw_spin_unlock_irq (&rnp->lock); }return fqs_state;
@@ -1566,7 +1566,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp) rdp =this_cpu_ptr (rsp->rda);rcu_advance_cbs (rsp, rnp, rdp);/* Reduce false positives below. */ if (cpu_needs_another_gp (rsp, rdp)) { - rsp->gp_flags =RCU_GP_FLAG_INIT ; +ACCESS_ONCE (rsp->gp_flags) =RCU_GP_FLAG_INIT ;trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum),TPS ("newreq" ));
@@ -1695,7 +1695,7 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,*/ return ; } - rsp->gp_flags =RCU_GP_FLAG_INIT ; +ACCESS_ONCE (rsp->gp_flags) =RCU_GP_FLAG_INIT ;trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum),TPS ("newreq" ));
@@ -2320,7 +2320,7 @@ static void force_quiescent_state(struct rcu_state *rsp)raw_spin_unlock_irqrestore (&rnp_old->lock, flags);return ;/* Someone beat us to it. */ } - rsp->gp_flags |=RCU_GP_FLAG_FQS ; +ACCESS_ONCE (rsp->gp_flags) |=RCU_GP_FLAG_FQS ;raw_spin_unlock_irqrestore (&rnp_old->lock, flags);wake_up (&rsp->gp_wq);/* Memory barrier implied by wake_up() path. */ }
Signature · 1 line
1.8.1.5
PATCH 2/19 rcu: Fix incorrect notes for code
From: Liu Ping Fan <[email protected]>
--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -2109,7 +2109,7 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) }#ifndef CONFIG_RCU_NOCB_CPU_ALL -/* Is the specified CPU a no-CPUs CPU? */ +/* Is the specified CPU a no-CBs CPU? */ bool rcu_is_nocb_cpu (int cpu) {if (have_rcu_nocb_mask)
Signature · 1 line
1.8.1.5
PATCH 3/19 rcu: Print negatives for stall-warning counter wraparound
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The print_other_cpu_stall() and print_cpu_stall() functions print grace-period numbers using an unsigned format, which means that the number one less than zero is a very large number. This commit therefore causes these numbers to be printed with a signed format in order to improve readability of the RCU CPU stall-warning output.
--- kernel/rcu/tree.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +5 -4
@@ -932,9 +932,9 @@ static void print_other_cpu_stall(struct rcu_state *rsp)print_cpu_stall_info_end ();for_each_possible_cpu (cpu) totqlen +=per_cpu_ptr (rsp->rda, cpu)->qlen; -pr_cont ("(detected by %d, t=%ld jiffies, g=%lu, c=%lu, q=%lu)\n" , +pr_cont ("(detected by %d, t=%ld jiffies, g=%ld, c=%ld, q=%lu)\n" ,smp_processor_id (), (long )(jiffies - rsp->gp_start), - rsp->gpnum, rsp->completed, totqlen); + (long )rsp->gpnum, (long )rsp->completed, totqlen);if (ndetected ==0 )pr_err ("INFO: Stall ended before state dump start\n" );else if (!trigger_all_cpu_backtrace ())
@@ -971,8 +971,9 @@ static void print_cpu_stall(struct rcu_state *rsp)print_cpu_stall_info_end ();for_each_possible_cpu (cpu) totqlen +=per_cpu_ptr (rsp->rda, cpu)->qlen; -pr_cont (" (t=%lu jiffies g=%lu c=%lu q=%lu)\n" , - jiffies - rsp->gp_start, rsp->gpnum, rsp->completed, totqlen); +pr_cont (" (t=%lu jiffies g=%ld c=%ld q=%lu)\n" , + jiffies - rsp->gp_start, + (long )rsp->gpnum, (long )rsp->completed, totqlen);if (!trigger_all_cpu_backtrace ())dump_stack ();
Signature · 1 line
1.8.1.5
PATCH 4/19 rcu: Update cpu_needs_another_gp() for futures from non-NOCB CPUs
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
In the old days, the only source of requests for future grace periods was NOCB CPUs. This has changed: CPUs routinely post requests for future grace periods in order to promote power efficiency and reduce OS jitter with minimal impact on grace-period latency. This commit therefore updates cpu_needs_another_gp() to invoke rcu_future_needs_gp() instead of rcu_nocb_needs_gp(). The latter is no longer used, so is now removed. This commit also adds tracing for the irq_work_queue() wakeup case.
--- kernel/rcu/tree.c | 39 +++++++++++++++++++++++++++++---------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_plugin.h | 18 ------------------ 3 files changed, 29 insertions(+), 29 deletions(-)
kernel/rcu/tree.c +29 -10
@@ -324,6 +324,28 @@ cpu_has_callbacks_ready_to_invoke(struct rcu_data *rdp) }/* +* Return the root node of the specified rcu_state structure. +*/ +static struct rcu_node *rcu_get_root (struct rcu_state *rsp) +{ +return &rsp->node[0 ]; +} + +/* +* Is there any need for future grace periods? +* Interrupts must be disabled. If the caller does not hold the root +* rnp_node structure's ->lock, the results are advisory only. +*/ +static int rcu_future_needs_gp (struct rcu_state *rsp) +{ +struct rcu_node *rnp =rcu_get_root (rsp); +int idx = (ACCESS_ONCE (rnp->completed) +1 ) &0x1 ; +int *fp = &rnp->need_future_gp[idx]; + +return ACCESS_ONCE (*fp); +} + +/* * Does the current CPU require a not-yet-started grace period? * The caller must have disabled interrupts to prevent races with * normal callback registry.
@@ -335,7 +357,7 @@ cpu_needs_another_gp(struct rcu_state *rsp, struct rcu_data *rdp)if (rcu_gp_in_progress (rsp))return 0 ;/* No, a grace period is already in progress. */ -if (rcu_nocb_needs_gp (rsp)) +if (rcu_future_needs_gp (rsp))return 1 ;/* Yes, a no-CBs CPU needs one. */ if (!rdp->nxttail[RCU_NEXT_TAIL ])return 0 ;/* No, this is a no-CBs (or offline) CPU. */
@@ -350,14 +372,6 @@ cpu_needs_another_gp(struct rcu_state *rsp, struct rcu_data *rdp) }/* -* Return the root node of the specified rcu_state structure. -*/ -static struct rcu_node *rcu_get_root (struct rcu_state *rsp) -{ -return &rsp->node[0 ]; -} - -/* * rcu_eqs_enter_common - current CPU is moving towards extended quiescent state * * If the new value of the ->dynticks_nesting counter now is zero,
@@ -1672,6 +1686,8 @@ static void rsp_wakeup(struct irq_work *work)/* Wake up rcu_gp_kthread() to start the grace period. */ wake_up (&rsp->gp_wq); +trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum), +"Workqueuewoken" ); }/*
@@ -1706,8 +1722,11 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,* the wakeup to interrupt context. And don't bother waking * up the running kthread. */ -if (current != rsp->gp_kthread) +if (current != rsp->gp_kthread) { +trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum), +"Workqueuewake" );irq_work_queue (&rsp->wakeup_work); + } }/*
kernel/rcu/tree.h +0 -1
@@ -547,7 +547,6 @@ static void print_cpu_stall_info(struct rcu_state *rsp, int cpu);static void print_cpu_stall_info_end (void );static void zero_cpu_stall_ticks (struct rcu_data *rdp);static void increment_cpu_stall_ticks (void ); -static int rcu_nocb_needs_gp (struct rcu_state *rsp);static void rcu_nocb_gp_set (struct rcu_node *rnp,int nrq);static void rcu_nocb_gp_cleanup (struct rcu_state *rsp,struct rcu_node *rnp);static void rcu_init_one_nocb (struct rcu_node *rnp);
kernel/rcu/tree_plugin.h +0 -18
@@ -2068,19 +2068,6 @@ static int __init parse_rcu_nocb_poll(char *arg)early_param ("rcu_nocb_poll" , parse_rcu_nocb_poll);/* -* Do any no-CBs CPUs need another grace period? -* -* Interrupts must be disabled. If the caller does not hold the root -* rnp_node structure's ->lock, the results are advisory only. -*/ -static int rcu_nocb_needs_gp (struct rcu_state *rsp) -{ -struct rcu_node *rnp =rcu_get_root (rsp); - -return rnp->need_future_gp[(ACCESS_ONCE (rnp->completed) +1 ) &0x1 ]; -} - -/* * Wake up any no-CBs CPUs' kthreads that were waiting on the just-ended * grace period. */
@@ -2402,11 +2389,6 @@ static bool init_nocb_callback_list(struct rcu_data *rdp)#else /* #ifdef CONFIG_RCU_NOCB_CPU */ -static int rcu_nocb_needs_gp (struct rcu_state *rsp) -{ -return 0 ; -} -static void rcu_nocb_gp_cleanup (struct rcu_state *rsp,struct rcu_node *rnp) { }
Signature · 1 line
1.8.1.5
PATCH 5/19 rcu: Remove unused rcu_data structure field
In reply to Paul E. McKenneyFrom: Iulia Manda <[email protected]>
The ->preemptible field in rcu_data is only initialized in the function rcu_init_percpu_data(), and never used. This commit therefore removes this field.
--- kernel/rcu/tree.c | 6 ++---- kernel/rcu/tree.h | 1 - 2 files changed, 2 insertions(+), 5 deletions(-)
kernel/rcu/tree.c +2 -4
@@ -3180,7 +3180,7 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp)* that this CPU cannot possibly have any RCU callbacks in flight yet. */ static void -rcu_init_percpu_data (int cpu,struct rcu_state *rsp,int preemptible) +rcu_init_percpu_data (int cpu,struct rcu_state *rsp) {unsigned long flags;unsigned long mask;
@@ -3193,7 +3193,6 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp, int preemptible)/* Set up local state, ensuring consistent view of global state. */ raw_spin_lock_irqsave (&rnp->lock, flags); rdp->beenonline =1 ;/* We have now been online. */ - rdp->preemptible = preemptible; rdp->qlen_last_fqs_check =0 ; rdp->n_force_qs_snap = rsp->n_force_qs; rdp->blimit = blimit;
@@ -3237,8 +3236,7 @@ static void rcu_prepare_cpu(int cpu)struct rcu_state *rsp;for_each_rcu_flavor (rsp) -rcu_init_percpu_data (cpu, rsp, -strcmp (rsp->name,"rcu_preempt" ) ==0 ); +rcu_init_percpu_data (cpu, rsp); }/*
kernel/rcu/tree.h +0 -1
@@ -252,7 +252,6 @@ struct rcu_data {bool passed_quiesce;/* User-mode/idle loop etc. */ bool qs_pending;/* Core waits for quiesc state. */ bool beenonline;/* CPU online at least once. */ -bool preemptible;/* Preemptible RCU? */ struct rcu_node *mynode;/* This CPU's leaf of hierarchy */ unsigned long grpmask;/* Mask to apply to leaf qsmask. */ #ifdef CONFIG_RCU_CPU_STALL_INFO
Signature · 1 line
1.8.1.5
PATCH 6/19 rcu: Protect uses of jiffies_stall field with ACCESS_ONCE()
In reply to Paul E. McKenneyFrom: Iulia Manda <[email protected]>
Some of the uses of the rcu_state structure's ->jiffies_stall field do not use ACCESS_ONCE(), despite there being unprotected accesses. This commit therefore uses the ACCESS_ONCE() macro to protect this field.
--- kernel/rcu/tree.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +6 -6
@@ -865,7 +865,7 @@ static void record_gp_stall_check_time(struct rcu_state *rsp) rsp->gp_start = j;smp_wmb ();/* Record start time before stall time. */ j1 =rcu_jiffies_till_stall_check (); - rsp->jiffies_stall = j + j1; +ACCESS_ONCE (rsp->jiffies_stall) = j + j1; rsp->jiffies_resched = j + j1 /2 ; }
@@ -904,12 +904,12 @@ static void print_other_cpu_stall(struct rcu_state *rsp)/* Only let one CPU complain about others per time interval. */ raw_spin_lock_irqsave (&rnp->lock, flags); - delta = jiffies - rsp->jiffies_stall; + delta = jiffies -ACCESS_ONCE (rsp->jiffies_stall);if (delta <RCU_STALL_RAT_DELAY || !rcu_gp_in_progress (rsp)) {raw_spin_unlock_irqrestore (&rnp->lock, flags);return ; } - rsp->jiffies_stall = jiffies +3 *rcu_jiffies_till_stall_check () +3 ; +ACCESS_ONCE (rsp->jiffies_stall) = jiffies +3 *rcu_jiffies_till_stall_check () +3 ;raw_spin_unlock_irqrestore (&rnp->lock, flags);/*
@@ -992,8 +992,8 @@ static void print_cpu_stall(struct rcu_state *rsp)dump_stack ();raw_spin_lock_irqsave (&rnp->lock, flags); -if (ULONG_CMP_GE (jiffies, rsp->jiffies_stall)) - rsp->jiffies_stall = jiffies + +if (ULONG_CMP_GE (jiffies,ACCESS_ONCE (rsp->jiffies_stall))) +ACCESS_ONCE (rsp->jiffies_stall) = jiffies +3 *rcu_jiffies_till_stall_check () +3 ;raw_spin_unlock_irqrestore (&rnp->lock, flags);
@@ -1077,7 +1077,7 @@ void rcu_cpu_stall_reset(void)struct rcu_state *rsp;for_each_rcu_flavor (rsp) - rsp->jiffies_stall = jiffies +ULONG_MAX /2 ; +ACCESS_ONCE (rsp->jiffies_stall) = jiffies +ULONG_MAX /2 ; }/*
Signature · 1 line
1.8.1.5
PATCH 7/19 rcu: Make callers awaken grace-period kthread
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The rcu_start_gp_advanced() function currently uses irq_work_queue() to defer wakeups of the RCU grace-period kthread. This deferring is necessary to avoid RCU-scheduler deadlocks involving the rcu_node structure's lock, meaning that RCU cannot call any of the scheduler's wake-up functions while holding one of these locks.
Unfortunately, the second and subsequent calls to irq_work_queue() are ignored, and the first call will be ignored (aside from queuing the work item) if the scheduler-clock tick is turned off. This is OK for many uses, especially those where irq_work_queue() is called from an interrupt or softirq handler, because in those cases the scheduler-clock-tick state will be re-evaluated, which will turn the scheduler-clock tick back on. On the next tick, any deferred work will then be processed.
However, this strategy does not always work for RCU, which can be invoked at process level from idle CPUs. In this case, the tick might never be turned back on, indefinitely defering a grace-period start request. Note that the RCU CPU stall detector cannot see this condition, because there is no RCU grace period in progress. Therefore, we can (and do!) see long tens-of-seconds stalls in grace-period handling. In theory, we could see a full grace-period hang, but rcutorture testing to date has seen only the tens-of-seconds stalls. Event tracing demonstrates that irq_work_queue() is being called repeatedly to no effect during these stalls: The "newreq" event appears repeatedly from a task that is not one of the grace-period kthreads.
In theory, irq_work_queue() might be fixed to avoid this sort of issue, but RCU's requirements are unusual and it is quite straightforward to pass wake-up responsibility up through RCU's call chain, so that the wakeup happens when the offending locks are released.
This commit therefore makes this change. The rcu_start_gp_advanced(), rcu_start_future_gp(), rcu_accelerate_cbs(), rcu_advance_cbs(), __note_gp_changes(), and rcu_start_gp() functions now return a boolean which indicates when a wake-up is needed. A new rcu_gp_kthread_wake() does the wakeup when it is necessary and safe to do so: No self-wakes, no wake-ups if the ->gp_flags field indicates there is no need (as in someone else did the wake-up before we got around to it), and no wake-ups before the grace-period kthread has been created.
--- kernel/rcu/tree.c | 137 +++++++++++++++++++++++++++++------------------ kernel/rcu/tree.h | 1 - kernel/rcu/tree_plugin.h | 10 +++- 3 files changed, 94 insertions(+), 54 deletions(-)
kernel/rcu/tree.c +86 -51
@@ -243,7 +243,7 @@ static ulong jiffies_till_next_fqs = ULONG_MAX;module_param (jiffies_till_first_fqs, ulong,0644 );module_param (jiffies_till_next_fqs, ulong,0644 ); -static void rcu_start_gp_advanced (struct rcu_state *rsp,struct rcu_node *rnp, +static bool rcu_start_gp_advanced (struct rcu_state *rsp,struct rcu_node *rnp,struct rcu_data *rdp); static void force_qs_rnp(struct rcu_state *rsp, int (*f)(struct rcu_data *rsp, bool *isidle,
@@ -1138,15 +1138,18 @@ static void trace_rcu_future_gp(struct rcu_node *rnp, struct rcu_data *rdp,/* * Start some future grace period, as needed to handle newly arrived * callbacks. The required future grace periods are recorded in each -* rcu_node structure's ->need_future_gp field. +* rcu_node structure's ->need_future_gp field. Returns true if there +* is reason to awaken the grace-period kthread. * * The caller must hold the specified rcu_node structure's ->lock. */ -static unsigned long __maybe_unused -rcu_start_future_gp (struct rcu_node *rnp,struct rcu_data *rdp) +static bool __maybe_unused +rcu_start_future_gp (struct rcu_node *rnp,struct rcu_data *rdp, +unsigned long *c_out) {unsigned long c;int i; +bool ret = false;struct rcu_node *rnp_root =rcu_get_root (rdp->rsp);/*
@@ -1157,7 +1160,7 @@ rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp)trace_rcu_future_gp (rnp, rdp, c,TPS ("Startleaf" ));if (rnp->need_future_gp[c &0x1 ]) {trace_rcu_future_gp (rnp, rdp, c,TPS ("Prestartleaf" )); -return c; + goto out; }/*
@@ -1171,7 +1174,7 @@ rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp) ACCESS_ONCE(rnp->gpnum) != ACCESS_ONCE(rnp->completed)) { rnp->need_future_gp[c &0x1 ]++;trace_rcu_future_gp (rnp, rdp, c,TPS ("Startedleaf" )); -return c; + goto out; }/*
@@ -1212,12 +1215,15 @@ rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp)trace_rcu_future_gp (rnp, rdp, c,TPS ("Startedleafroot" )); }else {trace_rcu_future_gp (rnp, rdp, c,TPS ("Startedroot" )); -rcu_start_gp_advanced (rdp->rsp, rnp_root, rdp); + ret =rcu_start_gp_advanced (rdp->rsp, rnp_root, rdp); } unlock_out:if (rnp != rnp_root)raw_spin_unlock (&rnp_root->lock); -return c; +out: +if (c_out !=NULL ) + *c_out = c; +return ret; }/*
@@ -1241,25 +1247,43 @@ static int rcu_future_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp) }/* +* Awaken the grace-period kthread for the specified flavor of RCU. +* Don't do a self-awaken, and don't bother awakening when there is +* nothing for the grace-period kthread to do (as in several CPUs +* raced to awaken, and we lost), and finally don't try to awaken +* a kthread that has not yet been created. +*/ +static void rcu_gp_kthread_wake (struct rcu_state *rsp) +{ +if (current == rsp->gp_kthread || + !ACCESS_ONCE (rsp->gp_flags) || + !rsp->gp_kthread) +return ; +wake_up (&rsp->gp_wq); +} + +/* * If there is room, assign a ->completed number to any callbacks on * this CPU that have not already been assigned. Also accelerate any * callbacks that were previously assigned a ->completed number that has * since proven to be too conservative, which can happen if callbacks get * assigned a ->completed number while RCU is idle, but with reference to * a non-root rcu_node structure. This function is idempotent, so it does -* not hurt to call it repeatedly. +* not hurt to call it repeatedly. Returns an flag saying that we should +* awaken the RCU grace-period kthread. * * The caller must hold rnp->lock with interrupts disabled. */ -static void rcu_accelerate_cbs (struct rcu_state *rsp,struct rcu_node *rnp, +static bool rcu_accelerate_cbs (struct rcu_state *rsp,struct rcu_node *rnp,struct rcu_data *rdp) {unsigned long c;int i; +bool ret;/* If the CPU has no callbacks, nothing to do. */ if (!rdp->nxttail[RCU_NEXT_TAIL ] || !*rdp->nxttail[RCU_DONE_TAIL ]) -return ; +return false;/* * Starting from the sublist containing the callbacks most
@@ -1288,7 +1312,7 @@ static void rcu_accelerate_cbs(struct rcu_state *rsp, struct rcu_node *rnp,* be grouped into. */ if (++i >=RCU_NEXT_TAIL ) -return ; +return false;/* * Assign all subsequent callbacks' ->completed number to the next
@@ -1300,13 +1324,14 @@ static void rcu_accelerate_cbs(struct rcu_state *rsp, struct rcu_node *rnp, rdp->nxtcompleted[i] = c; }/* Record any needed additional grace periods. */ -rcu_start_future_gp (rnp, rdp); + ret =rcu_start_future_gp (rnp, rdp,NULL );/* Trace depending on how much we were able to accelerate. */ if (!*rdp->nxttail[RCU_WAIT_TAIL ])trace_rcu_grace_period (rsp->name, rdp->gpnum,TPS ("AccWaitCB" ));else trace_rcu_grace_period (rsp->name, rdp->gpnum,TPS ("AccReadyCB" )); +return ret; }/*
@@ -1315,17 +1340,18 @@ static void rcu_accelerate_cbs(struct rcu_state *rsp, struct rcu_node *rnp,* assign ->completed numbers to any callbacks in the RCU_NEXT_TAIL * sublist. This function is idempotent, so it does not hurt to * invoke it repeatedly. As long as it is not invoked -too- often... +* Returns true if the RCU grace-period kthread needs to be awakened. * * The caller must hold rnp->lock with interrupts disabled. */ -static void rcu_advance_cbs (struct rcu_state *rsp,struct rcu_node *rnp, +static bool rcu_advance_cbs (struct rcu_state *rsp,struct rcu_node *rnp,struct rcu_data *rdp) {int i, j;/* If the CPU has no callbacks, nothing to do. */ if (!rdp->nxttail[RCU_NEXT_TAIL ] || !*rdp->nxttail[RCU_DONE_TAIL ]) -return ; +return false;/* * Find all callbacks whose ->completed numbers indicate that they
@@ -1349,26 +1375,30 @@ static void rcu_advance_cbs(struct rcu_state *rsp, struct rcu_node *rnp, }/* Classify any remaining callbacks. */ -rcu_accelerate_cbs (rsp, rnp, rdp); +return rcu_accelerate_cbs (rsp, rnp, rdp); }/* * Update CPU-local rcu_data state to record the beginnings and ends of * grace periods. The caller must hold the ->lock of the leaf rcu_node * structure corresponding to the current CPU, and must have irqs disabled. +* Returns true if the grace-period kthread needs to be awakened. */ -static void __note_gp_changes (struct rcu_state *rsp,struct rcu_node *rnp,struct rcu_data *rdp) +static bool __note_gp_changes (struct rcu_state *rsp,struct rcu_node *rnp, +struct rcu_data *rdp) { +bool ret; +/* Handle the ends of any preceding grace periods first. */ if (rdp->completed == rnp->completed) {/* No grace period end, so just accelerate recent callbacks. */ -rcu_accelerate_cbs (rsp, rnp, rdp); + ret =rcu_accelerate_cbs (rsp, rnp, rdp); }else {/* Advance callbacks. */ -rcu_advance_cbs (rsp, rnp, rdp); + ret =rcu_advance_cbs (rsp, rnp, rdp);/* Remember that we saw this grace-period completion. */ rdp->completed = rnp->completed;
@@ -1387,11 +1417,13 @@ static void __note_gp_changes(struct rcu_state *rsp, struct rcu_node *rnp, struc rdp->qs_pending = !!(rnp->qsmask & rdp->grpmask);zero_cpu_stall_ticks (rdp); } +return ret; }static void note_gp_changes (struct rcu_state *rsp,struct rcu_data *rdp) {unsigned long flags; +bool needwake;struct rcu_node *rnp;local_irq_save (flags);
@@ -1403,8 +1435,10 @@ static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp)return ; }smp_mb__after_unlock_lock (); -__note_gp_changes (rsp, rnp, rdp); + needwake =__note_gp_changes (rsp, rnp, rdp);raw_spin_unlock_irqrestore (&rnp->lock, flags); +if (needwake) +rcu_gp_kthread_wake (rsp); }/*
@@ -1468,7 +1502,7 @@ static int rcu_gp_init(struct rcu_state *rsp)WARN_ON_ONCE (rnp->completed != rsp->completed);ACCESS_ONCE (rnp->completed) = rsp->completed;if (rnp == rdp->mynode) -__note_gp_changes (rsp, rnp, rdp); + (void )__note_gp_changes (rsp, rnp, rdp);rcu_preempt_boost_start_gp (rnp); trace_rcu_grace_period_init(rsp->name, rnp->gpnum, rnp->level, rnp->grplo,
@@ -1528,6 +1562,7 @@ static int rcu_gp_fqs(struct rcu_state *rsp, int fqs_state_in)static void rcu_gp_cleanup (struct rcu_state *rsp) {unsigned long gp_duration; +bool needgp = false;int nocb =0 ;struct rcu_data *rdp;struct rcu_node *rnp =rcu_get_root (rsp);
@@ -1563,7 +1598,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)ACCESS_ONCE (rnp->completed) = rsp->gpnum; rdp =this_cpu_ptr (rsp->rda);if (rnp == rdp->mynode) -__note_gp_changes (rsp, rnp, rdp); + needgp =__note_gp_changes (rsp, rnp, rdp) || needgp;/* smp_mb() provided by prior unlock-lock pair. */ nocb +=rcu_future_gp_cleanup (rsp, rnp);raw_spin_unlock_irq (&rnp->lock);
@@ -1579,8 +1614,9 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)trace_rcu_grace_period (rsp->name, rsp->completed,TPS ("end" )); rsp->fqs_state =RCU_GP_IDLE ; rdp =this_cpu_ptr (rsp->rda); -rcu_advance_cbs (rsp, rnp, rdp);/* Reduce false positives below. */ -if (cpu_needs_another_gp (rsp, rdp)) { +/* Advance CBs to reduce false positives below. */ + needgp =rcu_advance_cbs (rsp, rnp, rdp) || needgp; +if (needgp ||cpu_needs_another_gp (rsp, rdp)) {ACCESS_ONCE (rsp->gp_flags) =RCU_GP_FLAG_INIT ; trace_rcu_grace_period(rsp->name, ACCESS_ONCE(rsp->gpnum),
@@ -1680,16 +1716,6 @@ static int __noreturn rcu_gp_kthread(void *arg) } } -static void rsp_wakeup (struct irq_work *work) -{ - struct rcu_state *rsp = container_of(work, struct rcu_state, wakeup_work); - -/* Wake up rcu_gp_kthread() to start the grace period. */ -wake_up (&rsp->gp_wq); -trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum), -"Workqueuewoken" ); -} -/* * Start a new RCU grace period if warranted, re-initializing the hierarchy * in preparation for detecting the next grace period. The caller must hold
@@ -1698,8 +1724,10 @@ static void rsp_wakeup(struct irq_work *work)* Note that it is legal for a dying CPU (which is marked as offline) to * invoke this function. This can happen when the dying CPU reports its * quiescent state. +* +* Returns true if the grace-period kthread must be awakened. */ -static void +static bool rcu_start_gp_advanced (struct rcu_state *rsp,struct rcu_node *rnp,struct rcu_data *rdp) {
@@ -1710,7 +1738,7 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,* or a grace period is already in progress. * Either way, don't start a new grace period. */ -return ; +return false; }ACCESS_ONCE (rsp->gp_flags) =RCU_GP_FLAG_INIT ; trace_rcu_grace_period(rsp->name, ACCESS_ONCE(rsp->gpnum),
@@ -1719,14 +1747,9 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,/* * We can't do wakeups while holding the rnp->lock, as that * could cause possible deadlocks with the rq->lock. Defer -* the wakeup to interrupt context. And don't bother waking -* up the running kthread. +* the wakeup to our caller. */ -if (current != rsp->gp_kthread) { -trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum), -"Workqueuewake" ); -irq_work_queue (&rsp->wakeup_work); - } +return true; }/*
@@ -1735,12 +1758,14 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,* is invoked indirectly from rcu_advance_cbs(), which would result in * endless recursion -- or would do so if it wasn't for the self-deadlock * that is encountered beforehand. +* +* Returns true if the grace-period kthread needs to be awakened. */ -static void -rcu_start_gp (struct rcu_state *rsp) +static bool rcu_start_gp (struct rcu_state *rsp) {struct rcu_data *rdp =this_cpu_ptr (rsp->rda);struct rcu_node *rnp =rcu_get_root (rsp); +bool ret = false;/* * If there is no grace period in progress right now, any
@@ -1750,8 +1775,9 @@ rcu_start_gp(struct rcu_state *rsp)* resulting in pointless grace periods. So, advance callbacks * then start the grace period! */ -rcu_advance_cbs (rsp, rnp, rdp); -rcu_start_gp_advanced (rsp, rnp, rdp); + ret =rcu_advance_cbs (rsp, rnp, rdp) || ret; + ret =rcu_start_gp_advanced (rsp, rnp, rdp) || ret; +return ret; }/*
@@ -1840,6 +1866,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_state *rsp, struct rcu_data *rdp) {unsigned long flags;unsigned long mask; +bool needwake;struct rcu_node *rnp; rnp = rdp->mynode;
@@ -1868,9 +1895,11 @@ rcu_report_qs_rdp(int cpu, struct rcu_state *rsp, struct rcu_data *rdp)* This GP can't end until cpu checks in, so all of our * callbacks can be processed during the next GP. */ -rcu_accelerate_cbs (rsp, rnp, rdp); + needwake =rcu_accelerate_cbs (rsp, rnp, rdp);rcu_report_qs_rnp (mask, rsp, rnp, flags);/* rlses rnp->lock */ +if (needwake) +rcu_gp_kthread_wake (rsp); } }
@@ -2354,6 +2383,7 @@ static void __rcu_process_callbacks(struct rcu_state *rsp) {unsigned long flags; +bool needwake;struct rcu_data *rdp =__this_cpu_ptr (rsp->rda);WARN_ON_ONCE (rdp->beenonline ==0 );
@@ -2365,8 +2395,10 @@ __rcu_process_callbacks(struct rcu_state *rsp)local_irq_save (flags);if (cpu_needs_another_gp (rsp, rdp)) {raw_spin_lock (&rcu_get_root (rsp)->lock);/* irqs disabled. */ -rcu_start_gp (rsp); + needwake =rcu_start_gp (rsp);raw_spin_unlock_irqrestore (&rcu_get_root (rsp)->lock, flags); +if (needwake) +rcu_gp_kthread_wake (rsp); }else {local_irq_restore (flags); }
@@ -2424,6 +2456,8 @@ static void invoke_rcu_core(void)static void __call_rcu_core (struct rcu_state *rsp,struct rcu_data *rdp,struct rcu_head *head,unsigned long flags) { +bool needwake; +/* * If called from an extended quiescent state, invoke the RCU * core in order to force a re-evaluation of RCU's idleness.
@@ -2453,8 +2487,10 @@ static void __call_rcu_core(struct rcu_state *rsp, struct rcu_data *rdp,raw_spin_lock (&rnp_root->lock);smp_mb__after_unlock_lock (); -rcu_start_gp (rsp); + needwake =rcu_start_gp (rsp);raw_spin_unlock (&rnp_root->lock); +if (needwake) +rcu_gp_kthread_wake (rsp); }else {/* Give the grace period a kick. */ rdp->blimit =LONG_MAX ;
@@ -3440,7 +3476,6 @@ static void __init rcu_init_one(struct rcu_state *rsp, rsp->rda = rda;init_waitqueue_head (&rsp->gp_wq); -init_irq_work (&rsp->wakeup_work, rsp_wakeup); rnp = rsp->level[rcu_num_lvls -1 ];for_each_possible_cpu (i) {while (i > rnp->grphi)
kernel/rcu/tree.h +0 -1
@@ -461,7 +461,6 @@ struct rcu_state {const char *name;/* Name of structure. */ char abbr;/* Abbreviated name. */ struct list_head flavors;/* List of RCU flavors. */ -struct irq_work wakeup_work;/* Postponed wakeups */ };/* Values for rcu_state structure's gp_flags field. */
kernel/rcu/tree_plugin.h +8 -2
@@ -1744,6 +1744,7 @@ int rcu_needs_cpu(int cpu, unsigned long *dj)static void rcu_prepare_for_idle (int cpu) {#ifndef CONFIG_RCU_NOCB_CPU_ALL +bool needwake;struct rcu_data *rdp;struct rcu_dynticks *rdtp = &per_cpu (rcu_dynticks, cpu);struct rcu_node *rnp;
@@ -1792,8 +1793,10 @@ static void rcu_prepare_for_idle(int cpu) rnp = rdp->mynode;raw_spin_lock (&rnp->lock);/* irqs already disabled. */ smp_mb__after_unlock_lock (); -rcu_accelerate_cbs (rsp, rnp, rdp); + needwake =rcu_accelerate_cbs (rsp, rnp, rdp);raw_spin_unlock (&rnp->lock);/* irqs remain disabled. */ +if (needwake) + rcu_gp_kthread_wake(rsp); }#endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */ }
@@ -2230,12 +2233,15 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)unsigned long c;bool d;unsigned long flags; +bool needwake;struct rcu_node *rnp = rdp->mynode;raw_spin_lock_irqsave (&rnp->lock, flags);smp_mb__after_unlock_lock (); - c =rcu_start_future_gp (rnp, rdp); + needwake =rcu_start_future_gp (rnp, rdp, &c);raw_spin_unlock_irqrestore (&rnp->lock, flags); +if (needwake) +rcu_gp_kthread_wake (rdp->rsp);/* * Wait for the grace period. Do so interruptibly to avoid messing
Signature · 1 line
1.8.1.5
PATCH 8/19 rcu: Remove "extern" from function declaration in include/linux/rcupdate.h
In reply to Paul E. McKenneyFrom: Iulia Manda <[email protected]>
Because functions have the extern storage class specifier by default, this keyword can be removed. It is redundant to use it explicitly.
--- include/linux/rcupdate.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/rcupdate.h +1 -1
@@ -328,7 +328,7 @@ extern struct lockdep_map rcu_lock_map;extern struct lockdep_map rcu_bh_lock_map;extern struct lockdep_map rcu_sched_lock_map;extern struct lockdep_map rcu_callback_map; -extern int debug_lockdep_rcu_enabled (void ); +int debug_lockdep_rcu_enabled (void );/** * rcu_read_lock_held() - might we be in RCU read-side critical section?
Signature · 1 line
1.8.1.5
PATCH 9/19 rcu: Protect uses of ->jiffies_stall with ACCESS_ONCE()
In reply to Paul E. McKenneyFrom: Himangi Saraogi <[email protected]>
Some of the accesses to the rcu_state structure's ->jiffies_stall field are unprotected. This patch protects them with ACCESS_ONCE(). The following coccinelle script was used to acheive this:
/* coccinelle script to protect uses of ->jiffies_stall with ACCESS_ONCE() */ @@
identifier a;
@@
(
ACCESS_ONCE(a->jiffies_stall)
|
- a->jiffies_stall
+ ACCESS_ONCE(a->jiffies_stall)
)
--- kernel/rcu/tiny_plugin.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/rcu/tiny_plugin.h +4 -4
@@ -144,7 +144,7 @@ static void check_cpu_stall(struct rcu_ctrlblk *rcp)return ; rcp->ticks_this_gp++; j = jiffies; - js = rcp->jiffies_stall; + js =ACCESS_ONCE (rcp->jiffies_stall);if (*rcp->curtail &&ULONG_CMP_GE (j, js)) { pr_err("INFO: %s stall on CPU (%lu ticks this GP) idle=%llx (t=%lu jiffies q=%ld)\n", rcp->name, rcp->ticks_this_gp, rcu_dynticks_nesting,
@@ -152,17 +152,17 @@ static void check_cpu_stall(struct rcu_ctrlblk *rcp)dump_stack (); }if (*rcp->curtail &&ULONG_CMP_GE (j, js)) - rcp->jiffies_stall = jiffies + +ACCESS_ONCE (rcp->jiffies_stall) = jiffies +3 *rcu_jiffies_till_stall_check () +3 ;else if (ULONG_CMP_GE (j, js)) - rcp->jiffies_stall = jiffies +rcu_jiffies_till_stall_check (); +ACCESS_ONCE (rcp->jiffies_stall) = jiffies +rcu_jiffies_till_stall_check (); }static void reset_cpu_stall_ticks (struct rcu_ctrlblk *rcp) { rcp->ticks_this_gp =0 ; rcp->gp_start = jiffies; - rcp->jiffies_stall = jiffies +rcu_jiffies_till_stall_check (); +ACCESS_ONCE (rcp->jiffies_stall) = jiffies +rcu_jiffies_till_stall_check (); }static void check_cpu_stalls (void )
Signature · 1 line
1.8.1.5
PATCH 10/19 rcu: Add event tracing to dyntick_save_progress_counter().
In reply to Paul E. McKenneyFrom: Andreea-Cristina Bernat <[email protected]>
This patch adds event tracing to dyntick_save_progress_counter() in the case where it returns 1. I used the tracepoint string "dti" because this function returns 1 in case the CPU is in dynticks idle mode.
--- kernel/rcu/tree.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +6 -1
@@ -772,7 +772,12 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp, { rdp->dynticks_snap =atomic_add_return (0 , &rdp->dynticks->dynticks);rcu_sysidle_check_cpu (rdp, isidle, maxj); -return (rdp->dynticks_snap &0x1 ) ==0 ; +if ((rdp->dynticks_snap &0x1 ) ==0 ) { +trace_rcu_fqs (rdp->rsp->name, rdp->gpnum, rdp->cpu,TPS ("dti" )); +return 1 ; + }else { +return 0 ; + } }/*
Signature · 1 line
1.8.1.5
PATCH 11/19 rcu: Replace NR_CPUS with nr_cpu_ids
In reply to Paul E. McKenneyFrom: Himangi Saraogi <[email protected]>
This patch replaces NR_CPUS with nr_cpu_ids as NR_CPUS should consider cpumask_var_t.
--- kernel/rcu/tree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree.c +2 -2
@@ -3461,8 +3461,8 @@ static void __init rcu_init_one(struct rcu_state *rsp, rnp->qsmaskinit =0 ; rnp->grplo = j * cpustride; rnp->grphi = (j +1 ) * cpustride -1 ; -if (rnp->grphi >=NR_CPUS ) - rnp->grphi =NR_CPUS -1 ; +if (rnp->grphi >= nr_cpu_ids) + rnp->grphi = nr_cpu_ids -1 ;if (i ==0 ) { rnp->grpnum =0 ; rnp->grpmask =0 ;
Signature · 1 line
1.8.1.5
PATCH 12/19 rcu: Consolidate kfree_call_rcu() to use rcu_state pointer
In reply to Paul E. McKenneyFrom: Andreea-Cristina Bernat <[email protected]>
kfree_call_rcu is defined two times. When defined under CONFIG_TREE_PREEMPT_RCU, it uses rcu_preempt_state. Otherwise, it uses rcu_sched_state.
This patch uses the rcu_state_pointer to combine the two definitions into one. The resulting function is placed after the closing of the preprocessor conditional CONFIG_TREE_PREEMPT_RCU.
--- kernel/rcu/tree.c | 14 ++++++++++++++ kernel/rcu/tree_plugin.h | 30 ------------------------------ 2 files changed, 14 insertions(+), 30 deletions(-)
kernel/rcu/tree.c +14 -0
@@ -2598,6 +2598,20 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))EXPORT_SYMBOL_GPL (call_rcu_bh);/* +* Queue an RCU callback for lazy invocation after a grace period. +* This will likely be later named something like "call_rcu_lazy()", +* but this change will require some way of tagging the lazy RCU +* callbacks in the list of pending callbacks. Until then, this +* function may only be called from __kfree_rcu(). +*/ +void kfree_call_rcu (struct rcu_head *head, +void (*func)(struct rcu_head *rcu)) +{ +__call_rcu (head, func, rcu_state,-1 ,1 ); +} +EXPORT_SYMBOL_GPL (kfree_call_rcu); + +/* * Because a context switch is a grace period for RCU-sched and RCU-bh, * any blocking grace-period wait automatically implies a grace period * if there is only one CPU online at any point time during execution
kernel/rcu/tree_plugin.h +0 -30
@@ -688,20 +688,6 @@ void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) }EXPORT_SYMBOL_GPL (call_rcu); -/* -* Queue an RCU callback for lazy invocation after a grace period. -* This will likely be later named something like "call_rcu_lazy()", -* but this change will require some way of tagging the lazy RCU -* callbacks in the list of pending callbacks. Until then, this -* function may only be called from __kfree_rcu(). -*/ -void kfree_call_rcu (struct rcu_head *head, -void (*func)(struct rcu_head *rcu)) -{ -__call_rcu (head, func, &rcu_preempt_state,-1 ,1 ); -} -EXPORT_SYMBOL_GPL (kfree_call_rcu); -/** * synchronize_rcu - wait until a grace period has elapsed. *
@@ -1080,22 +1066,6 @@ static void rcu_preempt_check_callbacks(int cpu) }/* -* Queue an RCU callback for lazy invocation after a grace period. -* This will likely be later named something like "call_rcu_lazy()", -* but this change will require some way of tagging the lazy RCU -* callbacks in the list of pending callbacks. Until then, this -* function may only be called from __kfree_rcu(). -* -* Because there is no preemptible RCU, we use RCU-sched instead. -*/ -void kfree_call_rcu (struct rcu_head *head, -void (*func)(struct rcu_head *rcu)) -{ -__call_rcu (head, func, &rcu_sched_state,-1 ,1 ); -} -EXPORT_SYMBOL_GPL (kfree_call_rcu); - -/* * Wait for an rcu-preempt grace period, but make it happen quickly. * But because preemptible RCU does not exist, map to rcu-sched. */
Signature · 1 line
1.8.1.5
PATCH 13/19 rcu: Merge rcu_sched_force_quiescent_state() with rcu_force_quiescent_state()
In reply to Paul E. McKenneyFrom: Andreea-Cristina Bernat <[email protected]>
This patch merges the function rcu_force_quiescent_state() with rcu_sched_force_quiescent_state(), using the rcu_state pointer. Firstly, the rcu_sched_force_quiescent_state() function is deleted from the file kernel/rcu/tree.c. Also, the rcu_force_quiescent_state() function that was calling force_quiescent_state with the argument rcu_preempt_state pointer was deleted as well. The new function that combines the old ones uses the rcu_state pointer and is located after rcu_batches_completed_bh() in kernel/rcu/tree.c.
--- kernel/rcu/tree.c | 9 +++++++++ kernel/rcu/tree_plugin.h | 19 ------------------- 2 files changed, 9 insertions(+), 19 deletions(-)
kernel/rcu/tree.c +9 -0
@@ -271,6 +271,15 @@ long rcu_batches_completed_bh(void)EXPORT_SYMBOL_GPL (rcu_batches_completed_bh);/* +* Force a quiescent state. +*/ +void rcu_force_quiescent_state (void ) +{ +force_quiescent_state (rcu_state); +} +EXPORT_SYMBOL_GPL (rcu_force_quiescent_state); + +/* * Force a quiescent state for RCU BH. */ void rcu_bh_force_quiescent_state (void )
kernel/rcu/tree_plugin.h +0 -19
@@ -149,15 +149,6 @@ long rcu_batches_completed(void)EXPORT_SYMBOL_GPL (rcu_batches_completed);/* -* Force a quiescent state for preemptible RCU. -*/ -void rcu_force_quiescent_state (void ) -{ -force_quiescent_state (&rcu_preempt_state); -} -EXPORT_SYMBOL_GPL (rcu_force_quiescent_state); - -/* * Record a preemptible-RCU quiescent state for the specified CPU. Note * that this just means that the task currently running on the CPU is * not in a quiescent state. There might be any number of tasks blocked
@@ -977,16 +968,6 @@ long rcu_batches_completed(void)EXPORT_SYMBOL_GPL (rcu_batches_completed);/* -* Force a quiescent state for RCU, which, because there is no preemptible -* RCU, becomes the same as rcu-sched. -*/ -void rcu_force_quiescent_state (void ) -{ -rcu_sched_force_quiescent_state (); -} -EXPORT_SYMBOL_GPL (rcu_force_quiescent_state); - -/* * Because preemptible RCU does not exist, we never have to check for * CPUs being in quiescent states. */
Signature · 1 line
1.8.1.5
PATCH 14/19 rcu: Document RCU_INIT_POINTER()'s lack of ordering guarantees
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Although rcu_assign_pointer() provides ordering guarantees, RCU_INIT_POINTER() does not. This commit makes that explicit in the docbook comment header.
--- include/linux/rcupdate.h | 3 +++ 1 file changed, 3 insertions(+)
include/linux/rcupdate.h +3 -0
@@ -949,6 +949,9 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)* pointers, but you must use rcu_assign_pointer() to initialize the * external-to-structure pointer -after- you have completely initialized * the reader-accessible portions of the linked structure. +* +* Note that unlike rcu_assign_pointer(), RCU_INIT_POINTER() provides no +* ordering guarantees for either the CPU or the compiler. */ #define RCU_INIT_POINTER (p, v) \ do { \
Signature · 1 line
1.8.1.5
PATCH 15/19 rcu: Bind RCU grace-period kthreads if NO_HZ_FULL
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, RCU binds the grace-period kthreads to the timekeeping CPU only if CONFIG_NO_HZ_FULL_SYSIDLE=y. This means that these kthreads must be bound manually when CONFIG_NO_HZ_FULL_SYSIDLE=n and CONFIG_NO_HZ_FULL=y: Otherwise, these kthreads will induce OS jitter on random CPUs. Given that we are trying to reduce the amount of manual tweaking required to make CONFIG_NO_HZ_FULL=y work nicely, this commit makes this binding happen when CONFIG_NO_HZ_FULL=y, even in cases where CONFIG_NO_HZ_FULL_SYSIDLE=n.
--- kernel/rcu/tree_plugin.h | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-)
kernel/rcu/tree_plugin.h +16 -18
@@ -2596,20 +2596,6 @@ static bool is_sysidle_rcu_state(struct rcu_state *rsp) }/* -* Bind the grace-period kthread for the sysidle flavor of RCU to the -* timekeeping CPU. -*/ -static void rcu_bind_gp_kthread (void ) -{ -int cpu =ACCESS_ONCE (tick_do_timer_cpu); - -if (cpu <0 || cpu >= nr_cpu_ids) -return ; -if (raw_smp_processor_id () != cpu) -set_cpus_allowed_ptr (current,cpumask_of (cpu)); -} - -/* * Return a delay in jiffies based on the number of CPUs, rcu_node * leaf fanout, and jiffies tick rate. The idea is to allow larger * systems more time to transition to full-idle state in order to
@@ -2819,10 +2805,6 @@ static bool is_sysidle_rcu_state(struct rcu_state *rsp)return false; } -static void rcu_bind_gp_kthread (void ) -{ -} -static void rcu_sysidle_report_gp (struct rcu_state *rsp,int isidle,unsigned long maxj) {
@@ -2853,3 +2835,19 @@ static bool rcu_nohz_full_cpu(struct rcu_state *rsp)#endif /* #ifdef CONFIG_NO_HZ_FULL */ return 0 ; } + +/* +* Bind the grace-period kthread for the sysidle flavor of RCU to the +* timekeeping CPU. +*/ +static void rcu_bind_gp_kthread (void ) +{ +#ifdef CONFIG_NO_HZ_FULL +int cpu =ACCESS_ONCE (tick_do_timer_cpu); + +if (cpu <0 || cpu >= nr_cpu_ids) +return ; +if (raw_smp_processor_id () != cpu) +set_cpus_allowed_ptr (current,cpumask_of (cpu)); +#endif /* #ifdef CONFIG_NO_HZ_FULL */ +}
Signature · 1 line
1.8.1.5
PATCH 16/19 rcu: Make large and small sysidle systems use same state machine
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, small systems move back into RCU_SYSIDLE_NOT from RCU_SYSIDLE_SHORT and large systems do not. This works because moving aggressively to RCU_SYSIDLE_NOT affects only performance, not correctness, and on small systems, the performance impact should be negligible. That said, this difference does make RCU a bit more complex, and RCU does not seem to be suffering from any lack of complexity. This commit therefore adjusts small-system operation to match that of large systems, so that the state never moves back to RCU_SYSIDLE_NOT from RCU_SYSIDLE_SHORT.
--- kernel/rcu/tree_plugin.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +2 -1
@@ -2659,7 +2659,8 @@ static void rcu_sysidle(unsigned long j)static void rcu_sysidle_cancel (void ) {smp_mb (); -ACCESS_ONCE (full_sysidle_state) =RCU_SYSIDLE_NOT ; +if (full_sysidle_state >RCU_SYSIDLE_SHORT ) +ACCESS_ONCE (full_sysidle_state) =RCU_SYSIDLE_NOT ; }/*
Signature · 1 line
1.8.1.5
PATCH 17/19 rcu: Remove duplicate resched_cpu() declaration
In reply to Paul E. McKenneyFrom: Pranith Kumar <[email protected]>
--- kernel/rcu/tree.c | 6 ------ 1 file changed, 6 deletions(-)
kernel/rcu/tree.c +0 -6
@@ -975,12 +975,6 @@ static void print_other_cpu_stall(struct rcu_state *rsp)force_quiescent_state (rsp);/* Kick them all. */ } -/* -* This function really isn't for public consumption, but RCU is special in -* that context switches can allow the state machine to make progress. -*/ -extern void resched_cpu (int cpu); -static void print_cpu_stall (struct rcu_state *rsp) {int cpu;
Signature · 1 line
1.8.1.5
PATCH 18/19 rcu: Replace __this_cpu_ptr() uses with raw_cpu_ptr()
In reply to Paul E. McKenneyFrom: Christoph Lameter <[email protected]>
__this_cpu_ptr is being phased out.
One special case is increment_cpu_stall_ticks(). A per cpu variable is incremented so use raw_cpu_inc().
--- kernel/rcu/tree.c | 6 +++--- kernel/rcu/tree_plugin.h | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
kernel/rcu/tree.c +3 -3
@@ -2008,7 +2008,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp,static void rcu_adopt_orphan_cbs (struct rcu_state *rsp,unsigned long flags) {int i; -struct rcu_data *rdp =__this_cpu_ptr (rsp->rda); +struct rcu_data *rdp =raw_cpu_ptr (rsp->rda);/* No-CBs CPUs are handled specially. */ if (rcu_nocb_adopt_orphan_cbs (rsp, rdp, flags))
@@ -2392,7 +2392,7 @@ __rcu_process_callbacks(struct rcu_state *rsp) {unsigned long flags;bool needwake; -struct rcu_data *rdp =__this_cpu_ptr (rsp->rda); +struct rcu_data *rdp =raw_cpu_ptr (rsp->rda);WARN_ON_ONCE (rdp->beenonline ==0 );
@@ -3066,7 +3066,7 @@ static void rcu_barrier_callback(struct rcu_head *rhp)static void rcu_barrier_func (void *type) {struct rcu_state *rsp = type; -struct rcu_data *rdp =__this_cpu_ptr (rsp->rda); +struct rcu_data *rdp =raw_cpu_ptr (rsp->rda);_rcu_barrier_trace (rsp,"IRQ" ,-1 , rsp->n_barrier_done);atomic_inc (&rsp->barrier_cpu_count);
kernel/rcu/tree_plugin.h +2 -2
@@ -1809,7 +1809,7 @@ static void rcu_oom_notify_cpu(void *unused)struct rcu_data *rdp;for_each_rcu_flavor (rsp) { - rdp =__this_cpu_ptr (rsp->rda); + rdp =raw_cpu_ptr (rsp->rda);if (rdp->qlen_lazy !=0 ) {atomic_inc (&oom_callback_count); rsp->call (&rdp->oom_head, rcu_oom_callback);
@@ -1951,7 +1951,7 @@ static void increment_cpu_stall_ticks(void)struct rcu_state *rsp;for_each_rcu_flavor (rsp) - __this_cpu_ptr(rsp->rda)->ticks_this_gp++; + raw_cpu_inc(rsp->rda->ticks_this_gp); }#else /* #ifdef CONFIG_RCU_CPU_STALL_INFO */
Signature · 1 line
1.8.1.5
PATCH 19/19 softirq: A single rcu_bh_qs() call per softirq set is enough
In reply to Paul E. McKenneyFrom: Eric Dumazet <[email protected]>
Calling rcu_bh_qs() after every softirq action is not really needed. What RCU needs is at least one rcu_bh_qs() per softirq round to note a quiescent state was passed for rcu_bh.
Note for Paul and myself : this could be inlined as a single instruction and avoid smp_processor_id()
(sone this_cpu_write(rcu_bh_data.passed_quiesce, 1))
--- kernel/softirq.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/softirq.c +1 -3
@@ -232,7 +232,6 @@ asmlinkage void __do_softirq(void)bool in_hardirq;__u32 pending;int softirq_bit; -int cpu;/* * Mask out PF_MEMALLOC s current task context is borrowed for the
@@ -247,7 +246,6 @@ asmlinkage void __do_softirq(void)__local_bh_disable_ip (_RET_IP_,SOFTIRQ_OFFSET ); in_hardirq =lockdep_softirq_start (); - cpu =smp_processor_id (); restart:/* Reset the pending bitmask before enabling irqs */ set_softirq_pending (0 );
@@ -276,11 +274,11 @@ restart: prev_count, preempt_count());preempt_count_set (prev_count); } -rcu_bh_qs (cpu); h++; pending >>= softirq_bit; } +rcu_bh_qs (smp_processor_id ());local_irq_disable (); pending =local_softirq_pending ();
Signature · 1 line
1.8.1.5
Quoted 52 lines · Paul E. McKenney
Hello!
This series provides miscellaneous fixes:
1. Apply
ACCESS_ONCE()to unprotected ->gp_flags accesses.2. Fix typo in comment, courtesy of Liu Ping Fan.
3. Make RCU CPU stall warnings print grace-period numbers in
signed format to improve readability of stall-warning output.4. Make
cpu_needs_another_gp()take future grace-period needs
into account.5. Remove unused ->preemptible field from the
rcu_datastructure,
courtesty of Iulia Manda.6. Apply
ACCESS_ONCE()to unprotected ->jiffies_stall accesses,
courtesty of Iulia Manda.7. Make callers responsible for grace-period kthread wakeup in
order to avoid potential silent grace-period stalls.8. Remove "extern" from RCU function declarations, courtesy of
Iulia Manda.9. Apply
ACCESS_ONCE()to additional ->jiffies_stall accesses,
courtesy of Himangi Saraogi.10. Add event tracing to
dyntick_save_progress_counter(), courtesy
of Andreea-Cristina Bernat.11. Make
rcu_init_one()usenr_cpu_idsinstead ofNR_CPUSfor
data-structure setup limit check, courtesy of Himangi Saraogi.12. Remove redundant
kfree_call_rcu()definition by using the
rcu_statepointer, courtesy of Andreea-Cristina Bernat.13. Merge
rcu_sched_force_quiescent_state()definition with
rcu_force_quiescent_state()by using thercu_statepointer, courtesy of Andreea-Cristina Bernat.14. Document RCU_INIT_POINTER()'s lack of ordering guarantees.
15. Automatically bind RCU's grace-period kthreads to timekeeping
CPU forNO_HZ_FULLbuilds.16. Make large and small sysidle systems use equivalent state machine.
17. Remove duplicate
resched_cpu()declaration, courtesy of
Pranith Kumar.
Quoted 6 lines
18. Replace deprecated
__this_cpu_ptr()uses withraw_cpu_ptr(),
courtesy of Christoph Lameter.19. Make softirq processing provide a quiescent state only once
per full pass over all softirqs rather than once per action, courtesy of Eric Dumazet.
For all 19:
Quoted 56 lines · Josh Triplett
Quoted 55 lines · Paul E. McKenney
Hello!
This series provides miscellaneous fixes:
1. Apply
ACCESS_ONCE()to unprotected ->gp_flags accesses.2. Fix typo in comment, courtesy of Liu Ping Fan.
3. Make RCU CPU stall warnings print grace-period numbers in
signed format to improve readability of stall-warning output.4. Make
cpu_needs_another_gp()take future grace-period needs
into account.5. Remove unused ->preemptible field from the
rcu_datastructure,
courtesty of Iulia Manda.6. Apply
ACCESS_ONCE()to unprotected ->jiffies_stall accesses,
courtesty of Iulia Manda.7. Make callers responsible for grace-period kthread wakeup in
order to avoid potential silent grace-period stalls.8. Remove "extern" from RCU function declarations, courtesy of
Iulia Manda.9. Apply
ACCESS_ONCE()to additional ->jiffies_stall accesses,
courtesy of Himangi Saraogi.10. Add event tracing to
dyntick_save_progress_counter(), courtesy
of Andreea-Cristina Bernat.11. Make
rcu_init_one()usenr_cpu_idsinstead ofNR_CPUSfor
data-structure setup limit check, courtesy of Himangi Saraogi.12. Remove redundant
kfree_call_rcu()definition by using the
rcu_statepointer, courtesy of Andreea-Cristina Bernat.13. Merge
rcu_sched_force_quiescent_state()definition with
rcu_force_quiescent_state()by using thercu_statepointer, courtesy of Andreea-Cristina Bernat.14. Document RCU_INIT_POINTER()'s lack of ordering guarantees.
15. Automatically bind RCU's grace-period kthreads to timekeeping
CPU forNO_HZ_FULLbuilds.16. Make large and small sysidle systems use equivalent state machine.
17. Remove duplicate
resched_cpu()declaration, courtesy of
Pranith Kumar.18. Replace deprecated
__this_cpu_ptr()uses withraw_cpu_ptr(),
courtesy of Christoph Lameter.
Quoted 6 lines
Quoted 3 lines
19. Make softirq processing provide a quiescent state only once
per full pass over all softirqs rather than once per action, courtesy of Eric Dumazet.For all 19:
Reviewed-by Josh Triplett <[email protected]>
And thank you for these reviews as well, applied!
Thanx, Paul