RCU_FAST_NO_HZ changes for 3.6
Subsystems (3)
cover RCU_FAST_NO_HZ changes for 3.6
Hello!
This patch series provides more adjustments to the (relatively) new large-system-safe implementation for RCU_FAST_NO_HZ:
1. Remove RCU_FAST_NO_HZ dependency on stop_machine() nature of
CPU hotplug.
2. Make RCU_FAST_NO_HZ tracing distinguish between short and
long idle intervals.
3. Move RCU_FAST_NO_HZ per-CPU state variables to the rcu_dynticks
per-CPU structure.
4. Precompute RCU_FAST_NO_HZ timer offsets so that the timers
will actually be paid attention to. This fixes the slow-boot problem that hit a few people.
5. Convert ftrace_dump() calls in idle entry and idle exit from
DUMP_ALL to DUMP_ORIG.
6. Fix erroneous TINY_PREEMPT_RCU assumption that rcu_preempt_needs_cpu()
is a quiescent state (it is not).
7. Round RCU_FAST_NO_HZ lazy timeout to nearest second to conserve
power on systems with synchronized scheduler-clock interrupts.
I am considering pushing #1-#4 into 3.5 for the slow-boot regression. If you object, please let me know.
Thanx, Paul
b/include/linux/rcupdate.h | 2 b/include/linux/rcutiny.h | 6 + b/include/linux/rcutree.h | 2 b/include/trace/events/rcu.h | 1 b/kernel/rcutiny_plugin.h | 2 b/kernel/rcutree.c | 2 b/kernel/rcutree.h | 14 +++ b/kernel/rcutree_plugin.h | 8 +- b/kernel/time/tick-sched.c | 7 + kernel/rcutree.c | 4 - kernel/rcutree_plugin.h | 167 ++++++++++++++++++++++--------------------- 11 files changed, 124 insertions(+), 91 deletions(-)
PATCH 1/8 rcu: RCU_FAST_NO_HZ detection of callback adoption
From: "Paul E. McKenney" <[email protected]>
In the present implementations of CPU hotplug, the outgoing CPU is guaranteed to run its stop-machine process on the way out, which will guarantee that RCU_FAST_NO_HZ forces the CPU out of dyntick-idle mode.
However, new versions of CPU hotplug might not work this way. This commit therefore removes this design constraint by explicitly notifying CPUs when they adopt non-lazy RCU callbacks.
--- kernel/rcutree.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/rcutree.c +2 -0
@@ -1397,6 +1397,8 @@ static void rcu_adopt_orphan_cbs(struct rcu_state *rsp) rdp->qlen_lazy += rsp->qlen_lazy; rdp->qlen += rsp->qlen; rdp->n_cbs_adopted += rsp->qlen; +if (rsp->qlen_lazy != rsp->qlen) +rcu_idle_count_callbacks_posted (); rsp->qlen_lazy =0 ; rsp->qlen =0 ;
Signature · 1 line
1.7.8
PATCH 2/8 rcu: Update RCU_FAST_NO_HZ tracing for lazy callbacks
From: "Paul E. McKenney" <[email protected]>
In the current code, a short dyntick-idle interval (where there is at least one non-lazy callback on the CPU) and a long dyntick-idle interval (where there are only lazy callbacks on the CPU) are traced identically, which can be less than helpful. This commit therefore emits different event traces in these two cases.
--- include/trace/events/rcu.h | 1 + kernel/rcutree_plugin.h | 8 +++++--- 2 files changed, 6 insertions(+), 3 deletions(-)
include/trace/events/rcu.h +1 -0
@@ -289,6 +289,7 @@ TRACE_EVENT(rcu_dyntick,* "In holdoff": Nothing to do, holding off after unsuccessful attempt. * "Begin holdoff": Attempt failed, don't retry until next jiffy. * "Dyntick with callbacks": Entering dyntick-idle despite callbacks. +* "Dyntick with lazy callbacks": Entering dyntick-idle w/lazy callbacks. * "More callbacks": Still more callbacks, try again to clear them out. * "Callbacks drained": All callbacks processed, off to dyntick idle! * "Timer": Timer fired to cause CPU to continue processing callbacks.
kernel/rcutree_plugin.h +5 -3
@@ -2165,15 +2165,17 @@ static void rcu_prepare_for_idle(int cpu) !rcu_pending(cpu) && !local_softirq_pending()) {/* Can we go dyntick-idle despite still having callbacks? */ -trace_rcu_prep_idle ("Dyntick with callbacks" );per_cpu (rcu_dyntick_drain, cpu) =0 ;per_cpu (rcu_dyntick_holdoff, cpu) = jiffies; -if (rcu_cpu_has_nonlazy_callbacks (cpu)) +if (rcu_cpu_has_nonlazy_callbacks (cpu)) { +trace_rcu_prep_idle ("Dyntick with callbacks" );per_cpu (rcu_idle_gp_timer_expires, cpu) = jiffies +RCU_IDLE_GP_DELAY ; -else + }else {per_cpu (rcu_idle_gp_timer_expires, cpu) = jiffies +RCU_IDLE_LAZY_GP_DELAY ; +trace_rcu_prep_idle ("Dyntick with lazy callbacks" ); + } tp = &per_cpu (rcu_idle_gp_timer, cpu);mod_timer_pinned (tp,per_cpu (rcu_idle_gp_timer_expires, cpu)); per_cpu(rcu_nonlazy_posted_snap, cpu) =
Signature · 1 line
1.7.8
PATCH 3/8 rcu: Move RCU_FAST_NO_HZ per-CPU variables to rcu_dynticks structure
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The RCU_FAST_NO_HZ code relies on a number of per-CPU variables. This works, but is hidden from someone scanning the data structures in rcutree.h. This commit therefore converts these per-CPU variables to fields in the per-CPU rcu_dynticks structures.
--- kernel/rcutree.h | 14 +++++++ kernel/rcutree_plugin.h | 99 +++++++++++++++++++++-------------------------- 2 files changed, 58 insertions(+), 55 deletions(-)
kernel/rcutree.h +14 -0
@@ -84,6 +84,20 @@ struct rcu_dynticks {/* Process level is worth LLONG_MAX/2. */ int dynticks_nmi_nesting;/* Track NMI nesting level. */ atomic_t dynticks;/* Even value for idle, else odd. */ +#ifdef CONFIG_RCU_FAST_NO_HZ +int dyntick_drain;/* Prepare-for-idle state variable. */ +unsigned long dyntick_holdoff; +/* No retries for the jiffy of failure. */ +struct timer_list idle_gp_timer; +/* Wake up CPU sleeping with callbacks. */ +unsigned long idle_gp_timer_expires; +/* When to wake up CPU (for repost). */ +bool idle_first_pass;/* First pass of attempt to go idle? */ +unsigned long nonlazy_posted; +/* # times non-lazy CBs posted to CPU. */ +unsigned long nonlazy_posted_snap; +/* idle-period nonlazy_posted snapshot. */ +#endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */ };/* RCU's kthread states for tracing. */
kernel/rcutree_plugin.h +44 -55
197 lines not on this page · raw message
Signature · 1 line
1.7.8
PATCH 4/8 rcu: Precompute RCU_FAST_NO_HZ timer offsets
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
When a CPU is entering dyntick-idle mode, tick_nohz_stop_sched_tick() calls rcu_needs_cpu() see if RCU needs that CPU, and, if not, computes the next wakeup time based on the timer wheels. Only later, when actually entering the idle loop, rcu_prepare_for_idle() will be invoked. In some cases, rcu_prepare_for_idle() will post timers to wake the CPU back up. But all for naught: The next wakeup time for the CPU has already been computed, and posting a timer afterwards does not force that wakeup time to be recomputed. This means that rcu_prepare_for_idle()'s have no effect.
This is not a problem on a busy system because something else will wake up the CPU soon enough. However, on lightly loaded systems, the CPU might stay asleep for a considerable length of time. If that CPU has a callback that the rest of the system is waiting on, the system might run very slowly or (in theory) even hang.
This commit avoids this problem by having rcu_needs_cpu() give tick_nohz_stop_sched_tick() an estimate of when RCU will need the CPU to wake back up, which tick_nohz_stop_sched_tick() takes into account when programming the CPU's wakeup time. An alternative approach is for rcu_prepare_for_idle() to use hrtimers instead of normal timers, but timers are much more efficient than are hrtimers for frequently and repeatedly posting and cancelling a given timer, which is exactly what RCU_FAST_NO_HZ does.
--- include/linux/rcutiny.h | 6 +++- include/linux/rcutree.h | 2 +- kernel/rcutree_plugin.h | 66 ++++++++++++++++++++++++++++++---------------- kernel/time/tick-sched.c | 7 ++++- 4 files changed, 54 insertions(+), 27 deletions(-)
include/linux/rcutiny.h +4 -2
@@ -87,8 +87,9 @@ static inline void kfree_call_rcu(struct rcu_head *head,#ifdef CONFIG_TINY_RCU -static inline int rcu_needs_cpu (int cpu) +static inline int rcu_needs_cpu (int cpu,unsigned long *delta_jiffies) { + *delta_jiffies =ULONG_MAX ;return 0 ; }
@@ -96,8 +97,9 @@ static inline int rcu_needs_cpu(int cpu)int rcu_preempt_needs_cpu (void ); -static inline int rcu_needs_cpu (int cpu) +static inline int rcu_needs_cpu (int cpu,unsigned long *delta_jiffies) { + *delta_jiffies =ULONG_MAX ;return rcu_preempt_needs_cpu (); }
include/linux/rcutree.h +1 -1
@@ -32,7 +32,7 @@extern void rcu_init (void );extern void rcu_note_context_switch (int cpu); -extern int rcu_needs_cpu (int cpu); +extern int rcu_needs_cpu (int cpu,unsigned long *delta_jiffies);extern void rcu_cpu_stall_reset (void );/*
kernel/rcutree_plugin.h +43 -23
85 lines not on this page · raw message
kernel/time/tick-sched.c +6 -1
@@ -274,6 +274,7 @@ EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);static void tick_nohz_stop_sched_tick (struct tick_sched *ts) {unsigned long seq, last_jiffies, next_jiffies, delta_jiffies; +unsigned long rcu_delta_jiffies;ktime_t last_update, expires, now;struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta;
@@ -322,7 +323,7 @@ static void tick_nohz_stop_sched_tick(struct tick_sched *ts) time_delta =timekeeping_max_deferment (); }while (read_seqretry (&xtime_lock, seq)); -if (rcu_needs_cpu (cpu) ||printk_needs_cpu (cpu) || +if (rcu_needs_cpu (cpu, &rcu_delta_jiffies) ||printk_needs_cpu (cpu) ||arch_needs_cpu (cpu)) { next_jiffies = last_jiffies +1 ; delta_jiffies =1 ;
@@ -330,6 +331,10 @@ static void tick_nohz_stop_sched_tick(struct tick_sched *ts)/* Get the next timer wheel timer */ next_jiffies =get_next_timer_interrupt (last_jiffies); delta_jiffies = next_jiffies - last_jiffies; +if (rcu_delta_jiffies < delta_jiffies) { + next_jiffies = last_jiffies + rcu_delta_jiffies; + delta_jiffies = rcu_delta_jiffies; + } }/* * Do not stop the tick, if we are only one off
Signature · 1 line
1.7.8
PATCH 5/8 rcu: Dump only the current CPU's buffers for idle-entry/exit warnings
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Problems in RCU idle entry and exit are almost always confined to the offending CPU. This commit therefore switches ftrace_dump() from DUMP_ALL to DUMP_ORIG.
--- kernel/rcutree.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/rcutree.c +2 -2
@@ -357,7 +357,7 @@ static void rcu_idle_enter_common(struct rcu_dynticks *rdtp, long long oldval)struct task_struct *idle =idle_task (smp_processor_id ());trace_rcu_dyntick ("Error on entry: not idle task" , oldval,0 ); -ftrace_dump (DUMP_ALL ); +ftrace_dump (DUMP_ORIG );WARN_ONCE (1 ,"Current pid: %d comm: %s / Idle pid: %d comm: %s" , current->pid, current->comm, idle->pid, idle->comm);/* must be idle task! */
@@ -467,7 +467,7 @@ static void rcu_idle_exit_common(struct rcu_dynticks *rdtp, long long oldval)trace_rcu_dyntick ("Error on exit: not idle task" , oldval, rdtp->dynticks_nesting); -ftrace_dump (DUMP_ALL ); +ftrace_dump (DUMP_ORIG );WARN_ONCE (1 ,"Current pid: %d comm: %s / Idle pid: %d comm: %s" , current->pid, current->comm, idle->pid, idle->comm);/* must be idle task! */
Signature · 1 line
1.7.8
PATCH 6/8 rcu: The rcu_needs_cpu() function is not a quiescent state
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The TINY_PREEMPT_RCU() function rcu_preempt_needs_cpu(), which is called from rcu_needs_cpu(), assumes that it is in a quiescent state with respect to the CPU. This is no longer the case. This commit therefore updates rcu_preempt_needs_cpu() to make it aware that it is not running in a quiescent state.
--- kernel/rcutiny_plugin.h | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-)
kernel/rcutiny_plugin.h +0 -2
@@ -846,8 +846,6 @@ EXPORT_SYMBOL_GPL(synchronize_rcu_expedited);*/ int rcu_preempt_needs_cpu (void ) { -if (!rcu_preempt_running_reader ()) -rcu_preempt_cpu_qs ();return rcu_preempt_ctrlblk.rcb.rcucblist !=NULL ; }
Signature · 1 line
1.7.8
PATCH 7/8 rcu: Round FAST_NO_HZ lazy timeout to nearest second
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, if several CPUs in the same package have all lazy RCU callbacks, their wakeups will be uncorrelated. If all the CPUs are in the same power domain (as is often the case), this will result in unnecessary power-ups of the package. This commit therefore uses round_jiffies() to round the timeouts to a second boundary, increasing the odds that they can be coalesced with each other or with other timeouts.
--- kernel/rcutree_plugin.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/rcutree_plugin.h +1 -1
@@ -2182,7 +2182,7 @@ static void rcu_prepare_for_idle(int cpu) jiffies +RCU_IDLE_GP_DELAY ; }else { rdtp->idle_gp_timer_expires = - jiffies +RCU_IDLE_LAZY_GP_DELAY ; +round_jiffies (jiffies +RCU_IDLE_LAZY_GP_DELAY );trace_rcu_prep_idle ("Dyntick with lazy callbacks" ); } tp = &rdtp->idle_gp_timer;
Signature · 1 line
1.7.8
PATCH 8/8 rcu: Fix qlen_lazy breakage
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Commit d8169d4c (Make __kfree_rcu() less dependent on compiler choices) created a macro out of an inline function in order to avoid build breakage for certain combinations of gcc flags. Unfortunately, it also converted a kfree_call_rcu() to a call_rcu(), which made the rcu_data structure's ->qlen_lazy field lose counts. This commit therefore changes the call_rcu() back to kfree_call_rcu().
--- include/linux/rcupdate.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/rcupdate.h +1 -1
@@ -936,7 +936,7 @@ void __kfree_rcu(struct rcu_head *head, unsigned long offset)#define __kfree_rcu (head, offset) \ do { \ BUILD_BUG_ON(!__is_kfree_rcu_offset(offset)); \ - call_rcu(head, (void (*)(struct rcu_head *))(unsigned long)(offset)); \ + kfree_call_rcu(head, (void (*)(struct rcu_head *))(unsigned long)(offset)); \ } while (0)/**
Signature · 1 line
1.7.8
Ingo Molnar
maintainerQuoted 21 lines · Paul E. McKenney
Hello!
This patch series provides more adjustments to the (relatively) new large-system-safe implementation for
RCU_FAST_NO_HZ:1. Remove
RCU_FAST_NO_HZdependency onstop_machine()nature of
CPU hotplug.
2. MakeRCU_FAST_NO_HZtracing distinguish between short and
long idle intervals.
3. MoveRCU_FAST_NO_HZper-CPU state variables to thercu_dynticks
per-CPU structure.
4. PrecomputeRCU_FAST_NO_HZtimer offsets so that the timers
will actually be paid attention to. This fixes the slow-boot problem that hit a few people.
5. Convertftrace_dump()calls in idle entry and idle exit from
DUMP_ALLtoDUMP_ORIG.
6. Fix erroneousTINY_PREEMPT_RCUassumption thatrcu_preempt_needs_cpu()
is a quiescent state (it is not).
7. RoundRCU_FAST_NO_HZlazy timeout to nearest second to conserve
power on systems with synchronized scheduler-clock interrupts.
Quoted 2 lines
I am considering pushing #1-#4 into 3.5 for the slow-boot regression. If you object, please let me know.
Sure, that's sensible - could get this to me ASAP so that we can send it to Linus before -rc2? #3 and #4 are pretty large so we want them upstream ASAP.
Thanks,
Ingo