nohz: Use sysidle detection to let the timekeeper sleep
Subsystems (3)
cover nohz: Use sysidle detection to let the timekeeper sleep
Hi,
This series makes the nohz subsystem eventually use the RCU full sysidle detection.
When we have CPUs running in full dynticks mode in the system, the CPU 0 handles the timekeeping duty on behalf of all other CPUs. Given that full dynticks can run anytime, CPU 0 stays periodic and never enter into dynticks idle mode. This is of course a powersaving issue.
Now making CPU 0 to become more power-friendly sounds like an easy task. After all we just need to allow it to enter in dynticks idle mode as soon as all full dynticks CPUs are idle (aka sysidle state, we'll probably need some more precise name as it only applies to full dynticks CPUs).
But sysidle state detection is actually difficult to get right. It must scale with growing number of CPUs, minimize IPIs and atomic operations on fast paths. Given that this detection already existed into a bit more generalized form through the existing RCU extended quiescent state detection, it has been implemented by specializing this code and adding a state machine on top of it.
(Thanks to Paul for this work, more details: https://lwn.net/Articles/558284/)
This feature which is enabled with CONFIG_NO_HZ_FULL_SYSIDLE=y, is working but is not yet plugged into the nohz subsystem. Namely we can detect states where all full dynticks CPUs are sleeping, but we don't yet take benefit from it by opportunistically stopping the tick of the timekeeper CPU 0.
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
* Allow all CPUs outside the nohz_full range to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.
* Let the timekeeper (including CPU 0) sleep when its duty is handed over to another CPU
* Allow timekeeper to sleep when all full dynticks CPUs are sleeping (plug nohz to RCU sysidle detection)
* Wake up timekeeper with an IPI when full dynticks CPUs exit sysidle state
* Wake up CPU 0 when a secondary timekeeper is offlined so that its duty gets migrated
For convenience, you can fetch this from:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
timers/full_sysidle-rfc
Thanks,
Frederic
---
Frederic Weisbecker (12):
tick: Rename tick_check_idle() to tick_irq_enter()
time: New helper to check CPU eligibility to handle timekeeping
rcu: Exclude all potential timekeepers from sysidle detection
tick: Use timekeeping_cpu() to elect the CPU handling timekeeping duty
rcu: Fix unraised IPI to timekeeping CPU
nohz: Introduce full dynticks' default timekeeping target
sched: Enable IPI reception on timekeeper under nohz full system
nohz: Get timekeeping max deferment outside jiffies_lock
nohz: Allow timekeeper's tick to stop when all full dynticks CPUs are idle
nohz: Hand over timekeeping duty on cpu offlining
nohz: Wake up timekeeper on exit from sysidle state
nohz: Allow all CPUs outside nohz_full range to do timekeeping
Alex Shi (1):
nohz_full: fix code style issue of tick_nohz_full_stop_tick
include/linux/tick.h | 38 ++++++++++++-- kernel/rcu/tree_plugin.h | 12 ++--- kernel/sched/core.c | 6 +-- kernel/softirq.c | 2 +- kernel/time/tick-common.c | 2 +- kernel/time/tick-sched.c | 128 +++++++++++++++++++++++++++++++++++----------- 6 files changed, 142 insertions(+), 46 deletions(-)
PATCH 1/13 tick: Rename tick_check_idle() to tick_irq_enter()
This makes the code more symetric against the existing tick functions called on irq exit: tick_irq_exit() and tick_nohz_irq_exit().
These function are also symetric as they mirror each other's action:
we start to account idle time on irq exit and we stop this accounting on irq entry. Also the tick is stopped on irq exit and timekeeping catches up with the tickless time elapsed until we reach irq entry.
This rename was suggested by Peter Zijlstra a long while ago but it got forgotten in the mass.
--- include/linux/tick.h | 6 +++--- kernel/softirq.c | 2 +- kernel/time/tick-sched.c | 8 ++++---- 3 files changed, 8 insertions(+), 8 deletions(-)
include/linux/tick.h +3 -3
@@ -104,7 +104,7 @@ extern struct cpumask *tick_get_broadcast_oneshot_mask(void);extern void tick_clock_notify (void );extern int tick_check_oneshot_change (int allow_nohz);extern struct tick_sched *tick_get_tick_sched (int cpu); -extern void tick_check_idle (void ); +extern void tick_irq_enter (void );extern int tick_oneshot_mode_active (void );# ifndef arch_needs_cpu# define arch_needs_cpu (cpu) (0)
@@ -112,7 +112,7 @@ extern int tick_oneshot_mode_active(void);# else static inline void tick_clock_notify (void ) { }static inline int tick_check_oneshot_change (int allow_nohz) {return 0 ; } -static inline void tick_check_idle (void ) { } +static inline void tick_irq_enter (void ) { }static inline int tick_oneshot_mode_active (void ) {return 0 ; }# endif
@@ -121,7 +121,7 @@ static inline void tick_init(void) { }static inline void tick_cancel_sched_timer (int cpu) { }static inline void tick_clock_notify (void ) { }static inline int tick_check_oneshot_change (int allow_nohz) {return 0 ; } -static inline void tick_check_idle (void ) { } +static inline void tick_irq_enter (void ) { }static inline int tick_oneshot_mode_active (void ) {return 0 ; }#endif /* !CONFIG_GENERIC_CLOCKEVENTS */
kernel/softirq.c +1 -1
@@ -318,7 +318,7 @@ void irq_enter(void)* here, as softirq will be serviced on return from interrupt. */ local_bh_disable (); -tick_check_idle (); +tick_irq_enter ();_local_bh_enable (); }
kernel/time/tick-sched.c +4 -4
@@ -1025,7 +1025,7 @@ static void tick_nohz_kick_tick(struct tick_sched *ts, ktime_t now)#endif } -static inline void tick_check_nohz_this_cpu (void ) +static inline void tick_nohz_irq_enter (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);ktime_t now;
@@ -1044,17 +1044,17 @@ static inline void tick_check_nohz_this_cpu(void)#else static inline void tick_nohz_switch_to_nohz (void ) { } -static inline void tick_check_nohz_this_cpu (void ) { } +static inline void tick_nohz_irq_enter (void ) { }#endif /* CONFIG_NO_HZ_COMMON */ /* * Called from irq_enter to notify about the possible interruption of idle() */ -void tick_check_idle (void ) +void tick_irq_enter (void ) {tick_check_oneshot_broadcast_this_cpu (); -tick_check_nohz_this_cpu (); +tick_nohz_irq_enter (); }/*
Signature · 1 line
1.8.3.1
tip-bot for Frederic Weisbecker
merge bot[tip:timers/urgent] tick: Rename tick_check_idle() to tick_irq_enter()
tick: Rename tick_check_idle() to tick_irq_enter()
Commit-ID: 5acac1be499d979e3aa463ea73a498888faefcbe Gitweb: http://git.kernel.org/tip/5acac1be499d979e3aa463ea73a498888faefcbe Author: Frederic Weisbecker <[email protected]> AuthorDate: Wed, 4 Dec 2013 18:28:20 +0100 Committer: Frederic Weisbecker <[email protected]> CommitDate: Wed, 15 Jan 2014 23:05:31 +0100
tick: Rename tick_check_idle() to tick_irq_enter()
This makes the code more symetric against the existing tick functions called on irq exit: tick_irq_exit() and tick_nohz_irq_exit().
These function are also symetric as they mirror each other's action:
we start to account idle time on irq exit and we stop this accounting on irq entry. Also the tick is stopped on irq exit and timekeeping catches up with the tickless time elapsed until we reach irq entry.
This rename was suggested by Peter Zijlstra a long while ago but it got forgotten in the mass.
--- include/linux/tick.h | 6 +++--- kernel/softirq.c | 2 +- kernel/time/tick-sched.c | 8 ++++---- 3 files changed, 8 insertions(+), 8 deletions(-)
include/linux/tick.h +3 -3
@@ -104,7 +104,7 @@ extern struct cpumask *tick_get_broadcast_oneshot_mask(void);extern void tick_clock_notify (void );extern int tick_check_oneshot_change (int allow_nohz);extern struct tick_sched *tick_get_tick_sched (int cpu); -extern void tick_check_idle (void ); +extern void tick_irq_enter (void );extern int tick_oneshot_mode_active (void );# ifndef arch_needs_cpu# define arch_needs_cpu (cpu) (0)
@@ -112,7 +112,7 @@ extern int tick_oneshot_mode_active(void);# else static inline void tick_clock_notify (void ) { }static inline int tick_check_oneshot_change (int allow_nohz) {return 0 ; } -static inline void tick_check_idle (void ) { } +static inline void tick_irq_enter (void ) { }static inline int tick_oneshot_mode_active (void ) {return 0 ; }# endif
@@ -121,7 +121,7 @@ static inline void tick_init(void) { }static inline void tick_cancel_sched_timer (int cpu) { }static inline void tick_clock_notify (void ) { }static inline int tick_check_oneshot_change (int allow_nohz) {return 0 ; } -static inline void tick_check_idle (void ) { } +static inline void tick_irq_enter (void ) { }static inline int tick_oneshot_mode_active (void ) {return 0 ; }#endif /* !CONFIG_GENERIC_CLOCKEVENTS */
kernel/softirq.c +1 -1
@@ -318,7 +318,7 @@ void irq_enter(void)* here, as softirq will be serviced on return from interrupt. */ local_bh_disable (); -tick_check_idle (); +tick_irq_enter ();_local_bh_enable (); }
kernel/time/tick-sched.c +4 -4
@@ -1023,7 +1023,7 @@ static void tick_nohz_kick_tick(struct tick_sched *ts, ktime_t now)#endif } -static inline void tick_check_nohz_this_cpu (void ) +static inline void tick_nohz_irq_enter (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);ktime_t now;
@@ -1042,17 +1042,17 @@ static inline void tick_check_nohz_this_cpu(void)#else static inline void tick_nohz_switch_to_nohz (void ) { } -static inline void tick_check_nohz_this_cpu (void ) { } +static inline void tick_nohz_irq_enter (void ) { }#endif /* CONFIG_NO_HZ_COMMON */ /* * Called from irq_enter to notify about the possible interruption of idle() */ -void tick_check_idle (void ) +void tick_irq_enter (void ) {tick_check_oneshot_broadcast_this_cpu (); -tick_check_nohz_this_cpu (); +tick_nohz_irq_enter (); }/*
PATCH 2/13 time: New helper to check CPU eligibility to handle timekeeping
In reply to Frederic WeisbeckerTraditionally, all online CPUs in the system are allowed to handle the timekeeping duty.
Now with the full dynticks subsystem, we don't want the busy CPUs running tickless to be bothered with housekeeping duties that require periodic interrupts because that would defeat the main purpose of running tickless in the first place.
As such, the following rules apply to define the timekeeping duty affinity:
* If we run in full dynticks mode, CPU 0 handles the timekeeping on behalf of all other CPUs in the system. No other CPU is allowed to take that duty. This behaviour will soon be subject to change though as we plan to balance this duty over all the CPUs outside the full dynticks range with CONFIG_NO_HZ_FULL=y in order to improve power consumption.
* On any other environment (strict periodic or dynticks idle kernel), the timekeeping duty can be handled by any CPU.
In order to enforce and consolidate this behaviour, provide an API that core subsystems can use to check if a CPU is allowed to handle the timekeeping duty.
This is going to be used by the timer subsystem before assigning a timekeeper and by RCU for the full sysidle detection.
--- include/linux/tick.h | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
include/linux/tick.h +21 -1
@@ -179,15 +179,35 @@ static inline bool tick_nohz_full_cpu(int cpu) return cpumask_test_cpu(cpu, tick_nohz_full_mask); } +/** +* tick_timeeping_cpu - check if a CPU is elligble to handle timekeeping duty +* @cpu: the cpu to check +* +* @return true if the CPU is elligible to perform timekeeping duty. +*/ +static inline bool tick_timekeeping_cpu (int cpu) +{ +/* +* If there are full dynticks CPUs around, +* CPU 0 must stay periodic to update timekeeping. +*/ +if (tick_nohz_full_enabled ()) +return cpu ==0 ; + +/* Otherwise any CPU is elligible for timekeeping duty */ +return true; +} +extern void tick_nohz_init (void );extern void __tick_nohz_full_check (void );extern void tick_nohz_full_kick (void );extern void tick_nohz_full_kick_all (void );extern void __tick_nohz_task_switch (struct task_struct *tsk); -#else +# else static inline void tick_nohz_init (void ) { }static inline bool tick_nohz_full_enabled (void ) {return false; }static inline bool tick_nohz_full_cpu (int cpu) {return false; } +static inline bool tick_timekeeping_cpu (int cpu) {return true; }static inline void __tick_nohz_full_check (void ) { }static inline void tick_nohz_full_kick (void ) { }static inline void tick_nohz_full_kick_all (void ) { }
Signature · 1 line
1.8.3.1
PATCH 3/13 rcu: Exclude all potential timekeepers from sysidle detection
In reply to Frederic WeisbeckerThe purpose of the full system idle detection is to notify the CPU handling the timekeeping when the rest of the system is idle so that it can sleep when nobody needs the jiffies nor GTOD to be maintained.
Now this machinery excludes CPU 0 itself from the range of the idle detection because if CPU 0 has any non-idle task to execute, it is going to restart its own tick since it's guaranteed to be outside the full dynticks range. And as it is the only eligible timekeeper when CONFIG_NO_HZ_FULL=y anyway, it can handle the timekeeping duty for and by itself.
Still we also plan to extend the timekeepers affinity and allow every CPU outside the full dynticks range to handle the timekeeping duty, not just CPU 0.
So once we reach that step, we can state that all CPUs that are not full dynticks can be excluded from the full system idle detection, simply because those CPUs share the same property than CPU 0 today. When a non full dynticks CPU needs to run some busy task, it restarts its tick and handles the timekeeping duty for its own needs as is currently done under CONFIG_NO_HZ_IDLE=y.
To prepare for this support in the sysidle detection, we can use the tick_timekeeping_cpu() API which checks if a CPU is allowed to handle timekeeping duty. If so we can conclude that it's not full dynticks and it can maintain timekeeping by itself and as such it can be excluded from the sysidle detection.
--- kernel/rcu/tree_plugin.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/rcu/tree_plugin.h +4 -4
@@ -2539,7 +2539,7 @@ static void rcu_sysidle_exit(struct rcu_dynticks *rdtp, int irq)* invoke rcu_sysidle_force_exit() directly if it does anything * more than take a scheduling-clock interrupt. */ -if (smp_processor_id () == tick_do_timer_cpu) +if (tick_timekeeping_cpu (smp_processor_id ()))return ;/* Update system-idle state: We are clearly no longer fully idle! */
@@ -2563,10 +2563,10 @@ static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,* is an offline or the timekeeping CPU, nothing to do. */ if (!*isidle || rdp->rsp != rcu_sysidle_state || -cpu_is_offline (rdp->cpu) || rdp->cpu == tick_do_timer_cpu) +cpu_is_offline (rdp->cpu) ||tick_timekeeping_cpu (rdp->cpu))return ;if (rcu_gp_in_progress (rdp->rsp)) -WARN_ON_ONCE (smp_processor_id () != tick_do_timer_cpu); +WARN_ON_ONCE (!tick_timekeeping_cpu (smp_processor_id ()));/* Pick up current idle and NMI-nesting counter and check. */ cur =atomic_read (&rdtp->dynticks_idle);
@@ -2729,7 +2729,7 @@ bool rcu_sys_is_idle(void)static struct rcu_sysidle_head rsh;int rss =ACCESS_ONCE (full_sysidle_state); -if (WARN_ON_ONCE (smp_processor_id () != tick_do_timer_cpu)) +if (WARN_ON_ONCE (!tick_timekeeping_cpu (smp_processor_id ())))return false;/* Handle small-system case by doing a full scan of CPUs. */
Signature · 1 line
1.8.3.1
Re: [PATCH 03/13] rcu: Exclude all potential timekeepers from sysidle detection
Quoted 28 lines · Frederic Weisbecker
The purpose of the full system idle detection is to notify the CPU handling the timekeeping when the rest of the system is idle so that it can sleep when nobody needs the jiffies nor GTOD to be maintained.
Now this machinery excludes CPU 0 itself from the range of the idle detection because if CPU 0 has any non-idle task to execute, it is going to restart its own tick since it's guaranteed to be outside the full dynticks range. And as it is the only eligible timekeeper when
CONFIG_NO_HZ_FULL=yanyway, it can handle the timekeeping duty for and by itself.Still we also plan to extend the timekeepers affinity and allow every CPU outside the full dynticks range to handle the timekeeping duty, not just CPU 0.
So once we reach that step, we can state that all CPUs that are not full dynticks can be excluded from the full system idle detection, simply because those CPUs share the same property than CPU 0 today. When a non full dynticks CPU needs to run some busy task, it restarts its tick and handles the timekeeping duty for its own needs as is currently done under
CONFIG_NO_HZ_IDLE=y.To prepare for this support in the sysidle detection, we can use the
tick_timekeeping_cpu()API which checks if a CPU is allowed to handle timekeeping duty. If so we can conclude that it's not full dynticks and it can maintain timekeeping by itself and as such it can be excluded from the sysidle detection.
Quoted 9 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>
A few comments below as well.
kernel/rcu/tree_plugin.h
Quoted 8 lines
--- kernel/rcu/tree_plugin.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)kernel/rcu/tree_plugin.h +0
-0
The rcu_sysidle_force_exit() function uses tick_do_timer_cpu, but presumably needs to continue doing so in order to whack the right CPU over the head. I am happy to defer worrying about the interaction with multiple timekeeping CPUs for the moment. ;-)
Quoted 12 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2539,7 +2539,7 @@ static void rcu_sysidle_exit(struct rcu_dynticks *rdtp, int irq)* invoke rcu_sysidle_force_exit() directly if it does anything * more than take a scheduling-clock interrupt. */ -if (smp_processor_id () == tick_do_timer_cpu) +if (tick_timekeeping_cpu (smp_processor_id ()))return ;/* Update system-idle state: We are clearly no longer fully idle! */ @@ -2563,10 +2563,10 @@ static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,* is an offline or the timekeeping CPU, nothing to do. */
kernel/rcu/tree_plugin.h static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle, · line 2572
Quoted 10 lines
kernel/rcu/tree_plugin.h +2
-2if (!*isidle || rdp->rsp != rcu_sysidle_state || -cpu_is_offline (rdp->cpu) || rdp->cpu == tick_do_timer_cpu) +cpu_is_offline (rdp->cpu) ||tick_timekeeping_cpu (rdp->cpu))return ;if (rcu_gp_in_progress (rdp->rsp)) -WARN_ON_ONCE (smp_processor_id () != tick_do_timer_cpu); +WARN_ON_ONCE (!tick_timekeeping_cpu (smp_processor_id ()));/* Pick up current idle and NMI-nesting counter and check. */ cur =atomic_read (&rdtp->dynticks_idle);
The rcu_bind_gp_kthread() uses tick_do_timer_cpu to figure out where to run. Is there some CPU mask that it should use instead once there can be multiple timekeeping CPUs?
Quoted 12 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2729,7 +2729,7 @@ bool rcu_sys_is_idle(void)static struct rcu_sysidle_head rsh;int rss =ACCESS_ONCE (full_sysidle_state); -if (WARN_ON_ONCE (smp_processor_id () != tick_do_timer_cpu)) +if (WARN_ON_ONCE (!tick_timekeeping_cpu (smp_processor_id ())))return false;/* Handle small-system case by doing a full scan of CPUs. */ Signature · 1 line
1.8.3.1
Re: [PATCH 03/13] rcu: Exclude all potential timekeepers from sysidle detection
Quoted 39 lines · Paul E. McKenney
Quoted 28 lines · Frederic Weisbecker
The purpose of the full system idle detection is to notify the CPU handling the timekeeping when the rest of the system is idle so that it can sleep when nobody needs the jiffies nor GTOD to be maintained.
Now this machinery excludes CPU 0 itself from the range of the idle detection because if CPU 0 has any non-idle task to execute, it is going to restart its own tick since it's guaranteed to be outside the full dynticks range. And as it is the only eligible timekeeper when
CONFIG_NO_HZ_FULL=yanyway, it can handle the timekeeping duty for and by itself.Still we also plan to extend the timekeepers affinity and allow every CPU outside the full dynticks range to handle the timekeeping duty, not just CPU 0.
So once we reach that step, we can state that all CPUs that are not full dynticks can be excluded from the full system idle detection, simply because those CPUs share the same property than CPU 0 today. When a non full dynticks CPU needs to run some busy task, it restarts its tick and handles the timekeeping duty for its own needs as is currently done under
CONFIG_NO_HZ_IDLE=y.To prepare for this support in the sysidle detection, we can use the
tick_timekeeping_cpu()API which checks if a CPU is allowed to handle timekeeping duty. If so we can conclude that it's not full dynticks and it can maintain timekeeping by itself and as such it can be excluded from the sysidle detection.Quoted 9 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>
Quoted 1 line
Reviewed-by Paul E. McKenney <[email protected]>
Thanks!
Quoted 7 lines
A few comments below as well.
Quoted 4 lines
--- kernel/rcu/tree_plugin.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Quoted 9 lines
kernel/rcu/tree_plugin.h
Quoted 4 lines
kernel/rcu/tree_plugin.h +0
-0The
rcu_sysidle_force_exit()function usestick_do_timer_cpu, but presumably needs to continue doing so in order to whack the right CPU over the head. I am happy to defer worrying about the interaction with multiple timekeeping CPUs for the moment. ;-)
So yeah, I changed that in "nohz: Wake up timekeeper on exit from sysidle state". We always target CPU 0 for the IPI.
That's to start simple as CPU 0 can't be offlined. But we certainly want to optimize that by kicking a potential timekeeper that is not idle.
But that require a lookup like get_timer_nohz_target() and some safety against CPU hotplug. So that needs more thought :)
Quoted 21 lines
Quoted 21 lines
kernel/rcu/tree_plugin.h +3
-3@@ -2539,7 +2539,7 @@ static void rcu_sysidle_exit(struct rcu_dynticks *rdtp, int irq)* invoke rcu_sysidle_force_exit() directly if it does anything * more than take a scheduling-clock interrupt. */ -if (smp_processor_id () == tick_do_timer_cpu) +if (tick_timekeeping_cpu (smp_processor_id ()))return ;/* Update system-idle state: We are clearly no longer fully idle! */ @@ -2563,10 +2563,10 @@ static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,* is an offline or the timekeeping CPU, nothing to do. */ if (!*isidle || rdp->rsp != rcu_sysidle_state || -cpu_is_offline (rdp->cpu) || rdp->cpu == tick_do_timer_cpu) +cpu_is_offline (rdp->cpu) ||tick_timekeeping_cpu (rdp->cpu))return ;if (rcu_gp_in_progress (rdp->rsp)) -WARN_ON_ONCE (smp_processor_id () != tick_do_timer_cpu); +WARN_ON_ONCE (!tick_timekeeping_cpu (smp_processor_id ()));
Quoted 6 lines
kernel/rcu/tree_plugin.h static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle, · line 2572
Quoted 2 lines
kernel/rcu/tree_plugin.h +0
-0/* Pick up current idle and NMI-nesting counter and check. */ cur =atomic_read (&rdtp->dynticks_idle);The
rcu_bind_gp_kthread()usestick_do_timer_cputo figure out where to run. Is there some CPU mask that it should use instead once there can be multiple timekeeping CPUs?
Good point, we'll need to build one, or use ~nohz_full_mask
Thanks.
Quoted 14 lines
Quoted 13 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2729,7 +2729,7 @@ bool rcu_sys_is_idle(void)static struct rcu_sysidle_head rsh;int rss =ACCESS_ONCE (full_sysidle_state); -if (WARN_ON_ONCE (smp_processor_id () != tick_do_timer_cpu)) +if (WARN_ON_ONCE (!tick_timekeeping_cpu (smp_processor_id ())))return false;/* Handle small-system case by doing a full scan of CPUs. */ Signature · 1 line
1.8.3.1
Peter Zijlstra
maintainerRe: [PATCH 03/13] rcu: Exclude all potential timekeepers from sysidle detection
Quoted 1 line · Frederic Weisbecker
That's to start simple as CPU 0 can't be offlined.
Yes it can, see CONFIG_BOOTPARAM_HOTPLUG_CPU0
Peter Zijlstra
maintainerRe: [PATCH 03/13] rcu: Exclude all potential timekeepers from sysidle detection
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Frederic Weisbecker
That's to start simple as CPU 0 can't be offlined.
Yes it can, see
CONFIG_BOOTPARAM_HOTPLUG_CPU0
Also, that's a traditional weirdness of x86. Other archs might not have this constraint at all and can hotplug any cpu.
So relying on the fact that cpu 0 cannot be hotplugged is broken.
Re: [PATCH 03/13] rcu: Exclude all potential timekeepers from sysidle detection
In reply to Peter ZijlstraQuoted 4 lines · Peter Zijlstra
Quoted 1 line · Frederic Weisbecker
That's to start simple as CPU 0 can't be offlined.
Yes it can, see
CONFIG_BOOTPARAM_HOTPLUG_CPU0
Ah, and cpu0_hotplug boot parameter even if CONFIG_BOOTPARAM_HOTPLUG_CPU0=n.
Good to know!
Thanx, Paul
Re: [PATCH 03/13] rcu: Exclude all potential timekeepers from sysidle detection
In reply to Peter ZijlstraQuoted 4 lines · Peter Zijlstra
Quoted 1 line · Frederic Weisbecker
That's to start simple as CPU 0 can't be offlined.
Yes it can, see
CONFIG_BOOTPARAM_HOTPLUG_CPU0
Yeah it can but the tick hotplug callback prevent that to happen on nohz full. You can check out tick_nohz_cpu_down_callback() for details.
I agree that rejecting CPU_DOWN_PREPARE requests is propably not the prettiest thing ever but that's the only solution that we have now.
There must _always_ be a timekeeper to maintain timekeeping duty as long as there are full dynticks CPUs around. So one of these potential timekeepers has to stay online. And CPU 0 is quite a handy candidate since rejecting any other CPU's offlining request may block suspend.
Now if it happens to be annoying for some reason, we can certainly change that behaviour in the future. This will involve forcing some full dynticks CPU to carry timekeeping but heh, if user does silly things :)
PATCH 4/13 tick: Use timekeeping_cpu() to elect the CPU handling timekeeping duty
In reply to Frederic WeisbeckerNow that we have an API to determine if a CPU is allowed to handle timekeeping duty, use it now on timekeeper selection time for clarity.
--- kernel/time/tick-common.c | 2 +- kernel/time/tick-sched.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
kernel/time/tick-common.c +1 -1
@@ -163,7 +163,7 @@ static void tick_setup_device(struct tick_device *td,* this cpu: */ if (tick_do_timer_cpu ==TICK_DO_TIMER_BOOT ) { -if (!tick_nohz_full_cpu (cpu)) +if (tick_timekeeping_cpu (cpu)) tick_do_timer_cpu = cpu;else tick_do_timer_cpu =TICK_DO_TIMER_NONE ;
kernel/time/tick-sched.c +1 -1
@@ -118,7 +118,7 @@ static void tick_sched_do_timer(ktime_t now)* jiffies_lock. */ if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE ) - && !tick_nohz_full_cpu (cpu)) + &&tick_timekeeping_cpu (cpu)) tick_do_timer_cpu = cpu;#endif
Signature · 1 line
1.8.3.1
Re: [PATCH 04/13] tick: Use timekeeping_cpu() to elect the CPU handling timekeeping duty
Quoted 12 lines · Frederic Weisbecker
Now that we have an API to determine if a CPU is allowed to handle timekeeping duty, use it now on timekeeper selection time for clarity.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>
Quoted 34 lines
--- kernel/time/tick-common.c | 2 +- kernel/time/tick-sched.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)kernel/time/tick-common.c +1
-1@@ -163,7 +163,7 @@ static void tick_setup_device(struct tick_device *td,* this cpu: */ if (tick_do_timer_cpu ==TICK_DO_TIMER_BOOT ) { -if (!tick_nohz_full_cpu (cpu)) +if (tick_timekeeping_cpu (cpu)) tick_do_timer_cpu = cpu;else tick_do_timer_cpu =TICK_DO_TIMER_NONE ;kernel/time/tick-sched.c +1
-1@@ -118,7 +118,7 @@ static void tick_sched_do_timer(ktime_t now)* jiffies_lock. */ if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE ) - && !tick_nohz_full_cpu (cpu)) + &&tick_timekeeping_cpu (cpu)) tick_do_timer_cpu = cpu;#endif Signature · 1 line
1.8.3.1
PATCH 5/13 rcu: Fix unraised IPI to timekeeping CPU
In reply to Frederic WeisbeckerThe plan with full system idle detection is to allow the timekeeper to sleep when all full dynticks CPUs are sleeping.
Then when a full dynticks CPU wakes up while the whole system is idle, it sends an IPI to the timekeeping CPU which then restarts its tick and polls on its timekeeping duty on behalf of all other CPUs in the system.
But we are using rcu_kick_nohz_cpu() to raise this IPI, which is wrong because this function is used to kick full dynticks CPUs when they run in the kernel for too long without reporting a quiescent state. And this function ignores targets that are not full dynticks, like our timekeeper.
To fix this, use the smp_send_reschedule() function directly.
--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -rcu_kick_nohz_cpu (tick_do_timer_cpu); +smp_send_reschedule (tick_do_timer_cpu);return ;/* We cleared it, done! */ } oldstate = newoldstate;
Signature · 1 line
1.8.3.1
Re: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 8 lines · Frederic Weisbecker
The plan with full system idle detection is to allow the timekeeper to sleep when all full dynticks CPUs are sleeping.
Then when a full dynticks CPU wakes up while the whole system is idle, it sends an IPI to the timekeeping CPU which then restarts its tick and polls on its timekeeping duty on behalf of all other CPUs in the system.
Quoted 7 lines
But we are using
rcu_kick_nohz_cpu()to raise this IPI, which is wrong because this function is used to kick full dynticks CPUs when they run in the kernel for too long without reporting a quiescent state. And this function ignores targets that are not full dynticks, like our timekeeper.To fix this, use the
smp_send_reschedule()function directly.
I guess the fact that you needed some change is reassuring. You know the old saying, "no bugs, no users". ;-)
Quoted 18 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree_plugin.h +0
-0
kernel/rcu/tree_plugin.h void rcu_sysidle_force_exit(void) · line 2491
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -rcu_kick_nohz_cpu (tick_do_timer_cpu); +smp_send_reschedule (tick_do_timer_cpu);
Hmmm...
We haven't done any sort of wakeup, and tick_nohz_full_cpu() should return false for tick_do_timer_cpu, and I don't see that we have done anything to make got_nohz_idle_kick() return true.
So the idea is that the fact of the interrupt is sufficient, and the target CPU will figure out that it must turn its scheduling-clock interrupt when returning from interrupt?
Or is something else going on here?
Thanx, Paul
Quoted 6 lines
kernel/rcu/tree_plugin.h +0
-0return ;/* We cleared it, done! */ } oldstate = newoldstate;Signature · 1 line
1.8.3.1
Re: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 15 lines · Paul E. McKenney
Quoted 14 lines · Frederic Weisbecker
The plan with full system idle detection is to allow the timekeeper to sleep when all full dynticks CPUs are sleeping.
Then when a full dynticks CPU wakes up while the whole system is idle, it sends an IPI to the timekeeping CPU which then restarts its tick and polls on its timekeeping duty on behalf of all other CPUs in the system.
But we are using
rcu_kick_nohz_cpu()to raise this IPI, which is wrong because this function is used to kick full dynticks CPUs when they run in the kernel for too long without reporting a quiescent state. And this function ignores targets that are not full dynticks, like our timekeeper.
Quoted 4 lines
Quoted 1 line
To fix this, use the
smp_send_reschedule()function directly.I guess the fact that you needed some change is reassuring. You know the old saying, "no bugs, no users". ;-)
;-)
Quoted 27 lines
Quoted 19 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree_plugin.h +0
-0kernel/rcu/tree_plugin.h void rcu_sysidle_force_exit(void) · line 2491
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -rcu_kick_nohz_cpu (tick_do_timer_cpu); +smp_send_reschedule (tick_do_timer_cpu);Hmmm...
Quoted 7 lines
We haven't done any sort of wakeup, and
tick_nohz_full_cpu()should return false fortick_do_timer_cpu, and I don't see that we have done anything to makegot_nohz_idle_kick()return true.So the idea is that the fact of the interrupt is sufficient, and the target CPU will figure out that it must turn its scheduling-clock interrupt when returning from interrupt?
Exactly, the interrupt alone is sufficient and the tick is reevaluated on irq_exit().
Quoted 12 lines
Or is something else going on here?
Thanx, Paul
Quoted 6 lines
kernel/rcu/tree_plugin.h +0
-0return ;/* We cleared it, done! */ } oldstate = newoldstate;Signature · 1 line
1.8.3.1
Re: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 54 lines · Frederic Weisbecker
Quoted 15 lines · Paul E. McKenney
Quoted 14 lines · Frederic Weisbecker
The plan with full system idle detection is to allow the timekeeper to sleep when all full dynticks CPUs are sleeping.
Then when a full dynticks CPU wakes up while the whole system is idle, it sends an IPI to the timekeeping CPU which then restarts its tick and polls on its timekeeping duty on behalf of all other CPUs in the system.
But we are using
rcu_kick_nohz_cpu()to raise this IPI, which is wrong because this function is used to kick full dynticks CPUs when they run in the kernel for too long without reporting a quiescent state. And this function ignores targets that are not full dynticks, like our timekeeper.Quoted 4 lines
Quoted 1 line
To fix this, use the
smp_send_reschedule()function directly.I guess the fact that you needed some change is reassuring. You know the old saying, "no bugs, no users". ;-)
;-)
Quoted 31 lines
Quoted 19 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/rcu/tree_plugin.h +0
-0kernel/rcu/tree_plugin.h void rcu_sysidle_force_exit(void) · line 2491
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -rcu_kick_nohz_cpu (tick_do_timer_cpu); +smp_send_reschedule (tick_do_timer_cpu);Hmmm...
We haven't done any sort of wakeup, and
tick_nohz_full_cpu()should return false fortick_do_timer_cpu, and I don't see that we have done anything to makegot_nohz_idle_kick()return true.
Quoted 6 lines
Quoted 3 lines
So the idea is that the fact of the interrupt is sufficient, and the target CPU will figure out that it must turn its scheduling-clock interrupt when returning from interrupt?
Exactly, the interrupt alone is sufficient and the tick is reevaluated on
irq_exit().
But if that is the case, why do you need the change to scheduler_ipi() in patch 07/13? Just having received any sort of IPI should suffice.
Thanx, Paul
Quoted 12 lines
Quoted 11 lines
Or is something else going on here?
Thanx, Paul
Quoted 6 lines
kernel/rcu/tree_plugin.h +0
-0return ;/* We cleared it, done! */ } oldstate = newoldstate;Signature · 1 line
1.8.3.1
Re: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 6 lines · Paul E. McKenney
Quoted 3 lines
Exactly, the interrupt alone is sufficient and the tick is reevaluated on
irq_exit().But if that is the case, why do you need the change to
scheduler_ipi()in patch 07/13? Just having received any sort of IPI should suffice.
Because scheduler_ipi() conditionally calls irq_enter() and irq_exit() (I wonder if that's a good idea btw, is that here to deal with spurious scheduler IPIs of some sort?)
And it's convenient because it can be called anywhere, even when irqs are disabled, unlike kernel/smp.c IPIs.
But I agree it's kind of an abuse of scheduler_ipi() here. Well I'm going to develop that in my answers to Peter as he has concerns precisely about that.
Thanks.
Peter Zijlstra
maintainerRe: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 3 lines · Frederic Weisbecker
Because
scheduler_ipi()conditionally callsirq_enter()andirq_exit()(I wonder if that's a good idea btw, is that here to deal with spurious scheduler IPIs of some sort?)
No its because the traditional scheduler IPI did all the work from the interrupt return path -- and like the comment in there says, for many of the IPIs that's still true.
So going through the architectures and making all scheduler_ipi callers do irq_enter/exit would actually make them slower.
Re: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
Because
scheduler_ipi()conditionally callsirq_enter()andirq_exit()(I wonder if that's a good idea btw, is that here to deal with spurious scheduler IPIs of some sort?)No its because the traditional scheduler IPI did all the work from the interrupt return path -- and like the comment in there says, for many of the IPIs that's still true.
So going through the architectures and making all
scheduler_ipicallers do irq_enter/exit would actually make them slower.
Fair point!
Thanks.
Peter Zijlstra
maintainerRe: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
In reply to Frederic Weisbeckerkernel/rcu/tree_plugin.h void rcu_sysidle_force_exit(void) · line 2491
Quoted 2 lines · Frederic Weisbecker
kernel/rcu/tree_plugin.h +1
-1-rcu_kick_nohz_cpu (tick_do_timer_cpu); +smp_send_reschedule (tick_do_timer_cpu);
Hurm.. so I don't really like this (same for the other patches doing the similar change).
Why not have a function called: wake_time_keeper_cpu(), and have that do the right thing?
Also, afaict, all you want is that remote cpu to run tick_nohz_full_check(), right?
So why are you (ab)using the scheduler_ipi for this at all? Why not have something like:
static DEFINE_PER_CPU(struct call_single_data, nohz_full_csd);
int init_nohz_full(void)
{
int cpu;
for_each_possible_cpu(cpu) {
struct call_single_data *csd = &per_cpu(nohz_full_csd, cpu);
csd->flags = 0;
csd->func = &tick_nohz_full_check;
csd->info = NULL;
}
return 0;
}void wake_time_keeper_cpu(void)
{
int cpu = pick_timekeeper_cpu();
struct single_call_data *csd = &per_cpu(nohz_full_csd, cpu);
__smp_call_function_single(cpu, csd, 0);
}Re: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 6 lines · Peter Zijlstra
kernel/rcu/tree_plugin.h void rcu_sysidle_force_exit(void) · line 2491
Quoted 2 lines · Frederic Weisbecker
kernel/rcu/tree_plugin.h +1
-1-rcu_kick_nohz_cpu (tick_do_timer_cpu); +smp_send_reschedule (tick_do_timer_cpu);Hurm.. so I don't really like this (same for the other patches doing the similar change).
Hehe, I knew you wouldn't like it :)
To be honest I'm not very happy with this change either.
Quoted 6 lines
Why not have a function called:
wake_time_keeper_cpu(), and have that do the right thing?Also, afaict, all you want is that remote cpu to run
tick_nohz_full_check(), right?
Not only. There are two kind of IPIs we are interested in in full dynticks environment:
1) Calls to tick_nohz_full_check() to make full dynticks CPUs to reevaluate their tick. This must be called from IRQ to avoid crazy locking scenarios from deep callers in place like scheduler internals.
So this one is currently implemented by scheduler_ipi() upstream.
2) Timekeeping kick: when a full dynticks CPU wakes up, it needs a housekeeping CPU to run the timekeeping duty on its behalf. But it's possible that all housekeeping CPUs are sleeping, so we need to kick one of them with an IPI.
In this case we don't need to call a specific handler, we are only interested in calling irq_exit(). Although, thinking about it more, having a real handler would make the code less complicated because then we can modify tick_do_timer_cpu locally. And that should simplify a few things.
Quoted 1 line
So why are you (ab)using the
scheduler_ipifor this at all?
So, I've been cowardly using it as a swiss army knife IPI ;-) After the BKL, the BKT (Big kernel timer), the BKI (big kernel IPI) ;-)
Quoted 20 lines
Why not have
something like:static
DEFINE_PER_CPU(struct call_single_data, nohz_full_csd);int
init_nohz_full(void){ int cpu; for_each_possible_cpu(cpu) { struct call_single_data *csd = &per_cpu(nohz_full_csd, cpu); csd->flags = 0; csd->func = &tick_nohz_full_check; csd->info = NULL; } return 0; }
Quoted 7 lines
void
wake_time_keeper_cpu(void){ int cpu = pick_timekeeper_cpu(); struct single_call_data *csd = &per_cpu(nohz_full_csd, cpu); __smp_call_function_single(cpu, csd, 0); }
That's because I can't call smp_call_function_single() with interrupts disabled.
BTW this warning in __smp_call_function_single() looks wrong:
WARN_ON_ONCE(cpu_online(smp_processor_id()) && wait && irqs_disabled()
&& !oops_in_progress)I think we want to warn on irqs_disabled() even if !wait, because we wait for csd lock anyway before raising the IPI.
Anyway so what I need is an IPI that can be raised with irqs disabled. And abusing the scheduler IPI with ugliness after the other is obsviously not what we want.
So I wonder if we could change these smp_call_function_*() to use llist and behave closer to what irq_work does.
Like: enqueue the llist_entry, if it's already enqueued then the remote CPU is going to execute the callback so we are good. But may be we lose some ordering guarantees in this case:
store A exec IPI
raise IPI load A
is perhaps not guaranteed anymore if the callback is already queued when we want to add it and thus we don't raise the IPI. Althouth llist_add() failure perhaps brings the atomic + full barrier that we need. So on the other side, llist_get() provides the atomic + barrier that are needed such that the remote CPU sees our latest state.
Now may be I'm missing something obvious there, but that would allow us to drop that csd thing.
I think that Christophe Hellwig had patches to drop csd and make smp.c lockless, I don't recall the details though.
Re: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
Quoted 2 lines · Frederic Weisbecker
I think that Christophe Hellwig had patches to drop csd and make
smp.clockless, I don't recall the details though.
I had a patch to use llists, but you still have the csd_lock opencoded lock. By coincidence Jan Kara just resent the patch a few minutes ago.
Peter Zijlstra
maintainerRe: [PATCH 05/13] rcu: Fix unraised IPI to timekeeping CPU
In reply to Frederic WeisbeckerQuoted 10 lines · Frederic Weisbecker
BTW this warning in
__smp_call_function_single()looks wrong:WARN_ON_ONCE(cpu_online(smp_processor_id()) && wait && irqs_disabled() && !oops_in_progress)I think we want to warn on
irqs_disabled()even if !wait, because we wait for csd lock anyway before raising the IPI.Anyway so what I need is an IPI that can be raised with irqs disabled. And abusing the scheduler IPI with ugliness after the other is obsviously not what we want.
Ah indeed, so you can actually make it work but its a little more involved.
struct foo {
struct call_single_data csd;
unsigned int msg;
};static DEFINE_PER_CPU_ALIGNED(struct foo, foo_csd);
/*
* set @bit in @msg, return true if its the first bit set */
static bool foo_set_msg(unsigned int *msg, int bit)
{
unsigned int old_msg, new_msg;
for (;;) {
old_msg = new_msg = *msg;
new_msg |= 1U << bit;
if (cmpxchg(msg, old_msg, new_msg) == old_msg)
break;
}
return old_msg == 0;
}/*
* clear @bit in @msg, return true if its the last bit cleared */
static bool foo_clear_msg(unsigned int *msg, int bit)
{
unsigned int old_msg, new_msg;
for (;;) {
old_msg = new_msg = *msg;
new_msg &= ~(1U << bit);
if (cmpxchg(msg, old_msg, new_msg) == old_msg)
break;
}
return new_msg == 0;
}/*
* handle @bit msg
*/
static void foo_handle_msg(int bit)
{
switch (bit) {
case 0:
/* .... */
break;
/* ... */
}
}static void foo_call(void *info)
{
struct foo *foo_csd = info;
int bit;
for (;;) {
for (bit = 0; bit < 32; bit++) {
if (ACCESS_ONCE(foo_csd->msg) & (1U << bit)) {
foo_handle_msg(bit);
if (foo_clear_msg(bit))
return;
}
}
}
}int foo_init(void)
{
int cpu;
for_each_possible_cpu(cpu) {
struct foo *foo_csd = &per_cpu(foo_csd, cpu);
foo_csd->csd.flags = 0;
foo_csd->csd.func = foo_call;
foo_csd->csd.info = foo_csd;
foo_csd->msg = 0;
}
return 0;
}void foo_msg(int cpu, int bit)
{
struct foo *foo_csd = &per_cpu(foo_csd, cpu);
/*
* Only send the IPI if this is the first msg bit set;
* otherwise the handler is already pending/running and
* will pick it up.
*
* This will not deadlock on csd_lock because the atomic
* msg manipulation ensures there's only ever one csd per cpu
* active.
*/
if (foo_set_msg(&foo_csd->msg, bit))
__smp_call_function_single(cpu, &foo_csd->csd, 0);
}PATCH 6/13 nohz: Introduce full dynticks' default timekeeping target
In reply to Frederic WeisbeckerWhen a full dynticks CPU wakes up while the whole rest of the system is idle, we need to wake up the CPU in charge of the timekeeping duty handling.
As of today, the CPU that maintains this duty is CPU 0 when CONFIG_NO_HZ_FULL=y. So referring to tick_do_timer_cpu like we currently do is correct. But this behaviour is subject to change in the future because we want to balance the timekeeping duty over all the CPUs outside the full dynticks range.
As such we now need to define a default timekeeping CPU which receives the timekeeping wakeup IPIs and which can't be offlined so that it's guaranteed to always be present for full dynticks CPUs housekeeping.
So lets stick to CPU 0 for this purpose. It's convenient because rejecting any other CPU's offlining request may result in suspend failure.
We can optimize this solution later by adaptively sending the IPI to a potential timekeeping CPU that is already running a non idle task.
--- include/linux/tick.h | 16 ++++++++++++++++ kernel/rcu/tree_plugin.h | 4 ++-- kernel/time/tick-sched.c | 2 +- 3 files changed, 19 insertions(+), 3 deletions(-)
include/linux/tick.h +16 -0
@@ -180,6 +180,22 @@ static inline bool tick_nohz_full_cpu(int cpu) }/** +* tick_timekeeping_default_cpu - seek timekeeping default CPU + +* @return the default target which we send an IPI to +* when a full dynticks CPU wakes up and exits from full +* system idle state. +* +* This target is always CPU 0 in full dynticks environment. +* If we were to pick up any other CPU, that would result in suspend +* failures due to rejected offlining request. +*/ +static inline int tick_timekeeping_default_cpu (void ) +{ +return 0 ; +} + +/** * tick_timeeping_cpu - check if a CPU is elligble to handle timekeeping duty * @cpu: the cpu to check *
kernel/rcu/tree_plugin.h +2 -2
@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -smp_send_reschedule (tick_do_timer_cpu); +smp_send_reschedule (tick_timekeeping_default_cpu ());return ;/* We cleared it, done! */ } oldstate = newoldstate;
@@ -2597,7 +2597,7 @@ static bool is_sysidle_rcu_state(struct rcu_state *rsp)*/ static void rcu_bind_gp_kthread (void ) { -int cpu =ACCESS_ONCE (tick_do_timer_cpu); +int cpu =tick_timekeeping_default_cpu ();if (cpu <0 || cpu >= nr_cpu_ids)return ;
kernel/time/tick-sched.c +1 -1
@@ -305,7 +305,7 @@ static int tick_nohz_cpu_down_callback(struct notifier_block *nfb,* If we handle the timekeeping duty for full dynticks CPUs, * we can't safely shutdown that CPU. */ -if (tick_nohz_full_running && tick_do_timer_cpu == cpu) +if (tick_nohz_full_running &&tick_timekeeping_default_cpu () == cpu)return NOTIFY_BAD ;break ; }
Signature · 1 line
1.8.3.1
Re: [PATCH 06/13] nohz: Introduce full dynticks' default timekeeping target
Quoted 10 lines · Frederic Weisbecker
When a full dynticks CPU wakes up while the whole rest of the system is idle, we need to wake up the CPU in charge of the timekeeping duty handling.
As of today, the CPU that maintains this duty is CPU 0 when
CONFIG_NO_HZ_FULL=y. So referring totick_do_timer_cpulike we currently do is correct. But this behaviour is subject to change in the future because we want to balance the timekeeping duty over all the CPUs outside the full dynticks range.
Quoted 7 lines
As such we now need to define a default timekeeping CPU which receives the timekeeping wakeup IPIs and which can't be offlined so that it's guaranteed to always be present for full dynticks CPUs housekeeping.
So lets stick to CPU 0 for this purpose. It's convenient because rejecting any other CPU's offlining request may result in suspend failure.
OK, so not multiple CPUs yet. Whew!
Well, at least you know some of my concerns with multiple timekeeping CPUs beforehand, which would not have happened had I reviewed the patches in order. ;-)
Thanx, Paul
Quoted 83 lines
We can optimize this solution later by adaptively sending the IPI to a potential timekeeping CPU that is already running a non idle task.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- include/linux/tick.h | 16 ++++++++++++++++ kernel/rcu/tree_plugin.h | 4 ++-- kernel/time/tick-sched.c | 2 +- 3 files changed, 19 insertions(+), 3 deletions(-)include/linux/tick.h +16
-0@@ -180,6 +180,22 @@ static inline bool tick_nohz_full_cpu(int cpu) }/** +* tick_timekeeping_default_cpu - seek timekeeping default CPU + +* @return the default target which we send an IPI to +* when a full dynticks CPU wakes up and exits from full +* system idle state. +* +* This target is always CPU 0 in full dynticks environment. +* If we were to pick up any other CPU, that would result in suspend +* failures due to rejected offlining request. +*/ +static inline int tick_timekeeping_default_cpu (void ) +{ +return 0 ; +} + +/** * tick_timeeping_cpu - check if a CPU is elligble to handle timekeeping duty * @cpu: the cpu to check * kernel/rcu/tree_plugin.h +2
-2@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -smp_send_reschedule (tick_do_timer_cpu); +smp_send_reschedule (tick_timekeeping_default_cpu ());return ;/* We cleared it, done! */ } oldstate = newoldstate;@@ -2597,7 +2597,7 @@ static bool is_sysidle_rcu_state(struct rcu_state *rsp)*/ static void rcu_bind_gp_kthread (void ) { -int cpu =ACCESS_ONCE (tick_do_timer_cpu); +int cpu =tick_timekeeping_default_cpu ();if (cpu <0 || cpu >= nr_cpu_ids)return ;kernel/time/tick-sched.c +1
-1@@ -305,7 +305,7 @@ static int tick_nohz_cpu_down_callback(struct notifier_block *nfb,* If we handle the timekeeping duty for full dynticks CPUs, * we can't safely shutdown that CPU. */ -if (tick_nohz_full_running && tick_do_timer_cpu == cpu) +if (tick_nohz_full_running &&tick_timekeeping_default_cpu () == cpu)return NOTIFY_BAD ;break ; }Signature · 1 line
1.8.3.1
PATCH 7/13 sched: Enable IPI reception on timekeeper under nohz full system
In reply to Frederic WeisbeckerWe need the default timekeeping CPU to be able to receive IPIs sent from full dynticks CPUs when they wake up from full system idle state.
Therefore we need an entrypoint from the scheduler IPI so that the need to poll on timekeeping duty is re-evaluated from irq_exit().
In order to achieve this, lets take the scheduler IPI everytime as long as there is at least one full dynticks CPU around. Full dynticks CPUs are interested too in taking scheduler IPIs to reevaluate their tick.
--- kernel/sched/core.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/sched/core.c +3 -3
@@ -1502,9 +1502,9 @@ void scheduler_ipi(void)if (tif_need_resched ())set_preempt_need_resched (); -if (llist_empty (&this_rq ()->wake_list) - && !tick_nohz_full_cpu (smp_processor_id ()) - && !got_nohz_idle_kick ()) +if (llist_empty (&this_rq ()->wake_list) && + !tick_nohz_full_enabled () && + !got_nohz_idle_kick ())return ;/*
Signature · 1 line
1.8.3.1
Re: [PATCH 07/13] sched: Enable IPI reception on timekeeper under nohz full system
Quoted 28 lines · Frederic Weisbecker
We need the default timekeeping CPU to be able to receive IPIs sent from full dynticks CPUs when they wake up from full system idle state.
Therefore we need an entrypoint from the scheduler IPI so that the need to poll on timekeeping duty is re-evaluated from
irq_exit().In order to achieve this, lets take the scheduler IPI everytime as long as there is at least one full dynticks CPU around. Full dynticks CPUs are interested too in taking scheduler IPIs to reevaluate their tick.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/sched/core.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c void scheduler_ipi(void) · line 1508
Quoted 10 lines
kernel/sched/core.c +3
-3@@ -1502,9 +1502,9 @@ void scheduler_ipi(void)if (tif_need_resched ())set_preempt_need_resched (); -if (llist_empty (&this_rq ()->wake_list) - && !tick_nohz_full_cpu (smp_processor_id ()) - && !got_nohz_idle_kick ()) +if (llist_empty (&this_rq ()->wake_list) && + !tick_nohz_full_enabled () && + !got_nohz_idle_kick ())return ;
OK, this is what I was missing in my question about whether the NO_HZ_FULL state was re-evaluated in the interrupt-return path.
Thanx, Paul
Quoted 4 lines
kernel/sched/core.c +0
-0/* Signature · 1 line
1.8.3.1
Re: [PATCH 07/13] sched: Enable IPI reception on timekeeper under nohz full system
Quoted 32 lines · Paul E. McKenney
Quoted 31 lines · Frederic Weisbecker
We need the default timekeeping CPU to be able to receive IPIs sent from full dynticks CPUs when they wake up from full system idle state.
Therefore we need an entrypoint from the scheduler IPI so that the need to poll on timekeeping duty is re-evaluated from
irq_exit().In order to achieve this, lets take the scheduler IPI everytime as long as there is at least one full dynticks CPU around. Full dynticks CPUs are interested too in taking scheduler IPIs to reevaluate their tick.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/sched/core.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/sched/core.c +0
-0@@ -1502,9 +1502,9 @@ void scheduler_ipi(void)if (tif_need_resched ())set_preempt_need_resched ();
Quoted 10 lines
kernel/sched/core.c void scheduler_ipi(void) · line 1508
Quoted 7 lines
kernel/sched/core.c +3
-3-if (llist_empty (&this_rq ()->wake_list) - && !tick_nohz_full_cpu (smp_processor_id ()) - && !got_nohz_idle_kick ()) +if (llist_empty (&this_rq ()->wake_list) && + !tick_nohz_full_enabled () && + !got_nohz_idle_kick ())return ;OK, this is what I was missing in my question about whether the
NO_HZ_FULLstate was re-evaluated in the interrupt-return path.
I tend to write my patchset by splitting every single logical bricks and then only in the end I enable the feature.
But that makes a tradeoff between patchset granularity and global overview. And in the end, may be it's unbalanced toward overview.
Notwithstanding bisectability.
I remember I had similar reactions when I posted the initial full nohz patchset.
OTOH it's not good to have big all-in-one patches. And granular patchsets like this are better to focus discussions on each isolated topics.
There is a hard balance to find out here.
Quoted 7 lines
Thanx, Paul
Quoted 4 lines
kernel/sched/core.c +0
-0/* Signature · 1 line
1.8.3.1
Re: [PATCH 07/13] sched: Enable IPI reception on timekeeper under nohz full system
Quoted 50 lines · Frederic Weisbecker
Quoted 32 lines · Paul E. McKenney
Quoted 31 lines · Frederic Weisbecker
We need the default timekeeping CPU to be able to receive IPIs sent from full dynticks CPUs when they wake up from full system idle state.
Therefore we need an entrypoint from the scheduler IPI so that the need to poll on timekeeping duty is re-evaluated from
irq_exit().In order to achieve this, lets take the scheduler IPI everytime as long as there is at least one full dynticks CPU around. Full dynticks CPUs are interested too in taking scheduler IPIs to reevaluate their tick.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/sched/core.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/sched/core.c +0
-0@@ -1502,9 +1502,9 @@ void scheduler_ipi(void)if (tif_need_resched ())set_preempt_need_resched ();Quoted 10 lines
kernel/sched/core.c void scheduler_ipi(void) · line 1508
Quoted 7 lines
kernel/sched/core.c +3
-3-if (llist_empty (&this_rq ()->wake_list) - && !tick_nohz_full_cpu (smp_processor_id ()) - && !got_nohz_idle_kick ()) +if (llist_empty (&this_rq ()->wake_list) && + !tick_nohz_full_enabled () && + !got_nohz_idle_kick ())return ;OK, this is what I was missing in my question about whether the
NO_HZ_FULLstate was re-evaluated in the interrupt-return path.I tend to write my patchset by splitting every single logical bricks and then only in the end I enable the feature.
But that makes a tradeoff between patchset granularity and global overview. And in the end, may be it's unbalanced toward overview.
Quoted 8 lines
Notwithstanding bisectability.
I remember I had similar reactions when I posted the initial full nohz patchset.
OTOH it's not good to have big all-in-one patches. And granular patchsets like this are better to focus discussions on each isolated topics.
There is a hard balance to find out here.
It is OK. I have no idea what possessed me to review the patches in reverse order. I normally review in forward order.
Thanx, Paul
Peter Zijlstra
maintainerRe: [PATCH 07/13] sched: Enable IPI reception on timekeeper under nohz full system
In reply to Frederic WeisbeckerQuoted 9 lines · Frederic Weisbecker
We need the default timekeeping CPU to be able to receive IPIs sent from full dynticks CPUs when they wake up from full system idle state.
Therefore we need an entrypoint from the scheduler IPI so that the need to poll on timekeeping duty is re-evaluated from
irq_exit().In order to achieve this, lets take the scheduler IPI everytime as long as there is at least one full dynticks CPU around. Full dynticks CPUs are interested too in taking scheduler IPIs to reevaluate their tick.
I should probably read the other patches in this series, but the above makes my brain go *beeeep* and throw an error.
PATCH 8/13 nohz: Get timekeeping max deferment outside jiffies_lock
In reply to Frederic WeisbeckerWe don't need to fetch the timekeeping max deferment under the jiffies_lock seqlock.
If the clocksource is updated concurrently while we stop the tick, stop machine is called and the tick will be reevaluated again along with uptodate jiffies and its related values.
--- kernel/time/tick-sched.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/time/tick-sched.c +2 -1
@@ -532,12 +532,13 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta; + time_delta =timekeeping_max_deferment (); +/* Read jiffies and the time when jiffies were updated last */ do { seq =read_seqbegin (&jiffies_lock); last_update = last_jiffies_update; last_jiffies = jiffies; - time_delta =timekeeping_max_deferment (); }while (read_seqretry (&jiffies_lock, seq)); if (rcu_needs_cpu(cpu, &rcu_delta_jiffies) ||
Signature · 1 line
1.8.3.1
tip-bot for Frederic Weisbecker
merge bot[tip:timers/urgent] nohz: Get timekeeping max deferment outside jiffies_lock
nohz: Get timekeeping max deferment outside jiffies_lock
Commit-ID: 855a0fc30b70d6ae681badd24d6625f9a9abb787 Gitweb: http://git.kernel.org/tip/855a0fc30b70d6ae681badd24d6625f9a9abb787 Author: Frederic Weisbecker <[email protected]> AuthorDate: Tue, 17 Dec 2013 00:16:37 +0100 Committer: Frederic Weisbecker <[email protected]> CommitDate: Wed, 15 Jan 2014 23:07:11 +0100
nohz: Get timekeeping max deferment outside jiffies_lock
We don't need to fetch the timekeeping max deferment under the jiffies_lock seqlock.
If the clocksource is updated concurrently while we stop the tick, stop machine is called and the tick will be reevaluated again along with uptodate jiffies and its related values.
--- kernel/time/tick-sched.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/time/tick-sched.c +2 -1
@@ -533,12 +533,13 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta; + time_delta =timekeeping_max_deferment (); +/* Read jiffies and the time when jiffies were updated last */ do { seq =read_seqbegin (&jiffies_lock); last_update = last_jiffies_update; last_jiffies = jiffies; - time_delta =timekeeping_max_deferment (); }while (read_seqretry (&jiffies_lock, seq)); if (rcu_needs_cpu(cpu, &rcu_delta_jiffies) ||
PATCH 9/13 nohz: Allow timekeeper's tick to stop when all full dynticks CPUs are idle
In reply to Frederic WeisbeckerWhen all full dynticks CPUs are idle, as detected by RCU's sysidle detection, there is no need to keep the timekeeping CPU's tick alive anymore. So lets shut it down when we meet this favourable state. The timekeeper will be notified with an IPI if any full dynticks CPU wakes up.
Also, since we plan to allow every CPUs outside the full dynticks range to handle the timekeeping duty, lets also allow the timekeeping duty to be balanced. The only requirement is that the last timekeeper can't shut down its idle tick further than 1 jiffie until some other CPU takes its duty or until all full dynticks CPUs go to sleep.
--- kernel/time/tick-sched.c | 67 ++++++++++++++++++++++++++++++++++++------------ 1 file changed, 50 insertions(+), 17 deletions(-)
kernel/time/tick-sched.c +50 -17
@@ -192,6 +192,49 @@ static bool can_stop_full_tick(void)return true; } +/* +* Fetch max deferment for the current clockevent source until it overflows. +* Also in full dynticks environment, make sure the current timekeeper +* stays periodic until some other CPU can take its timekeeping duty +* or until all full dynticks go to sleep. +*/ +static u64 tick_timekeeping_max_deferment (struct tick_sched *ts) +{ +int cpu; +u64 ret =KTIME_MAX ; + +/* +* Fast path for full dynticks off-case: skip to +* clockevent max deferment +*/ +if (!tick_nohz_full_enabled ()) +return timekeeping_max_deferment (); + + cpu =smp_processor_id (); + +/* Full dynticks CPU don't take timekeeping duty */ +if (!tick_timekeeping_cpu (cpu)) +return timekeeping_max_deferment (); + +/* +* If we are the timekeeper and all full dynticks CPUs are idle, +* then we can finally sleep. +*/ +if (tick_do_timer_cpu == cpu || + (tick_do_timer_cpu ==TICK_DO_TIMER_NONE && ts->do_timer_last ==1 )) { +if (!rcu_sys_is_idle ()) { +/* +* Stop tick for 1 jiffy. In practice we stay periodic +* but that let us possibly delegate our timekeeping duty +* to stop the tick for real in the future. +*/ + ret = tick_period.tv64; + } + } + +return min_t (u64, ret,timekeeping_max_deferment ()); +} +static void tick_nohz_restart_sched_tick (struct tick_sched *ts,ktime_t now);/*
@@ -352,7 +395,12 @@ void __init tick_nohz_init(void)cpulist_scnprintf (nohz_full_buf,sizeof (nohz_full_buf), tick_nohz_full_mask); pr_info("NO_HZ: Full dynticks CPUs: %s.\n", nohz_full_buf); } -#endif +# else /* CONFIG_NO_HZ_FULL */ +static u64 tick_timekeeping_max_deferment (struct tick_sched *ts) +{ +return timekeeping_max_deferment (); +} +#endif /* CONFIG_NO_HZ_FULL */ /* * NOHZ - aka dynamic tick functionality
@@ -532,7 +580,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta; - time_delta =timekeeping_max_deferment (); + time_delta =tick_timekeeping_max_deferment (ts);/* Read jiffies and the time when jiffies were updated last */ do {
@@ -726,21 +774,6 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)return false; } -if (tick_nohz_full_enabled ()) { -/* -* Keep the tick alive to guarantee timekeeping progression -* if there are full dynticks CPUs around -*/ -if (tick_do_timer_cpu == cpu) -return false; -/* -* Boot safety: make sure the timekeeping duty has been -* assigned before entering dyntick-idle mode, -*/ -if (tick_do_timer_cpu ==TICK_DO_TIMER_NONE ) -return false; - } -return true; }
Signature · 1 line
1.8.3.1
Re: [PATCH 09/13] nohz: Allow timekeeper's tick to stop when all full dynticks CPUs are idle
Quoted 11 lines · Frederic Weisbecker
When all full dynticks CPUs are idle, as detected by RCU's sysidle detection, there is no need to keep the timekeeping CPU's tick alive anymore. So lets shut it down when we meet this favourable state. The timekeeper will be notified with an IPI if any full dynticks CPU wakes up.
Also, since we plan to allow every CPUs outside the full dynticks range to handle the timekeeping duty, lets also allow the timekeeping duty to be balanced. The only requirement is that the last timekeeper can't shut down its idle tick further than 1 jiffie until some other CPU takes its duty or until all full dynticks CPUs go to sleep.
Some questions below...
Thanx, Paul
Quoted 42 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/time/tick-sched.c | 67 ++++++++++++++++++++++++++++++++++++------------ 1 file changed, 50 insertions(+), 17 deletions(-)kernel/time/tick-sched.c +21
-0@@ -192,6 +192,49 @@ static bool can_stop_full_tick(void)return true; } +/* +* Fetch max deferment for the current clockevent source until it overflows. +* Also in full dynticks environment, make sure the current timekeeper +* stays periodic until some other CPU can take its timekeeping duty +* or until all full dynticks go to sleep. +*/ +static u64 tick_timekeeping_max_deferment (struct tick_sched *ts) +{ +int cpu; +u64 ret =KTIME_MAX ; + +/* +* Fast path for full dynticks off-case: skip to +* clockevent max deferment +*/ +if (!tick_nohz_full_enabled ()) +return timekeeping_max_deferment (); + + cpu =smp_processor_id (); + +/* Full dynticks CPU don't take timekeeping duty */
kernel/time/tick-sched.c static bool can_stop_full_tick(void) · line 225
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+if (!tick_timekeeping_cpu (cpu)) +return timekeeping_max_deferment (); + +/* +* If we are the timekeeper and all full dynticks CPUs are idle, +* then we can finally sleep. +*/ +if (tick_do_timer_cpu == cpu || + (tick_do_timer_cpu ==TICK_DO_TIMER_NONE && ts->do_timer_last ==1 )) { +if (!rcu_sys_is_idle ()) {
So multiple CPUs could call rcu_sys_is_idle()? Seems like this could happen if tick_do_timer_cpu == TICK_DO_TIMER_NONE. This would be OK only if tick_timekeeping_cpu() returns true for one and only one of the CPUs at any given range of time -- and also that no one calls rcu_sys_is_idle() during a timekeeping CPU handoff.
If two different CPUs call rcu_sys_is_idle() anywhere nearly concurrently on a small system (CONFIG_NO_HZ_FULL_SYSIDLE_SMALL), rcu_sys_is_idle() will break and you will have voided your warranty. ;-)
Also, if tick_timekeeping_cpu() doesn't think that there is a timekeeping CPU, rcu_sys_is_idle() will always return false. I think that this is what you want to happen, just checking.
kernel/time/tick-sched.c static bool can_stop_full_tick(void) · line 232
Quoted 7 lines
kernel/time/tick-sched.c +7
-0+/* +* Stop tick for 1 jiffy. In practice we stay periodic +* but that let us possibly delegate our timekeeping duty +* to stop the tick for real in the future. +*/ + ret = tick_period.tv64; + }
Do we need to set tick_do_timer_cpu to cpu? Or is that handled elsewhere? (If this is the boot-safety feature deleted below, could we please have the comment back here?)
Quoted 56 lines
kernel/time/tick-sched.c +12
-17+ } + +return min_t (u64, ret,timekeeping_max_deferment ()); +} +static void tick_nohz_restart_sched_tick (struct tick_sched *ts,ktime_t now);/* @@ -352,7 +395,12 @@ void __init tick_nohz_init(void)cpulist_scnprintf (nohz_full_buf,sizeof (nohz_full_buf), tick_nohz_full_mask); pr_info("NO_HZ: Full dynticks CPUs: %s.\n", nohz_full_buf); } -#endif +# else /* CONFIG_NO_HZ_FULL */ +static u64 tick_timekeeping_max_deferment (struct tick_sched *ts) +{ +return timekeeping_max_deferment (); +} +#endif /* CONFIG_NO_HZ_FULL */ /* * NOHZ - aka dynamic tick functionality @@ -532,7 +580,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta; - time_delta =timekeeping_max_deferment (); + time_delta =tick_timekeeping_max_deferment (ts);/* Read jiffies and the time when jiffies were updated last */ do {@@ -726,21 +774,6 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)return false; } -if (tick_nohz_full_enabled ()) { -/* -* Keep the tick alive to guarantee timekeeping progression -* if there are full dynticks CPUs around -*/ -if (tick_do_timer_cpu == cpu) -return false; -/* -* Boot safety: make sure the timekeeping duty has been -* assigned before entering dyntick-idle mode, -*/ -if (tick_do_timer_cpu ==TICK_DO_TIMER_NONE ) -return false; - } -return true; }Signature · 1 line
1.8.3.1
Re: [PATCH 09/13] nohz: Allow timekeeper's tick to stop when all full dynticks CPUs are idle
Quoted 30 lines · Paul E. McKenney
Quoted 29 lines · Frederic Weisbecker
kernel/time/tick-sched.c +29
-0+/* +* Fetch max deferment for the current clockevent source until it overflows. +* Also in full dynticks environment, make sure the current timekeeper +* stays periodic until some other CPU can take its timekeeping duty +* or until all full dynticks go to sleep. +*/ +static u64 tick_timekeeping_max_deferment (struct tick_sched *ts) +{ +int cpu; +u64 ret =KTIME_MAX ; + +/* +* Fast path for full dynticks off-case: skip to +* clockevent max deferment +*/ +if (!tick_nohz_full_enabled ()) +return timekeeping_max_deferment (); + + cpu =smp_processor_id (); + +/* Full dynticks CPU don't take timekeeping duty */ +if (!tick_timekeeping_cpu (cpu)) +return timekeeping_max_deferment (); + +/* +* If we are the timekeeper and all full dynticks CPUs are idle, +* then we can finally sleep. +*/ + if (tick_do_timer_cpu == cpu ||
Quoted 8 lines
kernel/time/tick-sched.c static bool can_stop_full_tick(void) · line 225
Quoted 2 lines
kernel/time/tick-sched.c +2
-0+ (tick_do_timer_cpu == TICK_DO_TIMER_NONE && ts->do_timer_last == 1)) { +if (!rcu_sys_is_idle ()) {So multiple CPUs could call
rcu_sys_is_idle()? Seems like this could happen iftick_do_timer_cpu==TICK_DO_TIMER_NONE. This would be OK only iftick_timekeeping_cpu()returns true for one and only one of the CPUs at any given range of time -- and also that no one callsrcu_sys_is_idle()during a timekeeping CPU handoff.
Hmm yeah I fear we can have concurrent callers of this at a same time range.
Quoted 4 lines
If two different CPUs call
rcu_sys_is_idle()anywhere nearly concurrently on a small system (CONFIG_NO_HZ_FULL_SYSIDLE_SMALL),rcu_sys_is_idle()will break and you will have voided your warranty. ;-)
So it breaks because of concurrent state machine stepping on each other toes, right? Like one CPU has reached RCU_SYSIDLE_SHORT and another comes and see only RCU_SYSIDLE_NONE, so it can for example overwite to SHORT while the other CPU can be already far further the cmpxchg() sequence?
Aye, I need to think further on how to cope with that...
Quoted 4 lines
Also, if
tick_timekeeping_cpu()doesn't think that there is a timekeeping CPU,rcu_sys_is_idle()will always return false. I think that this is what you want to happen, just checking.
Ah right but that should be fine. tick_timekeeping_cpu() works for all potential timekeepers. Basically it's !tick_nohz_full_cpu(cpu).
Quoted 12 lines
kernel/time/tick-sched.c static bool can_stop_full_tick(void) · line 232
Quoted 8 lines
kernel/time/tick-sched.c +7
-0+/* +* Stop tick for 1 jiffy. In practice we stay periodic +* but that let us possibly delegate our timekeeping duty +* to stop the tick for real in the future. +*/ + ret = tick_period.tv64; + }Do we need to set
tick_do_timer_cputo cpu? Or is that handled elsewhere? (If this is the boot-safety feature deleted below, could we please have the comment back here?)
This is done in the patch that calls ..kick_timekeeping() from sysidle_exit().
Do you have another case in mind?
Re: [PATCH 09/13] nohz: Allow timekeeper's tick to stop when all full dynticks CPUs are idle
Quoted 40 lines · Frederic Weisbecker
Quoted 30 lines · Paul E. McKenney
Quoted 29 lines · Frederic Weisbecker
kernel/time/tick-sched.c +29
-0+/* +* Fetch max deferment for the current clockevent source until it overflows. +* Also in full dynticks environment, make sure the current timekeeper +* stays periodic until some other CPU can take its timekeeping duty +* or until all full dynticks go to sleep. +*/ +static u64 tick_timekeeping_max_deferment (struct tick_sched *ts) +{ +int cpu; +u64 ret =KTIME_MAX ; + +/* +* Fast path for full dynticks off-case: skip to +* clockevent max deferment +*/ +if (!tick_nohz_full_enabled ()) +return timekeeping_max_deferment (); + + cpu =smp_processor_id (); + +/* Full dynticks CPU don't take timekeeping duty */ +if (!tick_timekeeping_cpu (cpu)) +return timekeeping_max_deferment (); + +/* +* If we are the timekeeper and all full dynticks CPUs are idle, +* then we can finally sleep. +*/ + if (tick_do_timer_cpu == cpu ||Quoted 8 lines
kernel/time/tick-sched.c static bool can_stop_full_tick(void) · line 225
Quoted 2 lines
kernel/time/tick-sched.c +2
-0+ (tick_do_timer_cpu == TICK_DO_TIMER_NONE && ts->do_timer_last == 1)) { +if (!rcu_sys_is_idle ()) {So multiple CPUs could call
rcu_sys_is_idle()? Seems like this could happen iftick_do_timer_cpu==TICK_DO_TIMER_NONE. This would be OK only iftick_timekeeping_cpu()returns true for one and only one of the CPUs at any given range of time -- and also that no one callsrcu_sys_is_idle()during a timekeeping CPU handoff.
Quoted 1 line
Hmm yeah I fear we can have concurrent callers of this at a same time range.
OK, we do need to fix that.
Quoted 8 lines
Quoted 3 lines
If two different CPUs call
rcu_sys_is_idle()anywhere nearly concurrently on a small system (CONFIG_NO_HZ_FULL_SYSIDLE_SMALL),rcu_sys_is_idle()will break and you will have voided your warranty. ;-)So it breaks because of concurrent state machine stepping on each other toes, right? Like one CPU has reached
RCU_SYSIDLE_SHORTand another comes and see onlyRCU_SYSIDLE_NONE, so it can for example overwite to SHORT while the other CPU can be already far further thecmpxchg()sequence?
The ugliest possibility is in the CONFIG_NO_HZ_FULL_SYSIDLE_SMALL case. Three CPUs are independently scanning the CPUs, and by luck, each of these three CPUs see all the CPUs idle, even though CPUs are going idle and non-idle continuously throughout. After all three complete, they each advance the state. The state thus becomes RCU_SYSIDLE_FULL even though there are non-idle CPUs. The timekeeping tick therefore gets shut off even though there are non-idle CPUs. Very bad.
Quoted 1 line
Aye, I need to think further on how to cope with that...
One way would be to cmpxchg the ownership, and only the winner does the call. After that, the winner is tick_do_timer_cpu and no one else does the call. I guess you would want to skip the cmpxchg during early boot, but can't claim to understand your constraints.
Quoted 6 lines
Quoted 3 lines
Also, if
tick_timekeeping_cpu()doesn't think that there is a timekeeping CPU,rcu_sys_is_idle()will always return false. I think that this is what you want to happen, just checking.Ah right but that should be fine.
tick_timekeeping_cpu()works for all potential timekeepers. Basically it's !tick_nohz_full_cpu(cpu).
OK, so we do have to worry about multiple callers. Like you said above. ;-)
Quoted 12 lines
Quoted 11 lines
kernel/time/tick-sched.c static bool can_stop_full_tick(void) · line 232
Quoted 7 lines
kernel/time/tick-sched.c +7
-0+/* +* Stop tick for 1 jiffy. In practice we stay periodic +* but that let us possibly delegate our timekeeping duty +* to stop the tick for real in the future. +*/ + ret = tick_period.tv64; + }Do we need to set
tick_do_timer_cputo cpu? Or is that handled elsewhere? (If this is the boot-safety feature deleted below, could we please have the comment back here?)
Quoted 3 lines
This is done in the patch that calls ..kick_timekeeping() from
sysidle_exit().Do you have another case in mind?
Mostly just that if we did that determination here, we could avoid having multiple callers into the state machine.
Alternatively, I might be able to use a lock to serialize callers, but that would result in CPUs spinning needlessly in rcu_sys_is_idle(). Hence my suggestion of a cmpxchg() to elect a timekeeping CPU in this code.
Thanx, Paul
PATCH 10/13 nohz: Hand over timekeeping duty on cpu offlining
In reply to Frederic WeisbeckerWhen there are full dynticks CPUs around and the timekeeper goes offline, we have to hand over the timekeeping duty to another potential timekeeper.
The default timekeeper (aka CPU 0) is the perfect candidate for this task since it can't be offlined itself.
So lets send an IPI to the default timekeeping when the current timekeeper goes offline, so that the duty is relayed.
--- include/linux/tick.h | 2 ++ kernel/time/tick-sched.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+)
include/linux/tick.h +2 -0
@@ -218,6 +218,7 @@ extern void tick_nohz_init(void);extern void __tick_nohz_full_check (void );extern void tick_nohz_full_kick (void );extern void tick_nohz_full_kick_all (void ); +extern void tick_nohz_full_kick_timekeeping (void );extern void __tick_nohz_task_switch (struct task_struct *tsk);# else static inline void tick_nohz_init (void ) { }
@@ -227,6 +228,7 @@ static inline bool tick_timekeeping_cpu(int cpu) { return true; }static inline void __tick_nohz_full_check (void ) { }static inline void tick_nohz_full_kick (void ) { }static inline void tick_nohz_full_kick_all (void ) { } +static inline void tick_nohz_full_kick_timekeeping (void ) { }static inline void __tick_nohz_task_switch (struct task_struct *tsk) { }#endif
kernel/time/tick-sched.c +31 -0
@@ -217,6 +217,12 @@ static u64 tick_timekeeping_max_deferment(struct tick_sched *ts)return timekeeping_max_deferment ();/* +* Order tick_do_timer_cpu read against the IPI, pairs with +* tick_nohz_full_kick_timekeeping() +*/ +smp_rmb (); + +/* * If we are the timekeeper and all full dynticks CPUs are idle, * then we can finally sleep. */
@@ -293,6 +299,22 @@ void tick_nohz_full_kick_all(void)preempt_enable (); } +/** +* tick_nohz_full_kick_timekeeping - kick the default timekeeper +* +* kick the default timekeeper when a secondary timekeeper goes offline. +*/ +void tick_nohz_full_kick_timekeeping (void ) +{ + tick_do_timer_cpu =tick_timekeeping_default_cpu (); +/* +* Order tick_do_timer_cpu against the IPI, pairs with +* tick_timekeeping_max_deferment on irq exit. +*/ +smp_wmb (); +smp_send_reschedule (tick_timekeeping_default_cpu ()); +} +/* * Re-evaluate the need for the tick as we switch the current task. * It might need the tick due to per task/process properties:
@@ -351,6 +373,15 @@ static int tick_nohz_cpu_down_callback(struct notifier_block *nfb,if (tick_nohz_full_running &&tick_timekeeping_default_cpu () == cpu)return NOTIFY_BAD ;break ; + +case CPU_DYING : +/* +* Notify default timekeeper if we are giving up +* timekeeping duty +*/ +if (tick_nohz_full_running && tick_do_timer_cpu == cpu) +tick_nohz_full_kick_timekeeping (); +break ; }return NOTIFY_OK ; }
Signature · 1 line
1.8.3.1
Re: [PATCH 10/13] nohz: Hand over timekeeping duty on cpu offlining
Quoted 9 lines · Frederic Weisbecker
When there are full dynticks CPUs around and the timekeeper goes offline, we have to hand over the timekeeping duty to another potential timekeeper.
The default timekeeper (aka CPU 0) is the perfect candidate for this task since it can't be offlined itself.
So lets send an IPI to the default timekeeping when the current timekeeper goes offline, so that the duty is relayed.
A few comments below.
Thanx, Paul
Quoted 39 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- include/linux/tick.h | 2 ++ kernel/time/tick-sched.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+)include/linux/tick.h +2
-0@@ -218,6 +218,7 @@ extern void tick_nohz_init(void);extern void __tick_nohz_full_check (void );extern void tick_nohz_full_kick (void );extern void tick_nohz_full_kick_all (void ); +extern void tick_nohz_full_kick_timekeeping (void );extern void __tick_nohz_task_switch (struct task_struct *tsk);# else static inline void tick_nohz_init (void ) { }@@ -227,6 +228,7 @@ static inline bool tick_timekeeping_cpu(int cpu) { return true; }static inline void __tick_nohz_full_check (void ) { }static inline void tick_nohz_full_kick (void ) { }static inline void tick_nohz_full_kick_all (void ) { } +static inline void tick_nohz_full_kick_timekeeping (void ) { }static inline void __tick_nohz_task_switch (struct task_struct *tsk) { }#endif kernel/time/tick-sched.c +0
-0
kernel/time/tick-sched.c static u64 tick_timekeeping_max_deferment(struct tick_sched *ts) · line 223
Quoted 7 lines
kernel/time/tick-sched.c +4
-0@@ -217,6 +217,12 @@ static u64 tick_timekeeping_max_deferment(struct tick_sched *ts)return timekeeping_max_deferment ();/* +* Order tick_do_timer_cpu read against the IPI, pairs with +* tick_nohz_full_kick_timekeeping() +*/ +smp_rmb ();
If this is the handler for the smp_send_reschedule(), then the above memory barrier is not needed. (See my comment below.)
Quoted 12 lines
kernel/time/tick-sched.c +5
-0+ +/* * If we are the timekeeper and all full dynticks CPUs are idle, * then we can finally sleep. */ @@ -293,6 +299,22 @@ void tick_nohz_full_kick_all(void)preempt_enable (); } +/** +* tick_nohz_full_kick_timekeeping - kick the default timekeeper +*
kernel/time/tick-sched.c void tick_nohz_full_kick_all(void) · line 314
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+* kick the default timekeeper when a secondary timekeeper goes offline. +*/ +void tick_nohz_full_kick_timekeeping (void ) +{ + tick_do_timer_cpu =tick_timekeeping_default_cpu (); +/* +* Order tick_do_timer_cpu against the IPI, pairs with +* tick_timekeeping_max_deferment on irq exit. +*/ +smp_wmb ();
But the IPI is supposed to provide full ordering between the CPU invoking the IPI and the IPI handler, right? I do not believe that you need the above smp_wmb() -- though keeping the comment stating that you are relying on the implicit barrier in IPI would be good.
kernel/time/tick-sched.c void tick_nohz_full_kick_all(void) · line 315
Quoted 1 line
kernel/time/tick-sched.c +1
-0+smp_send_reschedule (tick_timekeeping_default_cpu ());
Again, smp_send_reschedule()'s IPI hander does not necessarily do anything if there is nothing for the scheduler to do, so any needed actions are taking in the return-from-interrupt code?
Quoted 24 lines
kernel/time/tick-sched.c +11
-0+} +/* * Re-evaluate the need for the tick as we switch the current task. * It might need the tick due to per task/process properties: @@ -351,6 +373,15 @@ static int tick_nohz_cpu_down_callback(struct notifier_block *nfb,if (tick_nohz_full_running &&tick_timekeeping_default_cpu () == cpu)return NOTIFY_BAD ;break ; + +case CPU_DYING : +/* +* Notify default timekeeper if we are giving up +* timekeeping duty +*/ +if (tick_nohz_full_running && tick_do_timer_cpu == cpu) +tick_nohz_full_kick_timekeeping (); +break ; }return NOTIFY_OK ; }Signature · 1 line
1.8.3.1
Re: [PATCH 10/13] nohz: Hand over timekeeping duty on cpu offlining
Quoted 25 lines · Paul E. McKenney
kernel/time/tick-sched.c static u64 tick_timekeeping_max_deferment(struct tick_sched *ts) · line 223
Quoted 12 lines
kernel/time/tick-sched.c +4
-0@@ -217,6 +217,12 @@ static u64 tick_timekeeping_max_deferment(struct tick_sched *ts)return timekeeping_max_deferment ();/* +* Order tick_do_timer_cpu read against the IPI, pairs with +* tick_nohz_full_kick_timekeeping() +*/ +smp_rmb ();If this is the handler for the
smp_send_reschedule(), then the above memory barrier is not needed. (See my comment below.)Quoted 9 lines
kernel/time/tick-sched.c +2
-0+ +/* * If we are the timekeeper and all full dynticks CPUs are idle, * then we can finally sleep. */ @@ -293,6 +299,22 @@ void tick_nohz_full_kick_all(void)preempt_enable (); }
Quoted 16 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_all(void) · line 314
Quoted 13 lines
kernel/time/tick-sched.c +13
-0+/** +* tick_nohz_full_kick_timekeeping - kick the default timekeeper +* +* kick the default timekeeper when a secondary timekeeper goes offline. +*/ +void tick_nohz_full_kick_timekeeping (void ) +{ + tick_do_timer_cpu =tick_timekeeping_default_cpu (); +/* +* Order tick_do_timer_cpu against the IPI, pairs with +* tick_timekeeping_max_deferment on irq exit. +*/ +smp_wmb ();But the IPI is supposed to provide full ordering between the CPU invoking the IPI and the IPI handler, right?
Great! I remember you told me so once but I wasn't sure. Thanks for the confirmation!
Quoted 3 lines
I do not believe that you need
the abovesmp_wmb()-- though keeping the comment stating that you are relying on the implicit barrier in IPI would be good.
Indeed, I'll keep the comment on what the code rely on concerning implicit ordering guarantees and then remove the explicit barriers.
Quoted 6 lines
kernel/time/tick-sched.c void tick_nohz_full_kick_all(void) · line 315
Quoted 2 lines
kernel/time/tick-sched.c +1
-0+smp_send_reschedule (tick_timekeeping_default_cpu ());Again, smp_send_reschedule()'s IPI hander does not necessarily do anything if there is nothing for the scheduler to do, so any needed actions are taking in the return-from-interrupt code?
Yep, this is all handled from irq exit:
irq_exit -> tick_nohz_irq_exit() -> tick_nohz_stop_sched_tick() -> tick_timekeeping_max_deferment()
The latter function performs the tick re-evaluation on top of the need or not for the current CPU to handle the timekeeping on behalf of others.
Thanks.
Quoted 26 lines
Quoted 25 lines
kernel/time/tick-sched.c +11
-0+} +/* * Re-evaluate the need for the tick as we switch the current task. * It might need the tick due to per task/process properties: @@ -351,6 +373,15 @@ static int tick_nohz_cpu_down_callback(struct notifier_block *nfb,if (tick_nohz_full_running &&tick_timekeeping_default_cpu () == cpu)return NOTIFY_BAD ;break ; + +case CPU_DYING : +/* +* Notify default timekeeper if we are giving up +* timekeeping duty +*/ +if (tick_nohz_full_running && tick_do_timer_cpu == cpu) +tick_nohz_full_kick_timekeeping (); +break ; }return NOTIFY_OK ; }Signature · 1 line
1.8.3.1
Peter Zijlstra
maintainerRe: [PATCH 10/13] nohz: Hand over timekeeping duty on cpu offlining
In reply to Frederic WeisbeckerQuoted 2 lines · Frederic Weisbecker
The default timekeeper (aka CPU 0) is the perfect candidate for this task since it can't be offlined itself.
Like said previously, yes it can!
Re: [PATCH 10/13] nohz: Hand over timekeeping duty on cpu offlining
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker
The default timekeeper (aka CPU 0) is the perfect candidate for this task since it can't be offlined itself.
Like said previously, yes it can!
I'll rephrase that in the changelog to precise that it's the tick hotplug handler which prevent it to offline.
PATCH 11/13 nohz: Wake up timekeeper on exit from sysidle state
In reply to Frederic WeisbeckerWhen a full dynticks CPU wakes up from sysidle state, which means that all full dynticks CPUs were previously sleeping, it's possible that all the potential timekeeping CPUs are sleeping as well and nobody maintains the associated duty.
But full dynticks CPUs don't run the tick by definition so we need to wake up a timekeeper such that it can handle the timekeeping duty on behalf of the freshly awoken full dyntick CPU.
To achieve this and ensure that this CPU won't deal with stale jiffies values, lets wake up the default timekeeper using the right API.
--- kernel/rcu/tree_plugin.h | 2 +- kernel/time/tick-sched.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -smp_send_reschedule (tick_timekeeping_default_cpu ()); +tick_nohz_full_kick_timekeeping ();return ;/* We cleared it, done! */ } oldstate = newoldstate;
kernel/time/tick-sched.c +2 -1
@@ -302,7 +302,8 @@ void tick_nohz_full_kick_all(void)/** * tick_nohz_full_kick_timekeeping - kick the default timekeeper * -* kick the default timekeeper when a secondary timekeeper goes offline. +* kick the default timekeeper when full dynticks CPUs exit full +* system idle state or when a secondary timekeeper goes offline. */ void tick_nohz_full_kick_timekeeping (void ) {
Signature · 1 line
1.8.3.1
Re: [PATCH 11/13] nohz: Wake up timekeeper on exit from sysidle state
Quoted 32 lines · Frederic Weisbecker
When a full dynticks CPU wakes up from sysidle state, which means that all full dynticks CPUs were previously sleeping, it's possible that all the potential timekeeping CPUs are sleeping as well and nobody maintains the associated duty.
But full dynticks CPUs don't run the tick by definition so we need to wake up a timekeeper such that it can handle the timekeeping duty on behalf of the freshly awoken full dyntick CPU.
To achieve this and ensure that this CPU won't deal with stale jiffies values, lets wake up the default timekeeper using the right API.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- kernel/time/tick-sched.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)kernel/rcu/tree_plugin.h +0
-0
kernel/rcu/tree_plugin.h void rcu_sysidle_force_exit(void) · line 2491
Quoted 5 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -smp_send_reschedule (tick_timekeeping_default_cpu ()); +tick_nohz_full_kick_timekeeping ();
OK, I guess I should look at the patches in order. So yes, it is no longer safe to just kick tick_do_timer_cpu. ;-)
Never mind my question on patch 12/13 in this series.
Thanx, Paul
Quoted 20 lines
kernel/rcu/tree_plugin.h +0
-0return ;/* We cleared it, done! */ } oldstate = newoldstate;kernel/time/tick-sched.c +2
-1@@ -302,7 +302,8 @@ void tick_nohz_full_kick_all(void)/** * tick_nohz_full_kick_timekeeping - kick the default timekeeper * -* kick the default timekeeper when a secondary timekeeper goes offline. +* kick the default timekeeper when full dynticks CPUs exit full +* system idle state or when a secondary timekeeper goes offline. */ void tick_nohz_full_kick_timekeeping (void ) {Signature · 1 line
1.8.3.1
Re: [PATCH 11/13] nohz: Wake up timekeeper on exit from sysidle state
Quoted 28 lines · Paul E. McKenney
Quoted 27 lines · Frederic Weisbecker
When a full dynticks CPU wakes up from sysidle state, which means that all full dynticks CPUs were previously sleeping, it's possible that all the potential timekeeping CPUs are sleeping as well and nobody maintains the associated duty.
But full dynticks CPUs don't run the tick by definition so we need to wake up a timekeeper such that it can handle the timekeeping duty on behalf of the freshly awoken full dyntick CPU.
To achieve this and ensure that this CPU won't deal with stale jiffies values, lets wake up the default timekeeper using the right API.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- kernel/rcu/tree_plugin.h | 2 +- kernel/time/tick-sched.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
Quoted 13 lines
kernel/rcu/tree_plugin.h void rcu_sysidle_force_exit(void) · line 2491
Quoted 10 lines
kernel/rcu/tree_plugin.h +1
-1@@ -2488,7 +2488,7 @@ void rcu_sysidle_force_exit(void) oldstate, RCU_SYSIDLE_NOT);if (oldstate == newoldstate && oldstate ==RCU_SYSIDLE_FULL_NOTED ) { -smp_send_reschedule (tick_timekeeping_default_cpu ()); +tick_nohz_full_kick_timekeeping ();OK, I guess I should look at the patches in order. So yes, it is no longer safe to just kick
tick_do_timer_cpu. ;-)
Hehe, I do linear reviews as well ;)
Indeed it's no longer safe, especially since tick_do_timer_cpu can be TICK_DO_TIMER_NONE. So this always kick CPU 0 instead (for now).
PATCH 12/13 nohz: Allow all CPUs outside nohz_full range to do timekeeping
In reply to Frederic WeisbeckerNow that we have all the infrastructure in place and ready to support timekeeping duty balanced across every non full dynticks CPUs, we can hereby extend the timekeeping duty affinity.
--- include/linux/tick.h | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)
include/linux/tick.h +1 -9
@@ -203,15 +203,7 @@ static inline int tick_timekeeping_default_cpu(void)*/ static inline bool tick_timekeeping_cpu (int cpu) { -/* -* If there are full dynticks CPUs around, -* CPU 0 must stay periodic to update timekeeping. -*/ -if (tick_nohz_full_enabled ()) -return cpu ==0 ; - -/* Otherwise any CPU is elligible for timekeeping duty */ -return true; +return !tick_nohz_full_cpu (cpu); }extern void tick_nohz_init (void );
Signature · 1 line
1.8.3.1
Re: [PATCH 12/13] nohz: Allow all CPUs outside nohz_full range to do timekeeping
Quoted 25 lines · Frederic Weisbecker
Now that we have all the infrastructure in place and ready to support timekeeping duty balanced across every non full dynticks CPUs, we can hereby extend the timekeeping duty affinity.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Steven Rostedt <[email protected]>Cc Paul E. McKenney <[email protected]>Cc John Stultz <[email protected]>Cc Alex Shi <[email protected]>Cc Kevin Hilman <[email protected]>--- include/linux/tick.h | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)include/linux/tick.h +0
-0@@ -203,15 +203,7 @@ static inline int tick_timekeeping_default_cpu(void)*/ static inline bool tick_timekeeping_cpu (int cpu) {
include/linux/tick.h static inline int tick_timekeeping_default_cpu(void) · line 206
Quoted 10 lines
include/linux/tick.h +1
-9-/* -* If there are full dynticks CPUs around, -* CPU 0 must stay periodic to update timekeeping. -*/ -if (tick_nohz_full_enabled ()) -return cpu ==0 ; - -/* Otherwise any CPU is elligible for timekeeping duty */ -return true; +return !tick_nohz_full_cpu (cpu);
OK, I guess the future is already here. ;-)
Is it still OK for RCU to kick tick_do_timer_cpu? Or are there race conditions that could result in the wrong CPU being kicked? Or is there some guarantee that I missed that says that the timekeeping CPU cannot change while in RCU_SYSIDLE_FULL_NOTED state?
Thanx, Paul
Quoted 6 lines
include/linux/tick.h +0
-0}extern void tick_nohz_init (void );Signature · 1 line
1.8.3.1
PATCH 13/13 nohz_full: fix code style issue of tick_nohz_full_stop_tick
In reply to Frederic WeisbeckerFrom: Alex Shi <[email protected]>
Code usually starts with 'tab' instead of 7 'space' in kernel
--- kernel/time/tick-sched.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
kernel/time/tick-sched.c +8 -8
@@ -758,18 +758,18 @@ out:static void tick_nohz_full_stop_tick (struct tick_sched *ts) {#ifdef CONFIG_NO_HZ_FULL -int cpu =smp_processor_id (); +int cpu =smp_processor_id (); -if (!tick_nohz_full_cpu (cpu) ||is_idle_task (current)) -return ; +if (!tick_nohz_full_cpu (cpu) ||is_idle_task (current)) +return ; -if (!ts->tick_stopped && ts->nohz_mode ==NOHZ_MODE_INACTIVE ) -return ; +if (!ts->tick_stopped && ts->nohz_mode ==NOHZ_MODE_INACTIVE ) +return ; -if (!can_stop_full_tick ()) -return ; +if (!can_stop_full_tick ()) +return ; -tick_nohz_stop_sched_tick (ts,ktime_get (), cpu); +tick_nohz_stop_sched_tick (ts,ktime_get (), cpu);#endif }
Signature · 1 line
1.8.3.1
Quoted 7 lines · Frederic Weisbecker
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
* Allow all CPUs outside the
nohz_fullrange to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.
If the system just has one nohz_full cpu running, it will need another cpu to do timerkeeper job. Then the system roughly needs 2 cpu living.
Quoted 7 lines
From powersaving POV, that is not good compare to normal nohz idle.
* Let the timekeeper (including CPU 0) sleep when its duty is handed over to another CPU
* Allow timekeeper to sleep when all full dynticks CPUs are sleeping (plug nohz to RCU sysidle detection)
Thanks Fredic!
It is much better on powersaving POV compare to current nohz_full. :)
Quoted 6 lines
* Wake up timekeeper with an IPI when full dynticks CPUs exit sysidle state
* Wake up CPU 0 when a secondary timekeeper is offlined so that its duty gets migrated
Signature · 2 lines
Thanks
AlexPeter Zijlstra
maintainerQuoted 3 lines · Alex Shi
On 12/18/2013 06:51 AM, Frederic Weisbecker wrote:
Thanks Fredic!
It is much better on powersaving POV compare to currentnohz_full. :)
If you're using nohz_full for powersaving, you're doing it wrong.
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Alex Shi
On 12/18/2013 06:51 AM, Frederic Weisbecker wrote:
Thanks Fredic!
It is much better on powersaving POV compare to currentnohz_full. :)If you're using
nohz_fullfor powersaving, you're doing it wrong.
They just want nohz_full not to totally break powersavings when the entire system is idle. ;-)
Thanx, Paul
Quoted 5 lines · Alex Shi
Quoted 4 lines · Frederic Weisbecker
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
Quoted 7 lines
Quoted 3 lines
* Allow all CPUs outside the
nohz_fullrange to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.If the system just has one
nohz_fullcpu running, it will need another cpu to do timerkeeper job. Then the system roughly needs 2 cpu living. From powersaving POV, that is not good compare to normal nohz idle.
Sure, but everything has a tradeoff :)
We could theoretically run with the timekeeper purely idle if the other CPU in full dynticks mode runs in userspace for a long while and seldom do syscalls and faults. Timekeeping could be updated on kernel/user boundaries in this case without much impact on performances.
But then there is one strict condition for that: it can't read the timeofday through the vdso but only through a syscall.
Then if we can meet that condition, then CPU 0 could as well be full dynticks.
Now that's real extreme HPC ;)
Quoted 16 lines · Frederic Weisbecker
Quoted 5 lines · Alex Shi
Quoted 4 lines · Frederic Weisbecker
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
Quoted 7 lines
Quoted 3 lines
* Allow all CPUs outside the
nohz_fullrange to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.If the system just has one
nohz_fullcpu running, it will need another cpu to do timerkeeper job. Then the system roughly needs 2 cpu living. From powersaving POV, that is not good compare to normal nohz idle.Sure, but everything has a tradeoff :)
Quoted 7 lines
We could theoretically run with the timekeeper purely idle if the other CPU in full dynticks mode runs in userspace for a long while and seldom do syscalls and faults. Timekeeping could be updated on kernel/user boundaries in this case without much impact on performances.
But then there is one strict condition for that: it can't read the timeofday through the vdso but only through a syscall.
Where's your ambition? :)
If the vdso timing functions could see that it's been too long since a real timekeeping update, they could fall back to a syscall. Otherwise, they could using rdtsc or whatever is in use.
--Andy
Quoted 22 lines · Andy Lutomirski
Quoted 21 lines · Frederic Weisbecker
Quoted 5 lines · Alex Shi
Quoted 4 lines · Frederic Weisbecker
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
Quoted 7 lines
Quoted 3 lines
* Allow all CPUs outside the
nohz_fullrange to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.If the system just has one
nohz_fullcpu running, it will need another cpu to do timerkeeper job. Then the system roughly needs 2 cpu living. From powersaving POV, that is not good compare to normal nohz idle.Sure, but everything has a tradeoff :)
We could theoretically run with the timekeeper purely idle if the other CPU in full dynticks mode runs in userspace for a long while and seldom do syscalls and faults. Timekeeping could be updated on kernel/user boundaries in this case without much impact on performances.
Quoted 8 lines
Quoted 2 lines
But then there is one strict condition for that: it can't read the timeofday through the vdso but only through a syscall.
Where's your ambition? :)
If the vdso timing functions could see that it's been too long since a real timekeeping update, they could fall back to a syscall. Otherwise, they could using rdtsc or whatever is in use.
One objection to that approach in the past has been that it injects avoidable latency into the worker CPUs. I suppose that you could argue that the cache misses due to a timekeeping-CPU update are not free, but then again, the syscall is likely to also incur a few cache misses as well.
I bet that the timekeeping-CPU approach wins, but it would be cool to see you prove me wrong.
Thanx, Paul
Quoted 32 lines · Paul E. McKenney
Quoted 22 lines · Andy Lutomirski
Quoted 21 lines · Frederic Weisbecker
Quoted 5 lines · Alex Shi
Quoted 4 lines · Frederic Weisbecker
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
Quoted 7 lines
Quoted 3 lines
* Allow all CPUs outside the
nohz_fullrange to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.If the system just has one
nohz_fullcpu running, it will need another cpu to do timerkeeper job. Then the system roughly needs 2 cpu living. From powersaving POV, that is not good compare to normal nohz idle.Sure, but everything has a tradeoff :)
We could theoretically run with the timekeeper purely idle if the other CPU in full dynticks mode runs in userspace for a long while and seldom do syscalls and faults. Timekeeping could be updated on kernel/user boundaries in this case without much impact on performances.
Quoted 8 lines
Quoted 2 lines
But then there is one strict condition for that: it can't read the timeofday through the vdso but only through a syscall.
Where's your ambition? :)
If the vdso timing functions could see that it's been too long since a real timekeeping update, they could fall back to a syscall. Otherwise, they could using rdtsc or whatever is in use.
Quoted 8 lines
One objection to that approach in the past has been that it injects avoidable latency into the worker CPUs. I suppose that you could argue that the cache misses due to a timekeeping-CPU update are not free, but then again, the syscall is likely to also incur a few cache misses as well.
I bet that the timekeeping-CPU approach wins, but it would be cool to see you prove me wrong.
There's already some (very vague) discussion about having a scheduled time at which the clock frequency and/or offset will change, and this wouldn't be a huge departure from that. The goal there is to avoid waiting for timekeeping if vclock_gettime runs concurrently with an update, but the same approach could apply here (albeit with one extra branch).
Anyway, syscalls aren't *that* expensive.
Alternatively, couldn't workloads like this just turn off NTP?
--Andy
Quoted 43 lines · Andy Lutomirski
Quoted 32 lines · Paul E. McKenney
Quoted 22 lines · Andy Lutomirski
Quoted 21 lines · Frederic Weisbecker
Quoted 5 lines · Alex Shi
Quoted 4 lines · Frederic Weisbecker
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
Quoted 7 lines
Quoted 3 lines
* Allow all CPUs outside the
nohz_fullrange to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.If the system just has one
nohz_fullcpu running, it will need another cpu to do timerkeeper job. Then the system roughly needs 2 cpu living. From powersaving POV, that is not good compare to normal nohz idle.Sure, but everything has a tradeoff :)
We could theoretically run with the timekeeper purely idle if the other CPU in full dynticks mode runs in userspace for a long while and seldom do syscalls and faults. Timekeeping could be updated on kernel/user boundaries in this case without much impact on performances.
Quoted 8 lines
Quoted 2 lines
But then there is one strict condition for that: it can't read the timeofday through the vdso but only through a syscall.
Where's your ambition? :)
If the vdso timing functions could see that it's been too long since a real timekeeping update, they could fall back to a syscall. Otherwise, they could using rdtsc or whatever is in use.
Quoted 8 lines
One objection to that approach in the past has been that it injects avoidable latency into the worker CPUs. I suppose that you could argue that the cache misses due to a timekeeping-CPU update are not free, but then again, the syscall is likely to also incur a few cache misses as well.
I bet that the timekeeping-CPU approach wins, but it would be cool to see you prove me wrong.
Quoted 8 lines
There's already some (very vague) discussion about having a scheduled time at which the clock frequency and/or offset will change, and this wouldn't be a huge departure from that. The goal there is to avoid waiting for timekeeping if
vclock_gettimeruns concurrently with an update, but the same approach could apply here (albeit with one extra branch).Anyway, syscalls aren't *that* expensive.
Like I said, it would be cool to see you prove me wrong, but that will need to be with patches and performance results rather than rhetoric.
Quoted 1 line
Alternatively, couldn't workloads like this just turn off NTP?
Some probably could, but others need accurate time.
Thanx, Paul
Quoted 51 lines · Paul E. McKenney
Quoted 50 lines · Andy Lutomirski
Quoted 32 lines · Paul E. McKenney
Quoted 22 lines · Andy Lutomirski
Quoted 21 lines · Frederic Weisbecker
Quoted 5 lines · Alex Shi
Quoted 4 lines · Frederic Weisbecker
So this is what this series brings, more details following:
* Some code, naming and whitespace cleanups
Quoted 7 lines
Quoted 3 lines
* Allow all CPUs outside the
nohz_fullrange to handle the timekeeping duty, not just CPU 0. Balancing the timekeeping duty should improve powersavings.If the system just has one
nohz_fullcpu running, it will need another cpu to do timerkeeper job. Then the system roughly needs 2 cpu living. From powersaving POV, that is not good compare to normal nohz idle.Sure, but everything has a tradeoff :)
We could theoretically run with the timekeeper purely idle if the other CPU in full dynticks mode runs in userspace for a long while and seldom do syscalls and faults. Timekeeping could be updated on kernel/user boundaries in this case without much impact on performances.
Quoted 8 lines
Quoted 2 lines
But then there is one strict condition for that: it can't read the timeofday through the vdso but only through a syscall.
Where's your ambition? :)
If the vdso timing functions could see that it's been too long since a real timekeeping update, they could fall back to a syscall. Otherwise, they could using rdtsc or whatever is in use.
Quoted 8 lines
One objection to that approach in the past has been that it injects avoidable latency into the worker CPUs. I suppose that you could argue that the cache misses due to a timekeeping-CPU update are not free, but then again, the syscall is likely to also incur a few cache misses as well.
I bet that the timekeeping-CPU approach wins, but it would be cool to see you prove me wrong.
There's already some (very vague) discussion about having a scheduled time at which the clock frequency and/or offset will change, and this wouldn't be a huge departure from that. The goal there is to avoid waiting for timekeeping if
vclock_gettimeruns concurrently with an update, but the same approach could apply here (albeit with one extra branch).
Quoted 4 lines
Quoted 1 line
Anyway, syscalls aren't *that* expensive.
Like I said, it would be cool to see you prove me wrong, but that will need to be with patches and performance results rather than rhetoric.
Fair enough. Don't hold your breath. (And certainly don't take this as an objection to these patches.)
--Andy