nohz/cpuset: Start discussions on nohz CPUs
Subsystems (8)
Steven Rostedt
reviewercover nohz/cpuset: Start discussions on nohz CPUs
A while ago Frederic posted a series of patches to get an idea on how to implement nohz cpusets. Where you can add a task to a cpuset and mark the set to be 'nohz'. When the task runs on a CPU and is the only task scheduled (nr_running == 1), the tick will stop. The idea is to give the task the least amount of kernel interference as possible. If the task doesn't do any system calls (and possibly even if it does), no timer interrupt will bother it. By using isocpus and nohz cpuset, a task would be able to achieve true cpu isolation.
This has been long asked for by those in the RT community. If a task requires uninterruptible CPU time, this would be able to give a task that, even without the full PREEMPT-RT patch set.
This patch set is not for inclusion. It is just to get the topic at the forefront again. The design requires more work and more discussion.
I ported Frederic's work to v3.7-rc3 and I'm posting it here so that people can comment on it. I just did the minimal to get it to compile and boot. I haven't done any real tests with it yet. I may have screwed some things up during the port, but that's OK, because the patch set will most likely require a rewrite anyway.
Please have a look, and lets get this out the door.
-- Steve
Frederic Weisbecker (31):
nohz: Move nohz load balancer selection into idle logic
cpuset: Set up interface for nohz flag
nohz: Try not to give the timekeeping duty to an adaptive tickless cpu
x86: New cpuset nohz irq vector
nohz: Adaptive tick stop and restart on nohz cpuset nohz/cpuset: Don't turn off the tick if rcu needs it nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued nohz/cpuset: Don't stop the tick if posix cpu timers are running nohz/cpuset: Restart tick when nohz flag is cleared on cpuset nohz/cpuset: Restart the tick if printk needs it
rcu: Restart the tick on non-responding adaptive nohz CPUs
rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU
nohz: Generalize tickless cpu time accounting
nohz/cpuset: Account user and system times in adaptive nohz mode nohz/cpuset: New API to flush cputimes on nohz cpusets nohz/cpuset: Flush cputime on threads in nohz cpusets when waiting leader nohz/cpuset: Flush cputimes on procfs stat file read nohz/cpuset: Flush cputimes for getrusage() and times() syscalls
x86: Syscall hooks for nohz cpusets
nohz: Don't restart the tick before scheduling to idle
sched: Comment on rq->clock correctness in ttwu_do_wakeup() in nohz
sched: Update rq clock on nohz CPU before migrating tasks
sched: Update rq clock on nohz CPU before setting fair group shares
sched: Update rq clock on tickless CPUs before calling check_preempt_curr()
sched: Update rq clock earlier in unthrottle_cfs_rq
sched: Update clock of nohz busiest rq before balancing
sched: Update rq clock before idle balancing
sched: Update nohz rq clock before searching busiest group on load balancing
rcu: Switch to extended quiescent state in userspace from nohz cpuset nohz/cpuset: Disable under some configs
nohz, not for merge: Add tickless tracing
Hakan Akkan (1):
nohz/cpuset: enable addition&removal of cpus while in adaptive nohz mode
----
arch/Kconfig | 3 + arch/x86/include/asm/entry_arch.h | 3 + arch/x86/include/asm/hw_irq.h | 7 + arch/x86/include/asm/irq_vectors.h | 2 + arch/x86/include/asm/smp.h | 11 +- arch/x86/kernel/entry_64.S | 4 + arch/x86/kernel/irqinit.c | 4 + arch/x86/kernel/ptrace.c | 11 + arch/x86/kernel/smp.c | 28 +++ fs/proc/array.c | 2 + include/linux/cpuset.h | 35 ++++ include/linux/kernel_stat.h | 2 + include/linux/posix-timers.h | 1 + include/linux/rcupdate.h | 1 + include/linux/sched.h | 10 +- include/linux/tick.h | 72 +++++-- init/Kconfig | 8 + kernel/cpuset.c | 144 ++++++++++++- kernel/exit.c | 8 + kernel/posix-cpu-timers.c | 12 ++ kernel/printk.c | 15 +- kernel/rcutree.c | 28 ++- kernel/sched/core.c | 82 +++++++- kernel/sched/cputime.c | 22 ++ kernel/sched/fair.c | 41 +++- kernel/sched/sched.h | 18 ++ kernel/softirq.c | 6 +- kernel/sys.c | 6 + kernel/time/tick-sched.c | 398 ++++++++++++++++++++++++++++++++---- kernel/time/timer_list.c | 3 +- kernel/timer.c | 2 +- 31 files changed, 912 insertions(+), 77 deletions(-)
Steven Rostedt
reviewerPATCH 1/32 nohz: Move nohz load balancer selection into idle logic
From: Frederic Weisbecker <[email protected]>
[ ** BUGGY PATCH: I need to put more thinking into this ** ]
We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.
--- kernel/time/tick-sched.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
kernel/time/tick-sched.c +6 -5
@@ -372,9 +372,6 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,* the scheduler tick in nohz_restart_sched_tick. */ if (!ts->tick_stopped) { -nohz_balance_enter_idle (cpu); -calc_load_enter_idle (); - ts->last_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ; }
@@ -466,8 +463,11 @@ static void __tick_nohz_idle_enter(struct tick_sched *ts) ts->idle_expires = expires; } -if (!was_stopped && ts->tick_stopped) +if (!was_stopped && ts->tick_stopped) { ts->idle_jiffies = ts->last_jiffies; +nohz_balance_enter_idle (cpu); +calc_load_enter_idle (); + } } }
@@ -573,7 +573,6 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now)tick_do_update_jiffies64 (now);update_cpu_load_nohz (); -calc_load_exit_idle ();touch_softlockup_watchdog ();/* * Cancel the scheduled timer and restore the tick
@@ -628,6 +627,8 @@ void tick_nohz_idle_exit(void)tick_nohz_stop_idle (cpu, now);if (ts->tick_stopped) { +nohz_balance_enter_idle (cpu); +calc_load_exit_idle ();tick_nohz_restart_sched_tick (ts, now);tick_nohz_account_idle_ticks (ts); }
Signature · 1 line
1.7.10.4
Re: [PATCH 01/32] nohz: Move nohz load balancer selection into idle logic
calc_load_exit_idle depends on updated jiffies, so you shouldn't move this before tick_do_update_jiffies64.
And why should we do nohz_balance_enter_idle in tick_nohz_idle_exit? It's nohz_balance_exit_idle here.
Regards,
Charles
On 10/30/2012 04:27 AM, Steven Rostedt wrote:
［snipped］
Quoted 17 lines
kernel/time/tick-sched.c +2
-1@@ -573,7 +573,6 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now)tick_do_update_jiffies64 (now);update_cpu_load_nohz (); -calc_load_exit_idle ();touch_softlockup_watchdog ();/* * Cancel the scheduled timer and restore the tick @@ -628,6 +627,8 @@ void tick_nohz_idle_exit(void)tick_nohz_stop_idle (cpu, now);if (ts->tick_stopped) { +nohz_balance_enter_idle (cpu); +calc_load_exit_idle ();tick_nohz_restart_sched_tick (ts, now);tick_nohz_account_idle_ticks (ts); }
Steven Rostedt
reviewerRe: [PATCH 01/32] nohz: Move nohz load balancer selection into idle logic
Quoted 2 lines · Charles Wang
calc_load_exit_idledepends on updated jiffies, so you shouldn't move this beforetick_do_update_jiffies64.
OK, so it should be moved to the end of the if block. Note, that was a change I made, as that function was added since Frederic did his code. I looked into the code and it seemed that it should be moved for just idle as well. But I agree with you that it should be after the jiffies update.
Quoted 2 lines
And why should we do
nohz_balance_enter_idleintick_nohz_idle_exit? It'snohz_balance_exit_idlehere.
OK, that's my fault as well. As Frederic's original patch just moved select_nohz_load_balance(0). But the commit c1cc017c59 "sched/nohz:
Clean up select_nohz_load_balancer()" replaced it with nohz_balance_enter_idle(cpu), and removed it on exit, there was nothing to replace for the current code.
Knowing this patch was considered "buggy" instead of just not moving it (and forgetting about it), I did the change to remind myself to talk about it :-)
My question is now, is there any reason to keep that call there? Or can we just remove it as well.
-- Steve
Quoted 24 lines
Regards,
CharlesOn 10/30/2012 04:27 AM, Steven Rostedt wrote:
［snipped］Review static void tick_nohz_restart_sched_tick(struct
Quoted 1 line
(quoted diff) +0
-0@@ -573,7 +573,6 @@ static void tick_nohz_restart_sched_tick(struct
tick_sched*ts,ktime_tnow)Quoted 16 lines
kernel/time/tick-sched.c +2
-1tick_do_update_jiffies64 (now);update_cpu_load_nohz (); -calc_load_exit_idle ();touch_softlockup_watchdog ();/* * Cancel the scheduled timer and restore the tick @@ -628,6 +627,8 @@ void tick_nohz_idle_exit(void)tick_nohz_stop_idle (cpu, now);if (ts->tick_stopped) { +nohz_balance_enter_idle (cpu); +calc_load_exit_idle ();tick_nohz_restart_sched_tick (ts, now);tick_nohz_account_idle_ticks (ts); }
Steven Rostedt
reviewerPATCH 2/32 cpuset: Set up interface for nohz flag
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Prepare the interface to implement the nohz cpuset flag. This flag, once set, will tell the system to try to shutdown the periodic timer tick when possible.
We use here a per cpu refcounter. As long as a CPU is contained into at least one cpuset that has the nohz flag set, it is part of the set of CPUs that run into adaptive nohz mode.
[ include build fix from Zen Lin ]
--- arch/Kconfig | 3 +++ include/linux/cpuset.h | 31 ++++++++++++++++++++++++++++ init/Kconfig | 8 ++++++++ kernel/cpuset.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 94 insertions(+), 1 deletion(-)
arch/Kconfig +3 -0
@@ -239,6 +239,9 @@ config HAVE_ARCH_JUMP_LABEL bool config HAVE_ARCH_MUTEX_CPU_RELAX + bool + +config HAVE_CPUSETS_NO_HZ bool config HAVE_RCU_TABLE_FREE
include/linux/cpuset.h +31 -0
@@ -13,6 +13,7 @@#include <linux/nodemask.h> #include <linux/cgroup.h> #include <linux/mm.h> +#include <linux/atomic.h> #ifdef CONFIG_CPUSETS
@@ -235,4 +236,34 @@ static inline bool put_mems_allowed(unsigned int seq)#endif /* !CONFIG_CPUSETS */ +#ifdef CONFIG_CPUSETS_NO_HZ + +DECLARE_PER_CPU (atomic_t, cpu_adaptive_nohz_ref); + +static inline bool cpuset_cpu_adaptive_nohz (int cpu) +{ +atomic_t *ref = &per_cpu (cpu_adaptive_nohz_ref, cpu); + +if (atomic_add_return (0 , ref) >0 ) +return true; + +return false; +} + +static inline bool cpuset_adaptive_nohz (void ) +{ +/* +* We probably want to do atomic_read() when we read +* locally to avoid the overhead of an ordered add. +* For that we have to do the dec of the ref locally as +* well. +*/ +return cpuset_cpu_adaptive_nohz (smp_processor_id ()); +} +#else +static inline bool cpuset_cpu_adaptive_nohz (int cpu) {return false; } +static inline bool cpuset_adaptive_nohz (void ) {return false; } + +#endif /* CONFIG_CPUSETS_NO_HZ */ +#endif /* _LINUX_CPUSET_H */
init/Kconfig +8 -0
@@ -749,6 +749,14 @@ config PROC_PID_CPUSET depends on CPUSETS default y +config CPUSETS_NO_HZ + bool "Tickless cpusets" + depends on CPUSETS && HAVE_CPUSETS_NO_HZ + help + This options let you apply a nohz property to a cpuset such + that the periodic timer tick tries to be avoided when possible on + the concerned CPUs. + config CGROUP_CPUACCT bool "Simple CPU accounting cgroup subsystem" help
kernel/cpuset.c +52 -1
101 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerRe: [PATCH 02/32] cpuset: Set up interface for nohz flag
Quoted 5 lines · Steven Rostedt
#ifdef
CONFIG_CPUSETSinclude/linux/cpuset.h +0
-0@@ -235,4 +236,34 @@ static inline bool put_mems_allowed(unsigned int seq)#endif /* !CONFIG_CPUSETS */
include/linux/cpuset.h static inline bool put_mems_allowed(unsigned int seq) · line 247
Quoted 10 lines
include/linux/cpuset.h +9
-0+#ifdef CONFIG_CPUSETS_NO_HZ + +DECLARE_PER_CPU (atomic_t, cpu_adaptive_nohz_ref); + +static inline bool cpuset_cpu_adaptive_nohz (int cpu) +{ +atomic_t *ref = &per_cpu (cpu_adaptive_nohz_ref, cpu); + +if (atomic_add_return (0 , ref) >0 )
I'm assuming you do the atomic_add_return() for the implicit memory barrier? Yuck!
Please comment this. I see that rcutree.c does the same thing without a comment. Bad Paul, bad!
include/linux/cpuset.h static inline bool put_mems_allowed(unsigned int seq) · line 259
Quoted 12 lines
include/linux/cpuset.h +12
-0+return true; + +return false; +} + +static inline bool cpuset_adaptive_nohz (void ) +{ +/* +* We probably want to do atomic_read() when we read +* locally to avoid the overhead of an ordered add. +* For that we have to do the dec of the ref locally as +* well.
Does it matter if we miss the dec? What other synchronization is used?
CPU 1 CPU 2 ------ ----- var = atomic_add_return(0, ref) atomic_dec(ref); if (var > 0)
returns true.
For local cases, as this seems to be in a fast path, we should use this_cpu_read() as well.
Quoted 42 lines
include/linux/cpuset.h +9
-0+*/ +return cpuset_cpu_adaptive_nohz (smp_processor_id ()); +} +#else +static inline bool cpuset_cpu_adaptive_nohz (int cpu) {return false; } +static inline bool cpuset_adaptive_nohz (void ) {return false; } + +#endif /* CONFIG_CPUSETS_NO_HZ */ +#endif /* _LINUX_CPUSET_H */ init/Kconfig +8
-0@@ -749,6 +749,14 @@ config PROC_PID_CPUSET depends on CPUSETS default y +config CPUSETS_NO_HZ + bool "Tickless cpusets" + depends on CPUSETS && HAVE_CPUSETS_NO_HZ + help + This options let you apply a nohz property to a cpuset such + that the periodic timer tick tries to be avoided when possible on + the concerned CPUs. + config CGROUP_CPUACCT bool "Simple CPU accounting cgroup subsystem" helpkernel/cpuset.c +1
-0@@ -145,6 +145,7 @@ typedef enum {CS_SCHED_LOAD_BALANCE ,CS_SPREAD_PAGE ,CS_SPREAD_SLAB , +CS_ADAPTIVE_NOHZ , } cpuset_flagbits_t;/* the type of hotplug event */
kernel/cpuset.c static inline int is_spread_slab(const struct cpuset *cs) · line 196
Quoted 7 lines
kernel/cpuset.c +4
-0@@ -189,6 +190,11 @@ static inline int is_spread_slab(const struct cpuset *cs)return test_bit (CS_SPREAD_SLAB , &cs->flags); } +static inline int is_adaptive_nohz (const struct cpuset *cs) +{ +return test_bit (CS_ADAPTIVE_NOHZ , &cs->flags); +}
We can move this into the #ifdef CONFIG_CPUSETS_NO_HZ as well, and have the #else version just return zero. Why use test_bit() when we already know the answer?
Quoted 24 lines
kernel/cpuset.c +17
-0+static struct cpuset top_cpuset = { .flags = ((1 <<CS_CPU_EXCLUSIVE ) | (1 <<CS_MEM_EXCLUSIVE )), };@@ -1190,6 +1196,32 @@ static void cpuset_change_flag(struct task_struct *tsk,cpuset_update_task_spread_flag (cgroup_cs (scan->cg), tsk); } +#ifdef CONFIG_CPUSETS_NO_HZ + +DEFINE_PER_CPU (atomic_t, cpu_adaptive_nohz_ref); + +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) +{ +int cpu; +int val; + +if (is_adaptive_nohz (old_cs) ==is_adaptive_nohz (cs)) +return ; + +for_each_cpu (cpu, cs->cpus_allowed) { +atomic_t *ref = &per_cpu (cpu_adaptive_nohz_ref, cpu); +if (is_adaptive_nohz (cs)) +atomic_inc (ref);
kernel/cpuset.c static void cpuset_change_flag(struct task_struct *tsk, · line 1224
Quoted 10 lines
kernel/cpuset.c +10
-0+ else + atomic_dec(ref); + } +} +#else +static inline void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) +{ +} +#endif +
-- Steve
Steven Rostedt
reviewerPATCH 3/32 nohz: Try not to give the timekeeping duty to an adaptive tickless cpu
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.
[TODO: We need to find a way to ensure there is always one non-nohz running CPU maintaining the timekeeping duty if every non-idle CPUs are adaptive tickless]
--- kernel/time/tick-sched.c | 52 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 41 insertions(+), 11 deletions(-)
kernel/time/tick-sched.c +41 -11
@@ -20,6 +20,7 @@#include <linux/profile.h> #include <linux/sched.h> #include <linux/module.h> +#include <linux/cpuset.h> #include <asm/irq_regs.h>
@@ -789,6 +790,45 @@ void tick_check_idle(int cpu)tick_check_nohz (cpu); } +#ifdef CONFIG_CPUSETS_NO_HZ + +/* +* Take the timer duty if nobody is taking care of it. +* If a CPU already does and and it's in a nohz cpuset, +* then take the charge so that it can switch to nohz mode. +*/ +static void tick_do_timer_check_handler (int cpu) +{ +int handler = tick_do_timer_cpu; + +if (unlikely (handler ==TICK_DO_TIMER_NONE )) { + tick_do_timer_cpu = cpu; + }else { +if (!cpuset_adaptive_nohz () && +cpuset_cpu_adaptive_nohz (handler)) + tick_do_timer_cpu = cpu; + } +} + +#else + +static void tick_do_timer_check_handler (int cpu) +{ +#ifdef CONFIG_NO_HZ +/* +* Check if the do_timer duty was dropped. We don't care about +* concurrency: This happens only when the cpu in charge went +* into a long sleep. If two cpus happen to assign themself to +* this duty, then the jiffies update is still serialized by +* xtime_lock. +*/ +if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE )) + tick_do_timer_cpu = cpu; +#endif +} + +#endif /* CONFIG_CPUSETS_NO_HZ */ +/* * High resolution timer specific code */
@@ -805,17 +845,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)ktime_t now =ktime_get ();int cpu =smp_processor_id (); -#ifdef CONFIG_NO_HZ -/* -* Check if the do_timer duty was dropped. We don't care about -* concurrency: This happens only when the cpu in charge went -* into a long sleep. If two cpus happen to assign themself to -* this duty, then the jiffies update is still serialized by -* xtime_lock. -*/ -if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE )) - tick_do_timer_cpu = cpu; -#endif +tick_do_timer_check_handler (cpu);/* Check, if the jiffies need an update */ if (tick_do_timer_cpu == cpu)
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerRe: [PATCH 03/32] nohz: Try not to give the timekeeping duty to an adaptive tickless cpu
Quoted 16 lines · Steven Rostedt
kernel/time/tick-sched.c | 52 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 41 insertions(+), 11 deletions(-)kernel/time/tick-sched.c +1
-0@@ -20,6 +20,7 @@#include <linux/profile.h> #include <linux/sched.h> #include <linux/module.h> +#include <linux/cpuset.h> #include <asm/irq_regs.h>
kernel/time/tick-sched.c void tick_check_idle(int cpu) · line 797
Quoted 8 lines
kernel/time/tick-sched.c +5
-0@@ -789,6 +790,45 @@ void tick_check_idle(int cpu)tick_check_nohz (cpu); } +#ifdef CONFIG_CPUSETS_NO_HZ + +/* +* Take the timer duty if nobody is taking care of it. +* If a CPU already does and and it's in a nohz cpuset,
and and
kernel/time/tick-sched.c void tick_check_idle(int cpu) · line 805
Quoted 8 lines
kernel/time/tick-sched.c +8
-0+* then take the charge so that it can switch to nohz mode. +*/ +static void tick_do_timer_check_handler (int cpu) +{ +int handler = tick_do_timer_cpu; + +if (unlikely (handler ==TICK_DO_TIMER_NONE )) { + tick_do_timer_cpu = cpu;
I take it that this is the forced (no one has it, so I'll take it?)
Perhaps we should do a:
WARN_ON(cpuset_cpu_adaptive_nohz(cpu));
kernel/time/tick-sched.c void tick_check_idle(int cpu) · line 809
Quoted 4 lines
kernel/time/tick-sched.c +4
-0+ }else { +if (!cpuset_adaptive_nohz () && +cpuset_cpu_adaptive_nohz (handler)) + tick_do_timer_cpu = cpu;
Shouldn't this be
if (!cpuset_cpu_adaptive_nohz(cpu) && ...
Otherwise, we should have it go to smp_processor_id()?
OK, looking further down, the only caller of it passes smp_processor_id() to the function. But we still shouldn't assume this. Either, have the function use smp_processor_id() explicitly, or let it use any cpu. Otherwise it just confuses people, and is prone to be buggy if in the future something calls it with a cpu that's not the local cpu.
-- Steve
Quoted 44 lines
kernel/time/tick-sched.c +23
-11+ } +} + +#else + +static void tick_do_timer_check_handler (int cpu) +{ +#ifdef CONFIG_NO_HZ +/* +* Check if the do_timer duty was dropped. We don't care about +* concurrency: This happens only when the cpu in charge went +* into a long sleep. If two cpus happen to assign themself to +* this duty, then the jiffies update is still serialized by +* xtime_lock. +*/ +if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE )) + tick_do_timer_cpu = cpu; +#endif +} + +#endif /* CONFIG_CPUSETS_NO_HZ */ +/* * High resolution timer specific code */ @@ -805,17 +845,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)ktime_t now =ktime_get ();int cpu =smp_processor_id (); -#ifdef CONFIG_NO_HZ -/* -* Check if the do_timer duty was dropped. We don't care about -* concurrency: This happens only when the cpu in charge went -* into a long sleep. If two cpus happen to assign themself to -* this duty, then the jiffies update is still serialized by -* xtime_lock. -*/ -if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE )) - tick_do_timer_cpu = cpu; -#endif +tick_do_timer_check_handler (cpu);/* Check, if the jiffies need an update */ if (tick_do_timer_cpu == cpu)
Steven Rostedt
reviewerPATCH 4/32 x86: New cpuset nohz irq vector
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
We need a way to send an IPI (remote or local) in order to asynchronously restart the tick for CPUs in nohz adaptive mode.
This must be asynchronous such that we can trigger it with irqs disabled. This must be usable as a self-IPI as well for example in cases where we want to avoid random dealock scenario while restarting the tick inline otherwise.
This only settles the x86 backend. The core tick restart function will be defined in a later patch.
[CHECKME: Perhaps we instead need to use irq work for self IPIs. But we also need a way to send async remote IPIs.]
--- arch/x86/include/asm/entry_arch.h | 3 +++ arch/x86/include/asm/hw_irq.h | 7 +++++++ arch/x86/include/asm/irq_vectors.h | 2 ++ arch/x86/include/asm/smp.h | 11 ++++++++++- arch/x86/kernel/entry_64.S | 4 ++++ arch/x86/kernel/irqinit.c | 4 ++++ arch/x86/kernel/smp.c | 24 ++++++++++++++++++++++++ 7 files changed, 54 insertions(+), 1 deletion(-)
arch/x86/include/asm/entry_arch.h +3 -0
@@ -10,6 +10,9 @@* through the ICC by us (IPIs) */ #ifdef CONFIG_SMP +#ifdef CONFIG_CPUSETS_NO_HZ +BUILD_INTERRUPT(cpuset_update_nohz_interrupt,CPUSET_UPDATE_NOHZ_VECTOR) +#endif BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR) BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)BUILD_INTERRUPT (call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR )
arch/x86/include/asm/hw_irq.h +7 -0
@@ -35,6 +35,10 @@ extern void spurious_interrupt(void);extern void thermal_interrupt (void );extern void reschedule_interrupt (void ); +#ifdef CONFIG_CPUSETS_NO_HZ +extern void cpuset_update_nohz_interrupt (void ); +#endif +extern void invalidate_interrupt (void );extern void invalidate_interrupt0 (void );extern void invalidate_interrupt1 (void );
@@ -152,6 +156,9 @@ extern asmlinkage void smp_irq_move_cleanup_interrupt(void);#endif #ifdef CONFIG_SMP extern void smp_reschedule_interrupt (struct pt_regs *); +#ifdef CONFIG_CPUSETS_NO_HZ +extern void smp_cpuset_update_nohz_interrupt (struct pt_regs *); +#endif extern void smp_call_function_interrupt (struct pt_regs *);extern void smp_call_function_single_interrupt (struct pt_regs *);#ifdef CONFIG_X86_32
arch/x86/include/asm/irq_vectors.h +2 -0
@@ -112,6 +112,8 @@/* Xen vector callback to receive events in a HVM domain */ #define XEN_HVM_EVTCHN_CALLBACK 0xf3 +#define CPUSET_UPDATE_NOHZ_VECTOR 0xf2 +/* * Local APIC timer IRQ vector is on a different priority level, * to work around the 'lost local interrupt if more than 2 IRQ
arch/x86/include/asm/smp.h +10 -1
@@ -71,7 +71,9 @@ struct smp_ops {void (*stop_other_cpus)(int wait);void (*smp_send_reschedule)(int cpu); - +#ifdef CONFIG_CPUSETS_NO_HZ +void (*smp_cpuset_update_nohz)(int cpu); +#endif int (*cpu_up)(unsigned cpu,struct task_struct *tidle);int (*cpu_disable)(void );void (*cpu_die)(unsigned int cpu);
@@ -140,6 +142,13 @@ static inline void smp_send_reschedule(int cpu) smp_ops.smp_send_reschedule (cpu); } +static inline void smp_cpuset_update_nohz (int cpu) +{ +#ifdef CONFIG_CPUSETS_NO_HZ + smp_ops.smp_cpuset_update_nohz (cpu); +#endif +} +static inline void arch_send_call_function_single_ipi (int cpu) { smp_ops.send_call_func_single_ipi (cpu);
arch/x86/kernel/entry_64.S +4 -0
@@ -1173,6 +1173,10 @@ apicinterrupt CALL_FUNCTION_VECTOR \ call_function_interrupt smp_call_function_interrupt apicinterrupt RESCHEDULE_VECTOR \ reschedule_interrupt smp_reschedule_interrupt +#ifdef CONFIG_CPUSETS_NO_HZ +apicinterrupt CPUSET_UPDATE_NOHZ_VECTOR \ + cpuset_update_nohz_interrupt smp_cpuset_update_nohz_interrupt +#endif #endif apicinterrupt ERROR_APIC_VECTOR \
arch/x86/kernel/irqinit.c +4 -0
@@ -171,6 +171,10 @@ static void __init smp_intr_init(void)*/ alloc_intr_gate (RESCHEDULE_VECTOR , reschedule_interrupt); +#ifdef CONFIG_CPUSETS_NO_HZ +alloc_intr_gate (CPUSET_UPDATE_NOHZ_VECTOR , cpuset_update_nohz_interrupt); +#endif +/* IPI for generic function call */ alloc_intr_gate (CALL_FUNCTION_VECTOR , call_function_interrupt);
arch/x86/kernel/smp.c +24 -0
@@ -126,6 +126,17 @@ static void native_smp_send_reschedule(int cpu) apic->send_IPI_mask (cpumask_of (cpu),RESCHEDULE_VECTOR ); } +#ifdef CONFIG_CPUSETS_NO_HZ +static void native_smp_cpuset_update_nohz (int cpu) +{ +if (unlikely (cpu_is_offline (cpu))) { +WARN_ON (1 ); +return ; + } + apic->send_IPI_mask (cpumask_of (cpu),CPUSET_UPDATE_NOHZ_VECTOR ); +} +#endif +void native_send_call_func_single_ipi (int cpu) { apic->send_IPI_mask (cpumask_of (cpu),CALL_FUNCTION_SINGLE_VECTOR );
@@ -259,6 +270,16 @@ void smp_reschedule_interrupt(struct pt_regs *regs)*/ } +#ifdef CONFIG_CPUSETS_NO_HZ +void smp_cpuset_update_nohz_interrupt (struct pt_regs *regs) +{ +ack_APIC_irq (); +irq_enter (); +inc_irq_stat (irq_call_count); +irq_exit (); +} +#endif +void smp_call_function_interrupt (struct pt_regs *regs) {ack_APIC_irq ();
@@ -292,6 +313,9 @@ struct smp_ops smp_ops = { .stop_other_cpus = native_stop_other_cpus, .smp_send_reschedule = native_smp_send_reschedule, +#ifdef CONFIG_CPUSETS_NO_HZ + .smp_cpuset_update_nohz = native_smp_cpuset_update_nohz, +#endif .cpu_up = native_cpu_up, .cpu_die = native_cpu_die,
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerRe: [PATCH 04/32] x86: New cpuset nohz irq vector
Quoted 12 lines · Steven Rostedt
plain text document attachment
(0004-x86-New-cpuset-nohz-irq-vector.patch)
From: Frederic Weisbecker <[email protected]>We need a way to send an IPI (remote or local) in order to asynchronously restart the tick for CPUs in nohz adaptive mode.
This must be asynchronous such that we can trigger it with irqs disabled. This must be usable as a self-IPI as well for example in cases where we want to avoid random dealock scenario while restarting the tick inline otherwise.
Quoted 5 lines
This only settles the x86 backend. The core tick restart function will be defined in a later patch.
[CHECKME: Perhaps we instead need to use irq work for self IPIs. But we also need a way to send async remote IPIs.]
Probably just use irq_work for self ipis, and normal ipis for other CPUs.
Also, what reason do we have to force a task out of nohz? IOW, do we really need this?
Also, perhaps we could just tag onto the schedule_ipi() function instead of having to create a new IPI for all archs?
-- Steve
Quoted 20 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc Ingo Molnar <[email protected]>Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>
Re: [PATCH 04/32] x86: New cpuset nohz irq vector
2012/10/30 Steven Rostedt <[email protected]>:
Quoted 13 lines · Steven Rostedt
Quoted 12 lines
plain text document attachment
(0004-x86-New-cpuset-nohz-irq-vector.patch)
From: Frederic Weisbecker <[email protected]>We need a way to send an IPI (remote or local) in order to asynchronously restart the tick for CPUs in nohz adaptive mode.
This must be asynchronous such that we can trigger it with irqs disabled. This must be usable as a self-IPI as well for example in cases where we want to avoid random dealock scenario while restarting the tick inline otherwise.
Quoted 8 lines
Quoted 5 lines
This only settles the x86 backend. The core tick restart function will be defined in a later patch.
[CHECKME: Perhaps we instead need to use irq work for self IPIs. But we also need a way to send async remote IPIs.]
Probably just use
irq_workfor self ipis, and normal ipis for other CPUs.
Right. And that's one more reason why we want to know if the arch implements irq work with self ipis or not. If the arch can't, then we just don't stop the tick.
Quoted 2 lines
Also, what reason do we have to force a task out of nohz? IOW, do we really need this?
When a posix CPU timer is enqueued, when a new task is enqueued, etc...
Quoted 3 lines
Also, perhaps we could just tag onto the
schedule_ipi()function instead of having to create a new IPI for all archs?
irq work should be just fine. No need to add more overhead on the schedule ipi I think.
Steven Rostedt
reviewerRe: [PATCH 04/32] x86: New cpuset nohz irq vector
Quoted 6 lines · Frederic Weisbecker
Quoted 2 lines
Probably just use
irq_workfor self ipis, and normal ipis for other CPUs.Right. And that's one more reason why we want to know if the arch implements irq work with self ipis or not. If the arch can't, then we just don't stop the tick.
We can just allow certain archs to have cpuset/nohz. Make it depend on features that you want (or makes nohz easier to implement).
Quoted 5 lines
Quoted 3 lines
Also, what reason do we have to force a task out of nohz? IOW, do we really need this?
When a posix CPU timer is enqueued, when a new task is enqueued, etc...
I was thinking about something other than itself. That is, who would enqueue a posix cpu timer on the cpu other than the task running with nohz on that cpu?
A new task would send the schedule ipi too. Which would enqueue the task and take the cpu out of nohz, no?
Quoted 7 lines
Quoted 4 lines
Also, perhaps we could just tag onto the
schedule_ipi()function instead of having to create a new IPI for all archs?irq work should be just fine. No need to add more overhead on the schedule ipi I think.
irq_work can send the work to another CPU right? This part I wasn't sure about.
-- Steve
Re: [PATCH 04/32] x86: New cpuset nohz irq vector
2012/10/31 Steven Rostedt <[email protected]>:
Quoted 11 lines · Frederic Weisbecker
Quoted 7 lines
Quoted 3 lines
Probably just use
irq_workfor self ipis, and normal ipis for other CPUs.Right. And that's one more reason why we want to know if the arch implements irq work with self ipis or not. If the arch can't, then we just don't stop the tick.
We can just allow certain archs to have cpuset/nohz. Make it depend on features that you want (or makes nohz easier to implement).
Right.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Also, what reason do we have to force a task out of nohz? IOW, do we really need this?
When a posix CPU timer is enqueued, when a new task is enqueued, etc...
I was thinking about something other than itself. That is, who would enqueue a posix cpu timer on the cpu other than the task running with nohz on that cpu?
If the posix cpu timer is process wide (ie: whole threadgroup) this can happen.
Quoted 2 lines
A new task would send the schedule ipi too. Which would enqueue the task and take the cpu out of nohz, no?
Not if it's enqueued locally. And in this case we don't want to restart the tick from the ttwu path in order to avoid funny locking scenario. So a self IPI would do the trick.
Quoted 5 lines
Quoted 2 lines
irq work should be just fine. No need to add more overhead on the schedule ipi I think.
irq_workcan send the work to another CPU right? This part I wasn't sure about.
"Claiming" a work itself can be a cross CPU competition: multiple CPUs may want to queue the work at the same time, only one should succeed. Once claimed though, the work can only been enqueued locally.
Steven Rostedt
reviewerPATCH 5/32 nohz: Adaptive tick stop and restart on nohz cpuset
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
When a CPU is included in a nohz cpuset, try to switch it to nohz mode from the interrupt exit path if it is running a single non-idle task.
Then restart the tick if necessary if we are enqueuing a second task while the timer is stopped, so that the scheduler tick is rearmed.
[TODO: Handle the many things done from scheduler_tick()]
[ Included build fix from Geoff Levand ]
--- arch/x86/kernel/smp.c | 2 ++ include/linux/sched.h | 6 ++++ include/linux/tick.h | 11 +++++- init/Kconfig | 2 +- kernel/sched/core.c | 24 +++++++++++++ kernel/sched/sched.h | 12 +++++++ kernel/softirq.c | 6 ++-- kernel/time/tick-sched.c | 86 +++++++++++++++++++++++++++++++++++++++++----- 8 files changed, 137 insertions(+), 12 deletions(-)
arch/x86/kernel/smp.c +2 -0
@@ -23,6 +23,7 @@#include <linux/interrupt.h> #include <linux/cpu.h> #include <linux/gfp.h> +#include <linux/tick.h> #include <asm/mtrr.h> #include <asm/tlbflush.h>
@@ -275,6 +276,7 @@ void smp_cpuset_update_nohz_interrupt(struct pt_regs *regs) {ack_APIC_irq ();irq_enter (); +tick_nohz_check_adaptive ();inc_irq_stat (irq_call_count);irq_exit (); }
include/linux/sched.h +6 -0
@@ -2753,6 +2753,12 @@ static inline void inc_syscw(struct task_struct *tsk)#define TASK_SIZE_OF (tsk) TASK_SIZE#endif +#ifdef CONFIG_CPUSETS_NO_HZ +extern bool sched_can_stop_tick (void ); +#else +static inline bool sched_can_stop_tick (void ) {return false; } +#endif +#ifdef CONFIG_MM_OWNER extern void mm_update_next_owner (struct mm_struct *mm);extern void mm_init_owner (struct mm_struct *mm,struct task_struct *p);
include/linux/tick.h +10 -1
@@ -124,11 +124,12 @@ static inline int tick_oneshot_mode_active(void) { return 0; }# ifdef CONFIG_NO_HZ extern void tick_nohz_idle_enter (void );extern void tick_nohz_idle_exit (void ); +extern void tick_nohz_restart_sched_tick (void );extern void tick_nohz_irq_exit (void );extern ktime_t tick_nohz_get_sleep_length (void );extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time);extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time); -# else +# else /* !NO_HZ */ static inline void tick_nohz_idle_enter (void ) { }static inline void tick_nohz_idle_exit (void ) { }
@@ -142,4 +143,12 @@ static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; }static inline u64 get_cpu_iowait_time_us (int cpu,u64 *unused) {return -1 ; }# endif /* !NO_HZ */ +#ifdef CONFIG_CPUSETS_NO_HZ +extern void tick_nohz_check_adaptive (void ); +extern void tick_nohz_post_schedule (void ); +#else /* !CPUSETS_NO_HZ */ +static inline void tick_nohz_check_adaptive (void ) { } +static inline void tick_nohz_post_schedule (void ) { } +#endif /* CPUSETS_NO_HZ */ +#endif
init/Kconfig +1 -1
@@ -751,7 +751,7 @@ config PROC_PID_CPUSET config CPUSETS_NO_HZ bool "Tickless cpusets" - depends on CPUSETS && HAVE_CPUSETS_NO_HZ + depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS help This options let you apply a nohz property to a cpuset such that the periodic timer tick tries to be avoided when possible on
kernel/sched/core.c +24 -0
@@ -1196,6 +1196,29 @@ static void update_avg(u64 *avg, u64 sample) }#endif +#ifdef CONFIG_CPUSETS_NO_HZ +bool sched_can_stop_tick (void ) +{ +struct rq *rq; + + rq =this_rq (); + +/* +* This is called right after cpuset_adaptive_nohz() that +* uses atomic_add_return() so that we are ordered against +* cpu_adaptive_nohz_ref. When inc_nr_running() sends an +* IPI to this CPU, we are guaranteed to see the update on +* nr_running. +*/ + +/* More than one running task need preemption */ +if (rq->nr_running >1 ) +return false; + +return true; +} +#endif +static void ttwu_stat (struct task_struct *p,int cpu,int wake_flags) {
@@ -1897,6 +1920,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* frame will be invalid. */ finish_task_switch (this_rq (), prev); +tick_nohz_post_schedule (); }/*
kernel/sched/sched.h +12 -0
@@ -1,6 +1,7 @@#include <linux/sched.h> #include <linux/mutex.h> +#include <linux/cpuset.h> #include <linux/spinlock.h> #include <linux/stop_machine.h>
@@ -927,6 +928,17 @@ static inline u64 steal_ticks(u64 steal)static inline void inc_nr_running (struct rq *rq) { rq->nr_running++; + +if (rq->nr_running ==2 ) { +/* +* cpuset_cpu_adaptive_nohz() uses atomic_add_return() +* to order against rq->nr_running updates. This way +* the CPU that receives the IPI is guaranteed to see +* the update on nr_running without the rq->lock. +*/ +if (cpuset_cpu_adaptive_nohz (rq->cpu)) +smp_cpuset_update_nohz (rq->cpu); + } }static inline void dec_nr_running (struct rq *rq)
kernel/softirq.c +4 -2
@@ -25,6 +25,7 @@#include <linux/smp.h> #include <linux/smpboot.h> #include <linux/tick.h> +#include <linux/cpuset.h> #define CREATE_TRACE_POINTS #include <trace/events/irq.h>
@@ -307,7 +308,8 @@ void irq_enter(void)int cpu =smp_processor_id ();rcu_irq_enter (); -if (is_idle_task (current) && !in_interrupt ()) { + +if ((is_idle_task (current) ||cpuset_adaptive_nohz ()) && !in_interrupt ()) {/* * Prevent raise_softirq from needlessly waking up ksoftirqd * here, as softirq will be serviced on return from interrupt.
@@ -349,7 +351,7 @@ void irq_exit(void)#ifdef CONFIG_NO_HZ /* Make sure that timer wheel updates are propagated */ -if (idle_cpu (smp_processor_id ()) && !in_interrupt () && !need_resched ()) +if (!in_interrupt ())tick_nohz_irq_exit ();#endif rcu_irq_exit ();
kernel/time/tick-sched.c +78 -8
140 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerRe: [PATCH 05/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 5 lines · Steven Rostedt
kernel/sched/core.c +0
-0@@ -1196,6 +1196,29 @@ static void update_avg(u64 *avg, u64 sample) }#endif
kernel/sched/core.c static void update_avg(u64 *avg, u64 sample) · line 1207
Quoted 10 lines
kernel/sched/core.c +9
-0+#ifdef CONFIG_CPUSETS_NO_HZ +bool sched_can_stop_tick (void ) +{ +struct rq *rq; + + rq =this_rq (); + +/* +* This is called right after cpuset_adaptive_nohz() that
See below (for this caller).
Quoted 45 lines
kernel/sched/core.c +15
-0+* uses atomic_add_return() so that we are ordered against +* cpu_adaptive_nohz_ref. When inc_nr_running() sends an +* IPI to this CPU, we are guaranteed to see the update on +* nr_running. +*/ + +/* More than one running task need preemption */ +if (rq->nr_running >1 ) +return false; + + return true; +} +#endif +static void ttwu_stat (struct task_struct *p,int cpu,int wake_flags) {@@ -1897,6 +1920,7 @@ context_switch(struct rq *rq, struct task_struct *prev,* frame will be invalid. */ finish_task_switch (this_rq (), prev); +tick_nohz_post_schedule (); }/* kernel/sched/sched.h +5
-0@@ -1,6 +1,7 @@#include <linux/sched.h> #include <linux/mutex.h> +#include <linux/cpuset.h> #include <linux/spinlock.h> #include <linux/stop_machine.h> @@ -927,6 +928,17 @@ static inline u64 steal_ticks(u64 steal)static inline void inc_nr_running (struct rq *rq) { rq->nr_running++; + +if (rq->nr_running ==2 ) { +/* +* cpuset_cpu_adaptive_nohz() uses atomic_add_return()
kernel/sched/sched.h static inline u64 steal_ticks(u64 steal) · line 944
Quoted 10 lines
kernel/sched/sched.h +7
-0+* to order against rq->nr_running updates. This way +* the CPU that receives the IPI is guaranteed to see +* the update on nr_running without the rq->lock. +*/ +if (cpuset_cpu_adaptive_nohz (rq->cpu)) +smp_cpuset_update_nohz (rq->cpu); + } }static inline void dec_nr_running (struct rq *rq)
Should we add one for dec_nr_running()? Or is this done elsewhere. I would think that there's a good chance that we can miss a chance to stop the tick.
Quoted 36 lines
kernel/softirq.c +4
-2@@ -25,6 +25,7 @@#include <linux/smp.h> #include <linux/smpboot.h> #include <linux/tick.h> +#include <linux/cpuset.h> #define CREATE_TRACE_POINTS #include <trace/events/irq.h> @@ -307,7 +308,8 @@ void irq_enter(void)int cpu =smp_processor_id ();rcu_irq_enter (); -if (is_idle_task (current) && !in_interrupt ()) { + +if ((is_idle_task (current) ||cpuset_adaptive_nohz ()) && !in_interrupt ()) {/* * Prevent raise_softirq from needlessly waking up ksoftirqd * here, as softirq will be serviced on return from interrupt. @@ -349,7 +351,7 @@ void irq_exit(void)#ifdef CONFIG_NO_HZ /* Make sure that timer wheel updates are propagated */ -if (idle_cpu (smp_processor_id ()) && !in_interrupt () && !need_resched ()) +if (!in_interrupt ())tick_nohz_irq_exit ();#endif rcu_irq_exit ();kernel/time/tick-sched.c +0
-0
kernel/time/tick-sched.c void tick_nohz_idle_enter(void) · line 521
Quoted 10 lines
kernel/time/tick-sched.c +7
-0@@ -512,6 +512,24 @@ void tick_nohz_idle_enter(void)local_irq_enable (); } +static void tick_nohz_cpuset_stop_tick (struct tick_sched *ts) +{ +#ifdef CONFIG_CPUSETS_NO_HZ +int cpu =smp_processor_id (); + +if (!cpuset_adaptive_nohz () ||is_idle_task (current)) +return ;
The above is most likely true. Lets remove the memory barrier in cpuset_adaptive_nohz(), just add an explicit one here, in the slow path.
/* Before checking the below conditions, we must first
* make sure that the cpuset/nohz is active, so we do
* not miss a deactivating IPI.
* ie. when nr_running == 2, an IPI is sent, and this
* code must see the nr_running changed after testing
* if the current CPU is adaptive nohz.
*/
smp_mb();
Quoted 90 lines
kernel/time/tick-sched.c +44
-7+ +if (!ts->tick_stopped && ts->nohz_mode ==NOHZ_MODE_INACTIVE ) +return ; + +if (!sched_can_stop_tick ()) +return ; + +tick_nohz_stop_sched_tick (ts,ktime_get (), cpu); +#endif +} +/** * tick_nohz_irq_exit - update next tick event from interrupt exit * @@ -524,10 +542,12 @@ void tick_nohz_irq_exit(void) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); -if (!ts->inidle) -return ; - -__tick_nohz_idle_enter (ts); +if (ts->inidle) { +if (!need_resched ()) +__tick_nohz_idle_enter (ts); + }else { +tick_nohz_cpuset_stop_tick (ts); + } }/** @@ -568,7 +588,7 @@ static void tick_nohz_restart(struct tick_sched *ts, ktime_t now) } } -static void tick_nohz_restart_sched_tick (struct tick_sched *ts,ktime_t now) +static void __tick_nohz_restart_sched_tick (struct tick_sched *ts,ktime_t now) {/* Update jiffies first */ tick_do_update_jiffies64 (now);@@ -584,6 +604,31 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now)tick_nohz_restart (ts, now); } +/** +* tick_nohz_restart_sched_tick - restart the tick for a tickless CPU +* +* Restart the tick when the CPU is in adaptive tickless mode. +*/ +void tick_nohz_restart_sched_tick (void ) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +unsigned long flags; +ktime_t now; + +local_irq_save (flags); + +if (!ts->tick_stopped) { +local_irq_restore (flags); +return ; + } + + now =ktime_get (); +__tick_nohz_restart_sched_tick (ts, now); + +local_irq_restore (flags); +} + +static void tick_nohz_account_idle_ticks (struct tick_sched *ts) {#ifndef CONFIG_VIRT_CPU_ACCOUNTING @@ -630,7 +675,7 @@ void tick_nohz_idle_exit(void)if (ts->tick_stopped) {nohz_balance_enter_idle (cpu);calc_load_exit_idle (); -tick_nohz_restart_sched_tick (ts, now); +__tick_nohz_restart_sched_tick (ts, now);tick_nohz_account_idle_ticks (ts); }@@ -791,7 +836,6 @@ void tick_check_idle(int cpu) }#ifdef CONFIG_CPUSETS_NO_HZ -/* * Take the timer duty if nobody is taking care of it. * If a CPU already does and and it's in a nohz cpuset,
kernel/time/tick-sched.c static void tick_do_timer_check_handler(int cpu) · line 861
Quoted 8 lines
kernel/time/tick-sched.c +5
-0@@ -810,6 +854,31 @@ static void tick_do_timer_check_handler(int cpu) } } +void tick_nohz_check_adaptive (void ) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +if (cpuset_adaptive_nohz ()) {
Add smp_mb() here too, with the same comment.
Another option may be to only add the memory barrier on the true case:
static inline bool cpuset_adaptive_nohz(void)
{
if (atomic_read(this_cpu_read(cpu_adaptive_nohz_ref)) > 0) {
/*
* In order not to miss cases where we need to enable
* the tick again, we must make sure that all new checks
* are visible after we find we are in adaptive nohz
* mode.
*/
smp_mb();
return true;
}
return false;
}The above will only force the expensive memory barrier when adaptive nohz is enabled. All other cases will avoid that overhead. If we miss disabling the tick, the next tick should disable it. But we must make sure that we enable it when needed.
But as this is called in several fast paths for everyone, we need to keep the overhead of the default case as low as possible.
Quoted 32 lines
kernel/time/tick-sched.c +21
-0+if (ts->tick_stopped && !is_idle_task (current)) { +if (!sched_can_stop_tick ()) +tick_nohz_restart_sched_tick (); + } + } +} + +void tick_nohz_post_schedule (void ) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +/* +* No need to disable irqs here. The worst that can happen +* is an irq that comes and restart the tick before us. +* tick_nohz_restart_sched_tick() is irq safe. +*/ +if (ts->tick_stopped) +tick_nohz_restart_sched_tick (); +} +#else static void tick_do_timer_check_handler (int cpu)@@ -856,6 +925,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)* no valid regs pointer */ if (regs) { +int user =user_mode (regs);/* * When we are idle and the tick is stopped, we have to touch * the watchdog as we might not schedule for a really long
kernel/time/tick-sched.c static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer) · line 942
Quoted 5 lines
kernel/time/tick-sched.c +1
-1@@ -869,7 +939,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)if (is_idle_task (current)) ts->idle_jiffies++; } -update_process_times (user_mode (regs)); +update_process_times (user);
What's the purpose of this change?
-- Steve
Quoted 3 lines
kernel/time/tick-sched.c +0
-0profile_tick (CPU_PROFILING ); }
Steven Rostedt
reviewerPATCH 6/32 nohz/cpuset: Dont turn off the tick if rcu needs it
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we are not necessarily going to enter into rcu extended quiescent state, so we may need to keep the tick to note current CPU's quiescent states.
[added build fix from Zen Lin]
--- include/linux/rcupdate.h | 1 + kernel/rcutree.c | 3 +-- kernel/time/tick-sched.c | 22 ++++++++++++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)
include/linux/rcupdate.h +1 -0
@@ -186,6 +186,7 @@ static inline int rcu_preempt_depth(void)extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern void rcu_check_callbacks (int cpu,int user); +extern int rcu_pending (int cpu);struct notifier_block ;extern void rcu_idle_enter (void );extern void rcu_idle_exit (void );
kernel/rcutree.c +1 -2
@@ -234,7 +234,6 @@ module_param(jiffies_till_next_fqs, ulong, 0644);static void force_qs_rnp (struct rcu_state *rsp,int (*f)(struct rcu_data *));static void force_quiescent_state (struct rcu_state *rsp); -static int rcu_pending (int cpu);/* * Return the number of RCU-sched batches processed thus far for debug & stats.
@@ -2429,7 +2428,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)* by the current CPU, returning 1 if so. This function is part of the * RCU implementation; it is -not- an exported member of the RCU API. */ -static int rcu_pending (int cpu) +int rcu_pending (int cpu) {struct rcu_state *rsp;
kernel/time/tick-sched.c +18 -4
@@ -512,9 +512,21 @@ void tick_nohz_idle_enter(void)local_irq_enable (); } +#ifdef CONFIG_CPUSETS_NO_HZ +static bool can_stop_adaptive_tick (void ) +{ +if (!sched_can_stop_tick ()) +return false; + +/* Is there a grace period to complete ? */ +if (rcu_pending (smp_processor_id ())) +return false; + +return true; +} +static void tick_nohz_cpuset_stop_tick (struct tick_sched *ts) { -#ifdef CONFIG_CPUSETS_NO_HZ int cpu =smp_processor_id ();if (!cpuset_adaptive_nohz () ||is_idle_task (current))
@@ -523,12 +535,14 @@ static void tick_nohz_cpuset_stop_tick(struct tick_sched *ts)if (!ts->tick_stopped && ts->nohz_mode ==NOHZ_MODE_INACTIVE )return ; -if (!sched_can_stop_tick ()) +if (!can_stop_adaptive_tick ())return ; tick_nohz_stop_sched_tick(ts, ktime_get(), cpu); -#endif } +#else +static void tick_nohz_cpuset_stop_tick (struct tick_sched *ts) { } +#endif /** * tick_nohz_irq_exit - update next tick event from interrupt exit
@@ -860,7 +874,7 @@ void tick_nohz_check_adaptive(void)if (cpuset_adaptive_nohz ()) {if (ts->tick_stopped && !is_idle_task (current)) { -if (!sched_can_stop_tick ()) +if (!can_stop_adaptive_tick ())tick_nohz_restart_sched_tick (); } }
Signature · 1 line
1.7.10.4
Re: [PATCH 06/32] nohz/cpuset: Dont turn off the tick if rcu needs it
Quoted 11 lines · Steven Rostedt
plain text document attachment (0006-nohz-cpuset-Don-t-turn-off-the-tick-if-rcu-needs-it.patch)
From: Frederic Weisbecker <[email protected]>If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we are not necessarily going to enter into rcu extended quiescent state, so we may need to keep the tick to note current CPU's quiescent states.
[added build fix from Zen Lin]
I don't see anything obviously troubling in this patch.
-- Steve
Quoted 21 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc Ingo Molnar <[email protected]>Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Steven Rostedt <[email protected]>
Steven Rostedt
reviewerPATCH 7/32 nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Wake up a CPU when a timer list timer is enqueued there and the CPU is in adaptive nohz mode. Sending an IPI to it makes it reconsidering the next timer to program on top of recent updates.
--- include/linux/sched.h | 4 ++-- kernel/sched/core.c | 28 +++++++++++++++++++++++++++- kernel/timer.c | 2 +- 3 files changed, 30 insertions(+), 4 deletions(-)
include/linux/sched.h +2 -2
@@ -1972,9 +1972,9 @@ static inline void idle_task_exit(void) {}#endif #if defined(CONFIG_NO_HZ ) && defined(CONFIG_SMP ) -extern void wake_up_idle_cpu (int cpu); +extern void wake_up_nohz_cpu (int cpu);#else -static inline void wake_up_idle_cpu (int cpu) { } +static inline void wake_up_nohz_cpu (int cpu) { }#endif extern unsigned int sysctl_sched_latency;
kernel/sched/core.c +27 -1
@@ -578,7 +578,7 @@ unlock:* account when the CPU goes back to idle and evaluates the timer * wheel for the next timer event. */ -void wake_up_idle_cpu (int cpu) +static void wake_up_idle_cpu (int cpu) {struct rq *rq =cpu_rq (cpu);
@@ -608,6 +608,32 @@ void wake_up_idle_cpu(int cpu)smp_send_reschedule (cpu); } +static bool wake_up_cpuset_nohz_cpu (int cpu) +{ +#ifdef CONFIG_CPUSETS_NO_HZ +/* +* If the current CPU doesn't see the target as nohz +* then it means the target hasn't seen itself nohz +* yet either. In this case we don't send an IPI to the +* target because it hasn't yet tried to stop the tick. +* But if the nohz flag is set concurrently, the target +* will find the newly enqueued timer once we release +* the base->lock. +*/ +if (cpuset_cpu_adaptive_nohz (cpu)) { +smp_cpuset_update_nohz (cpu); +return true; + } +#endif +return false; +} + +void wake_up_nohz_cpu (int cpu) +{ +if (!wake_up_cpuset_nohz_cpu (cpu)) +wake_up_idle_cpu (cpu); +} +static inline bool got_nohz_idle_kick (void ) {int cpu =smp_processor_id ();
kernel/timer.c +1 -1
@@ -936,7 +936,7 @@ void add_timer_on(struct timer_list *timer, int cpu)* makes sure that a CPU on the way to idle can not evaluate * the timer wheel. */ -wake_up_idle_cpu (cpu); +wake_up_nohz_cpu (cpu);spin_unlock_irqrestore (&base->lock, flags); }EXPORT_SYMBOL_GPL (add_timer_on);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 8/32 nohz/cpuset: Dont stop the tick if posix cpu timers are running
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
If either a per thread or a per process posix cpu timer is running, don't stop the tick.
TODO: restart the tick if it is stopped and a posix cpu timer is enqueued. Check we probably need a memory barrier for the per process posix timer that can be enqueued from another task of the group.
--- include/linux/posix-timers.h | 1 + kernel/posix-cpu-timers.c | 12 ++++++++++++ kernel/time/tick-sched.c | 4 ++++ 3 files changed, 17 insertions(+)
include/linux/posix-timers.h +1 -0
@@ -119,6 +119,7 @@ int posix_timer_event(struct k_itimer *timr, int si_private);void posix_cpu_timer_schedule (struct k_itimer *timer);void run_posix_cpu_timers (struct task_struct *task); +bool posix_cpu_timers_running (struct task_struct *tsk);void posix_cpu_timers_exit (struct task_struct *task);void posix_cpu_timers_exit_group (struct task_struct *task);
kernel/posix-cpu-timers.c +12 -0
@@ -6,6 +6,7 @@#include <linux/posix-timers.h> #include <linux/errno.h> #include <linux/math64.h> +#include <linux/cpuset.h> #include <asm/uaccess.h> #include <linux/kernel_stat.h> #include <trace/events/timer.h>
@@ -1274,6 +1275,17 @@ static inline int fastpath_timer_check(struct task_struct *tsk)return 0 ; } +bool posix_cpu_timers_running (struct task_struct *tsk) +{ +if (!task_cputime_zero (&tsk->cputime_expires)) +return true; + +if (tsk->signal->cputimer.running) +return true; + +return false; +} +/* * This is called from the timer interrupt handler. The irq handler has * already updated our counts. We need to check if any timers fire now.
kernel/time/tick-sched.c +4 -0
@@ -21,6 +21,7 @@#include <linux/sched.h> #include <linux/module.h> #include <linux/cpuset.h> +#include <linux/posix-timers.h> #include <asm/irq_regs.h>
@@ -518,6 +519,9 @@ static bool can_stop_adaptive_tick(void)if (!sched_can_stop_tick ())return false; +if (posix_cpu_timers_running (current)) +return false; +/* Is there a grace period to complete ? */ if (rcu_pending (smp_processor_id ()))return false;
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 9/32 nohz/cpuset: Restart tick when nohz flag is cleared on cpuset
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Issue an IPI to restart the tick on a CPU that belongs to a cpuset when its nohz flag gets cleared.
--- include/linux/cpuset.h | 2 ++ kernel/cpuset.c | 25 +++++++++++++++++++++++-- kernel/time/tick-sched.c | 8 ++++++++ 3 files changed, 33 insertions(+), 2 deletions(-)
include/linux/cpuset.h +2 -0
@@ -260,6 +260,8 @@ static inline bool cpuset_adaptive_nohz(void)*/ return cpuset_cpu_adaptive_nohz(smp_processor_id()); } + +extern void cpuset_exit_nohz_interrupt (void *unused);#else static inline bool cpuset_cpu_adaptive_nohz (int cpu) {return false; }static inline bool cpuset_adaptive_nohz (void ) {return false; }
kernel/cpuset.c +23 -2
@@ -1200,6 +1200,14 @@ static void cpuset_change_flag(struct task_struct *tsk,DEFINE_PER_CPU (atomic_t, cpu_adaptive_nohz_ref); +static void cpu_exit_nohz (int cpu) +{ +preempt_disable (); +smp_call_function_single (cpu, cpuset_exit_nohz_interrupt, +NULL , true); +preempt_enable (); +} +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) {int cpu;
@@ -1211,9 +1219,22 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)for_each_cpu (cpu, cs->cpus_allowed) {atomic_t *ref = &per_cpu (cpu_adaptive_nohz_ref, cpu);if (is_adaptive_nohz (cs)) -atomic_inc (ref); + val =atomic_inc_return (ref);else -atomic_dec (ref); + val =atomic_dec_return (ref); + +if (!val) { +/* +* The update to cpu_adaptive_nohz_ref must be +* visible right away. So that once we restart the tick +* from the IPI, it won't be stopped again due to cache +* update lag. +* FIXME: We probably need more to ensure this value is really +* visible right away. +*/ +smp_mb (); +cpu_exit_nohz (cpu); + } } }#else
kernel/time/tick-sched.c +8 -0
@@ -884,6 +884,14 @@ void tick_nohz_check_adaptive(void) } } +void cpuset_exit_nohz_interrupt (void *unused) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +if (ts->tick_stopped && !is_idle_task (current)) +tick_nohz_restart_adaptive (); +} +void tick_nohz_post_schedule (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerRe: [PATCH 09/32] nohz/cpuset: Restart tick when nohz flag is cleared on cpuset
Quoted 72 lines · Steven Rostedt
plain text document attachment (0009-nohz-cpuset-Restart-tick-when-nohz-flag-is-cleared-o.patch)
From: Frederic Weisbecker <[email protected]>Issue an IPI to restart the tick on a CPU that belongs to a cpuset when its nohz flag gets cleared.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc Ingo Molnar <[email protected]>Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>--- include/linux/cpuset.h | 2 ++ kernel/cpuset.c | 25 +++++++++++++++++++++++-- kernel/time/tick-sched.c | 8 ++++++++ 3 files changed, 33 insertions(+), 2 deletions(-)include/linux/cpuset.h +2
-0@@ -260,6 +260,8 @@ static inline bool cpuset_adaptive_nohz(void)*/ return cpuset_cpu_adaptive_nohz(smp_processor_id()); } + +extern void cpuset_exit_nohz_interrupt (void *unused);#else static inline bool cpuset_cpu_adaptive_nohz (int cpu) {return false; }static inline bool cpuset_adaptive_nohz (void ) {return false; }kernel/cpuset.c +9
-2@@ -1200,6 +1200,14 @@ static void cpuset_change_flag(struct task_struct *tsk,DEFINE_PER_CPU (atomic_t, cpu_adaptive_nohz_ref); +static void cpu_exit_nohz (int cpu) +{ +preempt_disable (); +smp_call_function_single (cpu, cpuset_exit_nohz_interrupt, +NULL , true); +preempt_enable (); +} +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) {int cpu;@@ -1211,9 +1219,22 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)for_each_cpu (cpu, cs->cpus_allowed) {atomic_t *ref = &per_cpu (cpu_adaptive_nohz_ref, cpu);if (is_adaptive_nohz (cs)) -atomic_inc (ref); + val =atomic_inc_return (ref);else -atomic_dec (ref);
kernel/cpuset.c static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs) · line 1233
Quoted 10 lines
kernel/cpuset.c +10
-0+ val =atomic_dec_return (ref); + +if (!val) { +/* +* The update to cpu_adaptive_nohz_ref must be +* visible right away. So that once we restart the tick +* from the IPI, it won't be stopped again due to cache +* update lag. +* FIXME: We probably need more to ensure this value is really +* visible right away.
What more do you want? stomp_machine()??
kernel/cpuset.c static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs) · line 1235
Quoted 2 lines
kernel/cpuset.c +2
-0+*/ +smp_mb ();
The atomic_inc_return() and atomic_dec_return() already imply a smp_mb().
Later patches change this code, so I wont dwell on this patch too much.
Quoted 10 lines
kernel/cpuset.c +2
-0+ cpu_exit_nohz(cpu); + } } }#else kernel/time/tick-sched.c +0
-0
kernel/time/tick-sched.c void tick_nohz_check_adaptive(void) · line 892
Quoted 9 lines
kernel/time/tick-sched.c +6
-0@@ -884,6 +884,14 @@ void tick_nohz_check_adaptive(void) } } +void cpuset_exit_nohz_interrupt (void *unused) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +if (ts->tick_stopped && !is_idle_task (current)) +tick_nohz_restart_adaptive ();
BTW, what a confusing name. "restart_adaptive()"? It sounds like we are going to restart the adaptive code, like restarting NOHZ.
-- Steve
Quoted 5 lines
kernel/time/tick-sched.c +2
-0+} +void tick_nohz_post_schedule (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);
Steven Rostedt
reviewerPATCH 10/32 nohz/cpuset: Restart the tick if printk needs it
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
If we are in nohz adaptive mode and printk is called, the tick is missing to wake up the logger. We need to restart the tick when that happens. Do this asynchronously by issuing a tick restart self IPI to avoid deadlocking with the current random locking chain.
--- kernel/printk.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
kernel/printk.c +14 -1
@@ -42,6 +42,7 @@#include <linux/notifier.h> #include <linux/rculist.h> #include <linux/poll.h> +#include <linux/cpuset.h> #include <asm/uaccess.h>
@@ -1977,8 +1978,20 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void ) { -if (waitqueue_active (&log_wait)) +unsigned long flags; + +if (waitqueue_active (&log_wait)) {this_cpu_or (printk_pending,PRINTK_PENDING_WAKEUP ); +/* Make it visible from any interrupt from now */ +barrier (); +/* +* It's safe to check that even if interrupts are not disabled. +* If we enable nohz adaptive mode concurrently, we'll see the +* printk_pending value and thus keep a periodic tick behaviour. +*/ +if (cpuset_adaptive_nohz ()) +smp_cpuset_update_nohz (smp_processor_id ()); + } }static void console_cont_flush (char *text,size_t size)
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerRe: [PATCH 10/32] nohz/cpuset: Restart the tick if printk needs it
Quoted 47 lines · Steven Rostedt
plain text document attachment (0010-nohz-cpuset-Restart-the-tick-if-printk-needs-it.patch)
From: Frederic Weisbecker <[email protected]>If we are in nohz adaptive mode and printk is called, the tick is missing to wake up the logger. We need to restart the tick when that happens. Do this asynchronously by issuing a tick restart self IPI to avoid deadlocking with the current random locking chain.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc Ingo Molnar <[email protected]>Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>--- kernel/printk.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)kernel/printk.c +1
-0@@ -42,6 +42,7 @@#include <linux/notifier.h> #include <linux/rculist.h> #include <linux/poll.h> +#include <linux/cpuset.h> #include <asm/uaccess.h> @@ -1977,8 +1978,20 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void )
kernel/printk.c int printk_needs_cpu(int cpu) · line 1988
Quoted 10 lines
kernel/printk.c +7
-1{ -if (waitqueue_active (&log_wait)) +unsigned long flags; + +if (waitqueue_active (&log_wait)) {this_cpu_or (printk_pending,PRINTK_PENDING_WAKEUP ); +/* Make it visible from any interrupt from now */ +barrier (); +/* +* It's safe to check that even if interrupts are not disabled.
Probably need to at least disable preemption. I don't see any requirement that wake_up_klogd() needs to be called with preemption disabled.
The this_cpu_or() doesn't care which CPU it triggers, but the enabling of nohz does.
-- Steve
Quoted 9 lines
kernel/printk.c +6
-0+* If we enable nohz adaptive mode concurrently, we'll see the +* printk_pending value and thus keep a periodic tick behaviour. +*/ +if (cpuset_adaptive_nohz ()) +smp_cpuset_update_nohz (smp_processor_id ()); + } }static void console_cont_flush (char *text,size_t size)
Re: [PATCH 10/32] nohz/cpuset: Restart the tick if printk needs it
2012/10/30 Steven Rostedt <[email protected]>:
Quoted 6 lines
Probably need to at least disable preemption. I don't see any requirement that
wake_up_klogd()needs to be called with preemption disabled.The
this_cpu_or()doesn't care which CPU it triggers, but the enabling of nohz does.
This patch is deemed to be replaced with the printk in nohz patchset I'm working on. But it indeed to disable preemption as well and its irq work should be made per cpu.
Steven Rostedt
reviewerPATCH 11/32 rcu: Restart the tick on non-responding adaptive nohz CPUs
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
When a CPU in adaptive nohz mode doesn't respond to complete a grace period, issue it a specific IPI so that it restarts the tick and chases a quiescent state.
--- kernel/rcutree.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
kernel/rcutree.c +18 -0
@@ -53,6 +53,7 @@#include <linux/delay.h> #include <linux/stop_machine.h> #include <linux/random.h> +#include <linux/cpuset.h> #include "rcutree.h" #include <trace/events/rcu.h>
@@ -798,6 +799,20 @@ static int dyntick_save_progress_counter(struct rcu_data *rdp)return (rdp->dynticks_snap &0x1 ) ==0 ; } +static void cpuset_update_rcu_cpu (int cpu) +{ +#ifdef CONFIG_CPUSETS_NO_HZ +unsigned long flags; + +local_irq_save (flags); + +if (cpuset_cpu_adaptive_nohz (cpu)) +smp_cpuset_update_nohz (cpu); + +local_irq_restore (flags); +#endif +} +/* * Return true if the specified CPU has passed through a quiescent * state by virtue of being in or having passed through an dynticks
@@ -845,6 +860,9 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) rdp->offline_fqs++;return 1 ; } + +cpuset_update_rcu_cpu (rdp->cpu); +return 0 ; }
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 12/32 rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
If we enqueue an rcu callback, we need the CPU tick to stay alive until we take care of those by completing the appropriate grace period.
Thus, when we call_rcu(), send a self IPI that checks rcu_needs_cpu() so that we restore a periodic tick behaviour that can take care of everything.
--- kernel/rcutree.c | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/rcutree.c +7 -0
@@ -2081,6 +2081,13 @@ static void invoke_rcu_core(void)static void __call_rcu_core (struct rcu_state *rsp,struct rcu_data *rdp,struct rcu_head *head,unsigned long flags) { +/* Restart the timer if needed to handle the callbacks */ +if (cpuset_adaptive_nohz ()) { +/* Make updates on nxtlist visible to self IPI */ +barrier (); +smp_cpuset_update_nohz (smp_processor_id ()); + } +/* * If called from an extended quiescent state, invoke the RCU * core in order to force a re-evaluation of RCU's idleness.
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 13/32 nohz: Generalize tickless cpu time accounting
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
When the CPU enters idle, it saves the jiffies stamp into ts->idle_jiffies, increment this value by one every time there is a timer interrupt and accounts "jiffies - ts->idle_jiffies" idle ticks when we exit idle. This way we still account the idle CPU time even if the tick is stopped.
This patch settles the ground to generalize this for user and system accounting. ts->idle_jiffies becomes ts->saved_jiffies and a new member ts->saved_jiffies_whence indicates from which domain we saved the jiffies: user, system or idle.
This is one more step toward making the tickless infrastructure usable further idle contexts.
For now this is only used by idle but further patches make use of it for user and system.
--- include/linux/kernel_stat.h | 2 ++ include/linux/tick.h | 45 ++++++++++++++++++++-------------- kernel/sched/cputime.c | 22 +++++++++++++++++ kernel/time/tick-sched.c | 57 ++++++++++++++++++++++++++++--------------- kernel/time/timer_list.c | 3 ++- 5 files changed, 90 insertions(+), 39 deletions(-)
include/linux/kernel_stat.h +2 -0
@@ -122,7 +122,9 @@ static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)extern unsigned long long task_delta_exec (struct task_struct *);extern void account_user_time (struct task_struct *,cputime_t ,cputime_t ); +extern void account_user_ticks (struct task_struct *,unsigned long );extern void account_system_time (struct task_struct *,int ,cputime_t ,cputime_t ); +extern void account_system_ticks (struct task_struct *,unsigned long );extern void account_steal_time (cputime_t );extern void account_idle_time (cputime_t );
include/linux/tick.h +27 -18
65 lines not on this page · raw message
kernel/sched/cputime.c +22 -0
@@ -166,6 +166,17 @@ void account_user_time(struct task_struct *p, cputime_t cputime,acct_update_integrals (p); } +void account_user_ticks (struct task_struct *p,unsigned long ticks) +{ +cputime_t delta_cputime, delta_scaled; + +if (ticks) { + delta_cputime =jiffies_to_cputime (ticks); + delta_scaled =cputime_to_scaled (ticks); +account_user_time (p, delta_cputime, delta_scaled); + } +} +/* * Account guest cpu time to a process. * @p: the process that the cpu time gets accounted to
@@ -243,6 +254,17 @@ void account_system_time(struct task_struct *p, int hardirq_offset,__account_system_time (p, cputime, cputime_scaled, index); } +void account_system_ticks (struct task_struct *p,unsigned long ticks) +{ +cputime_t delta_cputime, delta_scaled; + +if (ticks) { + delta_cputime =jiffies_to_cputime (ticks); + delta_scaled =cputime_to_scaled (ticks); +account_system_time (p,0 , delta_cputime, delta_scaled); + } +} +/* * Account for involuntary wait time. * @cputime: the cpu time spent in involuntary wait
kernel/time/tick-sched.c +37 -20
97 lines not on this page · raw message
kernel/time/timer_list.c +2 -1
@@ -169,7 +169,8 @@ static void print_cpu(struct seq_file *m, int cpu, u64 now)P (nohz_mode);P_ns (last_tick);P (tick_stopped); -P (idle_jiffies); +/* CHECKME: Do we want saved_jiffies_whence as well? */ +P (saved_jiffies);P (idle_calls);P (idle_sleeps);P_ns (idle_entrytime);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 14/32 nohz/cpuset: Account user and system times in adaptive nohz mode
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
If we are not running the tick, we are not anymore regularly counting the user/system cputime at every jiffies.
To solve this, save a snapshot of the jiffies when we stop the tick and keep track of where we saved it: user or system. On top of this, we account the cputime elapsed when we cross the kernel entry/exit boundaries and when we restart the tick.
--- include/linux/tick.h | 12 +++++ kernel/sched/core.c | 1 + kernel/time/tick-sched.c | 129 +++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 140 insertions(+), 2 deletions(-)
include/linux/tick.h +12 -0
@@ -153,11 +153,23 @@ static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }# endif /* !NO_HZ */ #ifdef CONFIG_CPUSETS_NO_HZ +extern void tick_nohz_enter_kernel (void ); +extern void tick_nohz_exit_kernel (void ); +extern void tick_nohz_enter_exception (struct pt_regs *regs); +extern void tick_nohz_exit_exception (struct pt_regs *regs);extern void tick_nohz_check_adaptive (void ); +extern void tick_nohz_pre_schedule (void );extern void tick_nohz_post_schedule (void ); +extern bool tick_nohz_account_tick (void );#else /* !CPUSETS_NO_HZ */ +static inline void tick_nohz_enter_kernel (void ) { } +static inline void tick_nohz_exit_kernel (void ) { } +static inline void tick_nohz_enter_exception (struct pt_regs *regs) { } +static inline void tick_nohz_exit_exception (struct pt_regs *regs) { }static inline void tick_nohz_check_adaptive (void ) { } +static inline void tick_nohz_pre_schedule (void ) { }static inline void tick_nohz_post_schedule (void ) { } +static inline bool tick_nohz_account_tick (void ) {return false; }#endif /* CPUSETS_NO_HZ */ #endif
kernel/sched/core.c +1 -0
@@ -1771,6 +1771,7 @@ prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) {trace_sched_switch (prev, next); +tick_nohz_pre_schedule ();sched_info_switch (prev, next);perf_event_task_sched_out (prev, next);fire_sched_out_preempt_notifiers (prev, next);
kernel/time/tick-sched.c +127 -2
173 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 15/32 nohz/cpuset: New API to flush cputimes on nohz cpusets
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Provide a new API that sends an IPI to every CPUs included in nohz cpusets in order to flush their cputimes. It's going to be useful for those that want to see accurate cputimes on a nohz cpuset.
--- include/linux/cpuset.h | 2 ++ include/linux/tick.h | 1 + kernel/cpuset.c | 34 +++++++++++++++++++++++++++++++++- kernel/time/tick-sched.c | 21 ++++++++++++++++----- 4 files changed, 52 insertions(+), 6 deletions(-)
include/linux/cpuset.h +2 -0
@@ -262,9 +262,11 @@ static inline bool cpuset_adaptive_nohz(void) }extern void cpuset_exit_nohz_interrupt (void *unused); +extern void cpuset_nohz_flush_cputimes (void );#else static inline bool cpuset_cpu_adaptive_nohz (int cpu) {return false; }static inline bool cpuset_adaptive_nohz (void ) {return false; } +static inline void cpuset_nohz_flush_cputimes (void ) { }#endif /* CONFIG_CPUSETS_NO_HZ */
include/linux/tick.h +1 -0
@@ -161,6 +161,7 @@ extern void tick_nohz_check_adaptive(void);extern void tick_nohz_pre_schedule (void );extern void tick_nohz_post_schedule (void );extern bool tick_nohz_account_tick (void ); +extern void tick_nohz_flush_current_times (bool restart_tick);#else /* !CPUSETS_NO_HZ */ static inline void tick_nohz_enter_kernel (void ) { }static inline void tick_nohz_exit_kernel (void ) { }
kernel/cpuset.c +33 -1
@@ -59,6 +59,7 @@#include <linux/mutex.h> #include <linux/workqueue.h> #include <linux/cgroup.h> +#include <linux/tick.h> /* * Workqueue for cpuset related tasks.
@@ -1200,6 +1201,23 @@ static void cpuset_change_flag(struct task_struct *tsk,DEFINE_PER_CPU (atomic_t, cpu_adaptive_nohz_ref); +static cpumask_t nohz_cpuset_mask; + +static void flush_cputime_interrupt (void *unused) +{ +tick_nohz_flush_current_times (false); +} + +void cpuset_nohz_flush_cputimes (void ) +{ +preempt_disable (); +smp_call_function_many (&nohz_cpuset_mask, flush_cputime_interrupt, +NULL , true); +preempt_enable (); +/* Make the utime/stime updates visible */ +smp_mb (); +} +static void cpu_exit_nohz (int cpu) {preempt_disable ();
@@ -1223,7 +1241,15 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)else val =atomic_dec_return (ref); -if (!val) { +if (val ==1 ) { +cpumask_set_cpu (cpu, &nohz_cpuset_mask); +/* +* The mask update needs to be visible right away +* so that this CPU is part of the cputime IPI +* update right now. +*/ +smp_mb (); + }else if (!val) {/* * The update to cpu_adaptive_nohz_ref must be * visible right away. So that once we restart the tick
@@ -1234,6 +1260,12 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)*/ smp_mb ();cpu_exit_nohz (cpu); +/* +* Now that the tick has been restarted and cputimes +* flushed, we don't need anymore to be part of the +* cputime flush IPI. +*/ +cpumask_clear_cpu (cpu, &nohz_cpuset_mask); } } }
kernel/time/tick-sched.c +16 -5
@@ -712,7 +712,6 @@ static void tick_nohz_account_ticks(struct tick_sched *ts)WARN_ON_ONCE (1 ); } } - ts->saved_jiffies_whence =JIFFIES_SAVED_NONE ; }/**
@@ -746,6 +745,7 @@ void tick_nohz_idle_exit(void)__tick_nohz_restart_sched_tick (ts, now);#ifndef CONFIG_VIRT_CPU_ACCOUNTING tick_nohz_account_ticks (ts); + ts->saved_jiffies_whence =JIFFIES_SAVED_NONE ;#endif }
@@ -988,9 +988,7 @@ static void tick_do_timer_check_handler(int cpu)static void tick_nohz_restart_adaptive (void ) { -struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); - -tick_nohz_account_ticks (ts); +tick_nohz_flush_current_times (true);tick_nohz_restart_sched_tick ();clear_thread_flag (TIF_NOHZ ); }
@@ -1030,7 +1028,7 @@ void tick_nohz_pre_schedule(void)* on the prev task. */ if (test_thread_flag (TIF_NOHZ )) { -tick_nohz_account_ticks (ts); +tick_nohz_flush_current_times (true);clear_thread_flag (TIF_NOHZ ); } }
@@ -1047,6 +1045,19 @@ void tick_nohz_post_schedule(void)if (ts->tick_stopped) tick_nohz_restart_sched_tick(); } + +void tick_nohz_flush_current_times (bool restart_tick) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +if (ts->tick_stopped) { +tick_nohz_account_ticks (ts); +if (restart_tick) + ts->saved_jiffies_whence =JIFFIES_SAVED_NONE ; +else + ts->saved_jiffies = jiffies; + } +}#else static void tick_do_timer_check_handler (int cpu)
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 16/32 nohz/cpuset: Flush cputime on threads in nohz cpusets when waiting leader
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
When we wait for a zombie task, flush the cputimes on nohz cpusets in case we are waiting for a group leader that has threads running in nohz CPUs. This way thread_group_times() doesn't report stale values.
<doubts>
If I understood well the code, by the time we call that thread_group_times(), we may have childs that are still running, so this is necessary. But I need to check deeper.
</doubts>
--- kernel/exit.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/exit.c +8 -0
@@ -53,6 +53,7 @@#include <linux/oom.h> #include <linux/writeback.h> #include <linux/shm.h> +#include <linux/cpuset.h> #include <asm/uaccess.h> #include <asm/unistd.h>
@@ -1634,6 +1635,13 @@ repeat: (!wo->wo_pid || hlist_empty(&wo->wo_pid->tasks[wo->wo_type]))) goto notask; +/* +* For cputime in sub-threads before adding them. +* Must be called outside tasklist_lock lock because write lock +* can be acquired under irqs disabled. +*/ +cpuset_nohz_flush_cputimes (); +set_current_state (TASK_INTERRUPTIBLE );read_lock (&tasklist_lock); tsk = current;
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 17/32 nohz/cpuset: Flush cputimes on procfs stat file read
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
When we read a process's procfs stat file, we need to flush the cputimes of the tasks running in nohz cpusets in case some childs in the thread group are running there.
--- fs/proc/array.c | 2 ++ 1 file changed, 2 insertions(+)
fs/proc/array.c +2 -0
@@ -406,6 +406,8 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns, cutime = cstime = utime = stime =0 ; cgtime = gtime =0 ; +/* For thread group times */ +cpuset_nohz_flush_cputimes ();if (lock_task_sighand (task, &flags)) {struct signal_struct *sig = task->signal;
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 18/32 nohz/cpuset: Flush cputimes for getrusage() and times() syscalls
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Both syscalls need to iterate through the thread group to get the cputimes. As some threads of the group may be running on nohz cpuset, we need to flush the cputimes there.
--- kernel/sys.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sys.c +6 -0
@@ -47,6 +47,7 @@#include <linux/syscalls.h> #include <linux/kprobes.h> #include <linux/user_namespace.h> +#include <linux/cpuset.h> #include <linux/kmsg_dump.h> /* Move somewhere else to avoid recompiling? */
@@ -1045,6 +1046,8 @@ void do_sys_times(struct tms *tms) {cputime_t tgutime, tgstime, cutime, cstime; +cpuset_nohz_flush_cputimes (); +spin_lock_irq (¤t->sighand->siglock);thread_group_times (current, &tgutime, &tgstime); cutime = current->signal->cutime;
@@ -1710,6 +1713,9 @@ static void k_getrusage(struct task_struct *p, int who, struct rusage *r) goto out; } +/* For thread_group_times */ +cpuset_nohz_flush_cputimes (); +if (!lock_task_sighand (p, &flags))return ;
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 19/32 x86: Syscall hooks for nohz cpusets
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Add syscall hooks to notify syscall entry and exit on CPUs running in adative nohz mode.
--- arch/x86/kernel/ptrace.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/x86/kernel/ptrace.c +11 -0
@@ -22,6 +22,7 @@#include <linux/perf_event.h> #include <linux/hw_breakpoint.h> #include <linux/rcupdate.h> +#include <linux/tick.h> #include <asm/uaccess.h> #include <asm/pgtable.h>
@@ -1461,6 +1462,10 @@ long syscall_trace_enter(struct pt_regs *regs) {long ret =0 ; +/* Notify nohz task syscall early so the rest can use rcu */ +/* (SDR: Does the rcu_user_exit() make this obsolete?) */ +tick_nohz_enter_kernel (); +rcu_user_exit ();/*
@@ -1528,4 +1533,10 @@ void syscall_trace_leave(struct pt_regs *regs)tracehook_report_syscall_exit (regs, step);rcu_user_enter (); +/* +* Notify nohz task exit syscall at last so the rest can +* use rcu. +* (SDR: does the above make this obsolete?) +*/ +tick_nohz_exit_kernel (); }
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 20/32 nohz/cpuset: enable addition&removal of cpus while in adaptive nohz mode
In reply to Steven RostedtFrom: Hakan Akkan <[email protected]>
Currently modifying cpuset.cpus mask of a cgroup does not update the reference counters for adaptive nohz mode if the cpuset already had cpuset.adaptive_nohz == 1. Fix it so that cpus can be added or removed from a adaptive_nohz cpuset.
--- kernel/cpuset.c | 111 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 73 insertions(+), 38 deletions(-)
kernel/cpuset.c +73 -38
@@ -868,6 +868,8 @@ static void update_tasks_cpumask(struct cpuset *cs, struct ptr_heap *heap)cgroup_scan_tasks (&scan); } +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs); +/** * update_cpumask - update the cpus_allowed mask of a cpuset and all tasks in it * @cs: the cpuset to consider
@@ -908,6 +910,11 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs,if (cpumask_equal (cs->cpus_allowed, trialcs->cpus_allowed))return 0 ; +/* +* Update adaptive nohz bits. +*/ +update_nohz_cpus (cs, trialcs); + retval =heap_init (&heap,PAGE_SIZE ,GFP_KERNEL ,NULL );if (retval)return retval;
@@ -1226,50 +1233,75 @@ static void cpu_exit_nohz(int cpu)preempt_enable (); } -static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) +static void update_cpu_nohz_flag (int cpu,int adjust) { -int cpu; +atomic_t *ref = &per_cpu (cpu_adaptive_nohz_ref, cpu);int val; + val =atomic_add_return (adjust, ref); + +if (val ==1 && adjust >0 ) { +cpumask_set_cpu (cpu, &nohz_cpuset_mask); +/* +* The mask update needs to be visible right away +* so that this CPU is part of the cputime IPI +* update right now. +*/ +smp_mb (); + }else if (!val) { +/* +* The update to cpu_adaptive_nohz_ref must be +* visible right away. So that once we restart the tick +* from the IPI, it won't be stopped again due to cache +* update lag. +* FIXME: We probably need more to ensure this value is really +* visible right away. +*/ +smp_mb (); +cpu_exit_nohz (cpu); +/* +* Now that the tick has been restarted and cputimes +* flushed, we don't need anymore to be part of the +* cputime flush IPI. +*/ +cpumask_clear_cpu (cpu, &nohz_cpuset_mask); + } +} + +static void update_nohz_flag (struct cpuset *old_cs,struct cpuset *cs) +{ +int cpu; +int adjust; +if (is_adaptive_nohz (old_cs) ==is_adaptive_nohz (cs))return ; -for_each_cpu (cpu, cs->cpus_allowed) { -atomic_t *ref = &per_cpu (cpu_adaptive_nohz_ref, cpu); -if (is_adaptive_nohz (cs)) - val =atomic_inc_return (ref); -else - val =atomic_dec_return (ref); - -if (val ==1 ) { -cpumask_set_cpu (cpu, &nohz_cpuset_mask); -/* -* The mask update needs to be visible right away -* so that this CPU is part of the cputime IPI -* update right now. -*/ -smp_mb (); - }else if (!val) { -/* -* The update to cpu_adaptive_nohz_ref must be -* visible right away. So that once we restart the tick -* from the IPI, it won't be stopped again due to cache -* update lag. -* FIXME: We probably need more to ensure this value is really -* visible right away. -*/ -smp_mb (); -cpu_exit_nohz (cpu); -/* -* Now that the tick has been restarted and cputimes -* flushed, we don't need anymore to be part of the -* cputime flush IPI. -*/ -cpumask_clear_cpu (cpu, &nohz_cpuset_mask); - } - } + adjust =is_adaptive_nohz (cs) ?1 :-1 ; +for_each_cpu (cpu, cs->cpus_allowed) +update_cpu_nohz_flag (cpu, adjust); +} + +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) +{ +int cpu; +cpumask_t cpus; + +/* +* Only bother if the cpuset has adaptive nohz +*/ +if (!is_adaptive_nohz (cs)) +return ; + +cpumask_xor (&cpus, old_cs->cpus_allowed, cs->cpus_allowed); + +for_each_cpu (cpu, &cpus) +update_cpu_nohz_flag (cpu, +cpumask_test_cpu (cpu, cs->cpus_allowed) ?1 :-1 ); }#else +static inline void update_nohz_flag (struct cpuset *old_cs,struct cpuset *cs) +{ +}static inline void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) { }
@@ -1340,7 +1372,7 @@ static int update_flag(cpuset_flagbits_t bit, struct cpuset *cs, spread_flag_changed = ((is_spread_slab (cs) !=is_spread_slab (trialcs)) || (is_spread_page (cs) !=is_spread_page (trialcs))); -update_nohz_cpus (cs, trialcs); +update_nohz_flag (cs, trialcs);mutex_lock (&callback_mutex); cs->flags = trialcs->flags;
@@ -1965,7 +1997,8 @@ static struct cgroup_subsys_state *cpuset_create(struct cgroup *cont)/* * If the cpuset being removed has its flag 'sched_load_balance' * enabled, then simulate turning sched_load_balance off, which -* will call async_rebuild_sched_domains(). +* will call async_rebuild_sched_domains(). Also update adaptive +* nohz flag. */ static void cpuset_destroy (struct cgroup *cont)
@@ -1975,6 +2008,8 @@ static void cpuset_destroy(struct cgroup *cont)if (is_sched_load_balance (cs))update_flag (CS_SCHED_LOAD_BALANCE , cs,0 ); +update_flag (CS_ADAPTIVE_NOHZ , cs,0 ); + number_of_cpusets--;free_cpumask_var (cs->cpus_allowed);kfree (cs);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 21/32 nohz: Dont restart the tick before scheduling to idle
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
If we were running adaptive tickless but then we schedule out and enter the idle task, we don't need to restart the tick because tick_nohz_idle_enter() is going to be called right away.
The only thing we need to do is to save the jiffies such that when we later restart the tick we can account the CPU time spent while idle was tickless.
--- kernel/time/tick-sched.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
kernel/time/tick-sched.c +11 -7
@@ -1036,14 +1036,18 @@ void tick_nohz_pre_schedule(void)void tick_nohz_post_schedule (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +unsigned long flags; -/* -* No need to disable irqs here. The worst that can happen -* is an irq that comes and restart the tick before us. -* tick_nohz_restart_sched_tick() is irq safe. -*/ -if (ts->tick_stopped) -tick_nohz_restart_sched_tick (); +local_irq_save (flags); +if (ts->tick_stopped) { +if (is_idle_task (current)) { + ts->saved_jiffies = jiffies; + ts->saved_jiffies_whence =JIFFIES_SAVED_IDLE ; + }else { +tick_nohz_restart_sched_tick (); + } + } +local_irq_restore (flags); }void tick_nohz_flush_current_times (bool restart_tick)
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 22/32 sched: Comment on rq->clock correctness in ttwu_do_wakeup() in nohz
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
--- kernel/sched/core.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/core.c +6 -0
@@ -1309,6 +1309,12 @@ ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags)if (p->sched_class->task_woken) p->sched_class->task_woken (rq, p); +/* +* For adaptive nohz case: We called ttwu_activate() +* which just updated the rq clock. There is an +* exception with p->on_rq != 0 but in this case +* we are not idle and rq->idle_stamp == 0 +*/ if (rq->idle_stamp) {u64 delta = rq->clock - rq->idle_stamp;u64 max =2 *sysctl_sched_migration_cost;
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 23/32 sched: Update rq clock on nohz CPU before migrating tasks
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Because the sched_class::put_prev_task() callback of rt and fair classes are referring to the rq clock to update their runtime statistics. A CPU running in tickless mode may carry a stale value. We need to update it there.
--- kernel/sched/core.c | 6 ++++++ kernel/sched/sched.h | 6 ++++++ 2 files changed, 12 insertions(+)
kernel/sched/core.c +6 -0
@@ -4851,6 +4851,12 @@ static void migrate_tasks(unsigned int dead_cpu)*/ rq->stop =NULL ; +/* +* ->put_prev_task() need to have an up-to-date value +* of rq->clock[_task] +*/ +update_nohz_rq_clock (rq); +for ( ; ; ) {/* * There's this thread running, bail when that's the only
kernel/sched/sched.h +6 -0
@@ -948,6 +948,12 @@ static inline void dec_nr_running(struct rq *rq)extern void update_rq_clock (struct rq *rq); +static inline void update_nohz_rq_clock (struct rq *rq) +{ +if (cpuset_cpu_adaptive_nohz (cpu_of (rq))) +update_rq_clock (rq); +} +extern void activate_task (struct rq *rq,struct task_struct *p,int flags);extern void deactivate_task (struct rq *rq,struct task_struct *p,int flags);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 24/32 sched: Update rq clock on nohz CPU before setting fair group shares
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
Because we may update the execution time (sched_group_set_shares()->
update_cfs_shares()->reweight_entity()->update_curr()) before
reweighting the entity after updating the group shares and this requires an uptodate version of the runqueue clock. Let's update it on the target CPU if it runs tickless because scheduler_tick() is not there to maintain it.
--- kernel/sched/fair.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/sched/fair.c +5 -0
@@ -5264,6 +5264,11 @@ int sched_group_set_shares(struct task_group *tg, unsigned long shares) se = tg->se[i];/* Propagate contribution to hierarchy */ raw_spin_lock_irqsave (&rq->lock, flags); +/* +* We may call update_curr() which needs an up-to-date +* version of rq clock if the CPU runs tickless. +*/ +update_nohz_rq_clock (rq);for_each_sched_entity (se)update_cfs_shares (group_cfs_rq (se));raw_spin_unlock_irqrestore (&rq->lock, flags);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 25/32 sched: Update rq clock on tickless CPUs before calling check_preempt_curr()
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
check_preempt_wakeup() of fair class needs an uptodate sched clock value to update runtime stats of the current task.
When a task is woken up, activate_task() is usually called right before ttwu_do_wakeup() unless the task is already in the runqueue. In this case we need to update the rq clock manually in case the CPU runs tickless because ttwu_do_wakeup() calls check_preempt_wakeup().
--- kernel/sched/core.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
kernel/sched/core.c +16 -1
@@ -1353,6 +1353,12 @@ static int ttwu_remote(struct task_struct *p, int wake_flags) rq =__task_rq_lock (p);if (p->on_rq) { +/* +* Ensure check_preempt_curr() won't deal with a stale value +* of rq clock if the CPU is tickless. BTW do we actually need +* check_preempt_curr() to be called here? +*/ +update_nohz_rq_clock (rq);ttwu_do_wakeup (rq, p, wake_flags); ret =1 ; }
@@ -1530,8 +1536,17 @@ static void try_to_wake_up_local(struct task_struct *p)if (!(p->state &TASK_NORMAL )) goto out; -if (!p->on_rq) +if (!p->on_rq) {ttwu_activate (rq, p,ENQUEUE_WAKEUP ); + }else { +/* +* Even if the task is on the runqueue we still +* need to ensure check_preempt_curr() won't +* deal with a stale rq clock value on a tickless +* CPU +*/ +update_nohz_rq_clock (rq); + }ttwu_do_wakeup (rq, p,0 );ttwu_stat (p,smp_processor_id (),0 );
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 26/32 sched: Update rq clock earlier in unthrottle_cfs_rq
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
In this function we are making use of rq->clock right before the update of the rq clock, let's just call update_rq_clock() just before that to avoid using a stale rq clock value.
--- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/fair.c +3 -2
@@ -1667,15 +1667,16 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)long task_delta; se = cfs_rq->tg->se[cpu_of (rq_of (cfs_rq))]; - cfs_rq->throttled =0 ; + +update_rq_clock (rq); +raw_spin_lock (&cfs_b->lock); cfs_b->throttled_time += rq->clock - cfs_rq->throttled_timestamp;list_del_rcu (&cfs_rq->throttled_list);raw_spin_unlock (&cfs_b->lock); cfs_rq->throttled_timestamp =0 ; -update_rq_clock (rq);/* update hierarchical throttle state */ walk_tg_tree_from (cfs_rq->tg, tg_nop, tg_unthrottle_up, (void *)rq);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 27/32 sched: Update clock of nohz busiest rq before balancing
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
move_tasks() and active_load_balance_cpu_stop() both need to have the busiest rq clock uptodate because they may end up calling can_migrate_task() that uses rq->clock_task to determine if the task running in the busiest runqueue is cache hot.
Hence if the busiest runqueue is tickless, update its clock before reading it.
--- kernel/sched/fair.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
kernel/sched/fair.c +17 -0
@@ -4231,6 +4231,7 @@ static int load_balance(int this_cpu, struct rq *this_rq, {int ld_moved, cur_ld_moved, active_balance =0 ;int lb_iterations, max_lb_iterations; +int clock_updated;struct sched_group *group;struct rq *busiest;unsigned long flags;
@@ -4274,6 +4275,7 @@ redo: ld_moved =0 ; lb_iterations =1 ; + clock_updated =0 ;if (busiest->nr_running >1 ) {/* * Attempt to move tasks. If find_busiest_group has found
@@ -4297,6 +4299,14 @@ more_balance:*/ cur_ld_moved =move_tasks (&env); ld_moved += cur_ld_moved; + +/* +* Move tasks may end up calling can_migrate_task() which +* requires an uptodate value of the rq clock. +*/ +update_nohz_rq_clock (busiest); + clock_updated =1 ; +double_rq_unlock (env.dst_rq, busiest);local_irq_restore (flags);
@@ -4392,6 +4402,13 @@ more_balance: busiest->active_balance =1 ; busiest->push_cpu = this_cpu; active_balance =1 ; +/* +* active_load_balance_cpu_stop may end up calling +* can_migrate_task() which requires an uptodate +* value of the rq clock. +*/ +if (!clock_updated) +update_nohz_rq_clock (busiest); }raw_spin_unlock_irqrestore (&busiest->lock, flags);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 28/32 sched: Update rq clock before idle balancing
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
idle_balance() is called from schedule() right before we schedule the idle task. It needs to record the idle timestamp at that time and for this the rq clock must be accurate. If the CPU is running tickless we need to update the rq clock manually.
--- kernel/sched/fair.c | 1 + 1 file changed, 1 insertion(+)
kernel/sched/fair.c +1 -0
@@ -4470,6 +4470,7 @@ void idle_balance(int this_cpu, struct rq *this_rq)int pulled_task =0 ;unsigned long next_balance = jiffies +HZ ; +update_nohz_rq_clock (this_rq); this_rq->idle_stamp = this_rq->clock;if (this_rq->avg_idle < sysctl_sched_migration_cost)
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 29/32 sched: Update nohz rq clock before searching busiest group on load balancing
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
While load balancing an rq target, we look for the busiest group. This operation may require an uptodate rq clock if we end up calling scale_rt_power(). To this end, update it manually if the target is running tickless.
DOUBT: don't we actually also need this in vanilla kernel, in case this_cpu is in dyntick-idle mode?
--- kernel/sched/fair.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/sched/fair.c +13 -0
@@ -4252,6 +4252,19 @@ static int load_balance(int this_cpu, struct rq *this_rq,schedstat_inc (sd, lb_count[idle]); +/* +* find_busiest_group() may need an uptodate cpu clock +* for find_busiest_group() (see scale_rt_power()). If +* the CPU is nohz, it's clock may be stale. +*/ +if (cpuset_cpu_adaptive_nohz (this_cpu)) { +local_irq_save (flags); +raw_spin_lock (&this_rq->lock); +update_rq_clock (this_rq); +raw_spin_unlock (&this_rq->lock); +local_irq_restore (flags); + } + redo: group =find_busiest_group (&env, balance);
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 30/32 rcu: Switch to extended quiescent state in userspace from nohz cpuset
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
When we switch to adaptive nohz mode and we run in userspace, we can still receive IPIs from the RCU core if a grace period has been started by another CPU because we need to take part of its completion.
However running in userspace is similar to that of running in idle because we don't make use of RCU there, thus we can be considered as running in RCU extended quiescent state. The benefit when running into that mode is that we are not anymore disturbed by needless IPIs coming from the RCU core.
To perform this, we just to use the RCU extended quiescent state APIs on the following points:
- kernel exit or tick stop in userspace: here we switch to extended quiescent state because we run in userspace without the tick.
- kernel entry or tick restart: here we exit the extended quiescent state because either we enter the kernel and we may make use of RCU read side critical section anytime, or we need the timer tick for some reason and that takes care of RCU grace period in a traditional way.
--- include/linux/tick.h | 3 +++ kernel/time/tick-sched.c | 27 +++++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 2 deletions(-)
include/linux/tick.h +3 -0
@@ -153,6 +153,8 @@ static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }# endif /* !NO_HZ */ #ifdef CONFIG_CPUSETS_NO_HZ +DECLARE_PER_CPU (int, nohz_task_ext_qs); +extern void tick_nohz_enter_kernel (void );extern void tick_nohz_exit_kernel (void );extern void tick_nohz_enter_exception (struct pt_regs *regs);
@@ -160,6 +162,7 @@ extern void tick_nohz_exit_exception(struct pt_regs *regs);extern void tick_nohz_check_adaptive (void );extern void tick_nohz_pre_schedule (void );extern void tick_nohz_post_schedule (void ); +extern void tick_nohz_cpu_exit_qs (void );extern bool tick_nohz_account_tick (void );extern void tick_nohz_flush_current_times (bool restart_tick);#else /* !CPUSETS_NO_HZ */
kernel/time/tick-sched.c +25 -2
@@ -571,10 +571,13 @@ static void tick_nohz_cpuset_stop_tick(struct tick_sched *ts)if (!was_stopped && ts->tick_stopped) {WARN_ON_ONCE (ts->saved_jiffies_whence !=JIFFIES_SAVED_NONE ); -if (user) +if (user) { ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; -else if (!current->mm) +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_user_enter_irq (); + }else if (!current->mm) { ts->saved_jiffies_whence =JIFFIES_SAVED_SYS ; + } ts->saved_jiffies = jiffies;set_thread_flag (TIF_NOHZ );
@@ -906,6 +909,8 @@ void tick_check_idle(int cpu) }#ifdef CONFIG_CPUSETS_NO_HZ +DEFINE_PER_CPU (int, nohz_task_ext_qs); +void tick_nohz_exit_kernel (void ) {unsigned long flags;
@@ -928,6 +933,9 @@ void tick_nohz_exit_kernel(void) ts->saved_jiffies = jiffies; ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_user_enter (); +local_irq_restore (flags); }
@@ -947,6 +955,11 @@ void tick_nohz_enter_kernel(void)WARN_ON_ONCE (!ts->tick_stopped);WARN_ON_ONCE (ts->saved_jiffies_whence !=JIFFIES_SAVED_USER ); +if (__get_cpu_var (nohz_task_ext_qs) ==1 ) { +__get_cpu_var (nohz_task_ext_qs) =0 ; +rcu_user_exit (); + } + delta_jiffies = jiffies - ts->saved_jiffies;account_user_ticks (current, delta_jiffies);
@@ -956,6 +969,14 @@ void tick_nohz_enter_kernel(void)local_irq_restore (flags); } +void tick_nohz_cpu_exit_qs (void ) +{ +if (__get_cpu_var (nohz_task_ext_qs)) { +rcu_user_exit_irq (); +__get_cpu_var (nohz_task_ext_qs) =0 ; + } +} +void tick_nohz_enter_exception (struct pt_regs *regs) {if (user_mode (regs))
@@ -991,6 +1012,7 @@ static void tick_nohz_restart_adaptive(void)tick_nohz_flush_current_times (true);tick_nohz_restart_sched_tick ();clear_thread_flag (TIF_NOHZ ); +tick_nohz_cpu_exit_qs (); }void tick_nohz_check_adaptive (void )
@@ -1030,6 +1052,7 @@ void tick_nohz_pre_schedule(void)if (test_thread_flag (TIF_NOHZ )) {tick_nohz_flush_current_times (true);clear_thread_flag (TIF_NOHZ ); +/* FIXME: warn if we are in RCU idle mode */ } }
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 31/32 nohz/cpuset: Disable under some configs
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
This shows the various things that are not yet handled by the nohz cpusets: perf events, irq work, irq time accounting.
But there are further things that have yet to be handled:
sched clock tick, runqueue clock, sched_class::task_tick(), rq clock, cpu load, complete handling of cputimes, ...
--- init/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
init/Kconfig +1 -1
@@ -751,7 +751,7 @@ config PROC_PID_CPUSET config CPUSETS_NO_HZ bool "Tickless cpusets" - depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS + depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS && !IRQ_TIME_ACCOUNTING help This options let you apply a nohz property to a cpuset such that the periodic timer tick tries to be avoided when possible on
Signature · 1 line
1.7.10.4
Steven Rostedt
reviewerPATCH 32/32 nohz, not for merge: Add tickless tracing
In reply to Steven RostedtFrom: Frederic Weisbecker <[email protected]>
add hoc tickless tracing
--- arch/x86/kernel/smp.c | 2 ++ kernel/cpuset.c | 1 + kernel/time/tick-sched.c | 7 +++++++ 3 files changed, 10 insertions(+)
arch/x86/kernel/smp.c +2 -0
@@ -264,6 +264,7 @@ finish:void smp_reschedule_interrupt (struct pt_regs *regs) {ack_APIC_irq (); +trace_printk ("IPI: Scheduler\n" );inc_irq_stat (irq_resched_count);scheduler_ipi ();/*
@@ -276,6 +277,7 @@ void smp_cpuset_update_nohz_interrupt(struct pt_regs *regs) {ack_APIC_irq ();irq_enter (); +trace_printk ("IPI: Nohz update\n" );tick_nohz_check_adaptive ();inc_irq_stat (irq_call_count);irq_exit ();
kernel/cpuset.c +1 -0
@@ -1212,6 +1212,7 @@ static cpumask_t nohz_cpuset_mask;static void flush_cputime_interrupt (void *unused) { +trace_printk ("IPI: flush cputime\n" );tick_nohz_flush_current_times (false); }
kernel/time/tick-sched.c +7 -0
@@ -376,6 +376,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,if (!ts->tick_stopped) { ts->last_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ; +trace_printk ("Stop tick\n" ); }/*
@@ -581,6 +582,7 @@ static void tick_nohz_cpuset_stop_tick(struct tick_sched *ts) ts->saved_jiffies = jiffies;set_thread_flag (TIF_NOHZ ); + trace_printk("set TIF_NOHZ\n"); } }#else
@@ -659,6 +661,7 @@ static void __tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now) ts->idle_exittime = now;tick_nohz_restart (ts, now); +trace_printk ("Restart sched tick\n" ); }/**
@@ -1012,6 +1015,7 @@ static void tick_nohz_restart_adaptive(void)tick_nohz_flush_current_times (true);tick_nohz_restart_sched_tick ();clear_thread_flag (TIF_NOHZ ); +trace_printk ("clear TIF_NOHZ\n" );tick_nohz_cpu_exit_qs (); }
@@ -1031,6 +1035,7 @@ void cpuset_exit_nohz_interrupt(void *unused) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +trace_printk ("IPI: Nohz exit\n" );if (ts->tick_stopped && !is_idle_task (current))tick_nohz_restart_adaptive (); }
@@ -1052,6 +1057,7 @@ void tick_nohz_pre_schedule(void)if (test_thread_flag (TIF_NOHZ )) {tick_nohz_flush_current_times (true);clear_thread_flag (TIF_NOHZ ); +trace_printk ("clear TIF_NOHZ\n" );/* FIXME: warn if we are in RCU idle mode */ } }
@@ -1147,6 +1153,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer) }update_process_times (user);profile_tick (CPU_PROFILING ); +trace_printk ("tick\n" ); }hrtimer_forward (timer, now, tick_period);
Signature · 1 line
1.7.10.4
Quoted 3 lines · Steven Rostedt
A while ago Frederic posted a series of patches to get an idea on how to implement nohz cpusets.
<snip>
Quoted 4 lines
By using
isocpus and nohz cpuset, a task would be able to achieve true cpu isolation.
Quoted 8 lines
This has been long asked for by those in the RT community. If a task requires uninterruptible CPU time, this would be able to give a task that, even without the full PREEMPT-RT patch set.
This patch set is not for inclusion. It is just to get the topic at the forefront again. The design requires more work and more discussion.
Three additional data points that might be of interest to the discussion:
1. AFAIK both Tilera and Cavium carry patch sets with similar functionality in their respective kernels, so the idea has some real world users already.
2. I tested a previous version of the same patch set (based on 3.3) together with some fixes* and got the same latency, in cycles, from a simple test program and a version of said program running bare metal with no OS. The same program running without this patch got 3 orders of magnitude higher latency. So, this certainly shows some great potential.
3. Even if you don't care about latency at all, on a massively multi-core (or hyperscale, as I've read some people call it now) systems, assigning a task to a single CPU can makes a lot of sense from a cache utilization perspective etc; if you that, this feature can give a performance boost to anything that is mostly CPU bound and perhaps for some workloads that are not so CPU bound as well. Specifically, many high performance computing type of workloads come to mind. So, this has the potential to be useful to both RT folks and HPC folks, I think.
[*] A newer version patch set:
http://www.spinics.net/lists/linux-mm/msg33860.html and disabling the part that sends IPI to update cputime for nohz/cpuset CPUs.
Thanks,
Gilad
--
Gilad Ben-Yossef
Chief Coffee Drinker
[email protected]
Israel Cell: +972-52-8260388
US Cell: +1-973-8260388
http://benyossef.com
"If you take a class in large-scale robotics, can you end up in a situation where the homework eats your dog?"
-- Jean-Baptiste Queru
Quoted 9 lines · Steven Rostedt
A while ago Frederic posted a series of patches to get an idea on how to implement nohz cpusets. Where you can add a task to a cpuset and mark the set to be 'nohz'. When the task runs on a CPU and is the only task scheduled (
nr_running== 1), the tick will stop. The idea is to give the task the least amount of kernel interference as possible. If the task doesn't do any system calls (and possibly even if it does), no timer interrupt will bother it. By using isocpus and nohz cpuset, a task would be able to achieve true cpu isolation.
I thought isolcpus was on the way out? If there is no timer interrupt then there will also be no scheduler activity. Why do we need both?
Also could we have this support without cpusets? There are multiple means to do system segmentation (f.e. cgroups) and something like hz control is pretty basic. Control via some cpumask like irq affinities in f.e.
/sys/devices/system/cpu/nohz
or a per cpu flag in
/sys/devices/system/cpu/cpu0/hz
would be easier and not be tied to something like cpusets.
also it would be best to sync this conceptually with the processors enabled for rcu processing.
Maybe have a series of cpumasks in /sys/devices/system/cpu/ ?
Quoted 3 lines
This has been long asked for by those in the RT community. If a task requires uninterruptible CPU time, this would be able to give a task that, even without the full PREEMPT-RT patch set.
Also those interested in low latency are very very interested in this feature in particular in support without any preempt support on in the kernel.
Steven Rostedt
reviewerQuoted 12 lines · Christoph Lameter
Quoted 9 lines · Steven Rostedt
A while ago Frederic posted a series of patches to get an idea on how to implement nohz cpusets. Where you can add a task to a cpuset and mark the set to be 'nohz'. When the task runs on a CPU and is the only task scheduled (
nr_running== 1), the tick will stop. The idea is to give the task the least amount of kernel interference as possible. If the task doesn't do any system calls (and possibly even if it does), no timer interrupt will bother it. By using isocpus and nohz cpuset, a task would be able to achieve true cpu isolation.
Quoted 2 lines
I thought isolcpus was on the way out? If there is no timer interrupt then there will also be no scheduler activity. Why do we need both?
I probably shouldn't have mentioned isolcpus. I was using that as something that is general to get everything off of a cpu (irq affinity for example).
Quoted 5 lines
Also could we have this support without cpusets? There are multiple means to do system segmentation (f.e. cgroups) and something like hz control is pretty basic. Control via some cpumask like irq affinities in f.e.
Quoted 7 lines
/sys/devices/system/cpu/nohzor a per cpu flag in
/sys/devices/system/cpu/cpu0/hzwould be easier and not be tied to something like cpusets.
Frederic will have to answer this. I was just starting with his patches. Note, we are holding off this work for now until Frederic's other work is done (the irq_work and printk updates).
Quoted 3 lines
also it would be best to sync this conceptually with the processors enabled for rcu processing.
Processors can be disabled for rcu processing? Or are you talking about Paul's new work of offloading rcu callbacks?
Quoted 11 lines
Maybe have a series of cpumasks in
/sys/devices/system/cpu/?Quoted 3 lines
This has been long asked for by those in the RT community. If a task requires uninterruptible CPU time, this would be able to give a task that, even without the full PREEMPT-RT patch set.
Also those interested in low latency are very very interested in this feature in particular in support without any preempt support on in the kernel.
Yep understood. We really need to get things rolling.
-- Steve
Quoted 13 lines · Steven Rostedt
Quoted 12 lines · Christoph Lameter
Quoted 9 lines · Steven Rostedt
A while ago Frederic posted a series of patches to get an idea on how to implement nohz cpusets. Where you can add a task to a cpuset and mark the set to be 'nohz'. When the task runs on a CPU and is the only task scheduled (
nr_running== 1), the tick will stop. The idea is to give the task the least amount of kernel interference as possible. If the task doesn't do any system calls (and possibly even if it does), no timer interrupt will bother it. By using isocpus and nohz cpuset, a task would be able to achieve true cpu isolation.
One other aspect that this patch probably needs to address is the cache localization of irq spinlocks.
At least in 3.6, with !CONFIG_SPARSE_IRQ
--
struct irq_desc irq_desc[NR_IRQS] __cacheline_aligned_in_smp = {
[0 ... NR_IRQS-1] = {
.handle_irq = handle_bad_irq,
.depth = 1,
.lock = __RAW_SPIN_LOCK_UNLOCKED(irq_desc->lock),
}
};--
You are likely to get a cache miss in the top half of your low latency CPU anytime some other CPU has taken a spinlock which lies within the same cache line.
Or is my understanding of the __cacheline_aligned_in_smp declaration wrong ?
Br,
David
Quoted 5 lines · Steven Rostedt
Quoted 2 lines
also it would be best to sync this conceptually with the processors enabled for rcu processing.
Processors can be disabled for rcu processing? Or are you talking about Paul's new work of offloading rcu callbacks?
Yes. Paul's new work to remove rcu processing from processors. That needs to be synced configuration wise somehow. It does not make sense to process rcu callbacks on processors where the timer tick does not work anymore.
Steven Rostedt
reviewerQuoted 11 lines · Christoph Lameter
Quoted 5 lines · Steven Rostedt
Quoted 2 lines
also it would be best to sync this conceptually with the processors enabled for rcu processing.
Processors can be disabled for rcu processing? Or are you talking about Paul's new work of offloading rcu callbacks?
Yes. Paul's new work to remove rcu processing from processors. That needs to be synced configuration wise somehow. It does not make sense to process rcu callbacks on processors where the timer tick does not work anymore.
Don't worry, Paul is working with us too ;-)
-- Steve
Quoted 11 lines · Christoph Lameter
Quoted 5 lines · Steven Rostedt
Quoted 2 lines
also it would be best to sync this conceptually with the processors enabled for rcu processing.
Processors can be disabled for rcu processing? Or are you talking about Paul's new work of offloading rcu callbacks?
Yes. Paul's new work to remove rcu processing from processors. That needs to be synced configuration wise somehow. It does not make sense to process rcu callbacks on processors where the timer tick does not work anymore.
In kernels built with CONFIG_FAST_NO_HZ=n, if there are callbacks, then there will be a tick, with or without Frederic's adaptive ticks. If CONFIG_FAST_NO_HZ=y, if there are callbacks but no tick, RCU will arrange for a timer to allow RCU processing to proceed as needed, but much longer than one tick in duration, and only until such time as the RCU callbacks drain.
So, yes, people who need absolutely all jitter to be banished at whatever cost would want both adaptive ticks and no-CBs CPUs, but not everyone who wants adaptive ticks would necessarily want the burden of choosing which CPUs get callbacks offloaded from and where they should be executed.
So I believe that these need to be controlled separately for the immediate future.
Thanx, Paul
Quoted 2 lines · Paul E. McKenney
So I believe that these need to be controlled separately for the immediate future.
Yes they do but the configurations are similar and it would be best if these were cpumasks in standard locations instead of being specified at boot time or in a cpuset.
Put the cpu masks into
/sys/devices/system/cpu/{nohz_cpus,rcu_cpus}
or so?
Quoted 9 lines · Christoph Lameter
Quoted 2 lines · Paul E. McKenney
So I believe that these need to be controlled separately for the immediate future.
Yes they do but the configurations are similar and it would be best if these were cpumasks in standard locations instead of being specified at boot time or in a cpuset.
Quoted 5 lines
Put the cpu masks into
/sys/devices/system/cpu/{nohz_cpus,rcu_cpus}or so?
The no-CBs mask would be read-only for some time -- changed only at boot. Longer term, I hope to allow run-time modification, but...
Thanx, Paul
Steven Rostedt
reviewerQuoted 3 lines · Paul E. McKenney
The no-CBs mask would be read-only for some time -- changed only at boot. Longer term, I hope to allow run-time modification, but...
but what? You're not looking to retire already are you? ;-)
-- Steve
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · Paul E. McKenney
The no-CBs mask would be read-only for some time -- changed only at boot. Longer term, I hope to allow run-time modification, but...
but what? You're not looking to retire already are you? ;-)
Not for a few decades. ;-)
But let's add the no-CBs mask to sysfs when I add the ability to run-time modify that mast.
Thanx, Paul
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines · Steven Rostedt
Quoted 2 lines · Paul E. McKenney
The no-CBs mask would be read-only for some time -- changed only at boot. Longer term, I hope to allow run-time modification, but...
Quoted 6 lines
Quoted 1 line
but what? You're not looking to retire already are you? ;-)
Not for a few decades. ;-)
But let's add the no-CBs mask to sysfs when I add the ability to run-time modify that mast.
Well we are creating a user ABi with the boot time option. It would be best to get it right out of the door.
2012/11/5 Christoph Lameter <[email protected]>:
Quoted 10 lines · Paul E. McKenney
Quoted 9 lines · Steven Rostedt
Quoted 6 lines · Paul E. McKenney
Quoted 3 lines
The no-CBs mask would be read-only for some time -- changed only at boot. Longer term, I hope to allow run-time modification, but...
but what? You're not looking to retire already are you? ;-)
Quoted 7 lines
Quoted 4 lines
Not for a few decades. ;-)
But let's add the no-CBs mask to sysfs when I add the ability to run-time modify that mast.
Well we are creating a user ABi with the boot time option. It would be best to get it right out of the door.
I believe that a static setting through a boot option is a nice first step already. Runtime tuning may involve dynamic migration and other headaches. The nocb patch is tricky enough to review ;)
2012/11/2 Christoph Lameter <[email protected]>:
Quoted 11 lines
Also could we have this support without cpusets? There are multiple means to do system segmentation (f.e. cgroups) and something like hz control is pretty basic. Control via some cpumask like irq affinities in f.e.
/sys/devices/system/cpu/nohzor a per cpu flag in
/sys/devices/system/cpu/cpu0/hzwould be easier and not be tied to something like cpusets.
You really don't want that cpuset interface, do you? ;-)
Yeah I think I agree with you. This adds a dependency to cpusets/cgroups, I wish we could avoid that if possible. Also cpuset may be a bit counter intuitive for this usecase. What if a cpu is included in both a nohz cpuset and a non-nohz cpuset? What is the behaviour to adopt? An OR on the nohz flag such that as long as the CPU is in at least one nohz cpuset, it's considered a nohz CPU? Or only shutdown the tick for the tasks attached in the nohz cpusets? Do we really want that per cgroup granularity and the overhead / complexity that comes along?
No I think we should stay simple and have a simple per CPU property for that, without involving cgroups aside.
So indeed a cpumask in /sys/devices/system/cpu/nohz looks like a better interface.
Quoted 7 lines
Quoted 3 lines
This has been long asked for by those in the RT community. If a task requires uninterruptible CPU time, this would be able to give a task that, even without the full PREEMPT-RT patch set.
Also those interested in low latency are very very interested in this feature in particular in support without any preempt support on in the kernel.
Sure, we are trying to make that full dyncticks approach as much generic as possible.