sched,time: fix irq time accounting with nohz_idle
Subsystems (2)
v4 cover sched,time: fix irq time accounting with nohz_idle
Currently irq time accounting only works in these cases:
1) purely ticke based accounting
2) nohz_full accounting, but only on housekeeping & nohz_full CPUs
3) architectures with native vtime accounting
On nohz_idle CPUs, which are probably the majority nowadays, irq time accounting is currently broken. This leads to systems reporting a dramatically lower amount of irq & softirq time than is actually spent handling them, with all the time spent while the system is in the idle task being accounted as idle.
This patch set seems to bring the amount of irq time reported by top (and /proc/stat) roughly in line with that measured when I do a "perf record -g -a" run to see what is using all that time.
The amount of irq time used, especially softirq, is shockingly high, to the point of me thinking this patch set may be wrong, but the numbers seem to match what perf is giving me...
These patches apply on top of Wanpeng Li's steal time patches.
CONFIG_IRQ_TIME_ACCOUNTING is now a config option that is available as a separate choice from tick based / nohz_idle / nohz_full mode, a suggested by Frederic Weisbecker.
Next up: look at the things that are using CPU time on an otherwise idle system, and see if I can make those a little faster :)
v2: address Peterz's concerns, some more cleanups
v3: rewrite the code along Frederic's suggestions, now cputime_t
is used everywhere
v4: greatly simplify the local_irq_save/restore optimisation, thanks
to Paolo pointing out irqs are already blocked by the callers
PATCH v4 1/3 sched,time: count actually elapsed irq & softirq time
From: Rik van Riel <[email protected]>
Currently, if there was any irq or softirq time during 'ticks' jiffies, the entire period will be accounted as irq or softirq time.
This is inaccurate if only a subset of the time was actually spent handling irqs, and could conceivably mis-count all of the ticks during a period as irq time, when there was some irq and some softirq time.
This can actually happen when irqtime_account_process_tick is called from account_idle_ticks, which can pass a larger number of ticks down all at once.
Fix this by changing irqtime_account_hi_update, irqtime_account_si_update, and steal_account_process_ticks to work with cputime_t time units, and return the amount of time spent in each mode.
Rename steal_account_process_ticks to steal_account_process_time, to reflect that time is now accounted in cputime_t, instead of ticks.
Additionally, have irqtime_account_process_tick take into account how much time was spent in each of steal, irq, and softirq time.
The latter could help improve the accuracy of cputime accounting when returning from idle on a NO_HZ_IDLE CPU.
Properly accounting how much time was spent in hardirq and softirq time will also allow the NO_HZ_FULL code to re-use these same functions for hardirq and softirq accounting.
--- include/asm-generic/cputime_nsecs.h | 2 + kernel/sched/cputime.c | 124 ++++++++++++++++++++++-------------- 2 files changed, 79 insertions(+), 47 deletions(-)
include/asm-generic/cputime_nsecs.h +2 -0
@@ -50,6 +50,8 @@ typedef u64 __nocast cputime64_t; (__force u64)(__ct)#define nsecs_to_cputime (__nsecs) \ (__force cputime_t)(__nsecs) +#define nsecs_to_cputime64 (__nsecs) \ + (__force cputime_t)(__nsecs)/*
kernel/sched/cputime.c +77 -47
207 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v4 2/3 nohz,cputime: replace VTIME_GEN irq time code with IRQ_TIME_ACCOUNTING code
In reply to [email protected]From: Rik van Riel <[email protected]>
The CONFIG_VIRT_CPU_ACCOUNTING_GEN irq time tracking code does not appear to currently work right.
On CPUs without nohz_full=, only tick based irq time sampling is done, which breaks down when dealing with a nohz_idle CPU.
On firewalls and similar systems, no ticks may happen on a CPU for a while, and the irq time spent may never get accounted properly. This can cause issues with capacity planning and power saving, which use the CPU statistics as inputs in decision making.
Replace the VTIME_GEN vtime irq time code, and replace it with the IRQ_TIME_ACCOUNTING code, when selected as a config option by the user.
--- include/linux/vtime.h | 32 ++++++++++++++------------------ init/Kconfig | 6 +++--- kernel/sched/cputime.c | 16 +++------------- 3 files changed, 20 insertions(+), 34 deletions(-)
include/linux/vtime.h +14 -18
@@ -14,6 +14,18 @@ struct task_struct;*/ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE static inline bool vtime_accounting_cpu_enabled (void ) {return true; } + +#ifdef __ARCH_HAS_VTIME_ACCOUNT +extern void vtime_account_irq_enter (struct task_struct *tsk); +#else +extern void vtime_common_account_irq_enter (struct task_struct *tsk); +static inline void vtime_account_irq_enter (struct task_struct *tsk) +{ +if (vtime_accounting_cpu_enabled ()) +vtime_common_account_irq_enter (tsk); +} +#endif /* __ARCH_HAS_VTIME_ACCOUNT */ +#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
@@ -64,17 +76,6 @@ extern void vtime_account_system(struct task_struct *tsk);extern void vtime_account_idle (struct task_struct *tsk);extern void vtime_account_user (struct task_struct *tsk); -#ifdef __ARCH_HAS_VTIME_ACCOUNT -extern void vtime_account_irq_enter (struct task_struct *tsk); -#else -extern void vtime_common_account_irq_enter (struct task_struct *tsk); -static inline void vtime_account_irq_enter (struct task_struct *tsk) -{ -if (vtime_accounting_cpu_enabled ()) -vtime_common_account_irq_enter (tsk); -} -#endif /* __ARCH_HAS_VTIME_ACCOUNT */ -#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ static inline void vtime_task_switch (struct task_struct *prev) { }
@@ -85,13 +86,8 @@ static inline void vtime_account_irq_enter(struct task_struct *tsk) { }#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern void arch_vtime_task_switch (struct task_struct *tsk); -extern void vtime_gen_account_irq_exit (struct task_struct *tsk); - -static inline void vtime_account_irq_exit (struct task_struct *tsk) -{ -if (vtime_accounting_cpu_enabled ()) -vtime_gen_account_irq_exit (tsk); -} +static inline void vtime_account_irq_enter (struct task_struct *tsk) { } +static inline void vtime_account_irq_exit (struct task_struct *tsk) { }extern void vtime_user_enter (struct task_struct *tsk);
init/Kconfig +3 -3
@@ -375,9 +375,11 @@ config VIRT_CPU_ACCOUNTING_GEN If unsure, say N. +endchoice + config IRQ_TIME_ACCOUNTING bool "Fine granularity task level IRQ time accounting" - depends on HAVE_IRQ_TIME_ACCOUNTING && !NO_HZ_FULL + depends on HAVE_IRQ_TIME_ACCOUNTING && !VIRT_CPU_ACCOUNTING_NATIVE help Select this option to enable fine granularity task irq time accounting. This is done by reading a timestamp on each
@@ -386,8 +388,6 @@ config IRQ_TIME_ACCOUNTING If in doubt, say N here. -endchoice - config BSD_PROCESS_ACCT bool "BSD Process Accounting" depends on MULTIUSER
kernel/sched/cputime.c +3 -13
@@ -711,14 +711,14 @@ static cputime_t vtime_delta(struct task_struct *tsk)static cputime_t get_vtime_delta (struct task_struct *tsk) {unsigned long now =READ_ONCE (jiffies); -cputime_t delta, steal; +cputime_t delta, other; delta =jiffies_to_cputime (now - tsk->vtime_snap); - steal =steal_account_process_time (delta); + other =account_other_time (delta);WARN_ON_ONCE (tsk->vtime_snap_whence ==VTIME_INACTIVE ); tsk->vtime_snap = now; -return delta - steal; +return delta - other; }static void __vtime_account_system (struct task_struct *tsk)
@@ -738,16 +738,6 @@ void vtime_account_system(struct task_struct *tsk)write_seqcount_end (&tsk->vtime_seqcount); } -void vtime_gen_account_irq_exit (struct task_struct *tsk) -{ -write_seqcount_begin (&tsk->vtime_seqcount); -if (vtime_delta (tsk)) -__vtime_account_system (tsk); -if (context_tracking_in_user ()) - tsk->vtime_snap_whence =VTIME_USER ; -write_seqcount_end (&tsk->vtime_seqcount); -} -void vtime_account_user (struct task_struct *tsk) {cputime_t delta_cpu;
Signature · 1 line
2.7.4
PATCH v4 3/3 time: drop local_irq_save/restore from irqtime_account_irq
In reply to [email protected]From: Rik van Riel <[email protected]>
Paolo pointed out that irqs are already blocked when irqtime_account_irq is called. That means there is no reason to call local_irq_save/restore again.
--- kernel/sched/cputime.c | 4 ---- 1 file changed, 4 deletions(-)
kernel/sched/cputime.c +0 -4
@@ -49,15 +49,12 @@ DEFINE_PER_CPU(seqcount_t, irq_time_seq);*/ void irqtime_account_irq (struct task_struct *curr) { -unsigned long flags;s64 delta;int cpu;if (!sched_clock_irqtime)return ; -local_irq_save (flags); - cpu =smp_processor_id (); delta =sched_clock_cpu (cpu) -__this_cpu_read (irq_start_time);__this_cpu_add (irq_start_time, delta);
@@ -75,7 +72,6 @@ void irqtime_account_irq(struct task_struct *curr)__this_cpu_add (cpu_softirq_time, delta);irq_time_write_end (); -local_irq_restore (flags); }EXPORT_SYMBOL_GPL (irqtime_account_irq);
Signature · 1 line
2.7.4
Re: [PATCH 3/3] time: drop local_irq_save/restore from irqtime_account_irq
Quoted 34 lines · On 11/07/2016 18:53, [email protected] wrote:
From: Rik van Riel <[email protected]>
Paolo pointed out that irqs are already blocked when
irqtime_account_irqis called. That means there is no reason to call local_irq_save/restore again.Signed-off-by Rik van Riel <[email protected]>Suggested-by Paolo Bonzini <[email protected]>--- kernel/sched/cputime.c | 4 ---- 1 file changed, 4 deletions(-)kernel/sched/cputime.c +0
-3@@ -49,15 +49,12 @@ DEFINE_PER_CPU(seqcount_t, irq_time_seq);*/ void irqtime_account_irq (struct task_struct *curr) { -unsigned long flags;s64 delta;int cpu;if (!sched_clock_irqtime)return ; -local_irq_save (flags); - cpu =smp_processor_id (); delta =sched_clock_cpu (cpu) -__this_cpu_read (irq_start_time);__this_cpu_add (irq_start_time, delta);
kernel/sched/cputime.c void irqtime_account_irq(struct task_struct *curr) · line 77
Quoted 7 lines
kernel/sched/cputime.c +0
-1@@ -75,7 +72,6 @@ void irqtime_account_irq(struct task_struct *curr)__this_cpu_add (cpu_softirq_time, delta);irq_time_write_end (); -local_irq_restore (flags); }EXPORT_SYMBOL_GPL (irqtime_account_irq);
Quoted 25 lines · On Mon, Jul 11, 2016 at 12:53:54PM -0400, [email protected] wrote:
Currently irq time accounting only works in these cases:
1) purely ticke based accounting
2)nohz_fullaccounting, but only on housekeeping &nohz_fullCPUs
3) architectures with native vtime accountingOn
nohz_idleCPUs, which are probably the majority nowadays, irq time accounting is currently broken. This leads to systems reporting a dramatically lower amount of irq & softirq time than is actually spent handling them, with all the time spent while the system is in the idle task being accounted as idle.This patch set seems to bring the amount of irq time reported by top (and
/proc/stat) roughly in line with that measured when I do a "perf record -g -a" run to see what is using all that time.The amount of irq time used, especially softirq, is shockingly high, to the point of me thinking this patch set may be wrong, but the numbers seem to match what perf is giving me...
These patches apply on top of Wanpeng Li's steal time patches.
CONFIG_IRQ_TIME_ACCOUNTINGis now a config option that is available as a separate choice from tick based /nohz_idle/nohz_fullmode, a suggested by Frederic Weisbecker.
Quoted 9 lines
Next up: look at the things that are using CPU time on an otherwise idle system, and see if I can make those a little faster :)
v2: address Peterz's concerns, some more cleanups
v3: rewrite the code along Frederic's suggestions, nowcputime_t
is used everywhere
v4: greatly simplify the local_irq_save/restore optimisation, thanks
to Paolo pointing out irqs are already blocked by the callers
Thanks Rick!
I'm applying the series with my patches and will do a pull request to Ingo.