tick/sched: Refactor idle cputime accounting
Subsystems (10)
Frederic Weisbecker
maintainerv4 cover tick/sched: Refactor idle cputime accounting
Hi,
After the issue reported here:
https://lore.kernel.org/all/[email protected]/
It occurs that the idle cputime accounting is a big mess that accumulates within two concurrent statistics, each having their own shortcomings:
* The accounting for online CPUs which is based on the delta between tick_nohz_start_idle() and tick_nohz_stop_idle().
Pros:
- Works when the tick is off
- Has nsecs granularity
Cons:
- Account idle steal time but doesn't substract it from idle cputime.
- Assumes CONFIG_IRQ_TIME_ACCOUNTING by not accounting IRQs but the IRQ time is simply ignored when CONFIG_IRQ_TIME_ACCOUNTING=n
- The windows between 1) idle task scheduling and the first call to tick_nohz_start_idle() and 2) idle task between the last tick_nohz_stop_idle() and the rest of the idle time are blindspots wrt. cputime accounting (though mostly insignificant amount)
- Relies on private fields outside of kernel stats, with specific accessors.
* The accounting for offline CPUs which is based on ticks and the jiffies delta during which the tick was stopped.
Pros:
- Handles steal time correctly
- Handle CONFIG_IRQ_TIME_ACCOUNTING=y and
CONFIG_IRQ_TIME_ACCOUNTING=n correctly.
- Handles the whole idle task
- Accounts directly to kernel stats, without midlayer accumulator.
Cons:
- Doesn't elapse when the tick is off, which doesn't make it suitable for online CPUs.
- Has TICK_NSEC granularity (jiffies)
- Needs to track the dyntick-idle ticks that were accounted and substract them from the total jiffies time spent while the tick was stopped. This is an ugly workaround.
Having two different accounting for a single context is not the only
problem: since those accountings are of different natures, it is possible to observe the global idle time going backward after a CPU goes offline, as reported by Xin Zhao.
Clean up the situation with introducing a hybrid approach that stays coherent, fixes the backward jumps and works for both online and offline CPUs:
* Tick based or native vtime accounting operate before the tick is stopped and resumes once the tick is restarted.
* When the idle loop starts, switch to dynticks-idle accounting as is done currently, except that the statistics accumulate directly to the relevant kernel stat fields.
* Private dyntick cputime accounting fields are removed.
* Works on both online and offline case.
* Move most of the relevant code to the common sched/cputime subsystem
* Handle CONFIG_IRQ_TIME_ACCOUNTING=n correctly such that the dynticks-idle accounting still elapses while on IRQs.
* Correctly substract idle steal cputime from idle time
Changes since v3 (among which a lot of relevant reviews from Sashiko):
- Add new tags
- Rebase on latest -rc1
- Add "tick/sched: Fix TOCTOU in nohz idle time fetch" (Sashiko)
- Fix buggy state refetch in kcpustat_cpu_fetch_vtime() (Sashiko)
- Fix build issue on powerpc (Christophe Leroy)
- Fix s390 lost steal time occuring on idle IRQs (call vtime_flush() on vtime_account_hardirq() and vtime_account_softirq()) (Sashiko)
- Fix build issue on s390
- Fix uninitialized idle_sleeptime_seq (Sashiko)
- Fix irqtime being disabled or enabled in the middle of an idle IRQ (Sashiko)
- Fix tick restart and then restop in the same idle loop (Sashiko)
- Fix "sched/cputime: Handle idle irqtime gracefully" changelog (Sashiko)
- Fix idle steal time substracted from the wrong index between idle and iowait kcpustat. (Sashiko)
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
timers/core-v4
HEAD: e64ba052ce04e363ff76d3cb8bedc5f812188acb
Thanks,
Frederic
---
Frederic Weisbecker (15):
tick/sched: Fix TOCTOU in nohz idle time fetch
sched/idle: Handle offlining first in idle loop
sched/cputime: Remove superfluous and error prone kcpustat_field() parameter sched/cputime: Correctly support generic vtime idle time powerpc/time: Prepare to stop elapsing in dynticks-idle s390/time: Prepare to stop elapsing in dynticks-idle tick/sched: Unify idle cputime accounting
tick/sched: Remove nohz disabled special case in cputime fetch tick/sched: Move dyntick-idle cputime accounting to cputime code tick/sched: Remove unused fields
tick/sched: Account tickless idle cputime only when tick is stopped tick/sched: Consolidate idle time fetching APIs
sched/cputime: Provide get_cpu_[idle|iowait]_time_us() off-case sched/cputime: Handle idle irqtime gracefully
sched/cputime: Handle dyntick-idle steal time correctly
arch/powerpc/kernel/time.c | 41 +++++ arch/s390/include/asm/idle.h | 2 + arch/s390/kernel/idle.c | 5 +- arch/s390/kernel/vtime.c | 75 ++++++++- drivers/cpufreq/cpufreq.c | 29 +--- drivers/cpufreq/cpufreq_governor.c | 6 +- drivers/macintosh/rack-meter.c | 2 +- fs/proc/stat.c | 40 +---- fs/proc/uptime.c | 8 +- include/linux/kernel_stat.h | 76 +++++++-- include/linux/tick.h | 4 - include/linux/vtime.h | 22 ++- kernel/rcu/tree.c | 9 +- kernel/rcu/tree_stall.h | 7 +- kernel/sched/core.c | 6 +- kernel/sched/cputime.c | 308 +++++++++++++++++++++++++++++++------ kernel/sched/idle.c | 13 +- kernel/time/tick-sched.c | 212 ++++++------------------- kernel/time/tick-sched.h | 12 -- kernel/time/timer_list.c | 6 +- scripts/gdb/linux/timerlist.py | 4 - 21 files changed, 529 insertions(+), 358 deletions(-)
Frederic Weisbecker
maintainerPATCH v4 1/15 tick/sched: Fix TOCTOU in nohz idle time fetch
When the nohz idle time is fetched, the current clock timestamp is taken outside the seqcount, which can result in such a race as reported by Sashiko:
get_cpu_sleep_time_us() tick_nohz_start_idle()
----------------------- ---------------------
now = ktime_get()
write_seqcount_begin(idle_sleeptime_seq);
idle_entrytime = ktime_get()
tick_sched_flag_set(ts, TS_FLAG_IDLE_ACTIVE);
write_seqcount_end(&ts->idle_sleeptime_seq);
read_seqcount_begin(idle_sleeptime_seq)
delta = now - idle_entrytime);
//!! But now < idle_entrytime
idle = *sleeptime + delta;
read_seqcount_retry(&ts->idle_sleeptime_seq, seq)Here the read side fetches the timestamp before the write side and its update. As a result the time delta computed on the read side is negative (ktime_t is signed) and breaks the cputime monotonicity guarantee.
This could possibly be fixed with reading the current clock timestamp inside the seqcount but the reader overhead might then increase. Also simply checking that the current timestamp is above the idle entry time is enough to prevent any issue of the like.
--- kernel/time/tick-sched.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
kernel/time/tick-sched.c +6 -5
@@ -797,15 +797,16 @@ static u64 get_cpu_sleep_time_us(struct tick_sched *ts, ktime_t *sleeptime, *last_update_time =ktime_to_us (now);do { +ktime_t delta =0 ; + seq =read_seqcount_begin (&ts->idle_sleeptime_seq);if (tick_sched_flag_test (ts,TS_FLAG_IDLE_ACTIVE ) && compute_delta) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - - idle =ktime_add (*sleeptime, delta); - }else { - idle = *sleeptime; +if (now > ts->idle_entrytime) + delta =ktime_sub (now, ts->idle_entrytime); } + + idle =ktime_add (*sleeptime, delta); }while (read_seqcount_retry (&ts->idle_sleeptime_seq, seq));return ktime_to_us (idle);
Signature · 1 line
2.53.0
[tip: timers/nohz] tick/sched: Fix TOCTOU in nohz idle time fetch
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 86db4084b4b5d1a074bcc66c108a4c9d266812d4 Gitweb: https://git.kernel.org/tip/86db4084b4b5d1a074bcc66c108a4c9d266812d4 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:33 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:24 +02:00
tick/sched: Fix TOCTOU in nohz idle time fetch
When the nohz idle time is fetched, the current clock timestamp is taken outside the seqcount, which can result in a race as reported by Sashiko:
get_cpu_sleep_time_us() tick_nohz_start_idle()
----------------------- ---------------------
now = ktime_get()
write_seqcount_begin(idle_sleeptime_seq);
idle_entrytime = ktime_get()
tick_sched_flag_set(ts, TS_FLAG_IDLE_ACTIVE);
write_seqcount_end(&ts->idle_sleeptime_seq);
read_seqcount_begin(idle_sleeptime_seq)
delta = now - idle_entrytime);
//!! But now < idle_entrytime
idle = *sleeptime + delta;
read_seqcount_retry(&ts->idle_sleeptime_seq, seq)Here the read side fetches the timestamp before the write side and its update. As a result the time delta computed on the read side is negative (ktime_t is signed) and breaks the cputime monotonicity guarantee.
This could possibly be fixed with reading the current clock timestamp inside the seqcount but the reader overhead might then increase. Also simply checking that the current timestamp is above the idle entry time is enough to prevent any issue of the like.
--- kernel/time/tick-sched.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
kernel/time/tick-sched.c +6 -5
@@ -797,15 +797,16 @@ static u64 get_cpu_sleep_time_us(struct tick_sched *ts, ktime_t *sleeptime, *last_update_time =ktime_to_us (now);do { +ktime_t delta =0 ; + seq =read_seqcount_begin (&ts->idle_sleeptime_seq);if (tick_sched_flag_test (ts,TS_FLAG_IDLE_ACTIVE ) && compute_delta) { -ktime_t delta =ktime_sub (now, ts->idle_entrytime); - - idle =ktime_add (*sleeptime, delta); - }else { - idle = *sleeptime; +if (now > ts->idle_entrytime) + delta =ktime_sub (now, ts->idle_entrytime); } + + idle =ktime_add (*sleeptime, delta); }while (read_seqcount_retry (&ts->idle_sleeptime_seq, seq));return ktime_to_us (idle);
Frederic Weisbecker
maintainerPATCH v4 2/15 sched/idle: Handle offlining first in idle loop
In reply to Frederic WeisbeckerOffline handling happens from within the inner idle loop, after the beginning of dyntick cputime accounting, nohz idle load balancing and TIF_NEED_RESCHED polling.
This is not necessary and even buggy because:
* There is no dyntick handling to do. And calling tick_nohz_idle_enter() messes up with the struct tick_sched reset that was performed on tick_sched_timer_dying().
* There is no nohz idle balancing to do.
* Polling on TIF_RESCHED is irrelevant at this stage, there are no more tasks allowed to run.
* No need to check if need_resched() before offline handling since stop_machine is done and all per-cpu kthread should be done with their job.
Therefore move the offline handling at the beginning of the idle loop. This will also ease the idle cputime unification later by not elapsing idle time while offline through the call to:
tick_nohz_idle_enter() -> tick_nohz_start_idle()
--- kernel/sched/idle.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
kernel/sched/idle.c +8 -5
@@ -280,6 +280,14 @@ static void do_idle(void)int cpu =smp_processor_id ();bool got_tick = false; +if (cpu_is_offline (cpu)) { +local_irq_disable (); +/* All per-CPU kernel threads should be done by now. */ +WARN_ON_ONCE (need_resched ()); +cpuhp_report_idle_dead (); +arch_cpu_idle_dead (); + } +/* * Check if we need to update blocked load */
@@ -331,11 +339,6 @@ static void do_idle(void)*/ local_irq_disable (); -if (cpu_is_offline (cpu)) { -cpuhp_report_idle_dead (); -arch_cpu_idle_dead (); - } -arch_cpu_idle_enter ();rcu_nocb_flush_deferred_wakeup ();
Signature · 1 line
2.53.0
[tip: timers/nohz] sched/idle: Handle offlining first in idle loop
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 0236aaf07b406100c8c3a6b78dba211f32449f49 Gitweb: https://git.kernel.org/tip/0236aaf07b406100c8c3a6b78dba211f32449f49 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:34 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
sched/idle: Handle offlining first in idle loop
Offline handling happens from within the inner idle loop, after the beginning of dyntick cputime accounting, nohz idle load balancing and TIF_NEED_RESCHED polling.
This is not necessary and even buggy because:
* There is no dyntick handling to do. And calling tick_nohz_idle_enter() messes up with the struct tick_sched reset that was performed on tick_sched_timer_dying().
* There is no nohz idle balancing to do.
* Polling on TIF_RESCHED is irrelevant at this stage, there are no more tasks allowed to run.
* No need to check if need_resched() before offline handling since stop_machine is done and all per-cpu kthread should be done with their job.
Therefore move the offline handling at the beginning of the idle loop. This will also ease the idle cputime unification later by not elapsing idle time while offline through the call to:
tick_nohz_idle_enter() -> tick_nohz_start_idle()
--- kernel/sched/idle.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
kernel/sched/idle.c +8 -5
@@ -280,6 +280,14 @@ static void do_idle(void)int cpu =smp_processor_id ();bool got_tick = false; +if (cpu_is_offline (cpu)) { +local_irq_disable (); +/* All per-CPU kernel threads should be done by now. */ +WARN_ON_ONCE (need_resched ()); +cpuhp_report_idle_dead (); +arch_cpu_idle_dead (); + } +/* * Check if we need to update blocked load */
@@ -331,11 +339,6 @@ static void do_idle(void)*/ local_irq_disable (); -if (cpu_is_offline (cpu)) { -cpuhp_report_idle_dead (); -arch_cpu_idle_dead (); - } -arch_cpu_idle_enter ();rcu_nocb_flush_deferred_wakeup ();
Frederic Weisbecker
maintainerPATCH v4 3/15 sched/cputime: Remove superfluous and error prone kcpustat_field() parameter
In reply to Frederic WeisbeckerThe first parameter to kcpustat_field() is a pointer to the cpu kcpustat to be fetched from. This parameter is error prone because a copy to a kcpustat could be passed by accident instead of the original one. Also the kcpustat structure can already be retrieved with the help of the mandatory CPU argument.
Remove the needless parameter.
--- drivers/cpufreq/cpufreq_governor.c | 6 +++--- drivers/macintosh/rack-meter.c | 2 +- include/linux/kernel_stat.h | 8 +++----- kernel/rcu/tree.c | 9 +++------ kernel/rcu/tree_stall.h | 7 +++---- kernel/sched/cputime.c | 5 ++--- 6 files changed, 15 insertions(+), 22 deletions(-)
drivers/cpufreq/cpufreq_governor.c +3 -3
@@ -105,7 +105,7 @@ void gov_update_cpu_data(struct dbs_data *dbs_data) j_cdbs->prev_cpu_idle =get_cpu_idle_time (j, &j_cdbs->prev_update_time, dbs_data->io_is_busy);if (dbs_data->ignore_nice_load) - j_cdbs->prev_cpu_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); + j_cdbs->prev_cpu_nice =kcpustat_field (CPUTIME_NICE , j); } } }
@@ -165,7 +165,7 @@ unsigned int dbs_update(struct cpufreq_policy *policy) j_cdbs->prev_cpu_idle = cur_idle_time;if (ignore_nice) { -u64 cur_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); +u64 cur_nice =kcpustat_field (CPUTIME_NICE , j); idle_time +=div_u64 (cur_nice - j_cdbs->prev_cpu_nice,NSEC_PER_USEC ); j_cdbs->prev_cpu_nice = cur_nice;
@@ -539,7 +539,7 @@ int cpufreq_dbs_governor_start(struct cpufreq_policy *policy) j_cdbs->prev_load =0 ;if (ignore_nice) - j_cdbs->prev_cpu_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); + j_cdbs->prev_cpu_nice =kcpustat_field (CPUTIME_NICE , j); } gov->start (policy);
drivers/macintosh/rack-meter.c +1 -1
@@ -87,7 +87,7 @@ static inline u64 get_cpu_idle_time(unsigned int cpu) kcpustat->cpustat[CPUTIME_IOWAIT ];if (rackmeter_ignore_nice) - retval +=kcpustat_field (kcpustat,CPUTIME_NICE , cpu); + retval +=kcpustat_field (CPUTIME_NICE , cpu);return retval; }
include/linux/kernel_stat.h +3 -5
@@ -100,14 +100,12 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu) }#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -extern u64 kcpustat_field (struct kernel_cpustat *kcpustat, -enum cpu_usage_stat usage,int cpu); +extern u64 kcpustat_field (enum cpu_usage_stat usage,int cpu);extern void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu);#else -static inline u64 kcpustat_field (struct kernel_cpustat *kcpustat, -enum cpu_usage_stat usage,int cpu) +static inline u64 kcpustat_field (enum cpu_usage_stat usage,int cpu) { -return kcpustat->cpustat[usage]; +return kcpustat_cpu (cpu).cpustat[usage]; }static inline void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu)
kernel/rcu/tree.c +3 -6
@@ -969,14 +969,11 @@ static int rcu_watching_snap_recheck(struct rcu_data *rdp)if (rcu_cpu_stall_cputime && rdp->snap_record.gp_seq != rdp->gp_seq) {int cpu = rdp->cpu;struct rcu_snap_record *rsrp; -struct kernel_cpustat *kcsp; - - kcsp = &kcpustat_cpu (cpu); rsrp = &rdp->snap_record; - rsrp->cputime_irq =kcpustat_field (kcsp,CPUTIME_IRQ , cpu); - rsrp->cputime_softirq =kcpustat_field (kcsp,CPUTIME_SOFTIRQ , cpu); - rsrp->cputime_system =kcpustat_field (kcsp,CPUTIME_SYSTEM , cpu); + rsrp->cputime_irq =kcpustat_field (CPUTIME_IRQ , cpu); + rsrp->cputime_softirq =kcpustat_field (CPUTIME_SOFTIRQ , cpu); + rsrp->cputime_system =kcpustat_field (CPUTIME_SYSTEM , cpu); rsrp->nr_hardirqs =kstat_cpu_irqs_sum (cpu) +arch_irq_stat_cpu (cpu); rsrp->nr_softirqs =kstat_cpu_softirqs_sum (cpu); rsrp->nr_csw =nr_context_switches_cpu (cpu);
kernel/rcu/tree_stall.h +3 -4
@@ -479,7 +479,6 @@ static void print_cpu_stat_info(int cpu) {struct rcu_snap_record rsr, *rsrp;struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu); -struct kernel_cpustat *kcsp = &kcpustat_cpu (cpu);if (!rcu_cpu_stall_cputime)return ;
@@ -488,9 +487,9 @@ static void print_cpu_stat_info(int cpu)if (rsrp->gp_seq != rdp->gp_seq)return ; - rsr.cputime_irq =kcpustat_field (kcsp,CPUTIME_IRQ , cpu); - rsr.cputime_softirq =kcpustat_field (kcsp,CPUTIME_SOFTIRQ , cpu); - rsr.cputime_system =kcpustat_field (kcsp,CPUTIME_SYSTEM , cpu); + rsr.cputime_irq =kcpustat_field (CPUTIME_IRQ , cpu); + rsr.cputime_softirq =kcpustat_field (CPUTIME_SOFTIRQ , cpu); + rsr.cputime_system =kcpustat_field (CPUTIME_SYSTEM , cpu);pr_err ("\t hardirqs softirqs csw/system\n" ); pr_err("\t number: %8lld %10d %12lld\n",
kernel/sched/cputime.c +2 -3
@@ -975,10 +975,9 @@ static int kcpustat_field_vtime(u64 *cpustat,return 0 ; } -u64 kcpustat_field (struct kernel_cpustat *kcpustat, -enum cpu_usage_stat usage,int cpu) +u64 kcpustat_field (enum cpu_usage_stat usage,int cpu) { -u64 *cpustat = kcpustat->cpustat; +u64 *cpustat =kcpustat_cpu (cpu).cpustat;u64 val = cpustat[usage];struct rq *rq;int err;
Signature · 1 line
2.53.0
[tip: timers/nohz] sched/cputime: Remove superfluous and error prone kcpustat_field() parameter
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 080b5c6d95034e46f5ed1abe98c06218a1386aef Gitweb: https://git.kernel.org/tip/080b5c6d95034e46f5ed1abe98c06218a1386aef Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:35 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
sched/cputime: Remove superfluous and error prone kcpustat_field() parameter
The first parameter to kcpustat_field() is a pointer to the cpu kcpustat to be fetched from. This parameter is error prone because a copy to a kcpustat could be passed by accident instead of the original one. Also the kcpustat structure can already be retrieved with the help of the mandatory CPU argument.
Remove the needless parameter.
--- drivers/cpufreq/cpufreq_governor.c | 6 +++--- drivers/macintosh/rack-meter.c | 2 +- include/linux/kernel_stat.h | 8 +++----- kernel/rcu/tree.c | 9 +++------ kernel/rcu/tree_stall.h | 7 +++---- kernel/sched/cputime.c | 5 ++--- 6 files changed, 15 insertions(+), 22 deletions(-)
drivers/cpufreq/cpufreq_governor.c +3 -3
@@ -105,7 +105,7 @@ void gov_update_cpu_data(struct dbs_data *dbs_data) j_cdbs->prev_cpu_idle =get_cpu_idle_time (j, &j_cdbs->prev_update_time, dbs_data->io_is_busy);if (dbs_data->ignore_nice_load) - j_cdbs->prev_cpu_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); + j_cdbs->prev_cpu_nice =kcpustat_field (CPUTIME_NICE , j); } } }
@@ -165,7 +165,7 @@ unsigned int dbs_update(struct cpufreq_policy *policy) j_cdbs->prev_cpu_idle = cur_idle_time;if (ignore_nice) { -u64 cur_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); +u64 cur_nice =kcpustat_field (CPUTIME_NICE , j); idle_time +=div_u64 (cur_nice - j_cdbs->prev_cpu_nice,NSEC_PER_USEC ); j_cdbs->prev_cpu_nice = cur_nice;
@@ -539,7 +539,7 @@ int cpufreq_dbs_governor_start(struct cpufreq_policy *policy) j_cdbs->prev_load =0 ;if (ignore_nice) - j_cdbs->prev_cpu_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); + j_cdbs->prev_cpu_nice =kcpustat_field (CPUTIME_NICE , j); } gov->start (policy);
drivers/macintosh/rack-meter.c +1 -1
@@ -87,7 +87,7 @@ static inline u64 get_cpu_idle_time(unsigned int cpu) kcpustat->cpustat[CPUTIME_IOWAIT ];if (rackmeter_ignore_nice) - retval +=kcpustat_field (kcpustat,CPUTIME_NICE , cpu); + retval +=kcpustat_field (CPUTIME_NICE , cpu);return retval; }
include/linux/kernel_stat.h +3 -5
@@ -100,14 +100,12 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu) }#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -extern u64 kcpustat_field (struct kernel_cpustat *kcpustat, -enum cpu_usage_stat usage,int cpu); +extern u64 kcpustat_field (enum cpu_usage_stat usage,int cpu);extern void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu);#else -static inline u64 kcpustat_field (struct kernel_cpustat *kcpustat, -enum cpu_usage_stat usage,int cpu) +static inline u64 kcpustat_field (enum cpu_usage_stat usage,int cpu) { -return kcpustat->cpustat[usage]; +return kcpustat_cpu (cpu).cpustat[usage]; }static inline void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu)
kernel/rcu/tree.c +3 -6
@@ -969,14 +969,11 @@ static int rcu_watching_snap_recheck(struct rcu_data *rdp)if (rcu_cpu_stall_cputime && rdp->snap_record.gp_seq != rdp->gp_seq) {int cpu = rdp->cpu;struct rcu_snap_record *rsrp; -struct kernel_cpustat *kcsp; - - kcsp = &kcpustat_cpu (cpu); rsrp = &rdp->snap_record; - rsrp->cputime_irq =kcpustat_field (kcsp,CPUTIME_IRQ , cpu); - rsrp->cputime_softirq =kcpustat_field (kcsp,CPUTIME_SOFTIRQ , cpu); - rsrp->cputime_system =kcpustat_field (kcsp,CPUTIME_SYSTEM , cpu); + rsrp->cputime_irq =kcpustat_field (CPUTIME_IRQ , cpu); + rsrp->cputime_softirq =kcpustat_field (CPUTIME_SOFTIRQ , cpu); + rsrp->cputime_system =kcpustat_field (CPUTIME_SYSTEM , cpu); rsrp->nr_hardirqs =kstat_cpu_irqs_sum (cpu) +arch_irq_stat_cpu (cpu); rsrp->nr_softirqs =kstat_cpu_softirqs_sum (cpu); rsrp->nr_csw =nr_context_switches_cpu (cpu);
kernel/rcu/tree_stall.h +3 -4
@@ -479,7 +479,6 @@ static void print_cpu_stat_info(int cpu) {struct rcu_snap_record rsr, *rsrp;struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu); -struct kernel_cpustat *kcsp = &kcpustat_cpu (cpu);if (!rcu_cpu_stall_cputime)return ;
@@ -488,9 +487,9 @@ static void print_cpu_stat_info(int cpu)if (rsrp->gp_seq != rdp->gp_seq)return ; - rsr.cputime_irq =kcpustat_field (kcsp,CPUTIME_IRQ , cpu); - rsr.cputime_softirq =kcpustat_field (kcsp,CPUTIME_SOFTIRQ , cpu); - rsr.cputime_system =kcpustat_field (kcsp,CPUTIME_SYSTEM , cpu); + rsr.cputime_irq =kcpustat_field (CPUTIME_IRQ , cpu); + rsr.cputime_softirq =kcpustat_field (CPUTIME_SOFTIRQ , cpu); + rsr.cputime_system =kcpustat_field (CPUTIME_SYSTEM , cpu);pr_err ("\t hardirqs softirqs csw/system\n" ); pr_err("\t number: %8lld %10d %12lld\n",
kernel/sched/cputime.c +2 -3
@@ -975,10 +975,9 @@ static int kcpustat_field_vtime(u64 *cpustat,return 0 ; } -u64 kcpustat_field (struct kernel_cpustat *kcpustat, -enum cpu_usage_stat usage,int cpu) +u64 kcpustat_field (enum cpu_usage_stat usage,int cpu) { -u64 *cpustat = kcpustat->cpustat; +u64 *cpustat =kcpustat_cpu (cpu).cpustat;u64 val = cpustat[usage];struct rq *rq;int err;
Frederic Weisbecker
maintainerPATCH v4 4/15 sched/cputime: Correctly support generic vtime idle time
In reply to Frederic WeisbeckerCurrently whether generic vtime is running or not, the idle cputime is fetched from the nohz accounting.
However generic vtime already does its own idle cputime accounting. Only the kernel stat accessors are not plugged to support it.
Read the idle generic vtime cputime when it's running, this will allow to later more clearly split nohz and vtime cputime accounting.
--- include/linux/vtime.h | 9 +++++++-- kernel/sched/cputime.c | 38 +++++++++++++++++++++++++++++--------- kernel/time/tick-sched.c | 12 +++++++++--- 3 files changed, 45 insertions(+), 14 deletions(-)
include/linux/vtime.h +7 -2
@@ -10,7 +10,6 @@*/ #ifdef CONFIG_VIRT_CPU_ACCOUNTING extern void vtime_account_kernel (struct task_struct *tsk); -extern void vtime_account_idle (struct task_struct *tsk);#endif /* !CONFIG_VIRT_CPU_ACCOUNTING */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
@@ -27,7 +26,13 @@ static inline void vtime_guest_exit(struct task_struct *tsk) { }static inline void vtime_init_idle (struct task_struct *tsk,int cpu) { }#endif +static inline bool vtime_generic_enabled_cpu (int cpu) +{ +return context_tracking_enabled_cpu (cpu); +} +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE +extern void vtime_account_idle (struct task_struct *tsk);extern void vtime_account_irq (struct task_struct *tsk,unsigned int offset);extern void vtime_account_softirq (struct task_struct *tsk);extern void vtime_account_hardirq (struct task_struct *tsk);
@@ -74,7 +79,7 @@ static inline bool vtime_accounting_enabled(void)static inline bool vtime_accounting_enabled_cpu (int cpu) { -return context_tracking_enabled_cpu (cpu); +return vtime_generic_enabled_cpu (cpu); }static inline bool vtime_accounting_enabled_this_cpu (void )
kernel/sched/cputime.c +29 -9
@@ -773,9 +773,9 @@ void vtime_guest_exit(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (vtime_guest_exit); -void vtime_account_idle (struct task_struct *tsk) +static void __vtime_account_idle (struct vtime *vtime) { -account_idle_time (get_vtime_delta (&tsk->vtime)); +account_idle_time (get_vtime_delta (vtime)); }void vtime_task_switch_generic (struct task_struct *prev)
@@ -784,7 +784,7 @@ void vtime_task_switch_generic(struct task_struct *prev)write_seqcount_begin (&vtime->seqcount);if (vtime->state ==VTIME_IDLE ) -vtime_account_idle (prev); +__vtime_account_idle (vtime);else __vtime_account_kernel (prev, vtime); vtime->state =VTIME_INACTIVE ;
@@ -926,6 +926,7 @@ static int kcpustat_field_vtime(u64 *cpustat, int cpu, u64 *val) { struct vtime *vtime = &tsk->vtime; +struct rq *rq =cpu_rq (cpu);unsigned int seq; do {
@@ -967,6 +968,14 @@ static int kcpustat_field_vtime(u64 *cpustat,if (state ==VTIME_GUEST &&task_nice (tsk) >0 ) *val += vtime->gtime +vtime_delta (vtime);break ; +case CPUTIME_IDLE : +if (state ==VTIME_IDLE && !atomic_read (&rq->nr_iowait)) + *val +=vtime_delta (vtime); +break ; +case CPUTIME_IOWAIT : +if (state ==VTIME_IDLE &&atomic_read (&rq->nr_iowait) >0 ) + *val +=vtime_delta (vtime); +break ;default :break ; }
@@ -1029,8 +1038,8 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst, *dst = *src; cpustat = dst->cpustat; -/* Task is sleeping, dead or idle, nothing to add */ -if (state <VTIME_SYS ) +/* Task is sleeping or dead, nothing to add */ +if (state <VTIME_IDLE )continue ; delta =vtime_delta (vtime);
@@ -1039,15 +1048,17 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst,* Task runs either in user (including guest) or kernel space, * add pending nohz time to the right place. */ -if (state ==VTIME_SYS ) { +switch (state) { +case VTIME_SYS : cpustat[CPUTIME_SYSTEM ] += vtime->stime + delta; - }else if (state ==VTIME_USER ) { +break ; +case VTIME_USER :if (task_nice (tsk) >0 ) cpustat[CPUTIME_NICE ] += vtime->utime + delta;else cpustat[CPUTIME_USER ] += vtime->utime + delta; - }else { -WARN_ON_ONCE (state !=VTIME_GUEST ); +break ; +case VTIME_GUEST :if (task_nice (tsk) >0 ) { cpustat[CPUTIME_GUEST_NICE ] += vtime->gtime + delta; cpustat[CPUTIME_NICE ] += vtime->gtime + delta;
@@ -1055,6 +1066,15 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst, cpustat[CPUTIME_GUEST ] += vtime->gtime + delta; cpustat[CPUTIME_USER ] += vtime->gtime + delta; } +break ; +case VTIME_IDLE : +if (atomic_read (&cpu_rq (cpu)->nr_iowait) >0 ) + cpustat[CPUTIME_IOWAIT ] += delta; +else + cpustat[CPUTIME_IDLE ] += delta; +break ; +default : +WARN_ON_ONCE (1 ); } }while (read_seqcount_retry (&vtime->seqcount, seq));
kernel/time/tick-sched.c +9 -3
@@ -783,9 +783,10 @@ static void tick_nohz_start_idle(struct tick_sched *ts)sched_clock_idle_sleep_event (); } -static u64 get_cpu_sleep_time_us (struct tick_sched *ts,ktime_t *sleeptime, +static u64 get_cpu_sleep_time_us (int cpu,enum cpu_usage_stat idx,ktime_t *sleeptime,bool compute_delta,u64 *last_update_time) { +struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu);ktime_t now, idle;unsigned int seq;
@@ -796,6 +797,11 @@ static u64 get_cpu_sleep_time_us(struct tick_sched *ts, ktime_t *sleeptime,if (last_update_time) *last_update_time =ktime_to_us (now); +if (vtime_generic_enabled_cpu (cpu)) { + idle =kcpustat_field (idx, cpu); +return ktime_to_us (idle); + } + do { ktime_t delta = 0;
@@ -834,7 +840,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); -return get_cpu_sleep_time_us (ts, &ts->idle_sleeptime, +return get_cpu_sleep_time_us (cpu,CPUTIME_IDLE , &ts->idle_sleeptime, !nr_iowait_cpu (cpu), last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_idle_time_us);
@@ -860,7 +866,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); -return get_cpu_sleep_time_us (ts, &ts->iowait_sleeptime, +return get_cpu_sleep_time_us (cpu,CPUTIME_IOWAIT , &ts->iowait_sleeptime,nr_iowait_cpu (cpu), last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_iowait_time_us);
Signature · 1 line
2.53.0
[tip: timers/nohz] sched/cputime: Correctly support generic vtime idle time
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 650a59805a9baeff76379ea9309df1395eb15a46 Gitweb: https://git.kernel.org/tip/650a59805a9baeff76379ea9309df1395eb15a46 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:36 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
sched/cputime: Correctly support generic vtime idle time
Currently whether generic vtime is running or not, the idle cputime is fetched from the nohz accounting.
However generic vtime already does its own idle cputime accounting. Only the kernel stat accessors are not plugged to support it.
Read the idle generic vtime cputime when it's running, this will allow to later more clearly split nohz and vtime cputime accounting.
--- include/linux/vtime.h | 9 +++++++-- kernel/sched/cputime.c | 38 +++++++++++++++++++++++++++++--------- kernel/time/tick-sched.c | 12 +++++++++--- 3 files changed, 45 insertions(+), 14 deletions(-)
include/linux/vtime.h +7 -2
@@ -10,7 +10,6 @@*/ #ifdef CONFIG_VIRT_CPU_ACCOUNTING extern void vtime_account_kernel (struct task_struct *tsk); -extern void vtime_account_idle (struct task_struct *tsk);#endif /* !CONFIG_VIRT_CPU_ACCOUNTING */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
@@ -27,7 +26,13 @@ static inline void vtime_guest_exit(struct task_struct *tsk) { }static inline void vtime_init_idle (struct task_struct *tsk,int cpu) { }#endif +static inline bool vtime_generic_enabled_cpu (int cpu) +{ +return context_tracking_enabled_cpu (cpu); +} +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE +extern void vtime_account_idle (struct task_struct *tsk);extern void vtime_account_irq (struct task_struct *tsk,unsigned int offset);extern void vtime_account_softirq (struct task_struct *tsk);extern void vtime_account_hardirq (struct task_struct *tsk);
@@ -74,7 +79,7 @@ static inline bool vtime_accounting_enabled(void)static inline bool vtime_accounting_enabled_cpu (int cpu) { -return context_tracking_enabled_cpu (cpu); +return vtime_generic_enabled_cpu (cpu); }static inline bool vtime_accounting_enabled_this_cpu (void )
kernel/sched/cputime.c +29 -9
@@ -773,9 +773,9 @@ void vtime_guest_exit(struct task_struct *tsk) }EXPORT_SYMBOL_GPL (vtime_guest_exit); -void vtime_account_idle (struct task_struct *tsk) +static void __vtime_account_idle (struct vtime *vtime) { -account_idle_time (get_vtime_delta (&tsk->vtime)); +account_idle_time (get_vtime_delta (vtime)); }void vtime_task_switch_generic (struct task_struct *prev)
@@ -784,7 +784,7 @@ void vtime_task_switch_generic(struct task_struct *prev)write_seqcount_begin (&vtime->seqcount);if (vtime->state ==VTIME_IDLE ) -vtime_account_idle (prev); +__vtime_account_idle (vtime);else __vtime_account_kernel (prev, vtime); vtime->state =VTIME_INACTIVE ;
@@ -926,6 +926,7 @@ static int kcpustat_field_vtime(u64 *cpustat, int cpu, u64 *val) { struct vtime *vtime = &tsk->vtime; +struct rq *rq =cpu_rq (cpu);unsigned int seq; do {
@@ -967,6 +968,14 @@ static int kcpustat_field_vtime(u64 *cpustat,if (state ==VTIME_GUEST &&task_nice (tsk) >0 ) *val += vtime->gtime +vtime_delta (vtime);break ; +case CPUTIME_IDLE : +if (state ==VTIME_IDLE && !atomic_read (&rq->nr_iowait)) + *val +=vtime_delta (vtime); +break ; +case CPUTIME_IOWAIT : +if (state ==VTIME_IDLE &&atomic_read (&rq->nr_iowait) >0 ) + *val +=vtime_delta (vtime); +break ;default :break ; }
@@ -1029,8 +1038,8 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst, *dst = *src; cpustat = dst->cpustat; -/* Task is sleeping, dead or idle, nothing to add */ -if (state <VTIME_SYS ) +/* Task is sleeping or dead, nothing to add */ +if (state <VTIME_IDLE )continue ; delta =vtime_delta (vtime);
@@ -1039,15 +1048,17 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst,* Task runs either in user (including guest) or kernel space, * add pending nohz time to the right place. */ -if (state ==VTIME_SYS ) { +switch (state) { +case VTIME_SYS : cpustat[CPUTIME_SYSTEM ] += vtime->stime + delta; - }else if (state ==VTIME_USER ) { +break ; +case VTIME_USER :if (task_nice (tsk) >0 ) cpustat[CPUTIME_NICE ] += vtime->utime + delta;else cpustat[CPUTIME_USER ] += vtime->utime + delta; - }else { -WARN_ON_ONCE (state !=VTIME_GUEST ); +break ; +case VTIME_GUEST :if (task_nice (tsk) >0 ) { cpustat[CPUTIME_GUEST_NICE ] += vtime->gtime + delta; cpustat[CPUTIME_NICE ] += vtime->gtime + delta;
@@ -1055,6 +1066,15 @@ static int kcpustat_cpu_fetch_vtime(struct kernel_cpustat *dst, cpustat[CPUTIME_GUEST ] += vtime->gtime + delta; cpustat[CPUTIME_USER ] += vtime->gtime + delta; } +break ; +case VTIME_IDLE : +if (atomic_read (&cpu_rq (cpu)->nr_iowait) >0 ) + cpustat[CPUTIME_IOWAIT ] += delta; +else + cpustat[CPUTIME_IDLE ] += delta; +break ; +default : +WARN_ON_ONCE (1 ); } }while (read_seqcount_retry (&vtime->seqcount, seq));
kernel/time/tick-sched.c +9 -3
@@ -783,9 +783,10 @@ static void tick_nohz_start_idle(struct tick_sched *ts)sched_clock_idle_sleep_event (); } -static u64 get_cpu_sleep_time_us (struct tick_sched *ts,ktime_t *sleeptime, +static u64 get_cpu_sleep_time_us (int cpu,enum cpu_usage_stat idx,ktime_t *sleeptime,bool compute_delta,u64 *last_update_time) { +struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu);ktime_t now, idle;unsigned int seq;
@@ -796,6 +797,11 @@ static u64 get_cpu_sleep_time_us(struct tick_sched *ts, ktime_t *sleeptime,if (last_update_time) *last_update_time =ktime_to_us (now); +if (vtime_generic_enabled_cpu (cpu)) { + idle =kcpustat_field (idx, cpu); +return ktime_to_us (idle); + } + do { ktime_t delta = 0;
@@ -834,7 +840,7 @@ u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); -return get_cpu_sleep_time_us (ts, &ts->idle_sleeptime, +return get_cpu_sleep_time_us (cpu,CPUTIME_IDLE , &ts->idle_sleeptime, !nr_iowait_cpu (cpu), last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_idle_time_us);
@@ -860,7 +866,7 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) {struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu); -return get_cpu_sleep_time_us (ts, &ts->iowait_sleeptime, +return get_cpu_sleep_time_us (cpu,CPUTIME_IOWAIT , &ts->iowait_sleeptime,nr_iowait_cpu (cpu), last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_iowait_time_us);
Frederic Weisbecker
maintainerPATCH v4 5/15 powerpc/time: Prepare to stop elapsing in dynticks-idle
In reply to Frederic WeisbeckerCurrently the tick subsystem stores the idle cputime accounting in private fields, allowing cohabitation with architecture idle vtime accounting. The former is fetched on online CPUs, the latter on offline CPUs.
For consolidation purpose, architecture vtime accounting will continue to account the cputime but will make a break when the idle tick is stopped. The dyntick cputime accounting will then be relayed by the tick subsystem so that the idle cputime is still seen advancing coherently even when the tick isn't there to flush the idle vtime.
Prepare for that and introduce three new APIs which will be used in subsequent patches:
_ vtime_dynticks_start() is deemed to be called when idle enters in
dyntick mode. The idle cputime that elapsed so far is accumulated.
- vtime_dynticks_stop() is deemed to be called when idle exits from dyntick mode. The vtime entry clocks are fast-forward to current time so that idle accounting restarts elapsing from now.
- vtime_reset() is deemed to be called from dynticks idle IRQ entry to fast-forward the clock to current time so that the IRQ time is still accounted by vtime while nohz cputime is paused.
Also accumulated vtime won't be flushed from dyntick-idle ticks to avoid accounting twice the idle cputime, along with nohz accounting.
--- arch/powerpc/kernel/time.c | 41 ++++++++++++++++++++++++++++++++++++++ include/linux/vtime.h | 6 ++++++ 2 files changed, 47 insertions(+)
arch/powerpc/kernel/time.c +41 -0
@@ -376,6 +376,47 @@ void vtime_task_switch(struct task_struct *prev) acct->starttime = acct0->starttime; } } + +#ifdef CONFIG_NO_HZ_COMMON +/** +* vtime_reset - Fast forward vtime entry clocks +* +* Called from dynticks idle IRQ entry to fast-forward the clocks to current time +* so that the IRQ time is still accounted by vtime while nohz cputime is paused. +*/ +void vtime_reset (void ) +{ +struct cpu_accounting_data *acct =get_accounting (current); + + acct->starttime =mftb (); +#ifdef CONFIG_ARCH_HAS_SCALED_CPUTIME + acct->startspurr =read_spurr (acct->starttime); +#endif +} + +/** +* vtime_dyntick_start - Inform vtime about entry to idle-dynticks +* +* Called when idle enters in dyntick mode. The idle cputime that elapsed so far +* is accumulated and the tick subsystem takes over the idle cputime accounting. +*/ +void vtime_dyntick_start (void ) +{ +vtime_account_idle (current); +} + +/** +* vtime_dyntick_stop - Inform vtime about exit from idle-dynticks +* +* Called when idle exits from dyntick mode. The vtime entry clocks are +* fast-forward to current time so that idle accounting restarts elapsing from +* now. +*/ +void vtime_dyntick_stop (void ) +{ +vtime_reset (); +} +#endif /* CONFIG_NO_HZ_COMMON */ #endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ void __no_kcsan __delay(unsigned long loops)
include/linux/vtime.h +6 -0
@@ -37,11 +37,17 @@ extern void vtime_account_irq(struct task_struct *tsk, unsigned int offset);extern void vtime_account_softirq (struct task_struct *tsk);extern void vtime_account_hardirq (struct task_struct *tsk);extern void vtime_flush (struct task_struct *tsk); +extern void vtime_reset (void ); +extern void vtime_dyntick_start (void ); +extern void vtime_dyntick_stop (void );#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ static inline void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { }static inline void vtime_account_softirq (struct task_struct *tsk) { }static inline void vtime_account_hardirq (struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { } +static inline void vtime_reset (void ) { } +static inline void vtime_dyntick_start (void ) { } +static inline void vtime_dyntick_stop (void ) { }#endif /*
Signature · 1 line
2.53.0
[tip: timers/nohz] powerpc/time: Prepare to stop elapsing in dynticks-idle
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: c8ba971cf8567d49eb5f43ee90c4e50424331c18 Gitweb: https://git.kernel.org/tip/c8ba971cf8567d49eb5f43ee90c4e50424331c18 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:37 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
powerpc/time: Prepare to stop elapsing in dynticks-idle
Currently the tick subsystem stores the idle cputime accounting in private fields, allowing cohabitation with architecture idle vtime accounting. The former is fetched on online CPUs, the latter on offline CPUs.
For consolidation purpose, architecture vtime accounting will continue to account the cputime but will make a break when the idle tick is stopped. The dyntick cputime accounting will then be relayed by the tick subsystem so that the idle cputime is still seen advancing coherently even when the tick isn't there to flush the idle vtime.
Prepare for that and introduce three new APIs which will be used in subsequent patches:
- vtime_dynticks_start() is deemed to be called when idle enters in dyntick mode. The idle cputime that elapsed so far is accumulated.
- vtime_dynticks_stop() is deemed to be called when idle exits from dyntick mode. The vtime entry clocks are fast-forward to current time so that idle accounting restarts elapsing from now.
- vtime_reset() is deemed to be called from dynticks idle IRQ entry to fast-forward the clock to current time so that the IRQ time is still accounted by vtime while nohz cputime is paused.
Also accumulated vtime won't be flushed from dyntick-idle ticks to avoid accounting twice the idle cputime, along with nohz accounting.
--- arch/powerpc/kernel/time.c | 41 +++++++++++++++++++++++++++++++++++++- include/linux/vtime.h | 6 +++++- 2 files changed, 47 insertions(+)
arch/powerpc/kernel/time.c +41 -0
@@ -376,6 +376,47 @@ void vtime_task_switch(struct task_struct *prev) acct->starttime = acct0->starttime; } } + +#ifdef CONFIG_NO_HZ_COMMON +/** +* vtime_reset - Fast forward vtime entry clocks +* +* Called from dynticks idle IRQ entry to fast-forward the clocks to current time +* so that the IRQ time is still accounted by vtime while nohz cputime is paused. +*/ +void vtime_reset (void ) +{ +struct cpu_accounting_data *acct =get_accounting (current); + + acct->starttime =mftb (); +#ifdef CONFIG_ARCH_HAS_SCALED_CPUTIME + acct->startspurr =read_spurr (acct->starttime); +#endif +} + +/** +* vtime_dyntick_start - Inform vtime about entry to idle-dynticks +* +* Called when idle enters in dyntick mode. The idle cputime that elapsed so far +* is accumulated and the tick subsystem takes over the idle cputime accounting. +*/ +void vtime_dyntick_start (void ) +{ +vtime_account_idle (current); +} + +/** +* vtime_dyntick_stop - Inform vtime about exit from idle-dynticks +* +* Called when idle exits from dyntick mode. The vtime entry clocks are +* fast-forward to current time so that idle accounting restarts elapsing from +* now. +*/ +void vtime_dyntick_stop (void ) +{ +vtime_reset (); +} +#endif /* CONFIG_NO_HZ_COMMON */ #endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ void __no_kcsan __delay(unsigned long loops)
include/linux/vtime.h +6 -0
@@ -37,11 +37,17 @@ extern void vtime_account_irq(struct task_struct *tsk, unsigned int offset);extern void vtime_account_softirq (struct task_struct *tsk);extern void vtime_account_hardirq (struct task_struct *tsk);extern void vtime_flush (struct task_struct *tsk); +extern void vtime_reset (void ); +extern void vtime_dyntick_start (void ); +extern void vtime_dyntick_stop (void );#else /* !CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ static inline void vtime_account_irq (struct task_struct *tsk,unsigned int offset) { }static inline void vtime_account_softirq (struct task_struct *tsk) { }static inline void vtime_account_hardirq (struct task_struct *tsk) { }static inline void vtime_flush (struct task_struct *tsk) { } +static inline void vtime_reset (void ) { } +static inline void vtime_dyntick_start (void ) { } +static inline void vtime_dyntick_stop (void ) { }#endif /*
Frederic Weisbecker
maintainerPATCH v4 6/15 s390/time: Prepare to stop elapsing in dynticks-idle
In reply to Frederic WeisbeckerCurrently the tick subsystem stores the idle cputime accounting in private fields, allowing cohabitation with architecture idle vtime accounting. The former is fetched on online CPUs, the latter on offline CPUs.
For consolidation purpose, architecture vtime accounting will continue to account the cputime but will make a break when the idle tick is stopped. The dyntick cputime accounting will then be relayed by the tick subsystem so that the idle cputime is still seen advancing coherently even when the tick isn't there to flush the idle vtime.
Prepare for that and introduce three new APIs which will be used in subsequent patches:
_ vtime_dynticks_start() is deemed to be called when idle enters in
dyntick mode. The idle cputime that elapsed so far is accumulated and accounted. Also idle time accounting is ignored.
- vtime_dynticks_stop() is deemed to be called when idle exits from dyntick mode. The vtime entry clocks are fast-forward to current time so that idle accounting restarts elapsing from now. Also idle time accounting is resumed.
- vtime_reset() is deemed to be called from dynticks idle IRQ entry to fast-forward the clock to current time so that the IRQ time is still accounted by vtime while nohz cputime is paused.
Also accumulated vtime won't be flushed from dyntick-idle ticks to avoid accounting twice the idle cputime, along with nohz accounting.
--- arch/s390/include/asm/idle.h | 2 + arch/s390/kernel/idle.c | 5 ++- arch/s390/kernel/vtime.c | 75 ++++++++++++++++++++++++++++++++---- 3 files changed, 73 insertions(+), 9 deletions(-)
arch/s390/include/asm/idle.h +2 -0
@@ -8,10 +8,12 @@ #ifndef _S390_IDLE_H #define _S390_IDLE_H +#include <linux/percpu-defs.h> #include <linux/types.h> #include <linux/device.h> struct s390_idle_data { + bool idle_dyntick; unsigned long idle_count; unsigned long idle_time; unsigned long clock_idle_enter;
arch/s390/kernel/idle.c +4 -1
@@ -31,7 +31,10 @@ void account_idle_time_irq(void)/* Account time spent with enabled wait psw loaded as idle time. */ __atomic64_add (idle_time, &idle->idle_time);__atomic64_add_const (1 , &idle->idle_count); -account_idle_time (cputime_to_nsecs (idle_time)); + +/* Dyntick idle time accounted by nohz/scheduler */ +if (!idle->idle_dyntick) +account_idle_time (cputime_to_nsecs (idle_time)); } void noinstr arch_cpu_idle(void)
arch/s390/kernel/vtime.c +67 -8
110 lines not on this page · raw message
Signature · 1 line
2.53.0
[tip: timers/nohz] s390/time: Prepare to stop elapsing in dynticks-idle
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: ad5a9e14ec8b4a868fea13a9dfa1fb38b2c35354 Gitweb: https://git.kernel.org/tip/ad5a9e14ec8b4a868fea13a9dfa1fb38b2c35354 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:38 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
s390/time: Prepare to stop elapsing in dynticks-idle
Currently the tick subsystem stores the idle cputime accounting in private fields, allowing cohabitation with architecture idle vtime accounting. The former is fetched on online CPUs, the latter on offline CPUs.
For consolidation purposes, architecture vtime accounting will continue to account the cputime but will make a break when the idle tick is stopped. The dyntick cputime accounting will then be relayed by the tick subsystem so that the idle cputime is still seen advancing coherently even when the tick isn't there to flush the idle vtime.
Prepare for that and introduce three new APIs which will be used in subsequent patches:
- vtime_dynticks_start() is deemed to be called when idle enters in dyntick mode. The idle cputime that elapsed so far is accumulated and accounted. Also idle time accounting is ignored.
- vtime_dynticks_stop() is deemed to be called when idle exits from dyntick mode. The vtime entry clocks are fast-forward to current time so that idle accounting restarts elapsing from now. Also idle time accounting is resumed.
- vtime_reset() is deemed to be called from dynticks idle IRQ entry to fast-forward the clock to current time so that the IRQ time is still accounted by vtime while nohz cputime is paused.
Also accumulated vtime won't be flushed from dyntick-idle ticks to avoid accounting twice the idle cputime, along with nohz accounting.
--- arch/s390/include/asm/idle.h | 2 +- arch/s390/kernel/idle.c | 5 +- arch/s390/kernel/vtime.c | 75 +++++++++++++++++++++++++++++++---- 3 files changed, 73 insertions(+), 9 deletions(-)
arch/s390/include/asm/idle.h +2 -0
@@ -8,10 +8,12 @@ #ifndef _S390_IDLE_H #define _S390_IDLE_H +#include <linux/percpu-defs.h> #include <linux/types.h> #include <linux/device.h> struct s390_idle_data { + bool idle_dyntick; unsigned long idle_count; unsigned long idle_time; unsigned long clock_idle_enter;
arch/s390/kernel/idle.c +4 -1
@@ -31,7 +31,10 @@ void account_idle_time_irq(void)/* Account time spent with enabled wait psw loaded as idle time. */ __atomic64_add (idle_time, &idle->idle_time);__atomic64_add_const (1 , &idle->idle_count); -account_idle_time (cputime_to_nsecs (idle_time)); + +/* Dyntick idle time accounted by nohz/scheduler */ +if (!idle->idle_dyntick) +account_idle_time (cputime_to_nsecs (idle_time)); } void noinstr arch_cpu_idle(void)
arch/s390/kernel/vtime.c +67 -8
109 lines not on this page · raw message
Frederic Weisbecker
maintainerPATCH v4 7/15 tick/sched: Unify idle cputime accounting
In reply to Frederic WeisbeckerThe non-vtime dynticks-idle cputime accounting is a big mess that accumulates within two concurrent statistics, each having their own shortcomings:
* The accounting for online CPUs which is based on the delta between tick_nohz_start_idle() and tick_nohz_stop_idle().
Pros:
- Works when the tick is off
- Has nsecs granularity
Cons:
- Account idle steal time but doesn't substract it from idle cputime.
- Assumes CONFIG_IRQ_TIME_ACCOUNTING by not accounting IRQs but the IRQ time is simply ignored when CONFIG_IRQ_TIME_ACCOUNTING=n
- The windows between 1) idle task scheduling and the first call to tick_nohz_start_idle() and 2) idle task between the last tick_nohz_stop_idle() and the rest of the idle time are blindspots wrt. cputime accounting (though mostly insignificant amount)
- Relies on private fields outside of kernel stats, with specific accessors.
* The accounting for offline CPUs which is based on ticks and the jiffies delta during which the tick was stopped.
Pros:
- Handles steal time correctly
- Handle CONFIG_IRQ_TIME_ACCOUNTING=y and
CONFIG_IRQ_TIME_ACCOUNTING=n correctly.
- Handles the whole idle task
- Accounts directly to kernel stats, without midlayer accumulator.
Cons:
- Doesn't elapse when the tick is off, which doesn't make it suitable for online CPUs.
- Has TICK_NSEC granularity (jiffies)
- Needs to track the dyntick-idle ticks that were accounted and substract them from the total jiffies time spent while the tick was stopped. This is an ugly workaround.
Having two different accounting for a single context is not the only
problem: since those accountings are of different natures, it is possible to observe the global idle time going backward after a CPU goes offline.
Clean up the situation with introducing a hybrid approach that stays coherent and works for both online and offline CPUs:
* Tick based or native vtime accounting operate before the idle loop is entered and resume once the idle loop prepares to exit.
* When the idle loop starts, switch to dynticks-idle accounting as is done currently, except that the statistics accumulate directly to the relevant kernel stat fields.
* Private dyntick cputime accounting fields are removed.
* Works on both online and offline case.
Further improvement will include:
* Only switch to dynticks-idle cputime accounting when the tick actually goes in dynticks mode.
* Handle CONFIG_IRQ_TIME_ACCOUNTING=n correctly such that the dynticks-idle accounting still elapses while on IRQs.
* Correctly substract idle steal cputime from idle time
--- include/linux/kernel_stat.h | 24 ++++++++++--- include/linux/vtime.h | 7 +++- kernel/sched/cputime.c | 62 ++++++++++++++++----------------- kernel/time/tick-sched.c | 69 ++++++++++--------------------------- 4 files changed, 75 insertions(+), 87 deletions(-)
include/linux/kernel_stat.h +20 -4
@@ -34,6 +34,9 @@ enum cpu_usage_stat { };struct kernel_cpustat { +#ifdef CONFIG_NO_HZ_COMMON +int idle_dyntick; +#endif u64 cpustat[NR_STATS ]; };
@@ -99,6 +102,20 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu)return kstat_cpu (cpu).irqs_sum; } +#ifdef CONFIG_NO_HZ_COMMON +extern void kcpustat_dyntick_start (void ); +extern void kcpustat_dyntick_stop (void ); +static inline bool kcpustat_idle_dyntick (void ) +{ +return __this_cpu_read (kernel_cpustat.idle_dyntick); +} +#else +static inline bool kcpustat_idle_dyntick (void ) +{ +return false; +} +#endif /* CONFIG_NO_HZ_COMMON */ +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern u64 kcpustat_field (enum cpu_usage_stat usage,int cpu);extern void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu);
@@ -113,7 +130,7 @@ static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu) *dst = kcpustat_cpu(cpu); } -#endif +#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ extern void account_user_time (struct task_struct *,u64 );extern void account_guest_time (struct task_struct *,u64 );
@@ -127,14 +144,13 @@ extern u64 get_idle_time(struct kernel_cpustat *kcs, int cpu);#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE static inline void account_process_tick (struct task_struct *tsk,int user) { -vtime_flush (tsk); +if (!kcpustat_idle_dyntick ()) +vtime_flush (tsk); }#else extern void account_process_tick (struct task_struct *,int user);#endif -extern void account_idle_ticks (unsigned long ticks); -#ifdef CONFIG_SCHED_CORE extern void __account_forceidle_time (struct task_struct *tsk,u64 delta);#endif
include/linux/vtime.h +6 -1
@@ -31,6 +31,11 @@ static inline bool vtime_generic_enabled_cpu(int cpu)return context_tracking_enabled_cpu (cpu); } +static inline bool vtime_generic_enabled_this_cpu (void ) +{ +return context_tracking_enabled_this_cpu (); +} +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE extern void vtime_account_idle (struct task_struct *tsk);extern void vtime_account_irq (struct task_struct *tsk,unsigned int offset);
@@ -90,7 +95,7 @@ static inline bool vtime_accounting_enabled_cpu(int cpu)static inline bool vtime_accounting_enabled_this_cpu (void ) { -return context_tracking_enabled_this_cpu (); +return vtime_generic_enabled_this_cpu (); }extern void vtime_task_switch_generic (struct task_struct *prev);
kernel/sched/cputime.c +30 -32
92 lines not on this page · raw message
kernel/time/tick-sched.c +19 -50
174 lines not on this page · raw message
Signature · 1 line
2.53.0
[tip: timers/nohz] tick/sched: Unify idle cputime accounting
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: cf6444c3e1bb7dd5974441bbd74840e9821d36f9 Gitweb: https://git.kernel.org/tip/cf6444c3e1bb7dd5974441bbd74840e9821d36f9 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:39 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
tick/sched: Unify idle cputime accounting
The non-vtime dynticks-idle cputime accounting is a big mess that accumulates within two concurrent statistics, each having their own shortcomings:
* The accounting for online CPUs which is based on the delta between tick_nohz_start_idle() and tick_nohz_stop_idle().
Pros:
- Works when the tick is off
- Has nsecs granularity
Cons:
- Account idle steal time but doesn't substract it from idle cputime.
- Assumes CONFIG_IRQ_TIME_ACCOUNTING by not accounting IRQs but the IRQ time is simply ignored when CONFIG_IRQ_TIME_ACCOUNTING=n
- The windows between 1) idle task scheduling and the first call to tick_nohz_start_idle() and 2) idle task between the last tick_nohz_stop_idle() and the rest of the idle time are blindspots wrt. cputime accounting (though mostly insignificant amount)
- Relies on private fields outside of kernel stats, with specific accessors.
* The accounting for offline CPUs which is based on ticks and the jiffies delta during which the tick was stopped.
Pros:
- Handles steal time correctly
- Handle CONFIG_IRQ_TIME_ACCOUNTING=y and
CONFIG_IRQ_TIME_ACCOUNTING=n correctly.
- Handles the whole idle task
- Accounts directly to kernel stats, without midlayer accumulator.
Cons:
- Doesn't elapse when the tick is off, which doesn't make it suitable for online CPUs.
- Has TICK_NSEC granularity (jiffies)
- Needs to track the dyntick-idle ticks that were accounted and substract them from the total jiffies time spent while the tick was stopped. This is an ugly workaround.
Having two different accounting for a single context is not the only
problem: since those accountings are of different natures, it is possible to observe the global idle time going backward after a CPU goes offline.
Clean up the situation with introducing a hybrid approach that stays coherent and works for both online and offline CPUs:
* Tick based or native vtime accounting operate before the idle loop is entered and resume once the idle loop prepares to exit.
* When the idle loop starts, switch to dynticks-idle accounting as is done currently, except that the statistics accumulate directly to the relevant kernel stat fields.
* Private dyntick cputime accounting fields are removed.
* Works on both online and offline case.
Further improvement will include:
* Only switch to dynticks-idle cputime accounting when the tick actually goes in dynticks mode.
* Handle CONFIG_IRQ_TIME_ACCOUNTING=n correctly such that the dynticks-idle accounting still elapses while on IRQs.
* Correctly substract idle steal cputime from idle time
--- include/linux/kernel_stat.h | 24 ++++++++++--- include/linux/vtime.h | 7 +++- kernel/sched/cputime.c | 62 +++++++++++++++----------------- kernel/time/tick-sched.c | 69 +++++++++--------------------------- 4 files changed, 75 insertions(+), 87 deletions(-)
include/linux/kernel_stat.h +20 -4
@@ -34,6 +34,9 @@ enum cpu_usage_stat { };struct kernel_cpustat { +#ifdef CONFIG_NO_HZ_COMMON +int idle_dyntick; +#endif u64 cpustat[NR_STATS ]; };
@@ -99,6 +102,20 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu)return kstat_cpu (cpu).irqs_sum; } +#ifdef CONFIG_NO_HZ_COMMON +extern void kcpustat_dyntick_start (void ); +extern void kcpustat_dyntick_stop (void ); +static inline bool kcpustat_idle_dyntick (void ) +{ +return __this_cpu_read (kernel_cpustat.idle_dyntick); +} +#else +static inline bool kcpustat_idle_dyntick (void ) +{ +return false; +} +#endif /* CONFIG_NO_HZ_COMMON */ +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern u64 kcpustat_field (enum cpu_usage_stat usage,int cpu);extern void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu);
@@ -113,7 +130,7 @@ static inline void kcpustat_cpu_fetch(struct kernel_cpustat *dst, int cpu) *dst = kcpustat_cpu(cpu); } -#endif +#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ extern void account_user_time (struct task_struct *,u64 );extern void account_guest_time (struct task_struct *,u64 );
@@ -127,14 +144,13 @@ extern u64 get_idle_time(struct kernel_cpustat *kcs, int cpu);#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE static inline void account_process_tick (struct task_struct *tsk,int user) { -vtime_flush (tsk); +if (!kcpustat_idle_dyntick ()) +vtime_flush (tsk); }#else extern void account_process_tick (struct task_struct *,int user);#endif -extern void account_idle_ticks (unsigned long ticks); -#ifdef CONFIG_SCHED_CORE extern void __account_forceidle_time (struct task_struct *tsk,u64 delta);#endif
include/linux/vtime.h +6 -1
@@ -31,6 +31,11 @@ static inline bool vtime_generic_enabled_cpu(int cpu)return context_tracking_enabled_cpu (cpu); } +static inline bool vtime_generic_enabled_this_cpu (void ) +{ +return context_tracking_enabled_this_cpu (); +} +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE extern void vtime_account_idle (struct task_struct *tsk);extern void vtime_account_irq (struct task_struct *tsk,unsigned int offset);
@@ -90,7 +95,7 @@ static inline bool vtime_accounting_enabled_cpu(int cpu)static inline bool vtime_accounting_enabled_this_cpu (void ) { -return context_tracking_enabled_this_cpu (); +return vtime_generic_enabled_this_cpu (); }extern void vtime_task_switch_generic (struct task_struct *prev);
kernel/sched/cputime.c +30 -32
92 lines not on this page · raw message
kernel/time/tick-sched.c +19 -50
174 lines not on this page · raw message
Frederic Weisbecker
maintainerPATCH v4 8/15 tick/sched: Remove nohz disabled special case in cputime fetch
In reply to Frederic WeisbeckerEven when nohz is not runtime enabled, the dynticks idle cputime accounting can run and the common idle cputime accessors are still relevant.
Remove the nohz disabled special case accordingly.
--- kernel/time/tick-sched.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
kernel/time/tick-sched.c +2 -5
@@ -795,9 +795,6 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx,ktime_t now, idle;unsigned int seq; -if (!tick_nohz_active) -return -1 ; - now =ktime_get ();if (last_update_time) *last_update_time =ktime_to_us (now);
@@ -839,7 +836,7 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx,* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if NOHZ is not enabled, else total idle time of the @cpu +* Return: -1 if generic vtime is enabled, else total idle time of the @cpu */ u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time) {
@@ -863,7 +860,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if NOHZ is not enabled, else total iowait time of @cpu +* Return: -1 if generic vtime is enabled, else total iowait time of @cpu */ u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time) {
Signature · 1 line
2.53.0
[tip: timers/nohz] tick/sched: Remove nohz disabled special case in cputime fetch
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: bd0c77cd46c63d02bc33dacdba56133ec1fe44a0 Gitweb: https://git.kernel.org/tip/bd0c77cd46c63d02bc33dacdba56133ec1fe44a0 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:40 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
tick/sched: Remove nohz disabled special case in cputime fetch
Even when nohz is not runtime enabled, the dynticks idle cputime accounting can run and the common idle cputime accessors are still relevant.
Remove the nohz disabled special case accordingly.
--- kernel/time/tick-sched.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
kernel/time/tick-sched.c +2 -5
@@ -795,9 +795,6 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx,ktime_t now, idle;unsigned int seq; -if (!tick_nohz_active) -return -1 ; - now =ktime_get ();if (last_update_time) *last_update_time =ktime_to_us (now);
@@ -839,7 +836,7 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx,* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if NOHZ is not enabled, else total idle time of the @cpu +* Return: -1 if generic vtime is enabled, else total idle time of the @cpu */ u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time) {
@@ -863,7 +860,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if NOHZ is not enabled, else total iowait time of @cpu +* Return: -1 if generic vtime is enabled, else total iowait time of @cpu */ u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time) {
Frederic Weisbecker
maintainerPATCH v4 9/15 tick/sched: Move dyntick-idle cputime accounting to cputime code
In reply to Frederic WeisbeckerAlthough the dynticks-idle cputime accounting is necessarily tied to the tick subsystem, the actual related accounting code has no business residing there and should be part of the scheduler cputime code.
Move away the relevant pieces and state machine to where they belong.
--- include/linux/kernel_stat.h | 14 +++- kernel/sched/core.c | 6 +- kernel/sched/cputime.c | 148 ++++++++++++++++++++++++++++++-- kernel/time/tick-sched.c | 163 +++++++----------------------------- 4 files changed, 188 insertions(+), 143 deletions(-)
include/linux/kernel_stat.h +10 -4
@@ -35,9 +35,12 @@ enum cpu_usage_stat {struct kernel_cpustat {#ifdef CONFIG_NO_HZ_COMMON -int idle_dyntick; +bool idle_dyntick; +bool idle_elapse; +seqcount_t idle_sleeptime_seq; +u64 idle_entrytime;#endif -u64 cpustat[NR_STATS ]; +u64 cpustat[NR_STATS ]; };struct kernel_stat {
@@ -103,8 +106,11 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu) }#ifdef CONFIG_NO_HZ_COMMON -extern void kcpustat_dyntick_start (void ); -extern void kcpustat_dyntick_stop (void ); +extern void kcpustat_dyntick_start (u64 now); +extern void kcpustat_dyntick_stop (u64 now); +extern void kcpustat_irq_enter (u64 now); +extern void kcpustat_irq_exit (u64 now); +static inline bool kcpustat_idle_dyntick (void ) {return __this_cpu_read (kernel_cpustat.idle_dyntick);
kernel/sched/core.c +5 -1
@@ -5517,7 +5517,11 @@ void sched_exec(void) } DEFINE_PER_CPU(struct kernel_stat, kstat); -DEFINE_PER_CPU(struct kernel_cpustat, kernel_cpustat); +DEFINE_PER_CPU(struct kernel_cpustat, kernel_cpustat) = { +#ifdef CONFIG_NO_HZ_COMMON + .idle_sleeptime_seq = SEQCNT_ZERO(kernel_cpustat.idle_sleeptime_seq) +#endif +};EXPORT_PER_CPU_SYMBOL (kstat);EXPORT_PER_CPU_SYMBOL (kernel_cpustat);
kernel/sched/cputime.c +141 -7
169 lines not on this page · raw message
kernel/time/tick-sched.c +32 -131
206 lines not on this page · raw message
Signature · 1 line
2.53.0
[tip: timers/nohz] tick/sched: Move dyntick-idle cputime accounting to cputime code
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: a5fe724e206ec7ff3ceb15b285d94316c7fe6c41 Gitweb: https://git.kernel.org/tip/a5fe724e206ec7ff3ceb15b285d94316c7fe6c41 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:41 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
tick/sched: Move dyntick-idle cputime accounting to cputime code
Although the dynticks-idle cputime accounting is necessarily tied to the tick subsystem, the actual related accounting code has no business residing there and should be part of the scheduler cputime code.
Move away the relevant pieces and state machine to where they belong.
--- include/linux/kernel_stat.h | 14 ++- kernel/sched/core.c | 6 +- kernel/sched/cputime.c | 148 ++++++++++++++++++++++++++++++-- kernel/time/tick-sched.c | 163 ++++++----------------------------- 4 files changed, 188 insertions(+), 143 deletions(-)
include/linux/kernel_stat.h +10 -4
@@ -35,9 +35,12 @@ enum cpu_usage_stat {struct kernel_cpustat {#ifdef CONFIG_NO_HZ_COMMON -int idle_dyntick; +bool idle_dyntick; +bool idle_elapse; +seqcount_t idle_sleeptime_seq; +u64 idle_entrytime;#endif -u64 cpustat[NR_STATS ]; +u64 cpustat[NR_STATS ]; };struct kernel_stat {
@@ -103,8 +106,11 @@ static inline unsigned long kstat_cpu_irqs_sum(unsigned int cpu) }#ifdef CONFIG_NO_HZ_COMMON -extern void kcpustat_dyntick_start (void ); -extern void kcpustat_dyntick_stop (void ); +extern void kcpustat_dyntick_start (u64 now); +extern void kcpustat_dyntick_stop (u64 now); +extern void kcpustat_irq_enter (u64 now); +extern void kcpustat_irq_exit (u64 now); +static inline bool kcpustat_idle_dyntick (void ) {return __this_cpu_read (kernel_cpustat.idle_dyntick);
kernel/sched/core.c +5 -1
@@ -5518,7 +5518,11 @@ void sched_exec(void) } DEFINE_PER_CPU(struct kernel_stat, kstat); -DEFINE_PER_CPU(struct kernel_cpustat, kernel_cpustat); +DEFINE_PER_CPU(struct kernel_cpustat, kernel_cpustat) = { +#ifdef CONFIG_NO_HZ_COMMON + .idle_sleeptime_seq = SEQCNT_ZERO(kernel_cpustat.idle_sleeptime_seq) +#endif +};EXPORT_PER_CPU_SYMBOL (kstat);EXPORT_PER_CPU_SYMBOL (kernel_cpustat);
kernel/sched/cputime.c +141 -7
169 lines not on this page · raw message
kernel/time/tick-sched.c +32 -131
206 lines not on this page · raw message
Frederic Weisbecker
maintainerPATCH v4 10/15 tick/sched: Remove unused fields
In reply to Frederic WeisbeckerRemove fields after the dyntick-idle cputime migration to scheduler code.
--- kernel/time/tick-sched.h | 12 ------------ kernel/time/timer_list.c | 6 +----- scripts/gdb/linux/timerlist.py | 4 ---- 3 files changed, 1 insertion(+), 21 deletions(-)
kernel/time/tick-sched.h +0 -12
@@ -44,9 +44,7 @@ struct tick_device {* to resume the tick timer operation in the timeline * when the CPU returns from nohz sleep. * @next_tick: Next tick to be fired when in dynticks mode. -* @idle_jiffies: jiffies at the entry to idle for idle time accounting * @idle_waketime: Time when the idle was interrupted -* @idle_sleeptime_seq: sequence counter for data consistency * @idle_entrytime: Time when the idle call was entered * @last_jiffies: Base jiffies snapshot when next event was last computed * @timer_expires_base: Base time clock monotonic for @timer_expires
@@ -55,9 +53,6 @@ struct tick_device {* @idle_expires: Next tick in idle, for debugging purpose only * @idle_calls: Total number of idle calls * @idle_sleeps: Number of idle calls, where the sched tick was stopped -* @idle_exittime: Time when the idle state was left -* @idle_sleeptime: Sum of the time slept in idle with sched tick stopped -* @iowait_sleeptime: Sum of the time slept in idle with sched tick stopped, with IO outstanding * @tick_dep_mask: Tick dependency mask - is set, if someone needs the tick * @check_clocks: Notification mechanism about clocksource changes */
@@ -73,12 +68,10 @@ struct tick_sched {struct hrtimer sched_timer;ktime_t last_tick;ktime_t next_tick; -unsigned long idle_jiffies;ktime_t idle_waketime;unsigned int got_idle_tick;/* Idle entry */ -seqcount_t idle_sleeptime_seq;ktime_t idle_entrytime;/* Tick stop */
@@ -90,11 +83,6 @@ struct tick_sched {unsigned long idle_calls;unsigned long idle_sleeps; -/* Idle exit */ -ktime_t idle_exittime; -ktime_t idle_sleeptime; -ktime_t iowait_sleeptime; -/* Full dynticks handling */ atomic_t tick_dep_mask;
kernel/time/timer_list.c +1 -5
@@ -152,14 +152,10 @@ static void print_cpu(struct seq_file *m, int cpu, u64 now)P_flag (highres,TS_FLAG_HIGHRES );P_ns (last_tick);P_flag (tick_stopped,TS_FLAG_STOPPED ); -P (idle_jiffies);P (idle_calls);P (idle_sleeps);P_ns (idle_entrytime);P_ns (idle_waketime); -P_ns (idle_exittime); -P_ns (idle_sleeptime); -P_ns (iowait_sleeptime);P (last_jiffies);P (next_timer);P_ns (idle_expires);
@@ -256,7 +252,7 @@ static void timer_list_show_tickdevices_header(struct seq_file *m)static inline void timer_list_header (struct seq_file *m,u64 now) { -SEQ_printf (m,"Timer List Version: v0.10\n" ); +SEQ_printf (m,"Timer List Version: v0.11\n" );SEQ_printf (m,"HRTIMER_MAX_CLOCK_BASES: %d\n" ,HRTIMER_MAX_CLOCK_BASES );SEQ_printf (m,"now at %Ld nsecs\n" , (unsigned long long )now);SEQ_printf (m,"\n" );
scripts/gdb/linux/timerlist.py +0 -4
@@ -90,14 +90,10 @@ def print_cpu(hrtimer_bases, cpu, max_clock_bases): text += f" .{'nohz':15s}: {int(bool(ts['flags'] & TS_FLAG_NOHZ))}\n" text += f" .{'last_tick':15s}: {ts['last_tick']}\n" text += f" .{'tick_stopped':15s}: {int(bool(ts['flags'] & TS_FLAG_STOPPED))}\n" - text += f" .{'idle_jiffies':15s}: {ts['idle_jiffies']}\n" text += f" .{'idle_calls':15s}: {ts['idle_calls']}\n" text += f" .{'idle_sleeps':15s}: {ts['idle_sleeps']}\n" text += f" .{'idle_entrytime':15s}: {ts['idle_entrytime']} nsecs\n" text += f" .{'idle_waketime':15s}: {ts['idle_waketime']} nsecs\n" - text += f" .{'idle_exittime':15s}: {ts['idle_exittime']} nsecs\n" - text += f" .{'idle_sleeptime':15s}: {ts['idle_sleeptime']} nsecs\n" - text += f" .{'iowait_sleeptime':15s}: {ts['iowait_sleeptime']} nsecs\n" text += f" .{'last_jiffies':15s}: {ts['last_jiffies']}\n" text += f" .{'next_timer':15s}: {ts['next_timer']}\n" text += f" .{'idle_expires':15s}: {ts['idle_expires']} nsecs\n"
Signature · 1 line
2.53.0
[tip: timers/nohz] tick/sched: Remove unused fields
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 29807c524d66e27762cdc8992c2cac89b4c3fda9 Gitweb: https://git.kernel.org/tip/29807c524d66e27762cdc8992c2cac89b4c3fda9 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:42 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:25 +02:00
tick/sched: Remove unused fields
Remove fields after the dyntick-idle cputime migration to scheduler code.
--- kernel/time/tick-sched.h | 12 ------------ kernel/time/timer_list.c | 6 +----- scripts/gdb/linux/timerlist.py | 4 ---- 3 files changed, 1 insertion(+), 21 deletions(-)
kernel/time/tick-sched.h +0 -12
@@ -44,9 +44,7 @@ struct tick_device {* to resume the tick timer operation in the timeline * when the CPU returns from nohz sleep. * @next_tick: Next tick to be fired when in dynticks mode. -* @idle_jiffies: jiffies at the entry to idle for idle time accounting * @idle_waketime: Time when the idle was interrupted -* @idle_sleeptime_seq: sequence counter for data consistency * @idle_entrytime: Time when the idle call was entered * @last_jiffies: Base jiffies snapshot when next event was last computed * @timer_expires_base: Base time clock monotonic for @timer_expires
@@ -55,9 +53,6 @@ struct tick_device {* @idle_expires: Next tick in idle, for debugging purpose only * @idle_calls: Total number of idle calls * @idle_sleeps: Number of idle calls, where the sched tick was stopped -* @idle_exittime: Time when the idle state was left -* @idle_sleeptime: Sum of the time slept in idle with sched tick stopped -* @iowait_sleeptime: Sum of the time slept in idle with sched tick stopped, with IO outstanding * @tick_dep_mask: Tick dependency mask - is set, if someone needs the tick * @check_clocks: Notification mechanism about clocksource changes */
@@ -73,12 +68,10 @@ struct tick_sched {struct hrtimer sched_timer;ktime_t last_tick;ktime_t next_tick; -unsigned long idle_jiffies;ktime_t idle_waketime;unsigned int got_idle_tick;/* Idle entry */ -seqcount_t idle_sleeptime_seq;ktime_t idle_entrytime;/* Tick stop */
@@ -90,11 +83,6 @@ struct tick_sched {unsigned long idle_calls;unsigned long idle_sleeps; -/* Idle exit */ -ktime_t idle_exittime; -ktime_t idle_sleeptime; -ktime_t iowait_sleeptime; -/* Full dynticks handling */ atomic_t tick_dep_mask;
kernel/time/timer_list.c +1 -5
@@ -152,14 +152,10 @@ static void print_cpu(struct seq_file *m, int cpu, u64 now)P_flag (highres,TS_FLAG_HIGHRES );P_ns (last_tick);P_flag (tick_stopped,TS_FLAG_STOPPED ); -P (idle_jiffies);P (idle_calls);P (idle_sleeps);P_ns (idle_entrytime);P_ns (idle_waketime); -P_ns (idle_exittime); -P_ns (idle_sleeptime); -P_ns (iowait_sleeptime);P (last_jiffies);P (next_timer);P_ns (idle_expires);
@@ -256,7 +252,7 @@ static void timer_list_show_tickdevices_header(struct seq_file *m)static inline void timer_list_header (struct seq_file *m,u64 now) { -SEQ_printf (m,"Timer List Version: v0.10\n" ); +SEQ_printf (m,"Timer List Version: v0.11\n" );SEQ_printf (m,"HRTIMER_MAX_CLOCK_BASES: %d\n" ,HRTIMER_MAX_CLOCK_BASES );SEQ_printf (m,"now at %Ld nsecs\n" , (unsigned long long )now);SEQ_printf (m,"\n" );
scripts/gdb/linux/timerlist.py +0 -4
@@ -90,14 +90,10 @@ def print_cpu(hrtimer_bases, cpu, max_clock_bases): text += f" .{'nohz':15s}: {int(bool(ts['flags'] & TS_FLAG_NOHZ))}\n" text += f" .{'last_tick':15s}: {ts['last_tick']}\n" text += f" .{'tick_stopped':15s}: {int(bool(ts['flags'] & TS_FLAG_STOPPED))}\n" - text += f" .{'idle_jiffies':15s}: {ts['idle_jiffies']}\n" text += f" .{'idle_calls':15s}: {ts['idle_calls']}\n" text += f" .{'idle_sleeps':15s}: {ts['idle_sleeps']}\n" text += f" .{'idle_entrytime':15s}: {ts['idle_entrytime']} nsecs\n" text += f" .{'idle_waketime':15s}: {ts['idle_waketime']} nsecs\n" - text += f" .{'idle_exittime':15s}: {ts['idle_exittime']} nsecs\n" - text += f" .{'idle_sleeptime':15s}: {ts['idle_sleeptime']} nsecs\n" - text += f" .{'iowait_sleeptime':15s}: {ts['iowait_sleeptime']} nsecs\n" text += f" .{'last_jiffies':15s}: {ts['last_jiffies']}\n" text += f" .{'next_timer':15s}: {ts['next_timer']}\n" text += f" .{'idle_expires':15s}: {ts['idle_expires']} nsecs\n"
Frederic Weisbecker
maintainerPATCH v4 11/15 tick/sched: Account tickless idle cputime only when tick is stopped
In reply to Frederic WeisbeckerThere is no real point in switching to dyntick-idle cputime accounting mode if the tick is not actually stopped. This just adds overhead, notably fetching the GTOD, on each idle exit and each idle IRQ entry for no reason during short idle trips.
--- kernel/time/tick-sched.c | 50 +++++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 24 deletions(-)
kernel/time/tick-sched.c +26 -24
@@ -1157,8 +1157,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) +if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +kcpustat_dyntick_start (ts->idle_entrytime);nohz_balance_enter_idle (cpu); + } }else {tick_nohz_retain_tick (ts); }
@@ -1200,7 +1202,6 @@ void tick_nohz_idle_enter(void)WARN_ON_ONCE (ts->timer_expires_base);tick_sched_flag_set (ts,TS_FLAG_INIDLE ); ts->idle_entrytime =ktime_get (); -kcpustat_dyntick_start (ts->idle_entrytime);tick_nohz_clock_sleep (ts);local_irq_enable ();
@@ -1230,9 +1231,10 @@ void tick_nohz_irq_exit(void)struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched);if (tick_sched_flag_test (ts,TS_FLAG_INIDLE )) { - ts->idle_entrytime =ktime_get (); -kcpustat_irq_exit (ts->idle_entrytime);tick_nohz_clock_sleep (ts); + ts->idle_entrytime =ktime_get (); +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) +kcpustat_irq_exit (ts->idle_entrytime); }else {tick_nohz_full_update_tick (ts); }
@@ -1333,8 +1335,17 @@ void tick_nohz_idle_restart_tick(void) {struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched); -if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) -tick_nohz_restart_sched_tick (ts,ktime_get ()); +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +/* +* Update entrytime here in case the tick restart is due to temporary +* polling on forced broadcast. The tick may be stopped again later within +* the same idle trip. The idle_entrytime was updated recently but make sure +* no tiny amount of idle time is accounted twice. +*/ + ts->idle_entrytime =ktime_get (); +kcpustat_dyntick_stop (ts->idle_entrytime); +tick_nohz_restart_sched_tick (ts, ts->idle_entrytime); + } }static void tick_nohz_idle_update_tick (struct tick_sched *ts,ktime_t now)
@@ -1364,7 +1375,6 @@ static void tick_nohz_idle_update_tick(struct tick_sched *ts, ktime_t now)void tick_nohz_idle_exit (void ) {struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched); -bool idle_active, tick_stopped;ktime_t now;local_irq_disable ();
@@ -1373,18 +1383,13 @@ void tick_nohz_idle_exit(void)WARN_ON_ONCE (ts->timer_expires_base);tick_sched_flag_clear (ts,TS_FLAG_INIDLE ); - idle_active =tick_sched_flag_test (ts,TS_FLAG_IDLE_ACTIVE ); - tick_stopped =tick_sched_flag_test (ts,TS_FLAG_STOPPED ); +tick_nohz_clock_wakeup (ts); -if (idle_active || tick_stopped) +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { now =ktime_get (); - -if (idle_active) -tick_nohz_clock_wakeup (ts); - -if (tick_stopped) +kcpustat_dyntick_stop (now);tick_nohz_idle_update_tick (ts, now); -kcpustat_dyntick_stop (now); + }local_irq_enable (); }
@@ -1439,15 +1444,13 @@ static inline void tick_nohz_irq_enter(void)struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched);ktime_t now; -if (!tick_sched_flag_test (ts,TS_FLAG_STOPPED |TS_FLAG_IDLE_ACTIVE )) +tick_nohz_clock_wakeup (ts); + +if (!tick_sched_flag_test (ts,TS_FLAG_STOPPED ))return ; now =ktime_get (); - -if (tick_sched_flag_test (ts,TS_FLAG_IDLE_ACTIVE )) { -tick_nohz_clock_wakeup (ts); -kcpustat_irq_enter (now); - } +kcpustat_irq_enter (now);/* * If all CPUs are idle we may need to update a stale jiffies value.
@@ -1456,8 +1459,7 @@ static inline void tick_nohz_irq_enter(void)* rare case (typically stop machine). So we must make sure we have a * last resort. */ -if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) - tick_nohz_update_jiffies(now); + tick_nohz_update_jiffies(now); }#else
Signature · 1 line
2.53.0
[tip: timers/nohz] tick/sched: Account tickless idle cputime only when tick is stopped
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 6a1f6a9dd0736257f5e5af32dd955d186cdc075d Gitweb: https://git.kernel.org/tip/6a1f6a9dd0736257f5e5af32dd955d186cdc075d Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:43 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:26 +02:00
tick/sched: Account tickless idle cputime only when tick is stopped
There is no real point in switching to dyntick-idle cputime accounting mode if the tick is not actually stopped. This just adds overhead, notably fetching the GTOD, on each idle exit and each idle IRQ entry for no reason during short idle trips.
--- kernel/time/tick-sched.c | 50 ++++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 24 deletions(-)
kernel/time/tick-sched.c +26 -24
@@ -1157,8 +1157,10 @@ void tick_nohz_idle_stop_tick(void) ts->idle_sleeps++; ts->idle_expires = expires; -if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) +if (!was_stopped &&tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +kcpustat_dyntick_start (ts->idle_entrytime);nohz_balance_enter_idle (cpu); + } }else {tick_nohz_retain_tick (ts); }
@@ -1200,7 +1202,6 @@ void tick_nohz_idle_enter(void)WARN_ON_ONCE (ts->timer_expires_base);tick_sched_flag_set (ts,TS_FLAG_INIDLE ); ts->idle_entrytime =ktime_get (); -kcpustat_dyntick_start (ts->idle_entrytime);tick_nohz_clock_sleep (ts);local_irq_enable ();
@@ -1230,9 +1231,10 @@ void tick_nohz_irq_exit(void)struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched);if (tick_sched_flag_test (ts,TS_FLAG_INIDLE )) { - ts->idle_entrytime =ktime_get (); -kcpustat_irq_exit (ts->idle_entrytime);tick_nohz_clock_sleep (ts); + ts->idle_entrytime =ktime_get (); +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) +kcpustat_irq_exit (ts->idle_entrytime); }else {tick_nohz_full_update_tick (ts); }
@@ -1333,8 +1335,17 @@ void tick_nohz_idle_restart_tick(void) {struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched); -if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) -tick_nohz_restart_sched_tick (ts,ktime_get ()); +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { +/* +* Update entrytime here in case the tick restart is due to temporary +* polling on forced broadcast. The tick may be stopped again later within +* the same idle trip. The idle_entrytime was updated recently but make sure +* no tiny amount of idle time is accounted twice. +*/ + ts->idle_entrytime =ktime_get (); +kcpustat_dyntick_stop (ts->idle_entrytime); +tick_nohz_restart_sched_tick (ts, ts->idle_entrytime); + } }static void tick_nohz_idle_update_tick (struct tick_sched *ts,ktime_t now)
@@ -1364,7 +1375,6 @@ static void tick_nohz_idle_update_tick(struct tick_sched *ts, ktime_t now)void tick_nohz_idle_exit (void ) {struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched); -bool idle_active, tick_stopped;ktime_t now;local_irq_disable ();
@@ -1373,18 +1383,13 @@ void tick_nohz_idle_exit(void)WARN_ON_ONCE (ts->timer_expires_base);tick_sched_flag_clear (ts,TS_FLAG_INIDLE ); - idle_active =tick_sched_flag_test (ts,TS_FLAG_IDLE_ACTIVE ); - tick_stopped =tick_sched_flag_test (ts,TS_FLAG_STOPPED ); +tick_nohz_clock_wakeup (ts); -if (idle_active || tick_stopped) +if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) { now =ktime_get (); - -if (idle_active) -tick_nohz_clock_wakeup (ts); - -if (tick_stopped) +kcpustat_dyntick_stop (now);tick_nohz_idle_update_tick (ts, now); -kcpustat_dyntick_stop (now); + }local_irq_enable (); }
@@ -1439,15 +1444,13 @@ static inline void tick_nohz_irq_enter(void)struct tick_sched *ts =this_cpu_ptr (&tick_cpu_sched);ktime_t now; -if (!tick_sched_flag_test (ts,TS_FLAG_STOPPED |TS_FLAG_IDLE_ACTIVE )) +tick_nohz_clock_wakeup (ts); + +if (!tick_sched_flag_test (ts,TS_FLAG_STOPPED ))return ; now =ktime_get (); - -if (tick_sched_flag_test (ts,TS_FLAG_IDLE_ACTIVE )) { -tick_nohz_clock_wakeup (ts); -kcpustat_irq_enter (now); - } +kcpustat_irq_enter (now);/* * If all CPUs are idle we may need to update a stale jiffies value.
@@ -1456,8 +1459,7 @@ static inline void tick_nohz_irq_enter(void)* rare case (typically stop machine). So we must make sure we have a * last resort. */ -if (tick_sched_flag_test (ts,TS_FLAG_STOPPED )) - tick_nohz_update_jiffies(now); + tick_nohz_update_jiffies(now); }#else
Frederic Weisbecker
maintainerPATCH v4 12/15 tick/sched: Consolidate idle time fetching APIs
In reply to Frederic WeisbeckerFetching the idle cputime is available through a variety of accessors all over the place depending on the different accounting flavours and needs:
- idle vtime generic accounting can be accessed by kcpustat_field(), kcpustat_cpu_fetch(), get_idle/iowait_time() and get_cpu_idle/iowait_time_us()
- dynticks-idle accounting can only be accessed by get_idle/iowait_time() or get_cpu_idle/iowait_time_us()
- CONFIG_NO_HZ_COMMON=n idle accounting can be accessed by kcpustat_field() kcpustat_cpu_fetch(), or get_idle/iowait_time() but not by get_cpu_idle/iowait_time_us()
Moreover get_idle/iowait_time() relies on get_cpu_idle/iowait_time_us() with a non-sensical conversion to microseconds and back to nanoseconds on the way.
Start consolidating the APIs with removing get_idle/iowait_time() and make kcpustat_field() and kcpustat_cpu_fetch() work for all cases.
--- fs/proc/stat.c | 40 +++--------------------- fs/proc/uptime.c | 8 ++--- include/linux/kernel_stat.h | 34 ++++++++++++++++++--- kernel/sched/cputime.c | 61 ++++++++++++++++++++++++------------- 4 files changed, 76 insertions(+), 67 deletions(-)
fs/proc/stat.c +4 -36
@@ -22,38 +22,6 @@#define arch_irq_stat () 0#endif -u64 get_idle_time (struct kernel_cpustat *kcs,int cpu) -{ -u64 idle, idle_usecs =-1ULL ; - -if (cpu_online (cpu)) - idle_usecs =get_cpu_idle_time_us (cpu,NULL ); - -if (idle_usecs ==-1ULL ) -/* !NO_HZ or cpu offline so we can rely on cpustat.idle */ - idle = kcs->cpustat[CPUTIME_IDLE ]; -else - idle = idle_usecs *NSEC_PER_USEC ; - -return idle; -} - -static u64 get_iowait_time (struct kernel_cpustat *kcs,int cpu) -{ -u64 iowait, iowait_usecs =-1ULL ; - -if (cpu_online (cpu)) - iowait_usecs =get_cpu_iowait_time_us (cpu,NULL ); - -if (iowait_usecs ==-1ULL ) -/* !NO_HZ or cpu offline so we can rely on cpustat.iowait */ - iowait = kcs->cpustat[CPUTIME_IOWAIT ]; -else - iowait = iowait_usecs *NSEC_PER_USEC ; - -return iowait; -} -static void show_irq_gap (struct seq_file *p,unsigned int gap) {static const char zeros[] =" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" ;
@@ -105,8 +73,8 @@ static int show_stat(struct seq_file *p, void *v) user += cpustat[CPUTIME_USER ]; nice += cpustat[CPUTIME_NICE ]; system += cpustat[CPUTIME_SYSTEM ]; - idle +=get_idle_time (&kcpustat, i); - iowait +=get_iowait_time (&kcpustat, i); + idle += cpustat[CPUTIME_IDLE ]; + iowait += cpustat[CPUTIME_IOWAIT ]; irq += cpustat[CPUTIME_IRQ ]; softirq += cpustat[CPUTIME_SOFTIRQ ]; steal += cpustat[CPUTIME_STEAL ];
@@ -146,8 +114,8 @@ static int show_stat(struct seq_file *p, void *v) user = cpustat[CPUTIME_USER ]; nice = cpustat[CPUTIME_NICE ]; system = cpustat[CPUTIME_SYSTEM ]; - idle =get_idle_time (&kcpustat, i); - iowait =get_iowait_time (&kcpustat, i); + idle = cpustat[CPUTIME_IDLE ]; + iowait = cpustat[CPUTIME_IOWAIT ]; irq = cpustat[CPUTIME_IRQ ]; softirq = cpustat[CPUTIME_SOFTIRQ ]; steal = cpustat[CPUTIME_STEAL ];
fs/proc/uptime.c +2 -6
@@ -18,12 +18,8 @@ static int uptime_proc_show(struct seq_file *m, void *v)int i; idle_nsec =0 ; -for_each_possible_cpu (i) { -struct kernel_cpustat kcs; - -kcpustat_cpu_fetch (&kcs, i); - idle_nsec +=get_idle_time (&kcs, i); - } +for_each_possible_cpu (i) + idle_nsec +=kcpustat_field (CPUTIME_IDLE , i);ktime_get_boottime_ts64 (&uptime);timens_add_boottime (&uptime);
include/linux/kernel_stat.h +30 -4
@@ -110,32 +110,59 @@ extern void kcpustat_dyntick_start(u64 now);extern void kcpustat_dyntick_stop (u64 now);extern void kcpustat_irq_enter (u64 now);extern void kcpustat_irq_exit (u64 now); +extern u64 kcpustat_field_idle (int cpu); +extern u64 kcpustat_field_iowait (int cpu);static inline bool kcpustat_idle_dyntick (void ) {return __this_cpu_read (kernel_cpustat.idle_dyntick); }#else +static inline u64 kcpustat_field_idle (int cpu) +{ +return kcpustat_cpu (cpu).cpustat[CPUTIME_IDLE ]; +} +static inline u64 kcpustat_field_iowait (int cpu) +{ +return kcpustat_cpu (cpu).cpustat[CPUTIME_IOWAIT ]; +} +static inline bool kcpustat_idle_dyntick (void ) {return false; }#endif /* CONFIG_NO_HZ_COMMON */ +/* Fetch cputime values when vtime is disabled on a CPU */ +static inline u64 kcpustat_field_default (enum cpu_usage_stat usage,int cpu) +{ +if (usage ==CPUTIME_IDLE ) +return kcpustat_field_idle (cpu); +if (usage ==CPUTIME_IOWAIT ) +return kcpustat_field_iowait (cpu); +return kcpustat_cpu (cpu).cpustat[usage]; +} + +static inline void kcpustat_cpu_fetch_default (struct kernel_cpustat *dst,int cpu) +{ + *dst =kcpustat_cpu (cpu); + dst->cpustat[CPUTIME_IDLE ] =kcpustat_field_idle (cpu); + dst->cpustat[CPUTIME_IOWAIT ] =kcpustat_field_iowait (cpu); +} +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern u64 kcpustat_field (enum cpu_usage_stat usage,int cpu);extern void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu);#else static inline u64 kcpustat_field (enum cpu_usage_stat usage,int cpu) { -return kcpustat_cpu (cpu).cpustat[usage]; +return kcpustat_field_default (usage, cpu); }static inline void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu) { - *dst =kcpustat_cpu (cpu); +kcpustat_cpu_fetch_default (dst, cpu); } -#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ extern void account_user_time (struct task_struct *,u64 );
@@ -145,7 +172,6 @@ extern void account_system_index_time(struct task_struct *, u64, enum cpu_usage_stat);extern void account_steal_time (u64 );extern void account_idle_time (u64 ); -extern u64 get_idle_time (struct kernel_cpustat *kcs,int cpu);#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE static inline void account_process_tick (struct task_struct *tsk,int user)
kernel/sched/cputime.c +40 -21
106 lines not on this page · raw message
Signature · 1 line
2.53.0
[tip: timers/nohz] tick/sched: Consolidate idle time fetching APIs
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 127b2eb44f36d5d7059f1af425b5800cb27440f9 Gitweb: https://git.kernel.org/tip/127b2eb44f36d5d7059f1af425b5800cb27440f9 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:44 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:26 +02:00
tick/sched: Consolidate idle time fetching APIs
Fetching the idle cputime is available through a variety of accessors all over the place depending on the different accounting flavours and needs:
- idle vtime generic accounting can be accessed by kcpustat_field(), kcpustat_cpu_fetch(), get_idle/iowait_time() and get_cpu_idle/iowait_time_us()
- dynticks-idle accounting can only be accessed by get_idle/iowait_time() or get_cpu_idle/iowait_time_us()
- CONFIG_NO_HZ_COMMON=n idle accounting can be accessed by kcpustat_field() kcpustat_cpu_fetch(), or get_idle/iowait_time() but not by get_cpu_idle/iowait_time_us()
Moreover get_idle/iowait_time() relies on get_cpu_idle/iowait_time_us() with a non-sensical conversion to microseconds and back to nanoseconds on the way.
Start consolidating the APIs with removing get_idle/iowait_time() and make kcpustat_field() and kcpustat_cpu_fetch() work for all cases.
--- fs/proc/stat.c | 40 ++---------------------- fs/proc/uptime.c | 8 +---- include/linux/kernel_stat.h | 34 +++++++++++++++++--- kernel/sched/cputime.c | 61 +++++++++++++++++++++++------------- 4 files changed, 76 insertions(+), 67 deletions(-)
fs/proc/stat.c +4 -36
@@ -22,38 +22,6 @@#define arch_irq_stat () 0#endif -u64 get_idle_time (struct kernel_cpustat *kcs,int cpu) -{ -u64 idle, idle_usecs =-1ULL ; - -if (cpu_online (cpu)) - idle_usecs =get_cpu_idle_time_us (cpu,NULL ); - -if (idle_usecs ==-1ULL ) -/* !NO_HZ or cpu offline so we can rely on cpustat.idle */ - idle = kcs->cpustat[CPUTIME_IDLE ]; -else - idle = idle_usecs *NSEC_PER_USEC ; - -return idle; -} - -static u64 get_iowait_time (struct kernel_cpustat *kcs,int cpu) -{ -u64 iowait, iowait_usecs =-1ULL ; - -if (cpu_online (cpu)) - iowait_usecs =get_cpu_iowait_time_us (cpu,NULL ); - -if (iowait_usecs ==-1ULL ) -/* !NO_HZ or cpu offline so we can rely on cpustat.iowait */ - iowait = kcs->cpustat[CPUTIME_IOWAIT ]; -else - iowait = iowait_usecs *NSEC_PER_USEC ; - -return iowait; -} -static void show_irq_gap (struct seq_file *p,unsigned int gap) {static const char zeros[] =" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0" ;
@@ -105,8 +73,8 @@ static int show_stat(struct seq_file *p, void *v) user += cpustat[CPUTIME_USER ]; nice += cpustat[CPUTIME_NICE ]; system += cpustat[CPUTIME_SYSTEM ]; - idle +=get_idle_time (&kcpustat, i); - iowait +=get_iowait_time (&kcpustat, i); + idle += cpustat[CPUTIME_IDLE ]; + iowait += cpustat[CPUTIME_IOWAIT ]; irq += cpustat[CPUTIME_IRQ ]; softirq += cpustat[CPUTIME_SOFTIRQ ]; steal += cpustat[CPUTIME_STEAL ];
@@ -146,8 +114,8 @@ static int show_stat(struct seq_file *p, void *v) user = cpustat[CPUTIME_USER ]; nice = cpustat[CPUTIME_NICE ]; system = cpustat[CPUTIME_SYSTEM ]; - idle =get_idle_time (&kcpustat, i); - iowait =get_iowait_time (&kcpustat, i); + idle = cpustat[CPUTIME_IDLE ]; + iowait = cpustat[CPUTIME_IOWAIT ]; irq = cpustat[CPUTIME_IRQ ]; softirq = cpustat[CPUTIME_SOFTIRQ ]; steal = cpustat[CPUTIME_STEAL ];
fs/proc/uptime.c +2 -6
@@ -18,12 +18,8 @@ static int uptime_proc_show(struct seq_file *m, void *v)int i; idle_nsec =0 ; -for_each_possible_cpu (i) { -struct kernel_cpustat kcs; - -kcpustat_cpu_fetch (&kcs, i); - idle_nsec +=get_idle_time (&kcs, i); - } +for_each_possible_cpu (i) + idle_nsec +=kcpustat_field (CPUTIME_IDLE , i);ktime_get_boottime_ts64 (&uptime);timens_add_boottime (&uptime);
include/linux/kernel_stat.h +30 -4
@@ -110,32 +110,59 @@ extern void kcpustat_dyntick_start(u64 now);extern void kcpustat_dyntick_stop (u64 now);extern void kcpustat_irq_enter (u64 now);extern void kcpustat_irq_exit (u64 now); +extern u64 kcpustat_field_idle (int cpu); +extern u64 kcpustat_field_iowait (int cpu);static inline bool kcpustat_idle_dyntick (void ) {return __this_cpu_read (kernel_cpustat.idle_dyntick); }#else +static inline u64 kcpustat_field_idle (int cpu) +{ +return kcpustat_cpu (cpu).cpustat[CPUTIME_IDLE ]; +} +static inline u64 kcpustat_field_iowait (int cpu) +{ +return kcpustat_cpu (cpu).cpustat[CPUTIME_IOWAIT ]; +} +static inline bool kcpustat_idle_dyntick (void ) {return false; }#endif /* CONFIG_NO_HZ_COMMON */ +/* Fetch cputime values when vtime is disabled on a CPU */ +static inline u64 kcpustat_field_default (enum cpu_usage_stat usage,int cpu) +{ +if (usage ==CPUTIME_IDLE ) +return kcpustat_field_idle (cpu); +if (usage ==CPUTIME_IOWAIT ) +return kcpustat_field_iowait (cpu); +return kcpustat_cpu (cpu).cpustat[usage]; +} + +static inline void kcpustat_cpu_fetch_default (struct kernel_cpustat *dst,int cpu) +{ + *dst =kcpustat_cpu (cpu); + dst->cpustat[CPUTIME_IDLE ] =kcpustat_field_idle (cpu); + dst->cpustat[CPUTIME_IOWAIT ] =kcpustat_field_iowait (cpu); +} +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern u64 kcpustat_field (enum cpu_usage_stat usage,int cpu);extern void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu);#else static inline u64 kcpustat_field (enum cpu_usage_stat usage,int cpu) { -return kcpustat_cpu (cpu).cpustat[usage]; +return kcpustat_field_default (usage, cpu); }static inline void kcpustat_cpu_fetch (struct kernel_cpustat *dst,int cpu) { - *dst =kcpustat_cpu (cpu); +kcpustat_cpu_fetch_default (dst, cpu); } -#endif /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ extern void account_user_time (struct task_struct *,u64 );
@@ -145,7 +172,6 @@ extern void account_system_index_time(struct task_struct *, u64, enum cpu_usage_stat);extern void account_steal_time (u64 );extern void account_idle_time (u64 ); -extern u64 get_idle_time (struct kernel_cpustat *kcs,int cpu);#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE static inline void account_process_tick (struct task_struct *tsk,int user)
kernel/sched/cputime.c +40 -21
106 lines not on this page · raw message
Frederic Weisbecker
maintainerPATCH v4 13/15 sched/cputime: Provide get_cpu_[idle|iowait]_time_us() off-case
In reply to Frederic WeisbeckerThe last reason why get_cpu_idle/iowait_time_us() may return -1 now is if the config doesn't support nohz.
The ad-hoc replacement solution by cpufreq is to compute jiffies minus the whole busy cputime. Although the intention should provide a coherent low resolution estimation of the idle and iowait time, the implementation is buggy because jiffies don't start at 0.
Just provide instead a real get_cpu_[idle|iowait]_time_us() offcase.
--- drivers/cpufreq/cpufreq.c | 29 +---------------------------- include/linux/kernel_stat.h | 3 +++ include/linux/tick.h | 4 ---- kernel/sched/cputime.c | 12 +++++++++--- 4 files changed, 13 insertions(+), 35 deletions(-)
drivers/cpufreq/cpufreq.c +1 -28
@@ -130,38 +130,11 @@ struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy) }EXPORT_SYMBOL_GPL (get_governor_parent_kobj); -static inline u64 get_cpu_idle_time_jiffy (unsigned int cpu,u64 *wall) -{ -struct kernel_cpustat kcpustat; -u64 cur_wall_time; -u64 idle_time; -u64 busy_time; - - cur_wall_time =jiffies64_to_nsecs (get_jiffies_64 ()); - -kcpustat_cpu_fetch (&kcpustat, cpu); - - busy_time = kcpustat.cpustat[CPUTIME_USER ]; - busy_time += kcpustat.cpustat[CPUTIME_SYSTEM ]; - busy_time += kcpustat.cpustat[CPUTIME_IRQ ]; - busy_time += kcpustat.cpustat[CPUTIME_SOFTIRQ ]; - busy_time += kcpustat.cpustat[CPUTIME_STEAL ]; - busy_time += kcpustat.cpustat[CPUTIME_NICE ]; - - idle_time = cur_wall_time - busy_time; -if (wall) - *wall =div_u64 (cur_wall_time,NSEC_PER_USEC ); - -return div_u64 (idle_time,NSEC_PER_USEC ); -} -u64 get_cpu_idle_time (unsigned int cpu,u64 *wall,int io_busy) {u64 idle_time =get_cpu_idle_time_us (cpu, io_busy ? wall :NULL ); -if (idle_time ==-1ULL ) -return get_cpu_idle_time_jiffy (cpu, wall); -else if (!io_busy) +if (!io_busy) idle_time +=get_cpu_iowait_time_us (cpu, wall);return idle_time;
include/linux/kernel_stat.h +3 -0
@@ -133,6 +133,9 @@ static inline bool kcpustat_idle_dyntick(void) }#endif /* CONFIG_NO_HZ_COMMON */ +extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time); +extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time); +/* Fetch cputime values when vtime is disabled on a CPU */ static inline u64 kcpustat_field_default (enum cpu_usage_stat usage,int cpu) {
include/linux/tick.h +0 -4
@@ -139,8 +139,6 @@ extern bool tick_nohz_idle_got_tick(void);extern ktime_t tick_nohz_get_next_hrtimer (void );extern ktime_t tick_nohz_get_sleep_length (ktime_t *delta_next);extern unsigned long tick_nohz_get_idle_calls_cpu (int cpu); -extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time); -extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time);#else /* !CONFIG_NO_HZ_COMMON */ #define tick_nohz_enabled (0)static inline bool tick_nohz_is_active (void ) {return false; }
@@ -162,8 +160,6 @@ static inline ktime_t tick_nohz_get_sleep_length(ktime_t *delta_next) *delta_next =TICK_NSEC ; return *delta_next; } -static inline u64 get_cpu_idle_time_us (int cpu,u64 *unused) {return -1 ; } -static inline u64 get_cpu_iowait_time_us (int cpu,u64 *unused) {return -1 ; }#endif /* !CONFIG_NO_HZ_COMMON */ /*
kernel/sched/cputime.c +9 -3
@@ -522,6 +522,13 @@ u64 kcpustat_field_iowait(int cpu) nr_iowait_cpu(cpu), ktime_get()); }EXPORT_SYMBOL_GPL (kcpustat_field_iowait); +#else +static u64 kcpustat_field_dyntick (int cpu,enum cpu_usage_stat idx, +bool compute_delta,ktime_t now) +{ +return kcpustat_cpu (cpu).cpustat[idx]; +} +#endif /* CONFIG_NO_HZ_COMMON */ static u64 get_cpu_sleep_time_us (int cpu,enum cpu_usage_stat idx,bool compute_delta,u64 *last_update_time)
@@ -557,7 +564,7 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx,* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if generic vtime is enabled, else total idle time of the @cpu +* Return: total idle time of the @cpu */ u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time) {
@@ -581,7 +588,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if generic vtime is enabled, else total iowait time of @cpu +* Return: total iowait time of @cpu */ u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time) {
@@ -589,7 +596,6 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) nr_iowait_cpu(cpu), last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_iowait_time_us); -#endif /* CONFIG_NO_HZ_COMMON */ /* * Use precise platform statistics if available:
Signature · 1 line
2.53.0
[tip: timers/nohz] sched/cputime: Provide get_cpu_[idle|iowait]_time_us() off-case
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 3b45b4f188f3a0ebd16ab71efd2ffcc7a16ad861 Gitweb: https://git.kernel.org/tip/3b45b4f188f3a0ebd16ab71efd2ffcc7a16ad861 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:45 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:26 +02:00
sched/cputime: Provide get_cpu_[idle|iowait]_time_us() off-case
The last reason why get_cpu_idle/iowait_time_us() may return -1 now is if the config doesn't support nohz.
The ad-hoc replacement solution by cpufreq is to compute jiffies minus the whole busy cputime. Although the intention should provide a coherent low resolution estimation of the idle and iowait time, the implementation is buggy because jiffies don't start at 0.
Just provide instead a real get_cpu_[idle|iowait]_time_us() offcase.
--- drivers/cpufreq/cpufreq.c | 29 +---------------------------- include/linux/kernel_stat.h | 3 +++ include/linux/tick.h | 4 ---- kernel/sched/cputime.c | 12 +++++++++--- 4 files changed, 13 insertions(+), 35 deletions(-)
drivers/cpufreq/cpufreq.c +1 -28
@@ -130,38 +130,11 @@ struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy) }EXPORT_SYMBOL_GPL (get_governor_parent_kobj); -static inline u64 get_cpu_idle_time_jiffy (unsigned int cpu,u64 *wall) -{ -struct kernel_cpustat kcpustat; -u64 cur_wall_time; -u64 idle_time; -u64 busy_time; - - cur_wall_time =jiffies64_to_nsecs (get_jiffies_64 ()); - -kcpustat_cpu_fetch (&kcpustat, cpu); - - busy_time = kcpustat.cpustat[CPUTIME_USER ]; - busy_time += kcpustat.cpustat[CPUTIME_SYSTEM ]; - busy_time += kcpustat.cpustat[CPUTIME_IRQ ]; - busy_time += kcpustat.cpustat[CPUTIME_SOFTIRQ ]; - busy_time += kcpustat.cpustat[CPUTIME_STEAL ]; - busy_time += kcpustat.cpustat[CPUTIME_NICE ]; - - idle_time = cur_wall_time - busy_time; -if (wall) - *wall =div_u64 (cur_wall_time,NSEC_PER_USEC ); - -return div_u64 (idle_time,NSEC_PER_USEC ); -} -u64 get_cpu_idle_time (unsigned int cpu,u64 *wall,int io_busy) {u64 idle_time =get_cpu_idle_time_us (cpu, io_busy ? wall :NULL ); -if (idle_time ==-1ULL ) -return get_cpu_idle_time_jiffy (cpu, wall); -else if (!io_busy) +if (!io_busy) idle_time +=get_cpu_iowait_time_us (cpu, wall);return idle_time;
include/linux/kernel_stat.h +3 -0
@@ -133,6 +133,9 @@ static inline bool kcpustat_idle_dyntick(void) }#endif /* CONFIG_NO_HZ_COMMON */ +extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time); +extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time); +/* Fetch cputime values when vtime is disabled on a CPU */ static inline u64 kcpustat_field_default (enum cpu_usage_stat usage,int cpu) {
include/linux/tick.h +0 -4
@@ -139,8 +139,6 @@ extern bool tick_nohz_idle_got_tick(void);extern ktime_t tick_nohz_get_next_hrtimer (void );extern ktime_t tick_nohz_get_sleep_length (ktime_t *delta_next);extern unsigned long tick_nohz_get_idle_calls_cpu (int cpu); -extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time); -extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time);#else /* !CONFIG_NO_HZ_COMMON */ #define tick_nohz_enabled (0)static inline bool tick_nohz_is_active (void ) {return false; }
@@ -162,8 +160,6 @@ static inline ktime_t tick_nohz_get_sleep_length(ktime_t *delta_next) *delta_next =TICK_NSEC ; return *delta_next; } -static inline u64 get_cpu_idle_time_us (int cpu,u64 *unused) {return -1 ; } -static inline u64 get_cpu_iowait_time_us (int cpu,u64 *unused) {return -1 ; }#endif /* !CONFIG_NO_HZ_COMMON */ /*
kernel/sched/cputime.c +9 -3
@@ -522,6 +522,13 @@ u64 kcpustat_field_iowait(int cpu) nr_iowait_cpu(cpu), ktime_get()); }EXPORT_SYMBOL_GPL (kcpustat_field_iowait); +#else +static u64 kcpustat_field_dyntick (int cpu,enum cpu_usage_stat idx, +bool compute_delta,ktime_t now) +{ +return kcpustat_cpu (cpu).cpustat[idx]; +} +#endif /* CONFIG_NO_HZ_COMMON */ static u64 get_cpu_sleep_time_us (int cpu,enum cpu_usage_stat idx,bool compute_delta,u64 *last_update_time)
@@ -557,7 +564,7 @@ static u64 get_cpu_sleep_time_us(int cpu, enum cpu_usage_stat idx,* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if generic vtime is enabled, else total idle time of the @cpu +* Return: total idle time of the @cpu */ u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time) {
@@ -581,7 +588,7 @@ EXPORT_SYMBOL_GPL(get_cpu_idle_time_us);* This time is measured via accounting rather than sampling, * and is as accurate as ktime_get() is. * -* Return: -1 if generic vtime is enabled, else total iowait time of @cpu +* Return: total iowait time of @cpu */ u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time) {
@@ -589,7 +596,6 @@ u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time) nr_iowait_cpu(cpu), last_update_time); }EXPORT_SYMBOL_GPL (get_cpu_iowait_time_us); -#endif /* CONFIG_NO_HZ_COMMON */ /* * Use precise platform statistics if available:
Frederic Weisbecker
maintainerPATCH v4 14/15 sched/cputime: Handle idle irqtime gracefully
In reply to Frederic WeisbeckerThe dyntick-idle cputime accounting always assumes that IRQ time accounting is enabled and consequently stops elapsing the idle time during dyntick-idle IRQs.
This doesn't mix up well with disabled IRQ time accounting because then idle IRQs become a cputime blind-spot. Also this feature is disabled on most configurations and the overhead of pausing dyntick-idle accounting while in idle IRQs could then be avoided.
Fix the situation with conditionally pausing dyntick-idle accounting during idle IRQs only iff either native vtime (which does IRQ time accounting) or generic IRQ time accounting are enabled.
Also make sure that the accumulated IRQ time is not accidentally substracted from later accounting.
--- kernel/sched/cputime.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
kernel/sched/cputime.c +17 -3
@@ -47,7 +47,8 @@ static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,u64_stats_update_begin (&irqtime->sync); cpustat[idx] += delta; irqtime->total += delta; - irqtime->tick_delta += delta; +if (!kcpustat_idle_dyntick ()) + irqtime->tick_delta += delta;u64_stats_update_end (&irqtime->sync); }
@@ -444,6 +445,10 @@ static void kcpustat_idle_stop(struct kernel_cpustat *kc, u64 now)static void kcpustat_idle_start (struct kernel_cpustat *kc,u64 now) { +/* Irqtime accounting might have been enabled in the middle of the IRQ */ +if (kc->idle_elapse) +return ; +write_seqcount_begin (&kc->idle_sleeptime_seq); kc->idle_entrytime = now; kc->idle_elapse = true;
@@ -478,7 +483,8 @@ void kcpustat_irq_enter(u64 now) {struct kernel_cpustat *kc = kcpustat_this_cpu; -if (!vtime_generic_enabled_this_cpu ()) +if (!vtime_generic_enabled_this_cpu () && + (irqtime_enabled () ||vtime_accounting_enabled_this_cpu ()))kcpustat_idle_stop (kc, now); }
@@ -486,7 +492,15 @@ void kcpustat_irq_exit(u64 now) {struct kernel_cpustat *kc = kcpustat_this_cpu; -if (!vtime_generic_enabled_this_cpu ()) +/* +* Generic vtime already does its own idle accounting. +* But irqtime accounting or arch vtime which also accounts IRQs +* need to pause nohz accounting. Resume nohz accounting as long +* as the irqtime config is enabled to handle case where irqtime +* accounting got runtime disabled in the middle of an IRQ. +*/ +if (!vtime_generic_enabled_this_cpu () && + (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu ()))kcpustat_idle_start (kc, now); }
Signature · 1 line
2.53.0
[tip: timers/nohz] sched/cputime: Handle idle irqtime gracefully
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 7198e3927a14535475a24cce559f41f97e6c0b66 Gitweb: https://git.kernel.org/tip/7198e3927a14535475a24cce559f41f97e6c0b66 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:46 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:26 +02:00
sched/cputime: Handle idle irqtime gracefully
The dyntick-idle cputime accounting always assumes that interrupt time accounting is enabled and consequently stops elapsing the idle time during dyntick-idle interrupts.
This doesn't mix up well with disabled interrupt time accounting because then idle interrupts become a cputime blind-spot. Also this feature is disabled on most configurations and the overhead of pausing dyntick-idle accounting while in idle interrupts could then be avoided.
Fix the situation with conditionally pausing dyntick-idle accounting during idle interrupts only iff either native vtime (which does interrupt time accounting) or generic interrupt time accounting are enabled.
Also make sure that the accumulated interrupt time is not accidentally substracted from later accounting.
--- kernel/sched/cputime.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
kernel/sched/cputime.c +17 -3
@@ -47,7 +47,8 @@ static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,u64_stats_update_begin (&irqtime->sync); cpustat[idx] += delta; irqtime->total += delta; - irqtime->tick_delta += delta; +if (!kcpustat_idle_dyntick ()) + irqtime->tick_delta += delta;u64_stats_update_end (&irqtime->sync); }
@@ -444,6 +445,10 @@ static void kcpustat_idle_stop(struct kernel_cpustat *kc, u64 now)static void kcpustat_idle_start (struct kernel_cpustat *kc,u64 now) { +/* Irqtime accounting might have been enabled in the middle of the IRQ */ +if (kc->idle_elapse) +return ; +write_seqcount_begin (&kc->idle_sleeptime_seq); kc->idle_entrytime = now; kc->idle_elapse = true;
@@ -478,7 +483,8 @@ void kcpustat_irq_enter(u64 now) {struct kernel_cpustat *kc = kcpustat_this_cpu; -if (!vtime_generic_enabled_this_cpu ()) +if (!vtime_generic_enabled_this_cpu () && + (irqtime_enabled () ||vtime_accounting_enabled_this_cpu ()))kcpustat_idle_stop (kc, now); }
@@ -486,7 +492,15 @@ void kcpustat_irq_exit(u64 now) {struct kernel_cpustat *kc = kcpustat_this_cpu; -if (!vtime_generic_enabled_this_cpu ()) +/* +* Generic vtime already does its own idle accounting. +* But irqtime accounting or arch vtime which also accounts IRQs +* need to pause nohz accounting. Resume nohz accounting as long +* as the irqtime config is enabled to handle case where irqtime +* accounting got runtime disabled in the middle of an IRQ. +*/ +if (!vtime_generic_enabled_this_cpu () && + (IS_ENABLED (CONFIG_IRQ_TIME_ACCOUNTING ) ||vtime_accounting_enabled_this_cpu ()))kcpustat_idle_start (kc, now); }
Frederic Weisbecker
maintainerPATCH v4 15/15 sched/cputime: Handle dyntick-idle steal time correctly
In reply to Frederic WeisbeckerThe dyntick-idle steal time is currently accounted when the tick restarts but the stolen idle time is not subtracted from the idle time that was already accounted. This is to avoid observing the idle time going backward as the dyntick-idle cputime accessors can't reliably know in advance the stolen idle time.
In order to maintain a forward progressing idle cputime while subtracting idle steal time from it, keep track of the previously accounted idle stolen time and substract it from _later_ idle cputime accounting.
--- include/linux/kernel_stat.h | 1 + kernel/sched/cputime.c | 28 +++++++++++++++++++++++----- 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/kernel_stat.h +1 -0
@@ -39,6 +39,7 @@ struct kernel_cpustat {bool idle_elapse;seqcount_t idle_sleeptime_seq;u64 idle_entrytime; +u64 idle_stealtime[2 ];#endif u64 cpustat[NR_STATS ]; };
kernel/sched/cputime.c +23 -5
@@ -425,19 +425,32 @@ static inline void irqtime_account_process_tick(struct task_struct *p, int user_static void kcpustat_idle_stop (struct kernel_cpustat *kc,u64 now) {u64 *cpustat = kc->cpustat; -u64 delta; +u64 delta, steal, steal_delta; +int iowait;if (!kc->idle_elapse)return ; + iowait =nr_iowait_cpu (smp_processor_id ()) >0 ; delta = now - kc->idle_entrytime; + steal =steal_account_process_time (delta); +/* +* Record the idle time after substracting the steal time from +* previous update sequence. Don't substract the steal time from +* the current update sequence to avoid readers moving backward. +*/ write_seqcount_begin (&kc->idle_sleeptime_seq); -if (nr_iowait_cpu (smp_processor_id ()) >0 ) + steal_delta =min_t (u64, kc->idle_stealtime[iowait], delta); + delta -= steal_delta; + kc->idle_stealtime[iowait] -= steal_delta; + +if (iowait) cpustat[CPUTIME_IOWAIT ] += delta;else cpustat[CPUTIME_IDLE ] += delta; + kc->idle_stealtime[iowait] += steal; kc->idle_entrytime = now; kc->idle_elapse = false;write_seqcount_end (&kc->idle_sleeptime_seq);
@@ -464,7 +477,6 @@ void kcpustat_dyntick_stop(u64 now)kcpustat_idle_stop (kc, now); kc->idle_dyntick = false;vtime_dyntick_stop (); -steal_account_process_time (ULONG_MAX ); } }
@@ -508,6 +520,7 @@ static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx, bool compute_delta, u64 now) {struct kernel_cpustat *kc = &kcpustat_cpu (cpu); +int iowait = idx ==CPUTIME_IOWAIT ;u64 *cpustat = kc->cpustat;unsigned int seq;u64 idle;
@@ -516,8 +529,13 @@ static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx, seq =read_seqcount_begin (&kc->idle_sleeptime_seq); idle = cpustat[idx]; -if (kc->idle_elapse && compute_delta && now > kc->idle_entrytime) - idle += (now - kc->idle_entrytime); + +if (kc->idle_elapse && compute_delta && now > kc->idle_entrytime) { +u64 delta = now - kc->idle_entrytime; + + delta -=min_t (u64, kc->idle_stealtime[iowait], delta); + idle += delta; + } }while (read_seqcount_retry (&kc->idle_sleeptime_seq, seq));return idle;
Signature · 1 line
2.53.0
[tip: timers/nohz] sched/cputime: Handle dyntick-idle steal time correctly
The following commit has been merged into the timers/nohz branch of tip:
The following commit has been merged into the timers/nohz branch of tip:
Commit-ID: 6199f9999a9b62b2b84a1bf5b52a9fd0bb8de5af Gitweb: https://git.kernel.org/tip/6199f9999a9b62b2b84a1bf5b52a9fd0bb8de5af Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 08 May 2026 15:16:47 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 02 Jun 2026 21:27:26 +02:00
sched/cputime: Handle dyntick-idle steal time correctly
The dyntick-idle steal time is currently accounted when the tick restarts but the stolen idle time is not subtracted from the idle time that was already accounted. This is to avoid observing the idle time going backward as the dyntick-idle cputime accessors can't reliably know in advance the stolen idle time.
In order to maintain a forward progressing idle cputime while subtracting idle steal time from it, keep track of the previously accounted idle stolen time and substract it from _later_ idle cputime accounting.
--- include/linux/kernel_stat.h | 1 + kernel/sched/cputime.c | 28 +++++++++++++++++++++++----- 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/kernel_stat.h +1 -0
@@ -39,6 +39,7 @@ struct kernel_cpustat {bool idle_elapse;seqcount_t idle_sleeptime_seq;u64 idle_entrytime; +u64 idle_stealtime[2 ];#endif u64 cpustat[NR_STATS ]; };
kernel/sched/cputime.c +23 -5
@@ -425,19 +425,32 @@ static inline void irqtime_account_process_tick(struct task_struct *p, int user_static void kcpustat_idle_stop (struct kernel_cpustat *kc,u64 now) {u64 *cpustat = kc->cpustat; -u64 delta; +u64 delta, steal, steal_delta; +int iowait;if (!kc->idle_elapse)return ; + iowait =nr_iowait_cpu (smp_processor_id ()) >0 ; delta = now - kc->idle_entrytime; + steal =steal_account_process_time (delta); +/* +* Record the idle time after substracting the steal time from +* previous update sequence. Don't substract the steal time from +* the current update sequence to avoid readers moving backward. +*/ write_seqcount_begin (&kc->idle_sleeptime_seq); -if (nr_iowait_cpu (smp_processor_id ()) >0 ) + steal_delta =min_t (u64, kc->idle_stealtime[iowait], delta); + delta -= steal_delta; + kc->idle_stealtime[iowait] -= steal_delta; + +if (iowait) cpustat[CPUTIME_IOWAIT ] += delta;else cpustat[CPUTIME_IDLE ] += delta; + kc->idle_stealtime[iowait] += steal; kc->idle_entrytime = now; kc->idle_elapse = false;write_seqcount_end (&kc->idle_sleeptime_seq);
@@ -464,7 +477,6 @@ void kcpustat_dyntick_stop(u64 now)kcpustat_idle_stop (kc, now); kc->idle_dyntick = false;vtime_dyntick_stop (); -steal_account_process_time (ULONG_MAX ); } }
@@ -508,6 +520,7 @@ static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx, bool compute_delta, u64 now) {struct kernel_cpustat *kc = &kcpustat_cpu (cpu); +int iowait = idx ==CPUTIME_IOWAIT ;u64 *cpustat = kc->cpustat;unsigned int seq;u64 idle;
@@ -516,8 +529,13 @@ static u64 kcpustat_field_dyntick(int cpu, enum cpu_usage_stat idx, seq =read_seqcount_begin (&kc->idle_sleeptime_seq); idle = cpustat[idx]; -if (kc->idle_elapse && compute_delta && now > kc->idle_entrytime) - idle += (now - kc->idle_entrytime); + +if (kc->idle_elapse && compute_delta && now > kc->idle_entrytime) { +u64 delta = now - kc->idle_entrytime; + + delta -=min_t (u64, kc->idle_stealtime[iowait], delta); + idle += delta; + } }while (read_seqcount_retry (&kc->idle_sleeptime_seq, seq));return idle;
Frederic Weisbecker
maintainerHi,
I don't see any further concern. What should we do with this? It could either go through the scheduler tree or the timer tree.
Thanks.
Quoted 161 lines · Frederic Weisbecker
Hi,
After the issue reported here:
https://lore.kernel.org/all/[email protected]/It occurs that the idle cputime accounting is a big mess that accumulates within two concurrent statistics, each having their own shortcomings:
* The accounting for online CPUs which is based on the delta between
tick_nohz_start_idle()andtick_nohz_stop_idle().Pros:
- Works when the tick is off- Has nsecs granularity
Cons:
- Account idle steal time but doesn't substract it from idle cputime.- Assumes
CONFIG_IRQ_TIME_ACCOUNTINGby not accounting IRQs but the IRQ time is simply ignored whenCONFIG_IRQ_TIME_ACCOUNTING=n- The windows between 1) idle task scheduling and the first call to
tick_nohz_start_idle()and 2) idle task between the lasttick_nohz_stop_idle()and the rest of the idle time are blindspots wrt. cputime accounting (though mostly insignificant amount)- Relies on private fields outside of kernel stats, with specific accessors.
* The accounting for offline CPUs which is based on ticks and the jiffies delta during which the tick was stopped.
Pros:
- Handles steal time correctly- Handle
CONFIG_IRQ_TIME_ACCOUNTING=yand
CONFIG_IRQ_TIME_ACCOUNTING=ncorrectly.- Handles the whole idle task
- Accounts directly to kernel stats, without midlayer accumulator.
Cons:
- Doesn't elapse when the tick is off, which doesn't make it suitable for online CPUs.- Has
TICK_NSECgranularity (jiffies)- Needs to track the dyntick-idle ticks that were accounted and substract them from the total jiffies time spent while the tick was stopped. This is an ugly workaround.
Having two different accounting for a single context is not the only
problem: since those accountings are of different natures, it is possible to observe the global idle time going backward after a CPU goes offline, as reported by Xin Zhao.Clean up the situation with introducing a hybrid approach that stays coherent, fixes the backward jumps and works for both online and offline CPUs:
* Tick based or native vtime accounting operate before the tick is stopped and resumes once the tick is restarted.
* When the idle loop starts, switch to dynticks-idle accounting as is done currently, except that the statistics accumulate directly to the relevant kernel stat fields.
* Private dyntick cputime accounting fields are removed.
* Works on both online and offline case.
* Move most of the relevant code to the common sched/cputime subsystem
* Handle
CONFIG_IRQ_TIME_ACCOUNTING=ncorrectly such that the dynticks-idle accounting still elapses while on IRQs.* Correctly substract idle steal cputime from idle time
Changes since v3 (among which a lot of relevant reviews from Sashiko):
- Add new tags
- Rebase on latest -rc1
- Add "tick/sched: Fix TOCTOU in nohz idle time fetch" (Sashiko)
- Fix buggy state refetch in
kcpustat_cpu_fetch_vtime()(Sashiko)- Fix build issue on powerpc (Christophe Leroy)
- Fix s390 lost steal time occuring on idle IRQs (call
vtime_flush()onvtime_account_hardirq()andvtime_account_softirq()) (Sashiko)- Fix build issue on s390
- Fix uninitialized
idle_sleeptime_seq(Sashiko)- Fix irqtime being disabled or enabled in the middle of an idle IRQ (Sashiko)
- Fix tick restart and then restop in the same idle loop (Sashiko)
- Fix "sched/cputime: Handle idle irqtime gracefully" changelog (Sashiko)
- Fix idle steal time substracted from the wrong index between idle and iowait kcpustat. (Sashiko)
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
timers/core-v4HEAD:
e64ba052ce04e363ff76d3cb8bedc5f812188acb
Thanks,
Frederic
---Frederic Weisbecker (15):
tick/sched: Fix TOCTOU in nohz idle time fetch
sched/idle: Handle offlining first in idle loop
sched/cputime: Remove superfluous and error pronekcpustat_field()parameter sched/cputime: Correctly support generic vtime idle time powerpc/time: Prepare to stop elapsing in dynticks-idle s390/time: Prepare to stop elapsing in dynticks-idle tick/sched: Unify idle cputime accounting
tick/sched: Remove nohz disabled special case in cputime fetch tick/sched: Move dyntick-idle cputime accounting to cputime code tick/sched: Remove unused fields
tick/sched: Account tickless idle cputime only when tick is stopped tick/sched: Consolidate idle time fetching APIs
sched/cputime: Provide get_cpu_[idle|iowait]_time_us() off-case sched/cputime: Handle idle irqtime gracefully
sched/cputime: Handle dyntick-idle steal time correctlyarch/powerpc/kernel/time.c | 41 +++++ arch/s390/include/asm/idle.h | 2 + arch/s390/kernel/idle.c | 5 +- arch/s390/kernel/vtime.c | 75 ++++++++- drivers/cpufreq/cpufreq.c | 29 +--- drivers/cpufreq/cpufreq_governor.c | 6 +- drivers/macintosh/rack-meter.c | 2 +- fs/proc/stat.c | 40 +---- fs/proc/uptime.c | 8 +- include/linux/kernel_stat.h | 76 +++++++-- include/linux/tick.h | 4 - include/linux/vtime.h | 22 ++- kernel/rcu/tree.c | 9 +- kernel/rcu/tree_stall.h | 7 +- kernel/sched/core.c | 6 +- kernel/sched/cputime.c | 308 +++++++++++++++++++++++++++++++------ kernel/sched/idle.c | 13 +- kernel/time/tick-sched.c | 212 ++++++------------------- kernel/time/tick-sched.h | 12 -- kernel/time/timer_list.c | 6 +- scripts/gdb/linux/timerlist.py | 4 - 21 files changed, 529 insertions(+), 358 deletions(-)
Signature · 2 lines
Frederic Weisbecker SUSE Labs