sched/nohz: Make the rest of kcpustat vtime aware
Subsystems (7)
cover sched/nohz: Make the rest of kcpustat vtime aware
So we fixed CPUTIME_SYSTEM on nohz_full, now is the turn for CPUTIME_USER, CPUTIME_NICE, CPUTIME_GUEST and CPUTIME_GUEST_NICE.
The tricky piece was to handle "nice-ness" correctly. I think I addressed most reviews (one year ago now) from peterz
(https://lore.kernel.org/lkml/[email protected]/)
Among which:
* Merge patches that didn't make sense together
* Clarify the changelog (I hope) and add comments
* Also hook nice updates on __setscheduler_params()
* Uninline a big function
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/kcpustat-v3
HEAD: cfd377b0f261f1107ba5f3d2cf68a8ac5f359943
Thanks,
Frederic
---
Frederic Weisbecker (9):
sched/cputime: Allow to pass cputime index on user/guest accounting sched/cputime: Standardize the kcpustat index based accounting functions sched/vtime: Handle nice updates under vtime
sched/cputime: Support other fields on kcpustat_field() sched/vtime: Bring all-in-one kcpustat accessor for vtime fields
procfs: Use all-in-one vtime aware kcpustat accessor
cpufreq: Use vtime aware kcpustat accessors for user time
leds: Use all-in-one vtime aware kcpustat accessor
rackmeter: Use vtime aware kcpustat accessor
arch/ia64/kernel/time.c | 6 +- arch/powerpc/kernel/time.c | 6 +- arch/s390/kernel/vtime.c | 2 +- drivers/cpufreq/cpufreq.c | 13 +- drivers/cpufreq/cpufreq_governor.c | 6 +- drivers/leds/trigger/ledtrig-activity.c | 9 +- drivers/macintosh/rack-meter.c | 7 +- fs/proc/stat.c | 20 +- include/linux/kernel_stat.h | 25 ++- include/linux/sched.h | 1 + include/linux/vtime.h | 2 + kernel/sched/core.c | 16 +- kernel/sched/cputime.c | 378 ++++++++++++++++++++++++++------ kernel/sched/sched.h | 18 ++ 14 files changed, 412 insertions(+), 97 deletions(-)
PATCH 1/9 sched/cputime: Allow to pass cputime index on user/guest accounting
When we account user or guest cputime, we decide to add the delta either to the nice fields or the normal fields of kcpustat and this depends on the nice value for the task passed in parameter.
Since we are going to track the nice-ness from vtime instead, we'll need to be able to pass custom kcpustat destination index fields to the accounting functions.
--- kernel/sched/cputime.c | 50 +++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 18 deletions(-)
kernel/sched/cputime.c +32 -18
@@ -109,6 +109,20 @@ static inline void task_group_account_field(struct task_struct *p, int index,cgroup_account_cputime_field (p, index, tmp); } +static void account_user_time_index (struct task_struct *p, +u64 cputime,enum cpu_usage_stat index) +{ +/* Add user time to process. */ + p->utime += cputime; +account_group_user_time (p, cputime); + +/* Add user time to cpustat. */ +task_group_account_field (p, index, cputime); + +/* Account for user time used */ +acct_account_cputime (p); +} +/* * Account user CPU time to a process. * @p: the process that the CPU time gets accounted to
@@ -116,27 +130,14 @@ static inline void task_group_account_field(struct task_struct *p, int index,*/ void account_user_time (struct task_struct *p,u64 cputime) { -int index; - -/* Add user time to process. */ - p->utime += cputime; -account_group_user_time (p, cputime); +enum cpu_usage_stat index; index = (task_nice (p) >0 ) ?CPUTIME_NICE :CPUTIME_USER ; - -/* Add user time to cpustat. */ -task_group_account_field (p, index, cputime); - -/* Account for user time used */ -acct_account_cputime (p); +account_user_time_index (p, cputime, index); } -/* -* Account guest CPU time to a process. -* @p: the process that the CPU time gets accounted to -* @cputime: the CPU time spent in virtual machine since the last update -*/ -void account_guest_time (struct task_struct *p,u64 cputime) +static void account_guest_time_index (struct task_struct *p, +u64 cputime,enum cpu_usage_stat index) {u64 *cpustat = kcpustat_this_cpu->cpustat;
@@ -146,7 +147,7 @@ void account_guest_time(struct task_struct *p, u64 cputime) p->gtime += cputime;/* Add guest time to cpustat. */ -if (task_nice (p) >0 ) { +if (index ==CPUTIME_GUEST_NICE ) { cpustat[CPUTIME_NICE ] += cputime; cpustat[CPUTIME_GUEST_NICE ] += cputime; }else {
@@ -155,6 +156,19 @@ void account_guest_time(struct task_struct *p, u64 cputime) } } +/* +* Account guest CPU time to a process. +* @p: the process that the CPU time gets accounted to +* @cputime: the CPU time spent in virtual machine since the last update +*/ +void account_guest_time (struct task_struct *p,u64 cputime) +{ +enum cpu_usage_stat index; + + index = (task_nice (p) >0 ) ?CPUTIME_GUEST_NICE :CPUTIME_GUEST ; +account_guest_time_index (p, cputime, index); +} +/* * Account system CPU time to a process and desired cpustat field * @p: the process that the CPU time gets accounted to
Signature · 1 line
2.23.0
PATCH 2/9 sched/cputime: Standardize the kcpustat index based accounting functions
In reply to Frederic WeisbeckerSanitize a bit the functions that do cputime accounting with custom kcpustat indexes:
* Rename account_system_index_time() to account_system_time_index() to comply with account_guest/user_time_index()
* Use proper enum cpu_usage_stat type in account_system_time()
* Reorder task_group_account_field() parameters to comply with those of account_*_time_index().
* Rename task_group_account_field()'s tmp parameter to cputime
* Precise the type of index in task_group_account_field(): enum cpu_usage_stat
--- arch/ia64/kernel/time.c | 6 +++--- arch/powerpc/kernel/time.c | 6 +++--- arch/s390/kernel/vtime.c | 2 +- include/linux/kernel_stat.h | 2 +- kernel/sched/cputime.c | 22 +++++++++++----------- 5 files changed, 19 insertions(+), 19 deletions(-)
arch/ia64/kernel/time.c +3 -3
@@ -79,17 +79,17 @@ void vtime_flush(struct task_struct *tsk)if (ti->stime) { delta =cycle_to_nsec (ti->stime); -account_system_index_time (tsk, delta,CPUTIME_SYSTEM ); +account_system_time_index (tsk, delta,CPUTIME_SYSTEM ); }if (ti->hardirq_time) { delta =cycle_to_nsec (ti->hardirq_time); -account_system_index_time (tsk, delta,CPUTIME_IRQ ); +account_system_time_index (tsk, delta,CPUTIME_IRQ ); }if (ti->softirq_time) { delta =cycle_to_nsec (ti->softirq_time); -account_system_index_time (tsk, delta,CPUTIME_SOFTIRQ ); +account_system_time_index (tsk, delta,CPUTIME_SOFTIRQ ); } ti->utime =0 ;
arch/powerpc/kernel/time.c +3 -3
@@ -418,14 +418,14 @@ void vtime_flush(struct task_struct *tsk)account_idle_time (cputime_to_nsecs (acct->idle_time));if (acct->stime) -account_system_index_time (tsk,cputime_to_nsecs (acct->stime), +account_system_time_index (tsk,cputime_to_nsecs (acct->stime),CPUTIME_SYSTEM );if (acct->hardirq_time) -account_system_index_time (tsk,cputime_to_nsecs (acct->hardirq_time), +account_system_time_index (tsk,cputime_to_nsecs (acct->hardirq_time),CPUTIME_IRQ );if (acct->softirq_time) -account_system_index_time (tsk,cputime_to_nsecs (acct->softirq_time), +account_system_time_index (tsk,cputime_to_nsecs (acct->softirq_time),CPUTIME_SOFTIRQ );vtime_flush_scaled (tsk, acct);
arch/s390/kernel/vtime.c +1 -1
@@ -115,7 +115,7 @@ static void account_system_index_scaled(struct task_struct *p, u64 cputime, enum cpu_usage_stat index) { p->stimescaled +=cputime_to_nsecs (scale_vtime (cputime)); -account_system_index_time (p,cputime_to_nsecs (cputime), index); +account_system_time_index (p,cputime_to_nsecs (cputime), index); }/*
include/linux/kernel_stat.h +1 -1
@@ -92,7 +92,7 @@ static inline u64 kcpustat_field(struct kernel_cpustat *kcpustat,extern void account_user_time (struct task_struct *,u64 );extern void account_guest_time (struct task_struct *,u64 );extern void account_system_time (struct task_struct *,int ,u64 ); -extern void account_system_index_time (struct task_struct *,u64 , +extern void account_system_time_index (struct task_struct *,u64 ,enum cpu_usage_stat );extern void account_steal_time (u64 );extern void account_idle_time (u64 );
kernel/sched/cputime.c +11 -11
@@ -95,8 +95,8 @@ static u64 irqtime_tick_accounted(u64 dummy)#endif /* !CONFIG_IRQ_TIME_ACCOUNTING */ -static inline void task_group_account_field (struct task_struct *p,int index, -u64 tmp) +static inline void task_group_account_field (struct task_struct *p,u64 cputime, +enum cpu_usage_stat index) {/* * Since all updates are sure to touch the root cgroup, we
@@ -104,9 +104,9 @@ static inline void task_group_account_field(struct task_struct *p, int index,* is the only cgroup, then nothing else should be necessary. * */ -__this_cpu_add (kernel_cpustat.cpustat[index], tmp); +__this_cpu_add (kernel_cpustat.cpustat[index], cputime); -cgroup_account_cputime_field (p, index, tmp); +cgroup_account_cputime_field (p, index, cputime); } static void account_user_time_index(struct task_struct *p,
@@ -117,7 +117,7 @@ static void account_user_time_index(struct task_struct *p,account_group_user_time (p, cputime);/* Add user time to cpustat. */ -task_group_account_field (p, index, cputime); +task_group_account_field (p, cputime, index);/* Account for user time used */ acct_account_cputime (p);
@@ -175,7 +175,7 @@ void account_guest_time(struct task_struct *p, u64 cputime)* @cputime: the CPU time spent in kernel space since the last update * @index: pointer to cpustat field that has to be updated */ -void account_system_index_time (struct task_struct *p, +void account_system_time_index (struct task_struct *p,u64 cputime,enum cpu_usage_stat index) {/* Add system time to process. */
@@ -183,7 +183,7 @@ void account_system_index_time(struct task_struct *p,account_group_system_time (p, cputime);/* Add system time to cpustat. */ -task_group_account_field (p, index, cputime); +task_group_account_field (p, cputime, index);/* Account for system time used */ acct_account_cputime (p);
@@ -197,7 +197,7 @@ void account_system_index_time(struct task_struct *p,*/ void account_system_time (struct task_struct *p,int hardirq_offset,u64 cputime) { -int index; +enum cpu_usage_stat index;if ((p->flags &PF_VCPU ) && (irq_count () - hardirq_offset ==0 )) {account_guest_time (p, cputime);
@@ -211,7 +211,7 @@ void account_system_time(struct task_struct *p, int hardirq_offset, u64 cputime)else index =CPUTIME_SYSTEM ; -account_system_index_time (p, cputime, index); +account_system_time_index (p, cputime, index); }/*
@@ -392,7 +392,7 @@ static void irqtime_account_process_tick(struct task_struct *p, int user_tick,* So, we have to handle it separately here. * Also, p->stime needs to be updated for ksoftirqd. */ -account_system_index_time (p, cputime,CPUTIME_SOFTIRQ ); +account_system_time_index (p, cputime,CPUTIME_SOFTIRQ ); }else if (user_tick) {account_user_time (p, cputime); }else if (p == rq->idle) {
@@ -400,7 +400,7 @@ static void irqtime_account_process_tick(struct task_struct *p, int user_tick, }else if (p->flags &PF_VCPU ) {/* System time or guest time */ account_guest_time (p, cputime); }else { -account_system_index_time (p, cputime,CPUTIME_SYSTEM ); +account_system_time_index (p, cputime,CPUTIME_SYSTEM ); } }
Signature · 1 line
2.23.0
PATCH 3/9 sched/vtime: Handle nice updates under vtime
In reply to Frederic WeisbeckerThe cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.
So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as CPUTIME_NICE because it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted as CPUTIME_USER and 5 seconds as CPUTIME_NICE.
In order to solve this, we now cover nice updates withing three cases:
* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.
* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.
* When a task's nice value gets updated while it is sleeping, the next context switch takes into account the new nice value in order to later record the vtime delta to the appropriate kcpustat index.
--- include/linux/sched.h | 1 + include/linux/vtime.h | 2 + kernel/sched/core.c | 16 +++-- kernel/sched/cputime.c | 136 ++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 18 ++++++ 5 files changed, 159 insertions(+), 14 deletions(-)
include/linux/sched.h +1 -0
@@ -262,6 +262,7 @@ enum vtime_state {struct vtime {seqcount_t seqcount;unsigned long long starttime; +int nice;enum vtime_state state;unsigned int cpu;u64 utime;
include/linux/vtime.h +2 -0
@@ -9,6 +9,7 @@struct task_struct ; +struct rq ;/* * vtime_accounting_enabled_this_cpu() definitions/declarations
@@ -74,6 +75,7 @@ extern void vtime_user_exit(struct task_struct *tsk);extern void vtime_guest_enter (struct task_struct *tsk);extern void vtime_guest_exit (struct task_struct *tsk);extern void vtime_init_idle (struct task_struct *tsk,int cpu); +extern void __vtime_set_nice (struct rq *rq,struct task_struct *tsk,int old_prio);#else /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN */ static inline void vtime_user_enter (struct task_struct *tsk) { }static inline void vtime_user_exit (struct task_struct *tsk) { }
kernel/sched/core.c +12 -4
@@ -4488,6 +4488,7 @@ void set_user_nice(struct task_struct *p, long nice)int old_prio, delta;struct rq_flags rf;struct rq *rq; +int old_static;if (task_nice (p) == nice || nice <MIN_NICE || nice >MAX_NICE )return ;
@@ -4498,6 +4499,8 @@ void set_user_nice(struct task_struct *p, long nice) rq =task_rq_lock (p, &rf);update_rq_clock (rq); + old_static = p->static_prio; +/* * The RT priorities are set via sched_setscheduler(), but we still * allow the 'normal' nice value to be set - but as expected
@@ -4532,7 +4535,9 @@ void set_user_nice(struct task_struct *p, long nice) }if (running)set_next_task (rq, p); + out_unlock: +vtime_set_nice (rq, p, old_static);task_rq_unlock (rq, p, &rf); }EXPORT_SYMBOL (set_user_nice);
@@ -4668,8 +4673,8 @@ static struct task_struct *find_process_by_pid(pid_t pid)*/ #define SETPARAM_POLICY -1 -static void __setscheduler_params (struct task_struct *p, -const struct sched_attr *attr) +static void __setscheduler_params (struct rq *rq,struct task_struct *p, +const struct sched_attr *attr) {int policy = attr->sched_policy;
@@ -4680,8 +4685,11 @@ static void __setscheduler_params(struct task_struct *p,if (dl_policy (policy))__setparam_dl (p, attr); -else if (fair_policy (policy)) +else if (fair_policy (policy)) { +int old_prio = p->static_prio; p->static_prio =NICE_TO_PRIO (attr->sched_nice); +vtime_set_nice (rq, p, old_prio); + }/* * __sched_setscheduler() ensures attr->sched_priority == 0 when
@@ -4704,7 +4712,7 @@ static void __setscheduler(struct rq *rq, struct task_struct *p,if (attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )return ; -__setscheduler_params (p, attr); +__setscheduler_params (rq, p, attr);/* * Keep a potential priority boosting if called from
kernel/sched/cputime.c +126 -10
165 lines not on this page · raw message
kernel/sched/sched.h +18 -0
@@ -1895,6 +1895,24 @@ static inline int sched_tick_offload_init(void) { return 0; }static inline void sched_update_tick_dependency (struct rq *rq) { }#endif +static inline void vtime_set_nice (struct rq *rq, +struct task_struct *p,int old_prio) +{ +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN +int cpu; + +if (!vtime_accounting_enabled ()) +return ; + + cpu =cpu_of (rq); + +if (!vtime_accounting_enabled_cpu (cpu)) +return ; + +__vtime_set_nice (rq, p, old_prio); +#endif +} +static inline void add_nr_running (struct rq *rq,unsigned count) {unsigned prev_nr = rq->nr_running;
Signature · 1 line
2.23.0
Peter Zijlstra
maintainerRe: [PATCH 3/9] sched/vtime: Handle nice updates under vtime
Quoted 56 lines · Frederic Weisbecker
kernel/sched/core.c +11
-3@@ -4488,6 +4488,7 @@ void set_user_nice(struct task_struct *p, long nice)int old_prio, delta;struct rq_flags rf;struct rq *rq; +int old_static;if (task_nice (p) == nice || nice <MIN_NICE || nice >MAX_NICE )return ;@@ -4498,6 +4499,8 @@ void set_user_nice(struct task_struct *p, long nice) rq =task_rq_lock (p, &rf);update_rq_clock (rq); + old_static = p->static_prio; +/* * The RT priorities are set via sched_setscheduler(), but we still * allow the 'normal' nice value to be set - but as expected @@ -4532,7 +4535,9 @@ void set_user_nice(struct task_struct *p, long nice) }if (running)set_next_task (rq, p); + out_unlock: +vtime_set_nice (rq, p, old_static);task_rq_unlock (rq, p, &rf); }EXPORT_SYMBOL (set_user_nice);@@ -4668,8 +4673,8 @@ static struct task_struct *find_process_by_pid(pid_t pid)*/ #define SETPARAM_POLICY -1 -static void __setscheduler_params (struct task_struct *p, -const struct sched_attr *attr) +static void __setscheduler_params (struct rq *rq,struct task_struct *p, +const struct sched_attr *attr) {int policy = attr->sched_policy;@@ -4680,8 +4685,11 @@ static void __setscheduler_params(struct task_struct *p,if (dl_policy (policy))__setparam_dl (p, attr); -else if (fair_policy (policy)) +else if (fair_policy (policy)) { +int old_prio = p->static_prio; p->static_prio =NICE_TO_PRIO (attr->sched_nice); +vtime_set_nice (rq, p, old_prio); + }/* * __sched_setscheduler() ensures attr->sched_priority == 0 when
kernel/sched/core.c static void __setscheduler(struct rq *rq, struct task_struct *p, · line 4718
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -4704,7 +4712,7 @@ static void __setscheduler(struct rq *rq, struct task_struct *p,if (attr->sched_flags &SCHED_FLAG_KEEP_PARAMS )return ; -__setscheduler_params (p, attr); +__setscheduler_params (rq, p, attr);/* * Keep a potential priority boosting if called from
Would it make sense to add a helper like:
static void__sched_set_nice(struct rq *rq, struct task_struct *t, long nice)
{
int old_prio;
old_prio = t->static_prio;
t->static_prio = NICE_TO_PRIO(nice);
vtime_set_nice(rq, t, old_prio);
}Peter Zijlstra
maintainerRe: [PATCH 3/9] sched/vtime: Handle nice updates under vtime
In reply to Frederic WeisbeckerQuoted 19 lines · Frederic Weisbecker
The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.
So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as
CPUTIME_NICEbecause it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted asCPUTIME_USERand 5 seconds asCPUTIME_NICE.In order to solve this, we now cover nice updates withing three cases:
* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.
Quoted 5 lines
* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.
But but but, I thought the idea was to _never_ send interrupts to NOHZ_FULL cpus ?!?
Peter Zijlstra
maintainerRe: [PATCH 3/9] sched/vtime: Handle nice updates under vtime
Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Frederic Weisbecker
The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.
So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as
CPUTIME_NICEbecause it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted asCPUTIME_USERand 5 seconds asCPUTIME_NICE.In order to solve this, we now cover nice updates withing three cases:
* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.
Quoted 8 lines
Quoted 5 lines
* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.
But but but, I thought the idea was to _never_ send interrupts to
NOHZ_FULLcpus ?!?
That is, isn't the cure worse than the problem? I mean, who bloody cares about silly accounting crud more than not getting interrupts on their NOHZ_FULL cpus.
Re: [PATCH 3/9] sched/vtime: Handle nice updates under vtime
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Frederic Weisbecker
The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.
So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as
CPUTIME_NICEbecause it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted asCPUTIME_USERand 5 seconds asCPUTIME_NICE.In order to solve this, we now cover nice updates withing three cases:
* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.
* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.
Quoted 6 lines
Quoted 2 lines
But but but, I thought the idea was to _never_ send interrupts to
NOHZ_FULLcpus ?!?That is, isn't the cure worse than the problem? I mean, who bloody cares about silly accounting crud more than not getting interrupts on their
NOHZ_FULLcpus.
Yeah indeed. I tend to sacrifice everything for correctness but perhaps we can live with small issues like nice accounting not being accounted to the right place if that can avoid disturbing nohz_full CPUs. Also who cares about renicing a task that is supposed to run alone.
So here is what I can do: I'll make a simplified version of that set which accounts on top of the task_nice() value found on accounting time (context switch, user exit, guest exit) and if some user ever complains, I can still bring back that IPI solution.
Thanks.
Peter Zijlstra
maintainerRe: [PATCH 3/9] sched/vtime: Handle nice updates under vtime
Quoted 35 lines · Frederic Weisbecker
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Frederic Weisbecker
The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.
So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as
CPUTIME_NICEbecause it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted asCPUTIME_USERand 5 seconds asCPUTIME_NICE.In order to solve this, we now cover nice updates withing three cases:
* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.
* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.
Quoted 6 lines
Quoted 2 lines
But but but, I thought the idea was to _never_ send interrupts to
NOHZ_FULLcpus ?!?That is, isn't the cure worse than the problem? I mean, who bloody cares about silly accounting crud more than not getting interrupts on their
NOHZ_FULLcpus.
Quoted 8 lines
Yeah indeed. I tend to sacrifice everything for correctness but perhaps we can live with small issues like nice accounting not being accounted to the right place if that can avoid disturbing
nohz_fullCPUs. Also who cares about renicing a task that is supposed to run alone.So here is what I can do: I'll make a simplified version of that set which accounts on top of the
task_nice()value found on accounting time (context switch, user exit, guest exit) and if some user ever complains, I can still bring back that IPI solution.
Makes sense, thanks!
PATCH 4/9 sched/cputime: Support other fields on kcpustat_field()
In reply to Frederic WeisbeckerNow that we support nice updates under vtime accounting, we can provide tickless values for kcpustat readers of user and guest time.
--- kernel/sched/cputime.c | 40 +++++++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 7 deletions(-)
kernel/sched/cputime.c +33 -7
@@ -1042,6 +1042,15 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime) }while (read_seqcount_retry (&vtime->seqcount, seq)); } +static u64 kcpustat_user_vtime (struct vtime *vtime) +{ +if (vtime->state ==VTIME_USER ) +return vtime->utime +vtime_delta (vtime); +else if (vtime->state ==VTIME_GUEST ) +return vtime->gtime +vtime_delta (vtime); +return 0 ; +} + static int kcpustat_field_vtime(u64 *cpustat, struct vtime *vtime, enum cpu_usage_stat usage,
@@ -1076,9 +1085,30 @@ static int kcpustat_field_vtime(u64 *cpustat, *val = cpustat[usage]; -if (vtime->state ==VTIME_SYS ) - *val += vtime->stime +vtime_delta (vtime); - +switch (usage) { +case CPUTIME_SYSTEM : +if (vtime->state ==VTIME_SYS ) + *val += vtime->stime +vtime_delta (vtime); +break ; +case CPUTIME_USER : +if (!vtime->nice) + *val +=kcpustat_user_vtime (vtime); +break ; +case CPUTIME_NICE : +if (vtime->nice) + *val +=kcpustat_user_vtime (vtime); +break ; +case CPUTIME_GUEST : +if (vtime->state ==VTIME_GUEST && !vtime->nice) + *val += vtime->gtime +vtime_delta (vtime); +break ; +case CPUTIME_GUEST_NICE : +if (vtime->state ==VTIME_GUEST && vtime->nice) + *val += vtime->gtime +vtime_delta (vtime); +break ; +default : +break ; + } }while (read_seqcount_retry (&vtime->seqcount, seq));return 0 ;
@@ -1095,10 +1125,6 @@ u64 kcpustat_field(struct kernel_cpustat *kcpustat,if (!vtime_accounting_enabled_cpu (cpu))return cpustat[usage]; -/* Only support sys vtime for now */ -if (usage !=CPUTIME_SYSTEM ) -return cpustat[usage]; - rq =cpu_rq (cpu);for (;;) {
Signature · 1 line
2.23.0
PATCH 5/9 sched/vtime: Bring all-in-one kcpustat accessor for vtime fields
In reply to Frederic WeisbeckerMany callsites want to fetch the values of system, user, user_nice, guest or guest_nice kcpustat fields altogether or at least a pair of these.
In that case calling kcpustat_field() for each requested field brings unecessary overhead when we could fetch all of them in a row.
So provide kcpustat_cputime() that fetches all vtime sensitive fields under the same RCU and seqcount block.
--- include/linux/kernel_stat.h | 23 ++++++ kernel/sched/cputime.c | 138 ++++++++++++++++++++++++++++++------ 2 files changed, 141 insertions(+), 20 deletions(-)
include/linux/kernel_stat.h +23 -0
@@ -78,15 +78,38 @@ static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)return kstat_cpu (cpu).irqs_sum; } + +static inline void kcpustat_cputime_raw (u64 *cpustat,u64 *user,u64 *nice, +u64 *system,u64 *guest,u64 *guest_nice) +{ + *user = cpustat[CPUTIME_USER ]; + *nice = cpustat[CPUTIME_NICE ]; + *system = cpustat[CPUTIME_SYSTEM ]; + *guest = cpustat[CPUTIME_GUEST ]; + *guest_nice = cpustat[CPUTIME_GUEST_NICE ]; +} +#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern u64 kcpustat_field (struct kernel_cpustat *kcpustat,enum cpu_usage_stat usage,int cpu); +extern void kcpustat_cputime (struct kernel_cpustat *kcpustat,int cpu, +u64 *user,u64 *nice,u64 *system, +u64 *guest,u64 *guest_nice);#else static inline u64 kcpustat_field (struct kernel_cpustat *kcpustat,enum cpu_usage_stat usage,int cpu) {return kcpustat->cpustat[usage]; } + +static inline void kcpustat_cputime (struct kernel_cpustat *kcpustat,int cpu, +u64 *user,u64 *nice,u64 *system, +u64 *guest,u64 *guest_nice) +{ +kcpustat_cputime_raw (kcpustat->cpustat, user, nice, + system, guest, guest_nice); +} +#endif extern void account_user_time (struct task_struct *,u64 );
kernel/sched/cputime.c +118 -20
154 lines not on this page · raw message
Signature · 1 line
2.23.0
PATCH 6/9 procfs: Use all-in-one vtime aware kcpustat accessor
In reply to Frederic WeisbeckerNow that we can read also user and guest time safely under vtime, use the relevant accessor to fix frozen kcpustat values on nohz_full CPUs.
--- fs/proc/stat.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
fs/proc/stat.c +10 -10
@@ -120,18 +120,21 @@ static int show_stat(struct seq_file *p, void *v)getboottime64 (&boottime);for_each_possible_cpu (i) { +u64 cpu_user, cpu_nice, cpu_sys, cpu_guest, cpu_guest_nice;struct kernel_cpustat *kcs = &kcpustat_cpu (i); - user += kcs->cpustat[CPUTIME_USER ]; - nice += kcs->cpustat[CPUTIME_NICE ]; - system +=kcpustat_field (kcs,CPUTIME_SYSTEM , i); +kcpustat_cputime (kcs, i, &cpu_user, &cpu_nice, + &cpu_sys, &cpu_guest, &cpu_guest_nice); + user += cpu_user; + nice += cpu_nice; + system += cpu_sys; idle +=get_idle_time (kcs, i); iowait +=get_iowait_time (kcs, i); irq += kcs->cpustat[CPUTIME_IRQ ]; softirq += kcs->cpustat[CPUTIME_SOFTIRQ ]; steal += kcs->cpustat[CPUTIME_STEAL ]; - guest += kcs->cpustat[CPUTIME_GUEST ]; - guest_nice += kcs->cpustat[CPUTIME_GUEST_NICE ]; + guest += cpu_guest; + guest_nice += guest_nice; sum +=kstat_cpu_irqs_sum (i); sum +=arch_irq_stat_cpu (i);
@@ -159,17 +162,14 @@ static int show_stat(struct seq_file *p, void *v)for_each_online_cpu (i) {struct kernel_cpustat *kcs = &kcpustat_cpu (i); +kcpustat_cputime (kcs, i, &user, &nice, + &system, &guest, &guest_nice);/* Copy values here to work around gcc-2.95.3, gcc-2.96 */ - user = kcs->cpustat[CPUTIME_USER ]; - nice = kcs->cpustat[CPUTIME_NICE ]; - system =kcpustat_field (kcs,CPUTIME_SYSTEM , i); idle =get_idle_time (kcs, i); iowait =get_iowait_time (kcs, i); irq = kcs->cpustat[CPUTIME_IRQ ]; softirq = kcs->cpustat[CPUTIME_SOFTIRQ ]; steal = kcs->cpustat[CPUTIME_STEAL ]; - guest = kcs->cpustat[CPUTIME_GUEST ]; - guest_nice = kcs->cpustat[CPUTIME_GUEST_NICE ];seq_printf (p,"cpu%d" , i);seq_put_decimal_ull (p," " ,nsec_to_clock_t (user));seq_put_decimal_ull (p," " ,nsec_to_clock_t (nice));
Signature · 1 line
2.23.0
PATCH 7/9 cpufreq: Use vtime aware kcpustat accessors for user time
In reply to Frederic WeisbeckerWe can now safely read user and guest kcpustat fields on nohz_full CPUs. Use the appropriate accessors.
--- drivers/cpufreq/cpufreq.c | 13 +++++++++---- drivers/cpufreq/cpufreq_governor.c | 6 +++--- 2 files changed, 12 insertions(+), 7 deletions(-)
drivers/cpufreq/cpufreq.c +9 -4
@@ -113,18 +113,23 @@ EXPORT_SYMBOL_GPL(get_governor_parent_kobj);static inline u64 get_cpu_idle_time_jiffy (unsigned int cpu,u64 *wall) { -u64 idle_time; +struct kernel_cpustat *kcpustat = &kcpustat_cpu (cpu); +u64 user, nice, sys, guest, guest_nice;u64 cur_wall_time; +u64 idle_time;u64 busy_time; cur_wall_time =jiffies64_to_nsecs (get_jiffies_64 ()); - busy_time =kcpustat_cpu (cpu).cpustat[CPUTIME_USER ]; - busy_time +=kcpustat_field (&kcpustat_cpu (cpu),CPUTIME_SYSTEM , cpu); +kcpustat_cputime (kcpustat, cpu, &user, &nice, &sys, + &guest, &guest_nice); + + busy_time = user; + busy_time += sys; busy_time +=kcpustat_cpu (cpu).cpustat[CPUTIME_IRQ ]; busy_time +=kcpustat_cpu (cpu).cpustat[CPUTIME_SOFTIRQ ]; busy_time +=kcpustat_cpu (cpu).cpustat[CPUTIME_STEAL ]; - busy_time +=kcpustat_cpu (cpu).cpustat[CPUTIME_NICE ]; + busy_time += nice; idle_time = cur_wall_time - busy_time;if (wall)
drivers/cpufreq/cpufreq_governor.c +3 -3
@@ -105,7 +105,7 @@ void gov_update_cpu_data(struct dbs_data *dbs_data) j_cdbs->prev_cpu_idle =get_cpu_idle_time (j, &j_cdbs->prev_update_time, dbs_data->io_is_busy);if (dbs_data->ignore_nice_load) - j_cdbs->prev_cpu_nice =kcpustat_cpu (j).cpustat[CPUTIME_NICE ]; + j_cdbs->prev_cpu_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); } } }
@@ -149,7 +149,7 @@ unsigned int dbs_update(struct cpufreq_policy *policy) j_cdbs->prev_cpu_idle = cur_idle_time;if (ignore_nice) { -u64 cur_nice =kcpustat_cpu (j).cpustat[CPUTIME_NICE ]; +u64 cur_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); idle_time +=div_u64 (cur_nice - j_cdbs->prev_cpu_nice,NSEC_PER_USEC ); j_cdbs->prev_cpu_nice = cur_nice;
@@ -530,7 +530,7 @@ int cpufreq_dbs_governor_start(struct cpufreq_policy *policy) j_cdbs->prev_load =0 ;if (ignore_nice) - j_cdbs->prev_cpu_nice =kcpustat_cpu (j).cpustat[CPUTIME_NICE ]; + j_cdbs->prev_cpu_nice =kcpustat_field (&kcpustat_cpu (j),CPUTIME_NICE , j); } gov->start (policy);
Signature · 1 line
2.23.0
PATCH 8/9 leds: Use all-in-one vtime aware kcpustat accessor
In reply to Frederic WeisbeckerWe can now safely read user kcpustat fields on nohz_full CPUs. Use the appropriate accessor.
--- drivers/leds/trigger/ledtrig-activity.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/leds/trigger/ledtrig-activity.c +6 -3
@@ -57,9 +57,12 @@ static void led_activity_function(struct timer_list *t) curr_used =0 ;for_each_possible_cpu (i) { - curr_used +=kcpustat_cpu (i).cpustat[CPUTIME_USER ] - +kcpustat_cpu (i).cpustat[CPUTIME_NICE ] - +kcpustat_field (&kcpustat_cpu (i),CPUTIME_SYSTEM , i) +u64 user, nice, sys, guest, guest_nice; + +kcpustat_cputime (&kcpustat_cpu (i), i, &user, &nice, &sys, + &guest, &guest_nice); + + curr_used += user + nice + sys +kcpustat_cpu (i).cpustat[CPUTIME_SOFTIRQ ] +kcpustat_cpu (i).cpustat[CPUTIME_IRQ ]; cpus++;
Signature · 1 line
2.23.0
PATCH 9/9 rackmeter: Use vtime aware kcpustat accessor
In reply to Frederic WeisbeckerNow that we have a vtime safe kcpustat accessor, use it to fetch CPUTIME_NICE and fix frozen kcpustat values on nohz_full CPUs.
--- drivers/macintosh/rack-meter.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/macintosh/rack-meter.c +4 -3
@@ -81,13 +81,14 @@ static int rackmeter_ignore_nice;*/ static inline u64 get_cpu_idle_time (unsigned int cpu) { +struct kernel_cpustat *kcpustat = &kcpustat_cpu (cpu);u64 retval; - retval =kcpustat_cpu (cpu).cpustat[CPUTIME_IDLE ] + -kcpustat_cpu (cpu).cpustat[CPUTIME_IOWAIT ]; + retval = kcpustat->cpustat[CPUTIME_IDLE ] + + kcpustat->cpustat[CPUTIME_IOWAIT ];if (rackmeter_ignore_nice) - retval +=kcpustat_cpu (cpu).cpustat[CPUTIME_NICE ]; + retval +=kcpustat_field (kcpustat,CPUTIME_NICE , cpu);return retval; }
Signature · 1 line
2.23.0
Quoted 8 lines · Frederic Weisbecker
So we fixed
CPUTIME_SYSTEMonnohz_full, now is the turn forCPUTIME_USER,CPUTIME_NICE,CPUTIME_GUESTandCPUTIME_GUEST_NICE.The tricky piece was to handle "nice-ness" correctly. I think I addressed most reviews (one year ago now) from peterz
(https://lore.kernel.org/lkml/[email protected]/)
Among which:
Quoted 4 lines
* Merge patches that didn't make sense together
* Clarify the changelog (I hope) and add comments
* Also hook nice updates on__setscheduler_params()
* Uninline a big function
Gentle ping :)