Skip to contentSkip to conversation

sched/nohz: Make the rest of kcpustat vtime aware

PATCH In review Frederic Weisbecker 2 participants 16 messages Nov 6, 2019 – Nov 18, 2019

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover sched/nohz: Make the rest of kcpustat vtime aware

So we fixed CPUTIME_SYSTEM on nohz_full, now is the turn for CPUTIME_USER, CPUTIME_NICE, CPUTIME_GUEST and CPUTIME_GUEST_NICE.

The tricky piece was to handle "nice-ness" correctly. I think I addressed most reviews (one year ago now) from peterz
(https://lore.kernel.org/lkml/[email protected]/)
Among which:

* Merge patches that didn't make sense together
* Clarify the changelog (I hope) and add comments
* Also hook nice updates on __setscheduler_params()
* Uninline a big function

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
nohz/kcpustat-v3

HEAD: cfd377b0f261f1107ba5f3d2cf68a8ac5f359943

Thanks,
Frederic
---

Frederic Weisbecker (9):
sched/cputime: Allow to pass cputime index on user/guest accounting sched/cputime: Standardize the kcpustat index based accounting functions sched/vtime: Handle nice updates under vtime
sched/cputime: Support other fields on kcpustat_field() sched/vtime: Bring all-in-one kcpustat accessor for vtime fields
procfs: Use all-in-one vtime aware kcpustat accessor
cpufreq: Use vtime aware kcpustat accessors for user time
leds: Use all-in-one vtime aware kcpustat accessor
rackmeter: Use vtime aware kcpustat accessor

 arch/ia64/kernel/time.c                 |   6 +-
 arch/powerpc/kernel/time.c              |   6 +-
 arch/s390/kernel/vtime.c                |   2 +-
 drivers/cpufreq/cpufreq.c               |  13 +-
 drivers/cpufreq/cpufreq_governor.c      |   6 +-
 drivers/leds/trigger/ledtrig-activity.c |   9 +-
 drivers/macintosh/rack-meter.c          |   7 +-
 fs/proc/stat.c                          |  20 +-
 include/linux/kernel_stat.h             |  25 ++-
 include/linux/sched.h                   |   1 +
 include/linux/vtime.h                   |   2 +
 kernel/sched/core.c                     |  16 +-
 kernel/sched/cputime.c                  | 378 ++++++++++++++++++++++++++------
 kernel/sched/sched.h                    |  18 ++
 14 files changed, 412 insertions(+), 97 deletions(-)
#1 lore raw

PATCH 1/9 sched/cputime: Allow to pass cputime index on user/guest accounting

When we account user or guest cputime, we decide to add the delta either to the nice fields or the normal fields of kcpustat and this depends on the nice value for the task passed in parameter.

Since we are going to track the nice-ness from vtime instead, we'll need to be able to pass custom kcpustat destination index fields to the accounting functions.

Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/sched/cputime.c | 50 +++++++++++++++++++++++++++---------------
 1 file changed, 32 insertions(+), 18 deletions(-)
kernel/sched/cputime.c +32 -18
index e0cd20693ef5..738ed7db615e 100644
@@ -109,6 +109,20 @@ static inline void task_group_account_field(struct task_struct *p, int index,
 	cgroup_account_cputime_field(p, index, tmp);
 }
 
+static void account_user_time_index(struct task_struct *p,
+				    u64 cputime, enum cpu_usage_stat index)
+{
+	/* Add user time to process. */
+	p->utime += cputime;
+	account_group_user_time(p, cputime);
+
+	/* Add user time to cpustat. */
+	task_group_account_field(p, index, cputime);
+
+	/* Account for user time used */
+	acct_account_cputime(p);
+}
+
 /*
  * Account user CPU time to a process.
  * @p: the process that the CPU time gets accounted to
@@ -116,27 +130,14 @@ static inline void task_group_account_field(struct task_struct *p, int index,
  */
 void account_user_time(struct task_struct *p, u64 cputime)
 {
-	int index;
-
-	/* Add user time to process. */
-	p->utime += cputime;
-	account_group_user_time(p, cputime);
+	enum cpu_usage_stat index;
 
 	index = (task_nice(p) > 0) ? CPUTIME_NICE : CPUTIME_USER;
-
-	/* Add user time to cpustat. */
-	task_group_account_field(p, index, cputime);
-
-	/* Account for user time used */
-	acct_account_cputime(p);
+	account_user_time_index(p, cputime, index);
 }
 
-/*
- * Account guest CPU time to a process.
- * @p: the process that the CPU time gets accounted to
- * @cputime: the CPU time spent in virtual machine since the last update
- */
-void account_guest_time(struct task_struct *p, u64 cputime)
+static void account_guest_time_index(struct task_struct *p,
+				     u64 cputime, enum cpu_usage_stat index)
 {
 	u64 *cpustat = kcpustat_this_cpu->cpustat;
 
@@ -146,7 +147,7 @@ void account_guest_time(struct task_struct *p, u64 cputime)
 	p->gtime += cputime;
 
 	/* Add guest time to cpustat. */
-	if (task_nice(p) > 0) {
+	if (index == CPUTIME_GUEST_NICE) {
 		cpustat[CPUTIME_NICE] += cputime;
 		cpustat[CPUTIME_GUEST_NICE] += cputime;
 	} else {
@@ -155,6 +156,19 @@ void account_guest_time(struct task_struct *p, u64 cputime)
 	}
 }
 
+/*
+ * Account guest CPU time to a process.
+ * @p: the process that the CPU time gets accounted to
+ * @cputime: the CPU time spent in virtual machine since the last update
+ */
+void account_guest_time(struct task_struct *p, u64 cputime)
+{
+	enum cpu_usage_stat index;
+
+	index = (task_nice(p) > 0) ? CPUTIME_GUEST_NICE : CPUTIME_GUEST;
+	account_guest_time_index(p, cputime, index);
+}
+
 /*
  * Account system CPU time to a process and desired cpustat field
  * @p: the process that the CPU time gets accounted to
Signature · 1 line
2.23.0
#2 lore raw

PATCH 2/9 sched/cputime: Standardize the kcpustat index based accounting functions

In reply to Frederic Weisbecker

Sanitize a bit the functions that do cputime accounting with custom kcpustat indexes:

* Rename account_system_index_time() to account_system_time_index() to comply with account_guest/user_time_index()

* Use proper enum cpu_usage_stat type in account_system_time()

* Reorder task_group_account_field() parameters to comply with those of account_*_time_index().

* Rename task_group_account_field()'s tmp parameter to cputime

* Precise the type of index in task_group_account_field(): enum cpu_usage_stat

Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 arch/ia64/kernel/time.c     |  6 +++---
 arch/powerpc/kernel/time.c  |  6 +++---
 arch/s390/kernel/vtime.c    |  2 +-
 include/linux/kernel_stat.h |  2 +-
 kernel/sched/cputime.c      | 22 +++++++++++-----------
 5 files changed, 19 insertions(+), 19 deletions(-)
arch/ia64/kernel/time.c +3 -3
index 91b4024c9351..836e17c8b004 100644
@@ -79,17 +79,17 @@ void vtime_flush(struct task_struct *tsk)
 
 	if (ti->stime) {
 		delta = cycle_to_nsec(ti->stime);
-		account_system_index_time(tsk, delta, CPUTIME_SYSTEM);
+		account_system_time_index(tsk, delta, CPUTIME_SYSTEM);
 	}
 
 	if (ti->hardirq_time) {
 		delta = cycle_to_nsec(ti->hardirq_time);
-		account_system_index_time(tsk, delta, CPUTIME_IRQ);
+		account_system_time_index(tsk, delta, CPUTIME_IRQ);
 	}
 
 	if (ti->softirq_time) {
 		delta = cycle_to_nsec(ti->softirq_time);
-		account_system_index_time(tsk, delta, CPUTIME_SOFTIRQ);
+		account_system_time_index(tsk, delta, CPUTIME_SOFTIRQ);
 	}
 
 	ti->utime = 0;
arch/powerpc/kernel/time.c +3 -3
index 84827da01d45..f03469fe398b 100644
@@ -418,14 +418,14 @@ void vtime_flush(struct task_struct *tsk)
 		account_idle_time(cputime_to_nsecs(acct->idle_time));
 
 	if (acct->stime)
-		account_system_index_time(tsk, cputime_to_nsecs(acct->stime),
+		account_system_time_index(tsk, cputime_to_nsecs(acct->stime),
 					  CPUTIME_SYSTEM);
 
 	if (acct->hardirq_time)
-		account_system_index_time(tsk, cputime_to_nsecs(acct->hardirq_time),
+		account_system_time_index(tsk, cputime_to_nsecs(acct->hardirq_time),
 					  CPUTIME_IRQ);
 	if (acct->softirq_time)
-		account_system_index_time(tsk, cputime_to_nsecs(acct->softirq_time),
+		account_system_time_index(tsk, cputime_to_nsecs(acct->softirq_time),
 					  CPUTIME_SOFTIRQ);
 
 	vtime_flush_scaled(tsk, acct);
arch/s390/kernel/vtime.c +1 -1
index 8df10d3c8f6c..3428f28d3df1 100644
@@ -115,7 +115,7 @@ static void account_system_index_scaled(struct task_struct *p, u64 cputime,
 					enum cpu_usage_stat index)
 {
 	p->stimescaled += cputime_to_nsecs(scale_vtime(cputime));
-	account_system_index_time(p, cputime_to_nsecs(cputime), index);
+	account_system_time_index(p, cputime_to_nsecs(cputime), index);
 }
 
 /*
include/linux/kernel_stat.h +1 -1
index 79781196eb25..1b9b97f6946e 100644
@@ -92,7 +92,7 @@ static inline u64 kcpustat_field(struct kernel_cpustat *kcpustat,
 extern void account_user_time(struct task_struct *, u64);
 extern void account_guest_time(struct task_struct *, u64);
 extern void account_system_time(struct task_struct *, int, u64);
-extern void account_system_index_time(struct task_struct *, u64,
+extern void account_system_time_index(struct task_struct *, u64,
 				      enum cpu_usage_stat);
 extern void account_steal_time(u64);
 extern void account_idle_time(u64);
kernel/sched/cputime.c +11 -11
index 738ed7db615e..687b8684e480 100644
@@ -95,8 +95,8 @@ static u64 irqtime_tick_accounted(u64 dummy)
 
 #endif /* !CONFIG_IRQ_TIME_ACCOUNTING */
 
-static inline void task_group_account_field(struct task_struct *p, int index,
-					    u64 tmp)
+static inline void task_group_account_field(struct task_struct *p, u64 cputime,
+					    enum cpu_usage_stat index)
 {
 	/*
 	 * Since all updates are sure to touch the root cgroup, we
@@ -104,9 +104,9 @@ static inline void task_group_account_field(struct task_struct *p, int index,
 	 * is the only cgroup, then nothing else should be necessary.
 	 *
 	 */
-	__this_cpu_add(kernel_cpustat.cpustat[index], tmp);
+	__this_cpu_add(kernel_cpustat.cpustat[index], cputime);
 
-	cgroup_account_cputime_field(p, index, tmp);
+	cgroup_account_cputime_field(p, index, cputime);
 }
 
 static void account_user_time_index(struct task_struct *p,
@@ -117,7 +117,7 @@ static void account_user_time_index(struct task_struct *p,
 	account_group_user_time(p, cputime);
 
 	/* Add user time to cpustat. */
-	task_group_account_field(p, index, cputime);
+	task_group_account_field(p, cputime, index);
 
 	/* Account for user time used */
 	acct_account_cputime(p);
@@ -175,7 +175,7 @@ void account_guest_time(struct task_struct *p, u64 cputime)
  * @cputime: the CPU time spent in kernel space since the last update
  * @index: pointer to cpustat field that has to be updated
  */
-void account_system_index_time(struct task_struct *p,
+void account_system_time_index(struct task_struct *p,
 			       u64 cputime, enum cpu_usage_stat index)
 {
 	/* Add system time to process. */
@@ -183,7 +183,7 @@ void account_system_index_time(struct task_struct *p,
 	account_group_system_time(p, cputime);
 
 	/* Add system time to cpustat. */
-	task_group_account_field(p, index, cputime);
+	task_group_account_field(p, cputime, index);
 
 	/* Account for system time used */
 	acct_account_cputime(p);
@@ -197,7 +197,7 @@ void account_system_index_time(struct task_struct *p,
  */
 void account_system_time(struct task_struct *p, int hardirq_offset, u64 cputime)
 {
-	int index;
+	enum cpu_usage_stat index;
 
 	if ((p->flags & PF_VCPU) && (irq_count() - hardirq_offset == 0)) {
 		account_guest_time(p, cputime);
@@ -211,7 +211,7 @@ void account_system_time(struct task_struct *p, int hardirq_offset, u64 cputime)
 	else
 		index = CPUTIME_SYSTEM;
 
-	account_system_index_time(p, cputime, index);
+	account_system_time_index(p, cputime, index);
 }
 
 /*
@@ -392,7 +392,7 @@ static void irqtime_account_process_tick(struct task_struct *p, int user_tick,
 		 * So, we have to handle it separately here.
 		 * Also, p->stime needs to be updated for ksoftirqd.
 		 */
-		account_system_index_time(p, cputime, CPUTIME_SOFTIRQ);
+		account_system_time_index(p, cputime, CPUTIME_SOFTIRQ);
 	} else if (user_tick) {
 		account_user_time(p, cputime);
 	} else if (p == rq->idle) {
@@ -400,7 +400,7 @@ static void irqtime_account_process_tick(struct task_struct *p, int user_tick,
 	} else if (p->flags & PF_VCPU) { /* System time or guest time */
 		account_guest_time(p, cputime);
 	} else {
-		account_system_index_time(p, cputime, CPUTIME_SYSTEM);
+		account_system_time_index(p, cputime, CPUTIME_SYSTEM);
 	}
 }
 
Signature · 1 line
2.23.0
#3 lore raw

PATCH 3/9 sched/vtime: Handle nice updates under vtime

In reply to Frederic Weisbecker

The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.

So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as CPUTIME_NICE because it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted as CPUTIME_USER and 5 seconds as CPUTIME_NICE.

In order to solve this, we now cover nice updates withing three cases:

* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.

* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.

* When a task's nice value gets updated while it is sleeping, the next context switch takes into account the new nice value in order to later record the vtime delta to the appropriate kcpustat index.

Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 include/linux/sched.h  |   1 +
 include/linux/vtime.h  |   2 +
 kernel/sched/core.c    |  16 +++--
 kernel/sched/cputime.c | 136 ++++++++++++++++++++++++++++++++++++++---
 kernel/sched/sched.h   |  18 ++++++
 5 files changed, 159 insertions(+), 14 deletions(-)
include/linux/sched.h +1 -0
index 988c4da00c31..44552b74bd44 100644
@@ -262,6 +262,7 @@ enum vtime_state {
 struct vtime {
 	seqcount_t		seqcount;
 	unsigned long long	starttime;
+	int			nice;
 	enum vtime_state	state;
 	unsigned int		cpu;
 	u64			utime;
include/linux/vtime.h +2 -0
index 2cdeca062db3..eb2fa3c00f3e 100644
@@ -9,6 +9,7 @@
 
 
 struct task_struct;
+struct rq;
 
 /*
  * vtime_accounting_enabled_this_cpu() definitions/declarations
@@ -74,6 +75,7 @@ extern void vtime_user_exit(struct task_struct *tsk);
 extern void vtime_guest_enter(struct task_struct *tsk);
 extern void vtime_guest_exit(struct task_struct *tsk);
 extern void vtime_init_idle(struct task_struct *tsk, int cpu);
+extern void __vtime_set_nice(struct rq *rq, struct task_struct *tsk, int old_prio);
 #else /* !CONFIG_VIRT_CPU_ACCOUNTING_GEN  */
 static inline void vtime_user_enter(struct task_struct *tsk) { }
 static inline void vtime_user_exit(struct task_struct *tsk) { }
kernel/sched/core.c +12 -4
index 7880f4f64d0e..fa56a1bdd5d8 100644
@@ -4488,6 +4488,7 @@ void set_user_nice(struct task_struct *p, long nice)
 	int old_prio, delta;
 	struct rq_flags rf;
 	struct rq *rq;
+	int old_static;
 
 	if (task_nice(p) == nice || nice < MIN_NICE || nice > MAX_NICE)
 		return;
@@ -4498,6 +4499,8 @@ void set_user_nice(struct task_struct *p, long nice)
 	rq = task_rq_lock(p, &rf);
 	update_rq_clock(rq);
 
+	old_static = p->static_prio;
+
 	/*
 	 * The RT priorities are set via sched_setscheduler(), but we still
 	 * allow the 'normal' nice value to be set - but as expected
@@ -4532,7 +4535,9 @@ void set_user_nice(struct task_struct *p, long nice)
 	}
 	if (running)
 		set_next_task(rq, p);
+
 out_unlock:
+	vtime_set_nice(rq, p, old_static);
 	task_rq_unlock(rq, p, &rf);
 }
 EXPORT_SYMBOL(set_user_nice);
@@ -4668,8 +4673,8 @@ static struct task_struct *find_process_by_pid(pid_t pid)
  */
 #define SETPARAM_POLICY	-1
 
-static void __setscheduler_params(struct task_struct *p,
-		const struct sched_attr *attr)
+static void __setscheduler_params(struct rq *rq, struct task_struct *p,
+				  const struct sched_attr *attr)
 {
 	int policy = attr->sched_policy;
 
@@ -4680,8 +4685,11 @@ static void __setscheduler_params(struct task_struct *p,
 
 	if (dl_policy(policy))
 		__setparam_dl(p, attr);
-	else if (fair_policy(policy))
+	else if (fair_policy(policy)) {
+		int old_prio = p->static_prio;
 		p->static_prio = NICE_TO_PRIO(attr->sched_nice);
+		vtime_set_nice(rq, p, old_prio);
+	}
 
 	/*
 	 * __sched_setscheduler() ensures attr->sched_priority == 0 when
@@ -4704,7 +4712,7 @@ static void __setscheduler(struct rq *rq, struct task_struct *p,
 	if (attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)
 		return;
 
-	__setscheduler_params(p, attr);
+	__setscheduler_params(rq, p, attr);
 
 	/*
 	 * Keep a potential priority boosting if called from
kernel/sched/cputime.c +126 -10

165 lines not on this page · raw message

kernel/sched/sched.h +18 -0
index 0db2c1b3361e..b870042e65ae 100644
@@ -1895,6 +1895,24 @@ static inline int sched_tick_offload_init(void) { return 0; }
 static inline void sched_update_tick_dependency(struct rq *rq) { }
 #endif
 
+static inline void vtime_set_nice(struct rq *rq,
+				  struct task_struct *p, int old_prio)
+{
+#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
+	int cpu;
+
+	if (!vtime_accounting_enabled())
+		return;
+
+	cpu = cpu_of(rq);
+
+	if (!vtime_accounting_enabled_cpu(cpu))
+		return;
+
+	__vtime_set_nice(rq, p, old_prio);
+#endif
+}
+
 static inline void add_nr_running(struct rq *rq, unsigned count)
 {
 	unsigned prev_nr = rq->nr_running;
Signature · 1 line
2.23.0
maintainer
#4 lore raw

Re: [PATCH 3/9] sched/vtime: Handle nice updates under vtime

Quoted 56 lines · Frederic Weisbecker
kernel/sched/core.c +11 -3
index 7880f4f64d0e..fa56a1bdd5d8 100644
@@ -4488,6 +4488,7 @@ void set_user_nice(struct task_struct *p, long nice)
 	int old_prio, delta;
 	struct rq_flags rf;
 	struct rq *rq;
+	int old_static;
 
 	if (task_nice(p) == nice || nice < MIN_NICE || nice > MAX_NICE)
 		return;
@@ -4498,6 +4499,8 @@ void set_user_nice(struct task_struct *p, long nice)
 	rq = task_rq_lock(p, &rf);
 	update_rq_clock(rq);
 
+	old_static = p->static_prio;
+
 	/*
 	 * The RT priorities are set via sched_setscheduler(), but we still
 	 * allow the 'normal' nice value to be set - but as expected
@@ -4532,7 +4535,9 @@ void set_user_nice(struct task_struct *p, long nice)
 	}
 	if (running)
 		set_next_task(rq, p);
+
 out_unlock:
+	vtime_set_nice(rq, p, old_static);
 	task_rq_unlock(rq, p, &rf);
 }
 EXPORT_SYMBOL(set_user_nice);
@@ -4668,8 +4673,8 @@ static struct task_struct *find_process_by_pid(pid_t pid)
  */
 #define SETPARAM_POLICY	-1
 
-static void __setscheduler_params(struct task_struct *p,
-		const struct sched_attr *attr)
+static void __setscheduler_params(struct rq *rq, struct task_struct *p,
+				  const struct sched_attr *attr)
 {
 	int policy = attr->sched_policy;
 
@@ -4680,8 +4685,11 @@ static void __setscheduler_params(struct task_struct *p,
 
 	if (dl_policy(policy))
 		__setparam_dl(p, attr);
-	else if (fair_policy(policy))
+	else if (fair_policy(policy)) {
+		int old_prio = p->static_prio;
 		p->static_prio = NICE_TO_PRIO(attr->sched_nice);
+		vtime_set_nice(rq, p, old_prio);
+	}
 
 	/*
 	 * __sched_setscheduler() ensures attr->sched_priority == 0 when

kernel/sched/core.c static void __setscheduler(struct rq *rq, struct task_struct *p, · line 4718

Quoted 8 lines
kernel/sched/core.c +1 -1
@@ -4704,7 +4712,7 @@ static void __setscheduler(struct rq *rq, struct task_struct *p,
 	if (attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)
 		return;
 
-	__setscheduler_params(p, attr);
+	__setscheduler_params(rq, p, attr);
 
 	/*
 	 * Keep a potential priority boosting if called from

Would it make sense to add a helper like:

static void
__sched_set_nice(struct rq *rq, struct task_struct *t, long nice)

{
	int old_prio;

	old_prio = t->static_prio;
	t->static_prio = NICE_TO_PRIO(nice);
	vtime_set_nice(rq, t, old_prio);
}
maintainer
#5 lore raw

Re: [PATCH 3/9] sched/vtime: Handle nice updates under vtime

In reply to Frederic Weisbecker
Quoted 19 lines · Frederic Weisbecker

The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.

So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as CPUTIME_NICE because it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted as CPUTIME_USER and 5 seconds as CPUTIME_NICE.

In order to solve this, we now cover nice updates withing three cases:

* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.

Quoted 5 lines

* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.

But but but, I thought the idea was to _never_ send interrupts to NOHZ_FULL cpus ?!?

maintainer
#6 lore raw

Re: [PATCH 3/9] sched/vtime: Handle nice updates under vtime

Quoted 20 lines · Peter Zijlstra
Quoted 19 lines · Frederic Weisbecker

The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.

So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as CPUTIME_NICE because it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted as CPUTIME_USER and 5 seconds as CPUTIME_NICE.

In order to solve this, we now cover nice updates withing three cases:

* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.

Quoted 8 lines
Quoted 5 lines

* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.

But but but, I thought the idea was to _never_ send interrupts to NOHZ_FULL cpus ?!?

That is, isn't the cure worse than the problem? I mean, who bloody cares about silly accounting crud more than not getting interrupts on their NOHZ_FULL cpus.

#7 lore raw

Re: [PATCH 3/9] sched/vtime: Handle nice updates under vtime

Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Frederic Weisbecker

The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.

So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as CPUTIME_NICE because it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted as CPUTIME_USER and 5 seconds as CPUTIME_NICE.

In order to solve this, we now cover nice updates withing three cases:

* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.

* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.

Quoted 6 lines
Quoted 2 lines

But but but, I thought the idea was to _never_ send interrupts to NOHZ_FULL cpus ?!?

That is, isn't the cure worse than the problem? I mean, who bloody cares about silly accounting crud more than not getting interrupts on their NOHZ_FULL cpus.

Yeah indeed. I tend to sacrifice everything for correctness but perhaps we can live with small issues like nice accounting not being accounted to the right place if that can avoid disturbing nohz_full CPUs. Also who cares about renicing a task that is supposed to run alone.

So here is what I can do: I'll make a simplified version of that set which accounts on top of the task_nice() value found on accounting time (context switch, user exit, guest exit) and if some user ever complains, I can still bring back that IPI solution.

Thanks.

maintainer
#8 lore raw

Re: [PATCH 3/9] sched/vtime: Handle nice updates under vtime

Quoted 35 lines · Frederic Weisbecker
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Frederic Weisbecker

The cputime niceness is determined while checking the target's nice value at cputime accounting time. Under vtime this happens on context switches, user exit and guest exit. But nice value updates while the target is running are not taken into account.

So if a task runs tickless for 10 seconds in userspace but it has been reniced after 5 seconds from -1 (not nice) to 1 (nice), on user exit vtime will account the whole 10 seconds as CPUTIME_NICE because it only sees the latest nice value at accounting time which is 1 here. Yet it's wrong, 5 seconds should be accounted as CPUTIME_USER and 5 seconds as CPUTIME_NICE.

In order to solve this, we now cover nice updates withing three cases:

* If the nice updater is the current task, although we are in kernel mode there can be pending user or guest time in the cache to flush under the prior nice state. Account these if any. Also toggle the vtime nice flag for further user/guest cputime accounting.

* If the target runs on a different CPU, we interrupt it with an IPI to update the vtime state in place. If the task is running in user or guest, the pending cputime is accounted under the prior nice state. Then the vtime nice flag is toggled for further user/guest cputime accounting.

Quoted 6 lines
Quoted 2 lines

But but but, I thought the idea was to _never_ send interrupts to NOHZ_FULL cpus ?!?

That is, isn't the cure worse than the problem? I mean, who bloody cares about silly accounting crud more than not getting interrupts on their NOHZ_FULL cpus.

Quoted 8 lines

Yeah indeed. I tend to sacrifice everything for correctness but perhaps we can live with small issues like nice accounting not being accounted to the right place if that can avoid disturbing nohz_full CPUs. Also who cares about renicing a task that is supposed to run alone.

So here is what I can do: I'll make a simplified version of that set which accounts on top of the task_nice() value found on accounting time (context switch, user exit, guest exit) and if some user ever complains, I can still bring back that IPI solution.

Makes sense, thanks!

#9 lore raw

PATCH 4/9 sched/cputime: Support other fields on kcpustat_field()

In reply to Frederic Weisbecker

Now that we support nice updates under vtime accounting, we can provide tickless values for kcpustat readers of user and guest time.

Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/sched/cputime.c | 40 +++++++++++++++++++++++++++++++++-------
 1 file changed, 33 insertions(+), 7 deletions(-)
kernel/sched/cputime.c +33 -7
index e09194415c35..bf4b61f71194 100644
@@ -1042,6 +1042,15 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)
 	} while (read_seqcount_retry(&vtime->seqcount, seq));
 }
 
+static u64 kcpustat_user_vtime(struct vtime *vtime)
+{
+	if (vtime->state == VTIME_USER)
+		return vtime->utime + vtime_delta(vtime);
+	else if (vtime->state == VTIME_GUEST)
+		return vtime->gtime + vtime_delta(vtime);
+	return 0;
+}
+
 static int kcpustat_field_vtime(u64 *cpustat,
 				struct vtime *vtime,
 				enum cpu_usage_stat usage,
@@ -1076,9 +1085,30 @@ static int kcpustat_field_vtime(u64 *cpustat,
 
 		*val = cpustat[usage];
 
-		if (vtime->state == VTIME_SYS)
-			*val += vtime->stime + vtime_delta(vtime);
-
+		switch (usage) {
+		case CPUTIME_SYSTEM:
+			if (vtime->state == VTIME_SYS)
+				*val += vtime->stime + vtime_delta(vtime);
+			break;
+		case CPUTIME_USER:
+			if (!vtime->nice)
+				*val += kcpustat_user_vtime(vtime);
+			break;
+		case CPUTIME_NICE:
+			if (vtime->nice)
+				*val += kcpustat_user_vtime(vtime);
+			break;
+		case CPUTIME_GUEST:
+			if (vtime->state == VTIME_GUEST && !vtime->nice)
+				*val += vtime->gtime + vtime_delta(vtime);
+			break;
+		case CPUTIME_GUEST_NICE:
+			if (vtime->state == VTIME_GUEST && vtime->nice)
+				*val += vtime->gtime + vtime_delta(vtime);
+			break;
+		default:
+			break;
+		}
 	} while (read_seqcount_retry(&vtime->seqcount, seq));
 
 	return 0;
@@ -1095,10 +1125,6 @@ u64 kcpustat_field(struct kernel_cpustat *kcpustat,
 	if (!vtime_accounting_enabled_cpu(cpu))
 		return cpustat[usage];
 
-	/* Only support sys vtime for now */
-	if (usage != CPUTIME_SYSTEM)
-		return cpustat[usage];
-
 	rq = cpu_rq(cpu);
 
 	for (;;) {
Signature · 1 line
2.23.0
#10 lore raw

PATCH 5/9 sched/vtime: Bring all-in-one kcpustat accessor for vtime fields

In reply to Frederic Weisbecker

Many callsites want to fetch the values of system, user, user_nice, guest or guest_nice kcpustat fields altogether or at least a pair of these.

In that case calling kcpustat_field() for each requested field brings unecessary overhead when we could fetch all of them in a row.

So provide kcpustat_cputime() that fetches all vtime sensitive fields under the same RCU and seqcount block.

Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 include/linux/kernel_stat.h |  23 ++++++
 kernel/sched/cputime.c      | 138 ++++++++++++++++++++++++++++++------
 2 files changed, 141 insertions(+), 20 deletions(-)
include/linux/kernel_stat.h +23 -0
index 1b9b97f6946e..c76daad2d8e2 100644
@@ -78,15 +78,38 @@ static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)
 	return kstat_cpu(cpu).irqs_sum;
 }
 
+
+static inline void kcpustat_cputime_raw(u64 *cpustat, u64 *user, u64 *nice,
+					u64 *system, u64 *guest, u64 *guest_nice)
+{
+	*user = cpustat[CPUTIME_USER];
+	*nice = cpustat[CPUTIME_NICE];
+	*system = cpustat[CPUTIME_SYSTEM];
+	*guest = cpustat[CPUTIME_GUEST];
+	*guest_nice = cpustat[CPUTIME_GUEST_NICE];
+}
+
 #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
 extern u64 kcpustat_field(struct kernel_cpustat *kcpustat,
 			  enum cpu_usage_stat usage, int cpu);
+extern void kcpustat_cputime(struct kernel_cpustat *kcpustat, int cpu,
+			     u64 *user, u64 *nice, u64 *system,
+			     u64 *guest, u64 *guest_nice);
 #else
 static inline u64 kcpustat_field(struct kernel_cpustat *kcpustat,
 				 enum cpu_usage_stat usage, int cpu)
 {
 	return kcpustat->cpustat[usage];
 }
+
+static inline void kcpustat_cputime(struct kernel_cpustat *kcpustat, int cpu,
+				    u64 *user, u64 *nice, u64 *system,
+				    u64 *guest, u64 *guest_nice)
+{
+	kcpustat_cputime_raw(kcpustat->cpustat, user, nice,
+			     system, guest, guest_nice);
+}
+
 #endif
 
 extern void account_user_time(struct task_struct *, u64);
kernel/sched/cputime.c +118 -20

154 lines not on this page · raw message

Signature · 1 line
2.23.0
#11 lore raw

PATCH 6/9 procfs: Use all-in-one vtime aware kcpustat accessor

In reply to Frederic Weisbecker

Now that we can read also user and guest time safely under vtime, use the relevant accessor to fix frozen kcpustat values on nohz_full CPUs.

Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 fs/proc/stat.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
fs/proc/stat.c +10 -10
index 5c6bd0ae3802..b2ee5418dece 100644
@@ -120,18 +120,21 @@ static int show_stat(struct seq_file *p, void *v)
 	getboottime64(&boottime);
 
 	for_each_possible_cpu(i) {
+		u64 cpu_user, cpu_nice, cpu_sys, cpu_guest, cpu_guest_nice;
 		struct kernel_cpustat *kcs = &kcpustat_cpu(i);
 
-		user += kcs->cpustat[CPUTIME_USER];
-		nice += kcs->cpustat[CPUTIME_NICE];
-		system += kcpustat_field(kcs, CPUTIME_SYSTEM, i);
+		kcpustat_cputime(kcs, i, &cpu_user, &cpu_nice,
+				 &cpu_sys, &cpu_guest, &cpu_guest_nice);
+		user += cpu_user;
+		nice += cpu_nice;
+		system += cpu_sys;
 		idle += get_idle_time(kcs, i);
 		iowait += get_iowait_time(kcs, i);
 		irq += kcs->cpustat[CPUTIME_IRQ];
 		softirq += kcs->cpustat[CPUTIME_SOFTIRQ];
 		steal += kcs->cpustat[CPUTIME_STEAL];
-		guest += kcs->cpustat[CPUTIME_GUEST];
-		guest_nice += kcs->cpustat[CPUTIME_GUEST_NICE];
+		guest += cpu_guest;
+		guest_nice += guest_nice;
 		sum += kstat_cpu_irqs_sum(i);
 		sum += arch_irq_stat_cpu(i);
 
@@ -159,17 +162,14 @@ static int show_stat(struct seq_file *p, void *v)
 	for_each_online_cpu(i) {
 		struct kernel_cpustat *kcs = &kcpustat_cpu(i);
 
+		kcpustat_cputime(kcs, i, &user, &nice,
+				 &system, &guest, &guest_nice);
 		/* Copy values here to work around gcc-2.95.3, gcc-2.96 */
-		user = kcs->cpustat[CPUTIME_USER];
-		nice = kcs->cpustat[CPUTIME_NICE];
-		system = kcpustat_field(kcs, CPUTIME_SYSTEM, i);
 		idle = get_idle_time(kcs, i);
 		iowait = get_iowait_time(kcs, i);
 		irq = kcs->cpustat[CPUTIME_IRQ];
 		softirq = kcs->cpustat[CPUTIME_SOFTIRQ];
 		steal = kcs->cpustat[CPUTIME_STEAL];
-		guest = kcs->cpustat[CPUTIME_GUEST];
-		guest_nice = kcs->cpustat[CPUTIME_GUEST_NICE];
 		seq_printf(p, "cpu%d", i);
 		seq_put_decimal_ull(p, " ", nsec_to_clock_t(user));
 		seq_put_decimal_ull(p, " ", nsec_to_clock_t(nice));
Signature · 1 line
2.23.0
#12 lore raw

PATCH 7/9 cpufreq: Use vtime aware kcpustat accessors for user time

In reply to Frederic Weisbecker

We can now safely read user and guest kcpustat fields on nohz_full CPUs. Use the appropriate accessors.

Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Viresh Kumar <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
---
 drivers/cpufreq/cpufreq.c          | 13 +++++++++----
 drivers/cpufreq/cpufreq_governor.c |  6 +++---
 2 files changed, 12 insertions(+), 7 deletions(-)
drivers/cpufreq/cpufreq.c +9 -4
index a37ebfd7e0e8..cdc44a854ae1 100644
@@ -113,18 +113,23 @@ EXPORT_SYMBOL_GPL(get_governor_parent_kobj);
 
 static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall)
 {
-	u64 idle_time;
+	struct kernel_cpustat *kcpustat = &kcpustat_cpu(cpu);
+	u64 user, nice, sys, guest, guest_nice;
 	u64 cur_wall_time;
+	u64 idle_time;
 	u64 busy_time;
 
 	cur_wall_time = jiffies64_to_nsecs(get_jiffies_64());
 
-	busy_time = kcpustat_cpu(cpu).cpustat[CPUTIME_USER];
-	busy_time += kcpustat_field(&kcpustat_cpu(cpu), CPUTIME_SYSTEM, cpu);
+	kcpustat_cputime(kcpustat, cpu, &user, &nice, &sys,
+			 &guest, &guest_nice);
+
+	busy_time = user;
+	busy_time += sys;
 	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_IRQ];
 	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SOFTIRQ];
 	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
-	busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_NICE];
+	busy_time += nice;
 
 	idle_time = cur_wall_time - busy_time;
 	if (wall)
drivers/cpufreq/cpufreq_governor.c +3 -3
index 4bb054d0cb43..f99ae45efaea 100644
@@ -105,7 +105,7 @@ void gov_update_cpu_data(struct dbs_data *dbs_data)
 			j_cdbs->prev_cpu_idle = get_cpu_idle_time(j, &j_cdbs->prev_update_time,
 								  dbs_data->io_is_busy);
 			if (dbs_data->ignore_nice_load)
-				j_cdbs->prev_cpu_nice = kcpustat_cpu(j).cpustat[CPUTIME_NICE];
+				j_cdbs->prev_cpu_nice = kcpustat_field(&kcpustat_cpu(j), CPUTIME_NICE, j);
 		}
 	}
 }
@@ -149,7 +149,7 @@ unsigned int dbs_update(struct cpufreq_policy *policy)
 		j_cdbs->prev_cpu_idle = cur_idle_time;
 
 		if (ignore_nice) {
-			u64 cur_nice = kcpustat_cpu(j).cpustat[CPUTIME_NICE];
+			u64 cur_nice = kcpustat_field(&kcpustat_cpu(j), CPUTIME_NICE, j);
 
 			idle_time += div_u64(cur_nice - j_cdbs->prev_cpu_nice, NSEC_PER_USEC);
 			j_cdbs->prev_cpu_nice = cur_nice;
@@ -530,7 +530,7 @@ int cpufreq_dbs_governor_start(struct cpufreq_policy *policy)
 		j_cdbs->prev_load = 0;
 
 		if (ignore_nice)
-			j_cdbs->prev_cpu_nice = kcpustat_cpu(j).cpustat[CPUTIME_NICE];
+			j_cdbs->prev_cpu_nice = kcpustat_field(&kcpustat_cpu(j), CPUTIME_NICE, j);
 	}
 
 	gov->start(policy);
Signature · 1 line
2.23.0
#13 lore raw

PATCH 8/9 leds: Use all-in-one vtime aware kcpustat accessor

In reply to Frederic Weisbecker

We can now safely read user kcpustat fields on nohz_full CPUs. Use the appropriate accessor.

Cc Jacek Anaszewski <[email protected]>
Cc Pavel Machek <[email protected]>
Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 drivers/leds/trigger/ledtrig-activity.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/leds/trigger/ledtrig-activity.c +6 -3
index ddfc5edd07c8..20250de1138d 100644
@@ -57,9 +57,12 @@ static void led_activity_function(struct timer_list *t)
 	curr_used = 0;
 
 	for_each_possible_cpu(i) {
-		curr_used += kcpustat_cpu(i).cpustat[CPUTIME_USER]
-			  +  kcpustat_cpu(i).cpustat[CPUTIME_NICE]
-			  +  kcpustat_field(&kcpustat_cpu(i), CPUTIME_SYSTEM, i)
+		u64 user, nice, sys, guest, guest_nice;
+
+		kcpustat_cputime(&kcpustat_cpu(i), i, &user, &nice, &sys,
+				 &guest, &guest_nice);
+
+		curr_used += user + nice + sys
 			  +  kcpustat_cpu(i).cpustat[CPUTIME_SOFTIRQ]
 			  +  kcpustat_cpu(i).cpustat[CPUTIME_IRQ];
 		cpus++;
Signature · 1 line
2.23.0
#14 lore raw

PATCH 9/9 rackmeter: Use vtime aware kcpustat accessor

In reply to Frederic Weisbecker

Now that we have a vtime safe kcpustat accessor, use it to fetch CPUTIME_NICE and fix frozen kcpustat values on nohz_full CPUs.

Cc Benjamin Herrenschmidt <[email protected]>
Cc Yauheni Kaliuta <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Wanpeng Li <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 drivers/macintosh/rack-meter.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/macintosh/rack-meter.c +4 -3
index 4134e580f786..60311e8d6240 100644
@@ -81,13 +81,14 @@ static int rackmeter_ignore_nice;
  */
 static inline u64 get_cpu_idle_time(unsigned int cpu)
 {
+	struct kernel_cpustat *kcpustat = &kcpustat_cpu(cpu);
 	u64 retval;
 
-	retval = kcpustat_cpu(cpu).cpustat[CPUTIME_IDLE] +
-		 kcpustat_cpu(cpu).cpustat[CPUTIME_IOWAIT];
+	retval = kcpustat->cpustat[CPUTIME_IDLE] +
+		 kcpustat->cpustat[CPUTIME_IOWAIT];
 
 	if (rackmeter_ignore_nice)
-		retval += kcpustat_cpu(cpu).cpustat[CPUTIME_NICE];
+		retval += kcpustat_field(kcpustat, CPUTIME_NICE, cpu);
 
 	return retval;
 }
Signature · 1 line
2.23.0
#15 lore raw
In reply to Frederic Weisbecker
Quoted 8 lines · Frederic Weisbecker

So we fixed CPUTIME_SYSTEM on nohz_full, now is the turn for CPUTIME_USER, CPUTIME_NICE, CPUTIME_GUEST and CPUTIME_GUEST_NICE.

The tricky piece was to handle "nice-ness" correctly. I think I addressed most reviews (one year ago now) from peterz
(https://lore.kernel.org/lkml/[email protected]/)
Among which:

Quoted 4 lines

* Merge patches that didn't make sense together
* Clarify the changelog (I hope) and add comments
* Also hook nice updates on __setscheduler_params()
* Uninline a big function

Gentle ping :)