per-cgroup cpu-stat
Subsystems (3)
v6 cover per-cgroup cpu-stat
From: Glauber Costa <[email protected]>
Hi all,
This is an attempt to provide userspace with enough information to reconstruct per-container version of files like "/proc/stat". In particular, we are interested in knowing the per-cgroup slices of user time, system time, wait time, number of processes, and a variety of statistics.
This task is made more complicated by the fact that multiple controllers are involved in collecting those statistics: cpu and cpuacct. So the first thing I am doing here, is ressurecting Tejun's patches that aim at deprecating cpuacct.
This is one of the major differences from earlier attempts: all data is provided by the cpu controller, resulting in greater simplicity. Please note, however, that this patchset only goes as far as deprecating it: the cpuacct can still be mounted separately from the cpu cgroup if the user so whishes.
This also tries to hook into the existing scheduler hierarchy walks instead of providing new ones.
Glauber Costa (8):
don't call cpuacct_charge in stop_task.c
sched: adjust exec_clock to use it as cpu usage metric
cpuacct: don't actually do anything.
sched: document the cpu cgroup.
sched: account guest time per-cgroup as well.
sched: record per-cgroup number of context switches
sched: change nr_context_switches calculation.
sched: introduce cgroup file stat_percpu
Peter Zijlstra (1):
sched: Push put_prev_task() into pick_next_task()
Tejun Heo (3):
cgroup: implement CFTYPE_NO_PREFIX
cgroup, sched: let cpu serve the same files as cpuacct cgroup, sched: deprecate cpuacct
Documentation/cgroups/cpu.txt | 100 +++++++++++ include/linux/cgroup.h | 1 + include/linux/sched.h | 8 +- init/Kconfig | 11 +- kernel/cgroup.c | 57 ++++++- kernel/sched/core.c | 387 ++++++++++++++++++++++++++++++++++++++++-- kernel/sched/cputime.c | 29 +++- kernel/sched/fair.c | 39 ++++- kernel/sched/idle_task.c | 9 +- kernel/sched/rt.c | 42 +++-- kernel/sched/sched.h | 28 ++- kernel/sched/stop_task.c | 8 +- 12 files changed, 672 insertions(+), 47 deletions(-) create mode 100644 Documentation/cgroups/cpu.txt
Signature · 1 line
1.8.1
PATCH v6 1/12 don't call cpuacct_charge in stop_task.c
From: Glauber Costa <[email protected]>
Commit 8f618968 changed stop_task to do the same bookkeping as the other classes. However, the call to cpuacct_charge() doesn't affect the scheduler decisions at all, and doesn't need to be moved over.
Moreover, being a kthread, the migration thread won't belong to any cgroup anyway, rendering this call quite useless.
--- kernel/sched/stop_task.c | 1 - 1 file changed, 1 deletion(-)
kernel/sched/stop_task.c +0 -1
@@ -68,7 +68,6 @@ static void put_prev_task_stop(struct rq *rq, struct task_struct *prev)account_group_exec_runtime (curr, delta_exec); curr->se.exec_start = rq->clock_task; -cpuacct_charge (curr, delta_exec); }static void task_tick_stop (struct rq *rq,struct task_struct *curr,int queued)
Signature · 1 line
1.8.1
PATCH v6 2/12 cgroup: implement CFTYPE_NO_PREFIX
In reply to Lord Glauber Costa of SealandFrom: Tejun Heo <[email protected]>
When cgroup files are created, cgroup core automatically prepends the name of the subsystem as prefix. This patch adds CFTYPE_NO_PREFIX which disables the automatic prefix.
This will be used to deprecate cpuacct which will make cpu create and serve the cpuacct files.
--- include/linux/cgroup.h | 1 + kernel/cgroup.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/cgroup.h +1 -0
@@ -277,6 +277,7 @@ struct cgroup_map_cb {/* cftype->flags */ #define CFTYPE_ONLY_ON_ROOT (1U << 0)/* only create on root cg */ #define CFTYPE_NOT_ON_ROOT (1U << 1)/* don't create on root cg */ +#define CFTYPE_NO_PREFIX (1U << 2)/* skip subsys prefix */ #define MAX_CFTYPE_NAME 64
kernel/cgroup.c +2 -1
@@ -2723,7 +2723,8 @@ static int cgroup_add_file(struct cgroup *cgrp, struct cgroup_subsys *subsys,simple_xattrs_init (&cft->xattrs); -if (subsys && !test_bit (ROOT_NOPREFIX , &cgrp->root->flags)) { +if (subsys && !(cft->flags &CFTYPE_NO_PREFIX ) && + !test_bit (ROOT_NOPREFIX , &cgrp->root->flags)) {strcpy (name, subsys->name);strcat (name,"." ); }
Signature · 1 line
1.8.1
PATCH v6 3/12 cgroup, sched: let cpu serve the same files as cpuacct
In reply to Lord Glauber Costa of SealandFrom: Tejun Heo <[email protected]>
cpuacct being on a separate hierarchy is one of the main cgroup related complaints from scheduler side and the consensus seems to be
* Allowing cpuacct to be a separate controller was a mistake. In general multiple controllers on the same type of resource should be avoided, especially accounting-only ones.
* Statistics provided by cpuacct are useful and should instead be served by cpu.
This patch makes cpu maintain and serve all cpuacct.* files and make cgroup core ignore cpuacct if it's co-mounted with cpu. This is a step in deprecating cpuacct. The next patch will allow disabling or dropping cpuacct without affecting userland too much.
Note that this creates some discrepancies in /proc/cgroups and /proc/PID/cgroup. The co-mounted cpuacct won't be reflected correctly there. cpuacct will eventually be removed completely probably except for the statistics filenames and I'd like to keep the amount of compatbility hackery to minimum as much as possible.
The cpu statistics implementation isn't optimized in any way. It's mostly verbatim copy from cpuacct. The goal is allowing quick disabling and removal of CONFIG_CGROUP_CPUACCT and creating a base on top of which cpu can implement proper optimization.
[ glommer: don't call *_charge in stop_task.c ]
--- kernel/cgroup.c | 13 ++++ kernel/sched/core.c | 173 +++++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/cputime.c | 19 +++++- kernel/sched/fair.c | 1 + kernel/sched/rt.c | 1 + kernel/sched/sched.h | 7 ++ 6 files changed, 212 insertions(+), 2 deletions(-)
kernel/cgroup.c +13 -0
@@ -1248,6 +1248,19 @@ static int parse_cgroupfs_options(char *data, struct cgroup_sb_opts *opts)/* Consistency checks */ /* +* cpuacct is deprecated and cpu will serve the same stat files. +* If co-mount with cpu is requested, ignore cpuacct. Note that +* this creates some discrepancies in /proc/cgroups and +* /proc/PID/cgroup. +* +* https://lkml.org/lkml/2012/9/13/542 +*/ +#if IS_ENABLED (CONFIG_CGROUP_SCHED ) &&IS_ENABLED (CONFIG_CGROUP_CPUACCT ) +if ((opts->subsys_bits & (1 << cpu_cgroup_subsys_id)) && + (opts->subsys_bits & (1 << cpuacct_subsys_id))) + opts->subsys_bits &= ~(1 << cpuacct_subsys_id); +#endif +/* * Option noprefix was introduced just for backward compatibility * with the old cpuset, so we allow noprefix only if mounting just * the cpuset subsystem.
kernel/sched/core.c +173 -0
215 lines not on this page · raw message
kernel/sched/cputime.c +17 -2
@@ -114,8 +114,10 @@ static int irqtime_account_si_update(void)static inline void task_group_account_field (struct task_struct *p,int index,u64 tmp) { +#ifdef CONFIG_CGROUP_SCHED +struct task_group *tg; +#endif #ifdef CONFIG_CGROUP_CPUACCT -struct kernel_cpustat *kcpustat;struct cpuacct *ca;#endif /*
@@ -125,7 +127,19 @@ static inline void task_group_account_field(struct task_struct *p, int index,* */ __get_cpu_var (kernel_cpustat).cpustat[index] += tmp; +#ifdef CONFIG_CGROUP_SCHED +rcu_read_lock (); + tg =container_of (task_subsys_state (p, cpu_cgroup_subsys_id), + struct task_group, css); +while (tg && (tg != &root_task_group)) { +struct kernel_cpustat *kcpustat =this_cpu_ptr (tg->cpustat); + + kcpustat->cpustat[index] += tmp; + tg = tg->parent; + } +rcu_read_unlock (); +#endif #ifdef CONFIG_CGROUP_CPUACCT if (unlikely (!cpuacct_subsys.active))return ;
@@ -133,7 +147,8 @@ static inline void task_group_account_field(struct task_struct *p, int index,rcu_read_lock (); ca =task_ca (p);while (ca && (ca != &root_cpuacct)) { - kcpustat =this_cpu_ptr (ca->cpustat); +struct kernel_cpustat *kcpustat =this_cpu_ptr (ca->cpustat); + kcpustat->cpustat[index] += tmp; ca =parent_ca (ca); }
kernel/sched/fair.c +1 -0
@@ -707,6 +707,7 @@ static void update_curr(struct cfs_rq *cfs_rq)struct task_struct *curtask =task_of (curr);trace_sched_stat_runtime (curtask, delta_exec, curr->vruntime); +task_group_charge (curtask, delta_exec);cpuacct_charge (curtask, delta_exec);account_group_exec_runtime (curtask, delta_exec); }
kernel/sched/rt.c +1 -0
@@ -935,6 +935,7 @@ static void update_curr_rt(struct rq *rq)account_group_exec_runtime (curr, delta_exec); curr->se.exec_start = rq->clock_task; +task_group_charge (curr, delta_exec);cpuacct_charge (curr, delta_exec);sched_rt_avg_update (rq, delta_exec);
kernel/sched/sched.h +7 -0
@@ -104,6 +104,10 @@ struct cfs_bandwidth {struct task_group {struct cgroup_subsys_state css; +/* statistics */ +u64 __percpu *cpuusage; +struct kernel_cpustat __percpu *cpustat; +#ifdef CONFIG_FAIR_GROUP_SCHED /* schedulable entities of this group on each cpu */ struct sched_entity **se;
@@ -590,6 +594,8 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)#endif } +extern void task_group_charge (struct task_struct *tsk,u64 cputime); +#else /* CONFIG_CGROUP_SCHED */ static inline void set_task_rq (struct task_struct *p,unsigned int cpu) { }
@@ -597,6 +603,7 @@ static inline struct task_group *task_group(struct task_struct *p) {return NULL ; } +static inline void task_group_charge (struct task_struct *tsk,u64 cputime) { }#endif /* CONFIG_CGROUP_SCHED */
Signature · 1 line
1.8.1
PATCH v6 4/12 cgroup, sched: deprecate cpuacct
In reply to Lord Glauber Costa of SealandFrom: Tejun Heo <[email protected]>
Now that cpu serves the same files as cpuacct and using cpuacct separately from cpu is deprecated, we can deprecate cpuacct. To avoid disturbing userland which has been co-mounting cpu and cpuacct, implement some hackery in cgroup core so that cpuacct co-mounting still works even if cpuacct is disabled.
The goal of this patch is to accelerate disabling and removal of cpuacct by decoupling kernel-side deprecation from userland changes. Userland is recommended to do the following.
* If /proc/cgroups lists cpuacct, always co-mount it with cpu under e.g. /sys/fs/cgroup/cpu.
* Optionally create symlinks for compatibility -
e.g. /sys/fs/cgroup/cpuacct and /sys/fs/cgroup/cpu,cpucct both pointing to /sys/fs/cgroup/cpu - whether cpuacct exists or not.
This compatibility hack will eventually go away.
[ [email protected]: subsys_bits => subsys_mask ]
--- init/Kconfig | 11 ++++++++++- kernel/cgroup.c | 47 ++++++++++++++++++++++++++++++++++++++++++----- kernel/sched/core.c | 2 ++ 3 files changed, 54 insertions(+), 6 deletions(-)
init/Kconfig +10 -1
@@ -815,11 +815,20 @@ config PROC_PID_CPUSET default y config CGROUP_CPUACCT - bool "Simple CPU accounting cgroup subsystem" + bool "DEPRECATED: Simple CPU accounting cgroup subsystem" + default n help Provides a simple Resource Controller for monitoring the total CPU consumed by the tasks in a cgroup. + This cgroup subsystem is deprecated. The CPU cgroup + subsystem serves the same accounting files and "cpuacct" + mount option is ignored if specified with "cpu". As long as + userland co-mounts cpu and cpuacct, disabling this + controller should be mostly unnoticeable - one notable + difference is that /proc/PID/cgroup won't list cpuacct + anymore. + config RESOURCE_COUNTERS bool "Resource counters" help
kernel/cgroup.c +42 -5
87 lines not on this page · raw message
kernel/sched/core.c +2 -0
@@ -8110,6 +8110,8 @@ struct cgroup_subsys cpu_cgroup_subsys = {#ifdef CONFIG_CGROUP_CPUACCT +#warning CONFIG_CGROUP_CPUACCT is deprecated, read the Kconfig help message +/* * CPU accounting code for task groups. *
Signature · 1 line
1.8.1
PATCH v6 5/12 sched: adjust exec_clock to use it as cpu usage metric
In reply to Lord Glauber Costa of SealandFrom: Glauber Costa <[email protected]>
exec_clock already provides per-group cpu usage metrics, and can be reused by cpuacct in case cpu and cpuacct are comounted.
However, it is only provided by tasks in fair class. Doing the same for rt is easy, and can be done in an already existing hierarchy loop. This is an improvement over the independent hierarchy walk executed by cpuacct.
--- kernel/sched/rt.c | 1 + kernel/sched/sched.h | 3 +++ 2 files changed, 4 insertions(+)
kernel/sched/rt.c +1 -0
@@ -945,6 +945,7 @@ static void update_curr_rt(struct rq *rq)for_each_sched_rt_entity (rt_se) { rt_rq =rt_rq_of_se (rt_se); +schedstat_add (rt_rq, exec_clock, delta_exec);if (sched_rt_runtime (rt_rq) !=RUNTIME_INF ) {raw_spin_lock (&rt_rq->rt_runtime_lock);
kernel/sched/sched.h +3 -0
@@ -210,6 +210,7 @@ struct cfs_rq {unsigned int nr_running, h_nr_running;u64 exec_clock; +u64 prev_exec_clock;u64 min_vruntime;#ifndef CONFIG_64BIT u64 min_vruntime_copy;
@@ -312,6 +313,8 @@ struct rt_rq {struct plist_head pushable_tasks;#endif int rt_throttled; +u64 exec_clock; +u64 prev_exec_clock;u64 rt_time;u64 rt_runtime;/* Nests inside the rq lock: */
Signature · 1 line
1.8.1
PATCH v6 6/12 cpuacct: don't actually do anything.
In reply to Lord Glauber Costa of SealandFrom: Glauber Costa <[email protected]>
All the information we have that is needed for cpuusage (and cpuusage_percpu) is present in schedstats. It is already recorded in a sane hierarchical way.
If we have CONFIG_SCHEDSTATS, we don't really need to do any extra work. All former functions become empty inlines.
--- kernel/sched/core.c | 102 ++++++++++++++++++++++++++++++++++++++++++--------- kernel/sched/sched.h | 10 +++-- 2 files changed, 90 insertions(+), 22 deletions(-)
kernel/sched/core.c +84 -18
@@ -7267,6 +7267,7 @@ void sched_move_task(struct task_struct *tsk)task_rq_unlock (rq, tsk, &flags); } +#ifndef CONFIG_SCHEDSTATS void task_group_charge (struct task_struct *tsk,u64 cputime) {struct task_group *tg;
@@ -7284,6 +7285,7 @@ void task_group_charge(struct task_struct *tsk, u64 cputime)rcu_read_unlock (); } +#endif #endif /* CONFIG_CGROUP_SCHED */ #if defined(CONFIG_RT_GROUP_SCHED ) || defined(CONFIG_CFS_BANDWIDTH )
@@ -7640,22 +7642,92 @@ cpu_cgroup_exit(struct cgroup *cgrp, struct cgroup *old_cgrp,sched_move_task (task); } -static u64 task_group_cpuusage_read (struct task_group *tg,int cpu) +/* +* Take rq->lock to make 64-bit write safe on 32-bit platforms. +*/ +static inline void lock_rq_dword (int cpu) { -u64 *cpuusage =per_cpu_ptr (tg->cpuusage, cpu); -u64 data; -#ifndef CONFIG_64BIT -/* -* Take rq->lock to make 64-bit read safe on 32-bit platforms. -*/ raw_spin_lock_irq (&cpu_rq (cpu)->lock); - data = *cpuusage; +#endif +} + +static inline void unlock_rq_dword (int cpu) +{ +#ifndef CONFIG_64BIT raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +#endif +} + +#ifdef CONFIG_SCHEDSTATS +#ifdef CONFIG_FAIR_GROUP_SCHED +static inline u64 cfs_exec_clock (struct task_group *tg,int cpu) +{ +return tg->cfs_rq[cpu]->exec_clock - tg->cfs_rq[cpu]->prev_exec_clock; +} + +static inline void cfs_exec_clock_reset (struct task_group *tg,int cpu) +{ + tg->cfs_rq[cpu]->prev_exec_clock = tg->cfs_rq[cpu]->exec_clock; +}#else - data = *cpuusage; +static inline u64 cfs_exec_clock (struct task_group *tg,int cpu) +{ +} + +static inline void cfs_exec_clock_reset (struct task_group *tg,int cpu) +{ +} +#endif +#ifdef CONFIG_RT_GROUP_SCHED +static inline u64 rt_exec_clock (struct task_group *tg,int cpu) +{ +return tg->rt_rq[cpu]->exec_clock - tg->rt_rq[cpu]->prev_exec_clock; +} + +static inline void rt_exec_clock_reset (struct task_group *tg,int cpu) +{ + tg->rt_rq[cpu]->prev_exec_clock = tg->rt_rq[cpu]->exec_clock; +} +#else +static inline u64 rt_exec_clock (struct task_group *tg,int cpu) +{ +return 0 ; +} + +static inline void rt_exec_clock_reset (struct task_group *tg,int cpu) +{ +}#endif +static u64 task_group_cpuusage_read (struct task_group *tg,int cpu) +{ +u64 ret =0 ; + +lock_rq_dword (cpu); + ret =cfs_exec_clock (tg, cpu) +rt_exec_clock (tg, cpu); +unlock_rq_dword (cpu); + +return ret; +} + +static void task_group_cpuusage_write (struct task_group *tg,int cpu,u64 val) +{ +lock_rq_dword (cpu); +cfs_exec_clock_reset (tg, cpu); +rt_exec_clock_reset (tg, cpu); +unlock_rq_dword (cpu); +} +#else +static u64 task_group_cpuusage_read (struct task_group *tg,int cpu) +{ +u64 *cpuusage =per_cpu_ptr (tg->cpuusage, cpu); +u64 data; + +lock_rq_dword (cpu); + data = *cpuusage; +unlock_rq_dword (cpu); +return data; }
@@ -7663,17 +7735,11 @@ static void task_group_cpuusage_write(struct task_group *tg, int cpu, u64 val) {u64 *cpuusage =per_cpu_ptr (tg->cpuusage, cpu); -#ifndef CONFIG_64BIT -/* -* Take rq->lock to make 64-bit write safe on 32-bit platforms. -*/ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +lock_rq_dword (cpu); *cpuusage = val; -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); -#else - *cpuusage = val; -#endif +unlock_rq_dword (cpu); } +#endif /* return total cpu usage (in nanoseconds) of a group */ static u64 cpucg_cpuusage_read (struct cgroup *cgrp,struct cftype *cft)
kernel/sched/sched.h +6 -4
@@ -597,8 +597,6 @@ static inline void set_task_rq(struct task_struct *p, unsigned int cpu)#endif } -extern void task_group_charge (struct task_struct *tsk,u64 cputime); -#else /* CONFIG_CGROUP_SCHED */ static inline void set_task_rq (struct task_struct *p,unsigned int cpu) { }
@@ -606,10 +604,14 @@ static inline struct task_group *task_group(struct task_struct *p) {return NULL ; } -static inline void task_group_charge (struct task_struct *tsk,u64 cputime) { } -#endif /* CONFIG_CGROUP_SCHED */ +#if defined(CONFIG_CGROUP_SCHED ) && !defined(CONFIG_SCHEDSTATS ) +extern void task_group_charge (struct task_struct *tsk,u64 cputime); +#else +static inline void task_group_charge (struct task_struct *tsk,u64 cputime) {} +#endif +static inline void __set_task_cpu (struct task_struct *p,unsigned int cpu) {set_task_rq (p, cpu);
Signature · 1 line
1.8.1
PATCH v6 7/12 sched: document the cpu cgroup.
In reply to Lord Glauber Costa of SealandFrom: Glauber Costa <[email protected]>
The CPU cgroup is so far, undocumented. Although data exists in the Documentation directory about its functioning, it is usually spread, and/or presented in the context of something else. This file consolidates all cgroup-related information about it.
--- Documentation/cgroups/cpu.txt | 82 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 Documentation/cgroups/cpu.txt
Documentation/cgroups/cpu.txt new file +82 -0
@@ -0,0 +1,82 @@ +CPU Controller +-------------- + +The CPU controller is responsible for grouping tasks together that will be +viewed by the scheduler as a single unit. The CFS scheduler will first divide +CPU time equally between all entities in the same level, and then proceed by +doing the same in the next level. Basic use cases for that are described in the +main cgroup documentation file, cgroups.txt. + +Users of this functionality should be aware that deep hierarchies will of +course impose scheduler overhead, since the scheduler will have to take extra +steps and look up additional data structures to make its final decision. + +Through the CPU controller, the scheduler is also able to cap the CPU +utilization of a particular group. This is particularly useful in environments +in which CPU is paid for by the hour, and one values predictability over +performance. + +CPU Accounting +-------------- + +The CPU cgroup will also provide additional files under the prefix "cpuacct". +Those files provide accounting statistics and were previously provided by the +separate cpuacct controller. Although the cpuacct controller will still be kept +around for compatibility reasons, its usage is discouraged. If both the CPU and +cpuacct controllers are present in the system, distributors are encouraged to +always mount them together. + +Files +----- + +The CPU controller exposes the following files to the user: + + - cpu.shares: The weight of each group living in the same hierarchy, that + translates into the amount of CPU it is expected to get. Upon cgroup creation, + each group gets assigned a default of 1024. The percentage of CPU assigned to + the cgroup is the value of shares divided by the sum of all shares in all + cgroups in the same level. + + - cpu.cfs_period_us: The duration in microseconds of each scheduler period, for + bandwidth decisions. This defaults to 100000us or 100ms. Larger periods will + improve throughput at the expense of latency, since the scheduler will be able + to sustain a cpu-bound workload for longer. The opposite of true for smaller + periods. Note that this only affects non-RT tasks that are scheduled by the + CFS scheduler. + +- cpu.cfs_quota_us: The maximum time in microseconds during each cfs_period_us + in for the current group will be allowed to run. For instance, if it is set to + half of cpu_period_us, the cgroup will only be able to peak run for 50 % of + the time. One should note that this represents aggregate time over all CPUs + in the system. Therefore, in order to allow full usage of two CPUs, for + instance, one should set this value to twice the value of cfs_period_us. + +- cpu.stat: statistics about the bandwidth controls. No data will be presented + if cpu.cfs_quota_us is not set. The file presents three + numbers: + nr_periods: how many full periods have been elapsed. + nr_throttled: number of times we exausted the full allowed bandwidth + throttled_time: total time the tasks were not run due to being overquota + + - cpu.rt_runtime_us and cpu.rt_period_us: Those files are the RT-tasks + analogous to the CFS files cfs_quota_us and cfs_period_us. One important + difference, though, is that while the cfs quotas are upper bounds that + won't necessarily be met, the rt runtimes form a stricter guarantee. + Therefore, no overlap is allowed. Implications of that are that given a + hierarchy with multiple children, the sum of all rt_runtime_us may not exceed + the runtime of the parent. Also, a rt_runtime_us of 0, means that no rt tasks + can ever be run in this cgroup. For more information about rt tasks runtime + assignments, see scheduler/sched-rt-group.txt + + - cpuacct.usage: The aggregate CPU time, in nanoseconds, consumed by all tasks + in this group. + + - cpuacct.usage_percpu: The CPU time, in nanoseconds, consumed by all tasks in + this group, separated by CPU. The format is an space-separated array of time + values, one for each present CPU. + + - cpuacct.stat: aggregate user and system time consumed by tasks in this group. + The format is + user: x + system: y +
Signature · 1 line
1.8.1
PATCH v6 8/12 sched: account guest time per-cgroup as well.
In reply to Lord Glauber Costa of SealandFrom: Glauber Costa <[email protected]>
We already track multiple tick statistics per-cgroup, using the task_group_account_field facility. This patch accounts guest_time in that manner as well.
--- kernel/sched/cputime.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
kernel/sched/cputime.c +4 -6
@@ -190,8 +190,6 @@ void account_user_time(struct task_struct *p, cputime_t cputime,static void account_guest_time (struct task_struct *p,cputime_t cputime,cputime_t cputime_scaled) { -u64 *cpustat = kcpustat_this_cpu->cpustat; -/* Add guest time to process. */ p->utime += cputime; p->utimescaled += cputime_scaled;
@@ -200,11 +198,11 @@ static void account_guest_time(struct task_struct *p, cputime_t cputime,/* Add guest time to cpustat. */ if (TASK_NICE (p) >0 ) { - cpustat[CPUTIME_NICE ] += (__force u64 ) cputime; - cpustat[CPUTIME_GUEST_NICE ] += (__force u64 ) cputime; +task_group_account_field (p,CPUTIME_NICE , (__force u64 ) cputime); +task_group_account_field (p,CPUTIME_GUEST , (__force u64 ) cputime); }else { - cpustat[CPUTIME_USER ] += (__force u64 ) cputime; - cpustat[CPUTIME_GUEST ] += (__force u64 ) cputime; +task_group_account_field (p,CPUTIME_USER , (__force u64 ) cputime); +task_group_account_field (p,CPUTIME_GUEST , (__force u64 ) cputime); } }
Signature · 1 line
1.8.1
PATCH v6 9/12 sched: Push put_prev_task() into pick_next_task()
In reply to Lord Glauber Costa of SealandFrom: Peter Zijlstra <[email protected]>
In order to avoid having to do put/set on a whole cgroup hierarchy when we context switch, push the put into pick_next_task() so that both operations are in the same function. Further changes then allow us to possibly optimize away redundant work.
[ [email protected]: incorporated mailing list feedback ]
--- include/linux/sched.h | 8 +++++++- kernel/sched/core.c | 20 +++++++------------- kernel/sched/fair.c | 6 +++++- kernel/sched/idle_task.c | 6 +++++- kernel/sched/rt.c | 27 ++++++++++++++++----------- kernel/sched/stop_task.c | 5 ++++- 6 files changed, 44 insertions(+), 28 deletions(-)
include/linux/sched.h +7 -1
@@ -1082,7 +1082,13 @@ struct sched_class {void (*check_preempt_curr) (struct rq *rq,struct task_struct *p,int flags); -struct task_struct * (*pick_next_task) (struct rq *rq); +/* +* It is the responsibility of the pick_next_task() method that will +* return the next task to call put_prev_task() on the @prev task or +* something equivalent. +*/ +struct task_struct * (*pick_next_task) (struct rq *rq, +struct task_struct *prev);void (*put_prev_task) (struct rq *rq,struct task_struct *p);#ifdef CONFIG_SMP
kernel/sched/core.c +7 -13
@@ -2807,18 +2807,11 @@ static inline void schedule_debug(struct task_struct *prev)schedstat_inc (this_rq (), sched_count); } -static void put_prev_task (struct rq *rq,struct task_struct *prev) -{ -if (prev->on_rq || rq->skip_clock_update <0 ) -update_rq_clock (rq); - prev->sched_class->put_prev_task (rq, prev); -} -/* * Pick up the highest-prio task: */ static inline struct task_struct * -pick_next_task (struct rq *rq) +pick_next_task (struct rq *rq,struct task_struct *prev) {const struct sched_class *class;struct task_struct *p;
@@ -2828,13 +2821,13 @@ pick_next_task(struct rq *rq)* the fair class we can call that function directly: */ if (likely (rq->nr_running == rq->cfs.h_nr_running)) { - p = fair_sched_class.pick_next_task (rq); + p = fair_sched_class.pick_next_task (rq, prev);if (likely (p))return p; }for_each_class (class) { - p = class->pick_next_task (rq); + p = class->pick_next_task (rq, prev);if (p)return p; }
@@ -2929,8 +2922,9 @@ need_resched:if (unlikely (!rq->nr_running))idle_balance (cpu, rq); -put_prev_task (rq, prev); - next =pick_next_task (rq); +if (prev->on_rq || rq->skip_clock_update <0 ) +update_rq_clock (rq); + next =pick_next_task (rq, prev);clear_tsk_need_resched (prev); rq->skip_clock_update =0 ;
@@ -4880,7 +4874,7 @@ static void migrate_tasks(unsigned int dead_cpu)if (rq->nr_running ==1 )break ; - next =pick_next_task (rq); + next =pick_next_task (rq,NULL );BUG_ON (!next); next->sched_class->put_prev_task (rq, next);
kernel/sched/fair.c +5 -1
@@ -3595,7 +3595,8 @@ preempt:set_last_buddy (se); } -static struct task_struct *pick_next_task_fair (struct rq *rq) +static struct task_struct * +pick_next_task_fair (struct rq *rq,struct task_struct *prev) {struct task_struct *p;struct cfs_rq *cfs_rq = &rq->cfs;
@@ -3604,6 +3605,9 @@ static struct task_struct *pick_next_task_fair(struct rq *rq)if (!cfs_rq->nr_running)return NULL ; +if (prev) + prev->sched_class->put_prev_task (rq, prev); + do { se = pick_next_entity(cfs_rq); set_next_entity(cfs_rq, se);
kernel/sched/idle_task.c +5 -1
@@ -22,8 +22,12 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int flresched_task (rq->idle); } -static struct task_struct *pick_next_task_idle (struct rq *rq) +static struct task_struct * +pick_next_task_idle (struct rq *rq,struct task_struct *prev) { +if (prev) + prev->sched_class->put_prev_task (rq, prev); +schedstat_inc (rq, sched_goidle);return rq->idle; }
kernel/sched/rt.c +16 -11
@@ -1368,15 +1368,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq) {struct sched_rt_entity *rt_se;struct task_struct *p; -struct rt_rq *rt_rq; - - rt_rq = &rq->rt; - -if (!rt_rq->rt_nr_running) -return NULL ; - -if (rt_rq_throttled (rt_rq)) -return NULL ; +struct rt_rq *rt_rq = &rq->rt; do { rt_se = pick_next_rt_entity(rq, rt_rq);
@@ -1390,9 +1382,22 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)return p; } -static struct task_struct *pick_next_task_rt (struct rq *rq) +static struct task_struct * +pick_next_task_rt (struct rq *rq,struct task_struct *prev) { -struct task_struct *p =_pick_next_task_rt (rq); +struct task_struct *p; +struct rt_rq *rt_rq = &rq->rt; + +if (!rt_rq->rt_nr_running) +return NULL ; + +if (rt_rq_throttled (rt_rq)) +return NULL ; + +if (prev) + prev->sched_class->put_prev_task (rq, prev); + + p =_pick_next_task_rt (rq);/* The running task is never eligible for pushing */ if (p)
kernel/sched/stop_task.c +4 -1
@@ -23,12 +23,15 @@ check_preempt_curr_stop(struct rq *rq, struct task_struct *p, int flags)/* we're never preempted */ } -static struct task_struct *pick_next_task_stop (struct rq *rq) +static struct task_struct * +pick_next_task_stop (struct rq *rq,struct task_struct *prev) {struct task_struct *stop = rq->stop;if (stop && stop->on_rq) { stop->se.exec_start = rq->clock_task; +if (prev) + prev->sched_class->put_prev_task (rq, prev);return stop; }
Signature · 1 line
1.8.1
PATCH v6 10/12 sched: record per-cgroup number of context switches
In reply to Lord Glauber Costa of SealandFrom: Glauber Costa <[email protected]>
Context switches are, to this moment, a property of the runqueue. When running containers, we would like to be able to present a separate figure for each container (or cgroup, in this context).
The chosen way to accomplish this is to increment a per cfs_rq or rt_rq, depending on the task, for each of the sched entities involved, up to the parent. It is trivial to note that for the parent cgroup, we always add 1 by doing this. Also, we are not introducing any hierarchy walks in here. An already existent walk is reused.
There are, however, two main issues:
1. the traditional context switch code only increment nr_switches when a different task is being inserted in the rq. Eventually, albeit not likely, we will pick the same task as before. Since for cfq and rt we only now which task will be next after the walk, we need to do the walk again, decrementing 1. Since this is by far not likely, it seems a fair price to pay.
2. Those figures do not include switches from and to the idle or stop task. Those need to be recorded separately, which will happen in a follow up patch.
--- kernel/sched/fair.c | 18 ++++++++++++++++++ kernel/sched/rt.c | 15 +++++++++++++-- kernel/sched/sched.h | 3 +++ 3 files changed, 34 insertions(+), 2 deletions(-)
kernel/sched/fair.c +18 -0
@@ -3609,6 +3609,8 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev) prev->sched_class->put_prev_task (rq, prev); do { + if (likely(prev)) + cfs_rq->nr_switches++; se = pick_next_entity(cfs_rq); set_next_entity(cfs_rq, se); cfs_rq = group_cfs_rq(se);
@@ -3618,6 +3620,22 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev)if (hrtick_enabled (rq))hrtick_start_fair (rq, p); +/* +* This condition is extremely unlikely, and most of the time will just +* consist of this unlikely branch, which is extremely cheap. But we +* still need to have it, because when we first loop through cfs_rq's, +* we can't possibly know which task we will pick. The call to +* set_next_entity above is not meant to mess up the tree in this case, +* so this should give us the same chain, in the same order. +*/ +if (unlikely (p == prev)) { + se = &p->se; +for_each_sched_entity (se) { + cfs_rq =cfs_rq_of (se); + cfs_rq->nr_switches--; + } + } +return p; }
kernel/sched/rt.c +13 -2
@@ -1364,13 +1364,16 @@ static struct sched_rt_entity *pick_next_rt_entity(struct rq *rq,return next; } -static struct task_struct *_pick_next_task_rt (struct rq *rq) +static struct task_struct * +_pick_next_task_rt (struct rq *rq,struct task_struct *prev) {struct sched_rt_entity *rt_se;struct task_struct *p;struct rt_rq *rt_rq = &rq->rt; do { + if (likely(prev)) + rt_rq->rt_nr_switches++; rt_se = pick_next_rt_entity(rq, rt_rq); BUG_ON(!rt_se); rt_rq = group_rt_rq(rt_se);
@@ -1379,6 +1382,14 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq) p =rt_task_of (rt_se); p->se.exec_start = rq->clock_task; +/* See fair.c for an explanation on this */ +if (unlikely (p == prev)) { +for_each_sched_rt_entity (rt_se) { + rt_rq =rt_rq_of_se (rt_se); + rt_rq->rt_nr_switches--; + } + } +return p; }
@@ -1397,7 +1408,7 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev)if (prev) prev->sched_class->put_prev_task (rq, prev); - p =_pick_next_task_rt (rq); + p =_pick_next_task_rt (rq, prev);/* The running task is never eligible for pushing */ if (p)
kernel/sched/sched.h +3 -0
@@ -229,6 +229,7 @@ struct cfs_rq {unsigned int nr_spread_over;#endif +u64 nr_switches;#ifdef CONFIG_SMP /* * Load-tracking only depends on SMP, FAIR_GROUP_SCHED dependency below may be
@@ -298,6 +299,8 @@ static inline int rt_bandwidth_enabled(void)struct rt_rq {struct rt_prio_array active;unsigned int rt_nr_running; +u64 rt_nr_switches; + #if defined CONFIG_SMP || defined CONFIG_RT_GROUP_SCHED struct { int curr; /* highest queued rt task prio */
Signature · 1 line
1.8.1
PATCH v6 11/12 sched: change nr_context_switches calculation.
In reply to Lord Glauber Costa of SealandFrom: Glauber Costa <[email protected]>
This patch changes the calculation of nr_context_switches. The variable "nr_switches" is now used to account for the number of transition to the idle task, or stop task. It is removed from the schedule() path.
The total calculation can be made using the fact that the transitions to fair and rt classes are recorded in the root_task_group. One can easily derive the total figure by adding those quantities together.
--- kernel/sched/core.c | 17 +++++++++++++++-- kernel/sched/idle_task.c | 3 +++ kernel/sched/stop_task.c | 2 ++ 3 files changed, 20 insertions(+), 2 deletions(-)
kernel/sched/core.c +15 -2
@@ -2001,13 +2001,27 @@ unsigned long nr_uninterruptible(void)return sum; } +#ifdef CONFIG_FAIR_GROUP_SCHED +#define cfs_nr_switches (tg, cpu) (tg)->cfs_rq[cpu]->nr_switches +#else +#define cfs_nr_switches (tg, cpu) cpu_rq(cpu)->cfs.nr_switches +#endif +#ifdef CONFIG_RT_GROUP_SCHED +#define rt_nr_switches (tg, cpu) (tg)->rt_rq[cpu]->rt_nr_switches +#else +#define rt_nr_switches (tg, cpu) cpu_rq(cpu)->rt.rt_nr_switches +#endif +unsigned long long nr_context_switches (void ) {int i;unsigned long long sum =0 ; -for_each_possible_cpu (i) +for_each_possible_cpu (i) { + sum +=cfs_nr_switches (&root_task_group, i); + sum +=rt_nr_switches (&root_task_group, i); sum +=cpu_rq (i)->nr_switches; + }return sum; }
@@ -2929,7 +2943,6 @@ need_resched: rq->skip_clock_update =0 ;if (likely (prev != next)) { - rq->nr_switches++; rq->curr = next; ++*switch_count;
kernel/sched/idle_task.c +3 -0
@@ -28,6 +28,9 @@ pick_next_task_idle(struct rq *rq, struct task_struct *prev)if (prev) prev->sched_class->put_prev_task (rq, prev); +if (prev != rq->idle) + rq->nr_switches++; +schedstat_inc (rq, sched_goidle);return rq->idle; }
kernel/sched/stop_task.c +2 -0
@@ -32,6 +32,8 @@ pick_next_task_stop(struct rq *rq, struct task_struct *prev) stop->se.exec_start = rq->clock_task;if (prev) prev->sched_class->put_prev_task (rq, prev); +if (prev != rq->stop) + rq->nr_switches++;return stop; }
Signature · 1 line
1.8.1
PATCH v6 12/12 sched: introduce cgroup file stat_percpu
In reply to Lord Glauber Costa of SealandFrom: Glauber Costa <[email protected]>
The file cpu.stat_percpu will show various scheduler related information, that are usually available to the top level through other files.
For instance, most of the meaningful data in /proc/stat is presented here. Given this file, a container can easily construct a local copy of /proc/stat for internal consumption.
The data we export is comprised of:
* all the tick information, previously available only through cpuacct, like user time, system time, etc.
* wait time, which can be used to construct analogous information to steal time in hypervisors,
* nr_switches and nr_running, which are cgroup-local versions of their global counterparts.
The file format consists of a one-line header that describes the fields being listed. No guarantee is given that the fields will be kept the same between kernel releases, and readers should always check the header in order to introspect it.
Each of the following lines will show the respective field value for each of the possible cpus in the system. All values are show in nanoseconds.
One example output for this file is:
cpu user nice system irq softirq guest guest_nice wait nr_switches nr_running cpu0 471000000 0 15000000 0 0 0 0 1996534 7205 1
cpu1 588000000 0 17000000 0 0 0 0 2848680 6510 1
cpu2 505000000 0 14000000 0 0 0 0 2350771 6183 1
cpu3 472000000 0 16000000 0 0 0 0 19766345 6277 2
--- Documentation/cgroups/cpu.txt | 18 +++++++ kernel/sched/core.c | 109 ++++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 14 ++++++ kernel/sched/sched.h | 11 ++++- 4 files changed, 150 insertions(+), 2 deletions(-)
Documentation/cgroups/cpu.txt +18 -0
@@ -68,6 +68,24 @@ The CPU controller exposes the following files to the user: can ever be run in this cgroup. For more information about rt tasks runtime assignments, see scheduler/sched-rt-group.txt + - cpu.stat_percpu: Various scheduler statistics for the current group. The + information provided in this file is akin to the one displayed in /proc/stat, + except for the fact that it is cgroup-aware. The file format consists of a + one-line header that describes the fields being listed. No guarantee is + given that the fields will be kept the same between kernel releases, and + readers should always check the header in order to introspect it. + + Each of the following lines will show the respective field value for + each of the possible cpus in the system. All values are show in + nanoseconds. One example output for this file is: + + cpu user nice system irq softirq guest guest_nice wait nr_switches nr_running + cpu0 471000000 0 15000000 0 0 0 0 1996534 7205 1 + cpu1 588000000 0 17000000 0 0 0 0 2848680 6510 1 + cpu2 505000000 0 14000000 0 0 0 0 2350771 6183 1 + cpu3 472000000 0 16000000 0 0 0 0 19766345 6277 2 + + - cpuacct.usage: The aggregate CPU time, in nanoseconds, consumed by all tasks in this group.
kernel/sched/core.c +109 -0
127 lines not on this page · raw message
kernel/sched/fair.c +14 -0
@@ -1104,6 +1104,20 @@ static void update_cfs_shares(struct cfs_rq *cfs_rq) reweight_entity(cfs_rq_of(se), se, shares); } + +#ifdef CONFIG_SCHEDSTATS +u64 cfs_read_wait (struct task_group *tg,int cpu) +{ +struct sched_entity *se = tg->se[cpu]; +struct cfs_rq *cfs_rq =cfs_rq_of (se); +u64 value = se->statistics.wait_sum; + +if (!se->statistics.wait_start) +return value; + +return value +rq_of (cfs_rq)->clock - se->statistics.wait_start; +} +#endif #else /* CONFIG_FAIR_GROUP_SCHED */ static inline void update_cfs_shares (struct cfs_rq *cfs_rq) {
kernel/sched/sched.h +9 -2
@@ -198,8 +198,16 @@ extern void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq, struct sched_rt_entity *rt_se, int cpu, struct sched_rt_entity *parent); -#else /* CONFIG_CGROUP_SCHED */ +#ifdef CONFIG_FAIR_GROUP_SCHED +extern u64 cfs_read_wait (struct task_group *tg,int cpu); +#else +static inline u64 cfs_read_wait (struct task_group *tg,int cpu) +{ +return 0 ; +} +#endif +#else /* CONFIG_CGROUP_SCHED */ struct cfs_bandwidth { };#endif /* CONFIG_CGROUP_SCHED */
@@ -1195,7 +1203,6 @@ extern void init_cfs_rq(struct cfs_rq *cfs_rq);extern void init_rt_rq (struct rt_rq *rt_rq,struct rq *rq);extern void account_cfs_bandwidth_used (int enabled,int was_enabled); -#ifdef CONFIG_NO_HZ enum rq_nohz_flag_bits { NOHZ_TICK_STOPPED,
Signature · 1 line
1.8.1