cpuacct: per-cgroup utime/stime statistics - v4
Subsystems (2)
PATCH 1/1 cpuacct: per-cgroup utime/stime statistics - v4
Hi Ingo,
Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?
Changelog:
v4
- Remove comments in cpuacct_update_stats() which explained why rcu_read_lock() was needed (as per Peter Zijlstra's review comments).
- Don't say that percpu_counter_read() is broken in Documentation/cpuacct.txt as per KAMEZAWA Hiroyuki's review comments.
v3
- Fix a small race in the cpuacct hierarchy walk.
v2
- stime and utime now exported in clock_t units instead of msecs.
- Addressed the code review comments from Balbir and Li Zefan.
- Moved to -tip tree.
v1
- Moved the stime/utime accounting to cpuacct controller.
Earlier versions
- http://lkml.org/lkml/2009/2/25/129
Regards,
Bharata.
cpuacct: Add stime and utime statistics
Add per-cgroup cpuacct controller statistics like the system and user time consumed by the group of tasks.
--- Documentation/cgroups/cpuacct.txt | 18 +++++++ kernel/sched.c | 88 +++++++++++++++++++++++++++++++++++--- 2 files changed, 100 insertions(+), 6 deletions(-)
Documentation/cgroups/cpuacct.txt +18 -0
@@ -30,3 +30,21 @@ The above steps create a new group g1 an process (bash) into it. CPU time consumed by this bash and its children can be obtained from g1/cpuacct.usage and the same is accumulated in /cgroups/cpuacct.usage also. + +cpuacct.stat file lists a few statistics which further divide the +CPU time obtained by the cgroup into user and system times. Currently +the following statistics are supported: + +utime: Time spent by tasks of the cgroup in user mode. +stime: Time spent by tasks of the cgroup in kernel mode. + +utime and stime are in USER_HZ unit. + +cpuacct controller uses percpu_counter interface to collect utime and +stime. This causes two side effects: + +- It is theoretically possible to see wrong values for stime and utime. + This is because percpu_counter_read() on 32bit systems isn't safe + against concurrent writes. +- It is possible to see slightly outdated values for stime and utime + due to the batch processing nature of percpu_counter.
kernel/sched.c +82 -6
156 lines not on this page · raw message
Quoted 7 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…
Hi Ingo,
Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?
Quoted 8 lines
Changelog:
v4
- Remove comments incpuacct_update_stats()which explained whyrcu_read_lock()was needed (as per Peter Zijlstra's review comments).
- Don't say thatpercpu_counter_read()is broken inDocumentation/cpuacct.txtas per KAMEZAWA Hiroyuki's review comments.
Broken -> isn't safe ? no difference ;)
Can't this help you ?
---
include/linux/percpu_counter.h +15 -0
@@ -62,6 +62,16 @@ static inline s64 percpu_counter_read(streturn fbc->count; } +static inline s64 percpu_counter_read_safe (struct percpu_counter *fbc) +{ +s64 ret; + +spin_lock (&fbc->lock); + ret = fbc->count; +spin_unlock (&fbc->lock); +return ret; +} +/* * It is possible for the percpu_counter_read() to return a small negative * number for some counter which should never be negative.
@@ -114,6 +124,11 @@ static inline s64 percpu_counter_read(streturn fbc->count; } +static inline s64 percpu_counter_read_safe (struct percpu_counter *fbc) +{ +return fbc->count; +} +static inline s64 percpu_counter_read_positive (struct percpu_counter *fbc) {return fbc->count;
Quoted 18 lines · KAMEZAWA Hiroyuki
Quoted 7 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…
Hi Ingo,
Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?
Quoted 8 lines
Changelog:
v4
- Remove comments incpuacct_update_stats()which explained whyrcu_read_lock()was needed (as per Peter Zijlstra's review comments).
- Don't say thatpercpu_counter_read()is broken inDocumentation/cpuacct.txtas per KAMEZAWA Hiroyuki's review comments.
Quoted 3 lines
Broken -> isn't safe ? no difference ;)
From your comment last time, I thought you meant calling it broken would
be harsh.
Quoted 28 lines
Can't this help you ?
Signed-off-by KAMEZAWA Hiroyuki <[email protected]>---
Index:mmotm-2.6.29-Mar21/include/linux/percpu_counter.h
===================================================================include/linux/percpu_counter.h +10
-0@@ -62,6 +62,16 @@ static inline s64 percpu_counter_read(streturn fbc->count; } +static inline s64 percpu_counter_read_safe (struct percpu_counter *fbc) +{ +s64 ret; + +spin_lock (&fbc->lock); + ret = fbc->count; +spin_unlock (&fbc->lock); +return ret; +} +/* * It is possible for the percpu_counter_read() to return a small negative * number for some counter which should never be negative. @@ -114,6 +124,11 @@ static inline s64 percpu_counter_read(streturn fbc->count;
include/linux/percpu_counter.h static inline s64 percpu_counter_read(st · line 134
Quoted 10 lines
include/linux/percpu_counter.h +5
-0} +static inline s64 percpu_counter_read_safe (struct percpu_counter *fbc) +{ +return fbc->count; +} +static inline s64 percpu_counter_read_positive (struct percpu_counter *fbc) {return fbc->count;
Yes it helps, but for the records, let me note that it makes the readside ~20% slower.
[I measured time spent by cpuacct_stats_show(), which is part of the read routine for cpuacct.stat by using kretprobe and for 100 reads (or 100 executions of cpuacct_stats_show()), I get the following numbers:percpu_counter_read() - 3166367 nspercpu_counter_read_safe() - 3793546 ns which is ~20% slower. ]
I would prefer that this patch should be included in its current form and we could separately fix percpu_counter_read() given that there has been an attempt in the past to fix this (http://lkml.org/lkml/2008/8/27/303)
Moreover, I don't know how much acceptable it is to work around the problem in percpu_counter_read() by introducing another variant percpu_counter_read_safe().
Regards,
Bharata.
Quoted 72 lines · On Mon, 23 Mar 2009 12:06:03 +0530 Bharata B Rao <[email protected]>…
Quoted 12 lines · KAMEZAWA Hiroyuki
Quoted 9 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…
Hi Ingo,
Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?
Changelog:
Quoted 7 lines
Quoted 6 lines
v4
- Remove comments incpuacct_update_stats()which explained whyrcu_read_lock()was needed (as per Peter Zijlstra's review comments).
- Don't say thatpercpu_counter_read()is broken inDocumentation/cpuacct.txtas per KAMEZAWA Hiroyuki's review comments.Broken -> isn't safe ? no difference ;)
From your comment last time, I thought you meant calling it broken would be harsh.
Quoted 28 lines
Can't this help you ?
Signed-off-by KAMEZAWA Hiroyuki <[email protected]>---
Index:mmotm-2.6.29-Mar21/include/linux/percpu_counter.h
===================================================================include/linux/percpu_counter.h +10
-0@@ -62,6 +62,16 @@ static inline s64 percpu_counter_read(streturn fbc->count; } +static inline s64 percpu_counter_read_safe (struct percpu_counter *fbc) +{ +s64 ret; + +spin_lock (&fbc->lock); + ret = fbc->count; +spin_unlock (&fbc->lock); +return ret; +} +/* * It is possible for the percpu_counter_read() to return a small negative * number for some counter which should never be negative. @@ -114,6 +124,11 @@ static inline s64 percpu_counter_read(streturn fbc->count;include/linux/percpu_counter.h static inline s64 percpu_counter_read(st · line 134
Quoted 10 lines
include/linux/percpu_counter.h +5
-0} +static inline s64 percpu_counter_read_safe (struct percpu_counter *fbc) +{ +return fbc->count; +} +static inline s64 percpu_counter_read_positive (struct percpu_counter *fbc) {return fbc->count;Yes it helps, but for the records, let me note that it makes the readside ~20% slower.
[I measured time spent by
cpuacct_stats_show(), which is part of the read routine for cpuacct.stat by using kretprobe and for 100 reads (or 100 executions ofcpuacct_stats_show()), I get the following numbers:percpu_counter_read()- 3166367 nspercpu_counter_read_safe()- 3793546 ns which is ~20% slower. ]
Quoted 8 lines
I would prefer that this patch should be included in its current form and we could separately fix
percpu_counter_read()given that there has been an attempt in the past to fix this (http://lkml.org/lkml/2008/8/27/303)Moreover, I don't know how much acceptable it is to work around the problem in
percpu_counter_read()by introducing another variantpercpu_counter_read_safe().
Hmm, ok, how about this way ?
Removet this.
+- It is possible to see slightly outdated values for stime and utime + due to the batch processing nature of percpu_counter.
Anyway, we get overhead of vfs_read(), interrupts, at el ;)
put into todo list.
+TODO:
+- It is theoretically possible to see wrong values for stime and utime. + This is because percpu_counter_read() on 32bit systems isn't safe + against concurrent writes. needs to be fixed.
Thanks,
-Kame
Quoted 5 lines · KAMEZAWA Hiroyuki
Removet this.
+- It is possible to see slightly outdated values for stime and utime + due to the batch processing nature of percpu_counter.
This will explain why the stime and utime values for the top cgroup is different from the ones seen from /proc/stat. Hence I would prefer to have a mention of it in the documentation.
[root@llm11 cgroups]# cat cpuacct.stat ; cat /proc/stat | grep cpu utime 3110
stime 8410
cpu 3151 2 8336 4051594 2189 0 133 0 0
...
Regards,
Bharata.
Quoted 11 lines · On Mon, 23 Mar 2009 13:17:50 +0530 Bharata B Rao <[email protected]>…
Quoted 5 lines · KAMEZAWA Hiroyuki
Removet this.
+- It is possible to see slightly outdated values for stime and utime + due to the batch processing nature of percpu_counter.This will explain why the stime and utime values for the top cgroup is different from the ones seen from
/proc/stat. Hence I would prefer to have a mention of it in the documentation.
i see.
Thanks,
-Kame
Quoted 9 lines
[root@llm11 cgroups]# cat cpuacct.stat ; cat
/proc/stat| grep cpu utime 3110
stime 8410
cpu 3151 2 8336 4051594 2189 0 133 0 0
...Regards,
Bharata.
kernel/sched.c static int cpuacct_percpu_seq_read(struc · line 9909
Quoted 13 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…
kernel/sched.c +13
-0+static int cpuacct_stats_show (struct cgroup *cgrp,struct cftype *cft, +struct cgroup_map_cb *cb) +{ +struct cpuacct *ca =cgroup_ca (cgrp); +int i; + +for (i =0 ; i <CPUACCT_STAT_NSTATS ; i++) { +s64 val =percpu_counter_read (&ca->cpustat[i]); + val =cputime_to_clock_t (val); + cb->fill (cb, cpuacct_stat_desc[i], val); + } +return 0 ; +}
I'd have expected `val' to have type clock_t. But clock_t is 32-bit on 32-bit x86.
Is it correct to pass a 64-bit value to a function which takes a 32-bit value and to then copy the 32-bit return value into a 64-bit variable?
Quoted 19 lines · Andrew Morton
kernel/sched.c static int cpuacct_percpu_seq_read(struc · line 9909
Quoted 13 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…
kernel/sched.c +13
-0+static int cpuacct_stats_show (struct cgroup *cgrp,struct cftype *cft, +struct cgroup_map_cb *cb) +{ +struct cpuacct *ca =cgroup_ca (cgrp); +int i; + +for (i =0 ; i <CPUACCT_STAT_NSTATS ; i++) { +s64 val =percpu_counter_read (&ca->cpustat[i]); + val =cputime_to_clock_t (val); + cb->fill (cb, cpuacct_stat_desc[i], val); + } +return 0 ; +}I'd have expected `val' to have type
clock_t. Butclock_tis 32-bit on 32-bit x86.
Quoted 2 lines
Is it correct to pass a 64-bit value to a function which takes a 32-bit value and to then copy the 32-bit return value into a 64-bit variable?
Yes, doesn't look neat. Since the accumulated stat(val here) is of 64bit type, I could use cputime64_to_clock_t(val) in the above code. Storing the resulting 32bit clock_t in a 64 bit variable(which will be used by cb->fill) should be fine I believe. Let me know if I got this right.
Regards,
Bharata.