Skip to contentSkip to conversation

cpuacct: per-cgroup utime/stime statistics - v4

PATCH In review Bharata B Rao 3 participants 8 messages Mar 23, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpuacct: per-cgroup utime/stime statistics - v4

Hi Ingo,

Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?

Changelog:

v4
- Remove comments in cpuacct_update_stats() which explained why rcu_read_lock() was needed (as per Peter Zijlstra's review comments).
- Don't say that percpu_counter_read() is broken in Documentation/cpuacct.txt as per KAMEZAWA Hiroyuki's review comments.

v3
- Fix a small race in the cpuacct hierarchy walk.

v2
- stime and utime now exported in clock_t units instead of msecs.
- Addressed the code review comments from Balbir and Li Zefan.
- Moved to -tip tree.

v1
- Moved the stime/utime accounting to cpuacct controller.

Earlier versions
- http://lkml.org/lkml/2009/2/25/129

Regards,
Bharata.

cpuacct: Add stime and utime statistics

Add per-cgroup cpuacct controller statistics like the system and user time consumed by the group of tasks.

---
 Documentation/cgroups/cpuacct.txt |   18 +++++++
 kernel/sched.c                    |   88 +++++++++++++++++++++++++++++++++++---
 2 files changed, 100 insertions(+), 6 deletions(-)
Documentation/cgroups/cpuacct.txt +18 -0
@@ -30,3 +30,21 @@ The above steps create a new group g1 an
 process (bash) into it. CPU time consumed by this bash and its children
 can be obtained from g1/cpuacct.usage and the same is accumulated in
 /cgroups/cpuacct.usage also.
+
+cpuacct.stat file lists a few statistics which further divide the
+CPU time obtained by the cgroup into user and system times. Currently
+the following statistics are supported:
+
+utime: Time spent by tasks of the cgroup in user mode.
+stime: Time spent by tasks of the cgroup in kernel mode.
+
+utime and stime are in USER_HZ unit.
+
+cpuacct controller uses percpu_counter interface to collect utime and
+stime. This causes two side effects:
+
+- It is theoretically possible to see wrong values for stime and utime.
+  This is because percpu_counter_read() on 32bit systems isn't safe
+  against concurrent writes.
+- It is possible to see slightly outdated values for stime and utime
+  due to the batch processing nature of percpu_counter.
kernel/sched.c +82 -6

156 lines not on this page · raw message

#1 lore raw
Quoted 7 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…

Hi Ingo,

Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?

Quoted 8 lines

Changelog:

v4
- Remove comments in cpuacct_update_stats() which explained why rcu_read_lock() was needed (as per Peter Zijlstra's review comments).
- Don't say that percpu_counter_read() is broken in Documentation/cpuacct.txt as per KAMEZAWA Hiroyuki's review comments.

Broken -> isn't safe ? no difference ;)
Can't this help you ?

---

include/linux/percpu_counter.h +15 -0
Index: mmotm-2.6.29-Mar21/include/linux/percpu_counter.h
===================================================================
@@ -62,6 +62,16 @@ static inline s64 percpu_counter_read(st
 	return fbc->count;
 }
 
+static inline s64 percpu_counter_read_safe(struct percpu_counter *fbc)
+{
+	s64 ret;
+
+	spin_lock(&fbc->lock);
+	ret = fbc->count;
+	spin_unlock(&fbc->lock);
+	return ret;
+}
+
 /*
  * It is possible for the percpu_counter_read() to return a small negative
  * number for some counter which should never be negative.
@@ -114,6 +124,11 @@ static inline s64 percpu_counter_read(st
 	return fbc->count;
 }
 
+static inline s64 percpu_counter_read_safe(struct percpu_counter *fbc)
+{
+	return fbc->count;
+}
+
 static inline s64 percpu_counter_read_positive(struct percpu_counter *fbc)
 {
 	return fbc->count;
#2 lore raw
Quoted 18 lines · KAMEZAWA Hiroyuki
Quoted 7 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…

Hi Ingo,

Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?

Quoted 8 lines

Changelog:

v4
- Remove comments in cpuacct_update_stats() which explained why rcu_read_lock() was needed (as per Peter Zijlstra's review comments).
- Don't say that percpu_counter_read() is broken in Documentation/cpuacct.txt as per KAMEZAWA Hiroyuki's review comments.

Quoted 3 lines

Broken -> isn't safe ? no difference ;)

From your comment last time, I thought you meant calling it broken would

be harsh.

Quoted 28 lines

Can't this help you ?

---
Index: mmotm-2.6.29-Mar21/include/linux/percpu_counter.h
===================================================================

include/linux/percpu_counter.h +10 -0
@@ -62,6 +62,16 @@ static inline s64 percpu_counter_read(st
 	return fbc->count;
 }
 
+static inline s64 percpu_counter_read_safe(struct percpu_counter *fbc)
+{
+	s64 ret;
+
+	spin_lock(&fbc->lock);
+	ret = fbc->count;
+	spin_unlock(&fbc->lock);
+	return ret;
+}
+
 /*
  * It is possible for the percpu_counter_read() to return a small negative
  * number for some counter which should never be negative.
@@ -114,6 +124,11 @@ static inline s64 percpu_counter_read(st
 	return fbc->count;

include/linux/percpu_counter.h static inline s64 percpu_counter_read(st · line 134

Quoted 10 lines
include/linux/percpu_counter.h +5 -0
 }
 
+static inline s64 percpu_counter_read_safe(struct percpu_counter *fbc)
+{
+	return fbc->count;
+}
+
 static inline s64 percpu_counter_read_positive(struct percpu_counter *fbc)
 {
 	return fbc->count;

Yes it helps, but for the records, let me note that it makes the readside ~20% slower.

[I measured time spent by cpuacct_stats_show(), which is part of the read routine for cpuacct.stat by using kretprobe and for 100 reads (or 100 executions of cpuacct_stats_show()), I get the following numbers:
percpu_counter_read() - 3166367 ns
percpu_counter_read_safe() - 3793546 ns which is ~20% slower. ]

I would prefer that this patch should be included in its current form and we could separately fix percpu_counter_read() given that there has been an attempt in the past to fix this (http://lkml.org/lkml/2008/8/27/303)

Moreover, I don't know how much acceptable it is to work around the problem in percpu_counter_read() by introducing another variant percpu_counter_read_safe().

Regards,
Bharata.

#3 lore raw
Quoted 72 lines · On Mon, 23 Mar 2009 12:06:03 +0530 Bharata B Rao <[email protected]>…
Quoted 12 lines · KAMEZAWA Hiroyuki
Quoted 9 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…

Hi Ingo,

Here is the v4 of the cpuacct statistics patch to obtain per-cgroup system and user times with appropriate tags and complete changelog. This applies against the latest -tip plus the cpuacct hierarchy fix v2 which I posted just now. Could you please include this in -tip ?

Changelog:

Quoted 7 lines
Quoted 6 lines

v4
- Remove comments in cpuacct_update_stats() which explained why rcu_read_lock() was needed (as per Peter Zijlstra's review comments).
- Don't say that percpu_counter_read() is broken in Documentation/cpuacct.txt as per KAMEZAWA Hiroyuki's review comments.

Broken -> isn't safe ? no difference ;)

From your comment last time, I thought you meant calling it broken would be harsh.

Quoted 28 lines

Can't this help you ?

---
Index: mmotm-2.6.29-Mar21/include/linux/percpu_counter.h
===================================================================

include/linux/percpu_counter.h +10 -0
@@ -62,6 +62,16 @@ static inline s64 percpu_counter_read(st
 	return fbc->count;
 }
 
+static inline s64 percpu_counter_read_safe(struct percpu_counter *fbc)
+{
+	s64 ret;
+
+	spin_lock(&fbc->lock);
+	ret = fbc->count;
+	spin_unlock(&fbc->lock);
+	return ret;
+}
+
 /*
  * It is possible for the percpu_counter_read() to return a small negative
  * number for some counter which should never be negative.
@@ -114,6 +124,11 @@ static inline s64 percpu_counter_read(st
 	return fbc->count;

include/linux/percpu_counter.h static inline s64 percpu_counter_read(st · line 134

Quoted 10 lines
include/linux/percpu_counter.h +5 -0
 }
 
+static inline s64 percpu_counter_read_safe(struct percpu_counter *fbc)
+{
+	return fbc->count;
+}
+
 static inline s64 percpu_counter_read_positive(struct percpu_counter *fbc)
 {
 	return fbc->count;

Yes it helps, but for the records, let me note that it makes the readside ~20% slower.

[I measured time spent by cpuacct_stats_show(), which is part of the read routine for cpuacct.stat by using kretprobe and for 100 reads (or 100 executions of cpuacct_stats_show()), I get the following numbers:
percpu_counter_read() - 3166367 ns
percpu_counter_read_safe() - 3793546 ns which is ~20% slower. ]

Quoted 8 lines

I would prefer that this patch should be included in its current form and we could separately fix percpu_counter_read() given that there has been an attempt in the past to fix this (http://lkml.org/lkml/2008/8/27/303)

Moreover, I don't know how much acceptable it is to work around the problem in percpu_counter_read() by introducing another variant percpu_counter_read_safe().

Hmm, ok, how about this way ?

Removet this.

+- It is possible to see slightly outdated values for stime and utime
+  due to the batch processing nature of percpu_counter.

Anyway, we get overhead of vfs_read(), interrupts, at el ;)

put into todo list.

+TODO:

+- It is theoretically possible to see wrong values for stime and utime.
+  This is because percpu_counter_read() on 32bit systems isn't safe
+  against concurrent writes. needs to be fixed.

Thanks,
-Kame

#4 lore raw
Quoted 5 lines · KAMEZAWA Hiroyuki

Removet this.

+- It is possible to see slightly outdated values for stime and utime
+  due to the batch processing nature of percpu_counter.

This will explain why the stime and utime values for the top cgroup is different from the ones seen from /proc/stat. Hence I would prefer to have a mention of it in the documentation.

[root@llm11 cgroups]# cat cpuacct.stat ; cat /proc/stat | grep cpu utime 3110
stime 8410
cpu 3151 2 8336 4051594 2189 0 133 0 0
...

Regards,
Bharata.

#5 lore raw
Quoted 11 lines · On Mon, 23 Mar 2009 13:17:50 +0530 Bharata B Rao <[email protected]>…
Quoted 5 lines · KAMEZAWA Hiroyuki

Removet this.

+- It is possible to see slightly outdated values for stime and utime
+  due to the batch processing nature of percpu_counter.

This will explain why the stime and utime values for the top cgroup is different from the ones seen from /proc/stat. Hence I would prefer to have a mention of it in the documentation.

i see.

Thanks,
-Kame

Quoted 9 lines

[root@llm11 cgroups]# cat cpuacct.stat ; cat /proc/stat | grep cpu utime 3110
stime 8410
cpu 3151 2 8336 4051594 2189 0 133 0 0
...

Regards,
Bharata.

#6 lore raw
In reply to Bharata B Rao

kernel/sched.c static int cpuacct_percpu_seq_read(struc · line 9909

Quoted 13 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…
kernel/sched.c +13 -0
+static int cpuacct_stats_show(struct cgroup *cgrp, struct cftype *cft,
+		struct cgroup_map_cb *cb)
+{
+	struct cpuacct *ca = cgroup_ca(cgrp);
+	int i;
+
+	for (i = 0; i < CPUACCT_STAT_NSTATS; i++) {
+		s64 val = percpu_counter_read(&ca->cpustat[i]);
+		val = cputime_to_clock_t(val);
+		cb->fill(cb, cpuacct_stat_desc[i], val);
+	}
+	return 0;
+}

I'd have expected `val' to have type clock_t. But clock_t is 32-bit on 32-bit x86.

Is it correct to pass a 64-bit value to a function which takes a 32-bit value and to then copy the 32-bit return value into a 64-bit variable?

#7 lore raw
Quoted 19 lines · Andrew Morton

kernel/sched.c static int cpuacct_percpu_seq_read(struc · line 9909

Quoted 13 lines · On Mon, 23 Mar 2009 10:05:38 +0530 Bharata B Rao <[email protected]>…
kernel/sched.c +13 -0
+static int cpuacct_stats_show(struct cgroup *cgrp, struct cftype *cft,
+		struct cgroup_map_cb *cb)
+{
+	struct cpuacct *ca = cgroup_ca(cgrp);
+	int i;
+
+	for (i = 0; i < CPUACCT_STAT_NSTATS; i++) {
+		s64 val = percpu_counter_read(&ca->cpustat[i]);
+		val = cputime_to_clock_t(val);
+		cb->fill(cb, cpuacct_stat_desc[i], val);
+	}
+	return 0;
+}

I'd have expected `val' to have type clock_t. But clock_t is 32-bit on 32-bit x86.

Quoted 2 lines

Is it correct to pass a 64-bit value to a function which takes a 32-bit value and to then copy the 32-bit return value into a 64-bit variable?

Yes, doesn't look neat. Since the accumulated stat(val here) is of 64bit type, I could use cputime64_to_clock_t(val) in the above code. Storing the resulting 32bit clock_t in a 64 bit variable(which will be used by cb->fill) should be fine I believe. Let me know if I got this right.

Regards,
Bharata.