Enable CPU frequency and power tracking in cpuacct cgroup

v2:
Rebased off of Thomas Renninger's patch for cgroups_cpuacct refactoring, which is based off of linus's tree. Thomas it might be easier to merge our patches if you take these patches and put them in a series ontop of your original patches (presuming there are no objections to your patch).

This patch series introduces cpu frequency and power tracking for cpuacct cgroups. A similar patch set was discussed a while back and it was concluded that due to varying architectures (ppc, x86 with overboot) you cannot account for frequencies and their power consumption generically in sched.c, thus we have platform specific hooks the cpuacct can call into (if available).

This patch series is not 3 instead of 4. I have left out the power implementation for OMAP due to implementation conflicts in linux-next.

Mike Chan (3):
scheduler: cpuacct: Enable platform hooks to track cpuusage for CPU
frequencies
scheduler: cpuacct: Enable platform callbacks for cpuacct power
tracking
omap: cpu: Implement callbacks for cpu frequency tracking in cpuacct

 Documentation/cgroups/cpuacct.txt |    7 ++++
 arch/arm/plat-omap/cpu-omap.c     |   67 +++++++++++++++++++++++++++++++++++-
 include/linux/cpuacct.h           |   43 +++++++++++++++++++++++
 kernel/cgroup_cpuaccount.c        |   69 +++++++++++++++++++++++++++++++++++++
 4 files changed, 185 insertions(+), 1 deletions(-)
 create mode 100644 include/linux/cpuacct.h

V2:
- Rebased off of Thomas Renninger's cgroup_cpuacct refactoring.
- Renamed function struct to cpuacct_chrage_calls from cpuacct_cpufreq_calls

Introduce new platform callback hooks for cpuacct for tracking CPU frequencies

Not all platforms / architectures have a set CPU_FREQ_TABLE defined for CPU transition speeds. In order to track time spent in at various CPU frequencies, we enable platform callbacks from cpuacct for this accounting.

Architectures that support overclock boosting, or don't have pre-defined frequency tables can implement their own bucketing system that makes sense given their cpufreq scaling abilities.

New file:
cpuacct.cpufreq reports the CPU time (in nanoseconds) spent at each CPU frequency.

---
 Documentation/cgroups/cpuacct.txt |    4 +++
 include/linux/cpuacct.h           |   41 +++++++++++++++++++++++++++++++
 kernel/cgroup_cpuaccount.c        |   49 +++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/cpuacct.h

index 8b93094..600d2d0 100644
@@ -40,6 +40,10 @@ system: Time spent by tasks of the cgroup in kernel mode.
 
 user and system are in USER_HZ unit.
 
+cpuacct.cpufreq file gives CPU time (in nanoseconds) spent at each CPU
+frequency. Platform hooks must be implemented inorder to properly track
+time at each CPU frequency.
+
 cpuacct controller uses percpu_counter interface to collect user and
 system times. This has two side effects:
 

1.7.0.1

V2:
- Rebased off Thomass Renninger's cgroup_cpuacct refactoring

Platform must register cpu power function that return power in milliWatt seconds.

New file:
cpuacct.power reports the power consumed in milliWatt seconds

---
 Documentation/cgroups/cpuacct.txt |    3 +++
 include/linux/cpuacct.h           |    4 +++-
 kernel/cgroup_cpuaccount.c        |   24 ++++++++++++++++++++++--
 3 files changed, 28 insertions(+), 3 deletions(-)

index 600d2d0..84e471b 100644
@@ -44,6 +44,9 @@ cpuacct.cpufreq file gives CPU time (in nanoseconds) spent at each CPU
 frequency. Platform hooks must be implemented inorder to properly track
 time at each CPU frequency.
 
+cpuacct.power file gives CPU power consumed (in milliWatt seconds). Platform
+must provide and implement power callback functions.
+
 cpuacct controller uses percpu_counter interface to collect user and
 system times. This has two side effects:
 

index 6205d29..c17a634 100644
@@ -31,7 +31,9 @@ struct cpuacct_charge_calls {
 	 */
 	void (*init) (void **cpuacct_data);
 	void (*charge) (void *cpuacct_data,  u64 cputime, unsigned int cpu);
-	void (*show) (void *cpuacct_data, struct cgroup_map_cb *cb);
+	void (*cpufreq_show) (void *cpuacct_data, struct cgroup_map_cb *cb);
+	/* Returns power consumed in milliWatt seconds */
+	u64 (*power_usage) (void *cpuacct_data);
 };
 
 int cpuacct_charge_register(struct cpuacct_charge_calls *fn);

index 11799a7..d9bf889 100644
@@ -226,12 +226,28 @@ static int cpuacct_cpufreq_show(struct cgroup *cgrp, struct cftype *cft,
 		struct cgroup_map_cb *cb)
 {
 	struct cpuacct *ca = cgroup_ca(cgrp);
-	if (ca->cpufreq_fn && ca->cpufreq_fn->show)
-		ca->cpufreq_fn->show(ca->cpuacct_data, cb);
+	if (ca->cpufreq_fn && ca->cpufreq_fn->cpufreq_show)
+		ca->cpufreq_fn->cpufreq_show(ca->cpuacct_data, cb);
 
 	return 0;
 }
 
+/* return total cpu power usage (milliWatt second) of a group */
+static u64 cpuacct_powerusage_read(struct cgroup *cgrp, struct cftype *cft)
+{
+	int i;
+	struct cpuacct *ca = cgroup_ca(cgrp);
+	u64 totalpower = 0;
+
+	if (ca->cpufreq_fn && ca->cpufreq_fn->power_usage)
+		for_each_present_cpu(i) {
+			totalpower += ca->cpufreq_fn->power_usage(
+					ca->cpuacct_data);
+		}
+
+	return totalpower;
+}
+
 static struct cftype files[] = {
 	{
 		.name = "usage",

@@ -250,6 +266,10 @@ static struct cftype files[] = {
 		.name =  "cpufreq",
 		.read_map = cpuacct_cpufreq_show,
 	},
+	{
+		.name = "power",
+		.read_u64 = cpuacct_powerusage_read
+	},
 };
 
 static int cpuacct_populate(struct cgroup_subsys *ss, struct cgroup *cgrp)

1.7.0.1

Implement OMAP platform specific scheduler callbacks for tracking cpu frequencies per cpuacct cgroup.

---
 arch/arm/plat-omap/cpu-omap.c |   67 ++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 66 insertions(+), 1 deletions(-)

index 6d3d333..176417a 100644
@@ -21,6 +21,8 @@
 #include <linux/err.h>
 #include <linux/clk.h>
 #include <linux/io.h>
+#include <linux/slab.h>
+#include <linux/cpuacct.h>
 
 #include <mach/hardware.h>
 #include <plat/clock.h>

@@ -38,6 +40,10 @@ static struct cpufreq_frequency_table *freq_table;
 
 static struct clk *mpu_clk;
 
+#ifdef CONFIG_CGROUP_CPUACCT
+static int freq_index;
+#endif
+
 /* TODO: Add support for SDRAM timing changes */
 
 int omap_verify_speed(struct cpufreq_policy *policy)

@@ -96,6 +102,11 @@ static int omap_target(struct cpufreq_policy *policy,
 	       freqs.old, freqs.new);
 #endif
 	ret = clk_set_rate(mpu_clk, freqs.new * 1000);
+#ifdef CONFIG_CGROUP_CPUACCT
+	/* Update freq_index before cpufreq transition post notification. */
+	cpufreq_frequency_table_target(policy, freq_table, freqs.new,
+			CPUFREQ_RELATION_L, &freq_index);
+#endif
 	cpufreq_notify_transition(&freqs, CPUFREQ_POSTCHANGE);
 
 	return ret;

@@ -125,7 +136,14 @@ static int __init omap_cpu_init(struct cpufreq_policy *policy)
 		policy->cpuinfo.max_freq = clk_round_rate(mpu_clk,
 							VERY_HI_RATE) / 1000;
 	}
-
+#ifdef CONFIG_CGROUP_CPUACCT
+	/*
+	 * Update freq_index, since we are using the extact frequency
+	 * we can use any relation.
+	 */
+	cpufreq_frequency_table_target(policy, freq_table, policy->cur,
+			CPUFREQ_RELATION_L, &freq_index);
+#endif
 	/* FIXME: what's the actual transition time? */
 	policy->cpuinfo.transition_latency = 300 * 1000;
 

@@ -169,3 +187,50 @@ arch_initcall(omap_cpufreq_init);
  * cpufreq_frequency_table_put_attr()
  */
 
+#ifdef CONFIG_CGROUP_CPUACCT
+/*
+ * Omap platform calls for cpuacct frequency accounting.
+ */
+
+static void omap_cpuacct_freq_init(void **cpuacct_data)
+{
+	/* MAX_VDD1_OPP is gone, define a table size to fit available values */
+	*cpuacct_data = kzalloc(sizeof(u64) * 8, GFP_KERNEL);
+}
+
+/* Called with rcu_read_lock() held. */
+static void omap_cpuacct_freq_charge(void *cpuacct_data, u64 cputime, unsigned int cpu)
+{
+	u64 *cpuacct_freq = cpuacct_data;
+	if (freq_index < 0)
+		return;
+
+	cpuacct_freq[freq_index] += cputime;
+}
+
+static void omap_cpuacct_freq_show(void *cpuacct_data, struct cgroup_map_cb *cb)
+{
+	int i;
+	char buf[32];
+	u64 *cpuacct_freq = cpuacct_data;
+	for (i = 0; freq_table[i].frequency != CPUFREQ_TABLE_END; i++) {
+		snprintf(buf, sizeof(buf), "%u", freq_table[i].frequency);
+		cb->fill(cb, buf, cpuacct_freq[i]);
+	}
+}
+
+static struct cpuacct_charge_calls omap_cpuacct_cpufreq = {
+	.init = omap_cpuacct_freq_init,
+	.charge = omap_cpuacct_freq_charge,
+	.cpufreq_show = omap_cpuacct_freq_show,
+};
+
+static int __init omap_cpuacct_init(void)
+{
+	freq_index = -1;
+	cpuacct_charge_register(&omap_cpuacct_cpufreq);
+	return 0;
+}
+
+early_initcall(omap_cpuacct_init);
+#endif

1.7.0.1

Hi Mike,

thanks.
A general comment:
I don't know much about the cgroup stuff.
I am also not sure how exactly power can be measured on this arch based on frequency accounting (there also were some threads I was not aware of?) A signed-off-by or reviewed-by from someone who is more involved in this omap stuff would probably not that bad.

Still I Iike this interface and I could imagine others hook into it as well, for whatever has to be cpu cgroup accounted.

My two cents...,

Thomas

Perhaps Paul Menage or Balbir Singh can look and sign off on the API's?

If you know how much time was spent at each frequency executing code, you can calculate how much power was consumed if the platform (with hooks) provide power numbers (in milliWatts) for the power at frequency X.

I did some initial testing on Motorola Droid comparing to a power meter and I got within 2% variance.

OMAP was the closest with mainline support I could provide an example how to use these hooks.

I'm hoping for some blessing from some people on the linux-omap list for that. However can we possibly just stack the first two patches to get the API in? This will make it easier to fixup the omap hooks if they don't get in.

-- Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

This looks like a great enhancement to me.

Speaking for OMAP PM... I'd suggest getting the generic stuff upstream (or into -next) soon and then work out the OMAP specifics after.

Since the OMAP OPP layer is going through some churn (but stabilizing and will be submitted for 2.6.36), I'd suggest we queue the OMAP-specific parts of this along with the OPP layer changes.

Kevin

So it looks like there is no objections to this API and I'm OK with dropping the omap hooks for now until things are settled in 2.6.36. So are things good with Thomas' re-factoring patch for cpuacct as well as the first 2 patches?

-- Mike