perf: show package power consumption in perf

Hi, all,

RAPL(running average power limit) is a new feature which provides mechanisms to enforce power consumption limit, on some new processors.

Generally speaking, by using RAPL, OS can set a power budget in a certain time window, and let Hardware to throttle the processor P/T-state to meet this energy limitation.

RAPL also provides a new MSR, i.e. MSR_PKG_ENERGY_STATUS, which reports the total amount of energy consumed by the package.

I'm not sure if to support RAPL or not, but anyway, it sounds like a good idea to export the energy status in perf.

So a new perf pmu and event to show the package energy consumed is introduced in this patch.

Here is what I get after applying the three patches,

#./perf stat -e energy test
Performance counter stats for 'test':

202 Joules cost by package
7.926001238 seconds time elapsed

Note that this patch set is made based on Peter's perf-pmu branch,
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/linux-2.6-perf.git
which provides better interfaces to register/unregister a new pmu.

any comment are welcome. :)

(quoted diff) +7 -0

+static int __init intel_rapl_init(void)
+{
+	/*
+	 * RAPL features are only supported on processors have a CPUID
+	 * signature with DisplayFamily_DisplayModel of 06_2AH, 06_2DH
+	 */
+	if (boot_cpu_data.x86 != 0x06 ||

Quoted 27 lines · Zhang Rui

Hi, all,

RAPL(running average power limit) is a new feature which provides mechanisms to enforce power consumption limit, on some new processors.

Generally speaking, by using RAPL, OS can set a power budget in a certain time window, and let Hardware to throttle the processor P/T-state to meet this energy limitation.

RAPL also provides a new MSR, i.e. MSR_PKG_ENERGY_STATUS, which reports the total amount of energy consumed by the package.

I'm not sure if to support RAPL or not, but anyway, it sounds like a good idea to export the energy status in perf.

So a new perf pmu and event to show the package energy consumed is introduced in this patch.

Here is what I get after applying the three patches,

#./perf stat -e energy test
Performance counter stats for 'test':

202 Joules cost by package
7.926001238 seconds time elapsed

Quoted 5 lines

Note that this patch set is made based on Peter's perf-pmu branch,
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/linux-2.6-perf.git
which provides better interfaces to register/unregister a new pmu.

any comment are welcome. :)

Nice,.. however:

- if it is a pure read-only counter without sampling support, expose it as such, don't fudge in the hrtimer stuff. Simply fail to create a sampling event.

SH has the same problem for its 'normal' PMU, the solution is to use event groups, Matt was looking at adding support to perf-record for that, if creating a sampling event fails, fall back to {hrtimer, $event} groups.

Quoted 37 lines · Peter Zijlstra

Quoted 27 lines · Zhang Rui

Hi, all,

RAPL(running average power limit) is a new feature which provides mechanisms to enforce power consumption limit, on some new processors.

Generally speaking, by using RAPL, OS can set a power budget in a certain time window, and let Hardware to throttle the processor P/T-state to meet this energy limitation.

RAPL also provides a new MSR, i.e. MSR_PKG_ENERGY_STATUS, which reports the total amount of energy consumed by the package.

I'm not sure if to support RAPL or not, but anyway, it sounds like a good idea to export the energy status in perf.

So a new perf pmu and event to show the package energy consumed is introduced in this patch.

Here is what I get after applying the three patches,

#./perf stat -e energy test
Performance counter stats for 'test':

202 Joules cost by package
7.926001238 seconds time elapsed

Quoted 5 lines

Note that this patch set is made based on Peter's perf-pmu branch,
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/linux-2.6-perf.git
which provides better interfaces to register/unregister a new pmu.

any comment are welcome. :)

Nice,.. however:

Quoted 8 lines

- if it is a pure read-only counter without sampling support, expose it as such, don't fudge in the hrtimer stuff. Simply fail to create a sampling event.

SH has the same problem for its 'normal' PMU, the solution is to use event groups, Matt was looking at adding support to perf-record for that, if creating a sampling event fails, fall back to {hrtimer, $event} groups.

I had a quick look over the patches and Peter is right - the group events stuff would probably fit quite well here. Unfortunately, due to holidays and things, I haven't been able to get them finished yet. I'll get on that ASAP.

Quoted 5 lines · Matt Fleming

I had a quick look over the patches and Peter is right - the group events stuff would probably fit quite well here. Unfortunately, due to holidays and things, I haven't been able to get them finished yet. I'll get on that ASAP.

Hi, Matt

What's the "group events stuff"?
Is there some discussion on LKML or elsewhere I can have a look at?

Thanks,
Lin Ming

Quoted 12 lines · Lin Ming

Quoted 5 lines · Matt Fleming

I had a quick look over the patches and Peter is right - the group events stuff would probably fit quite well here. Unfortunately, due to holidays and things, I haven't been able to get them finished yet. I'll get on that ASAP.

Hi, Matt

What's the "group events stuff"?
Is there some discussion on LKML or elsewhere I can have a look at?

Quoted 2 lines

Thanks,
Lin Ming

The relevant information can be found here in this thread,
http://lkml.org/lkml/2010/8/4/174. I'm working on some patches for this but they're not finished yet. I can probably get something to show by next week.

The discussion started because the performance counters on SH do not generate an interrupt on overflow, so we need to periodically sample them. Am I correct in thinking that the energy counters also do not generate an interrupt on overflow and that's why you wrote the event as a software event?

Quoted 7 lines · Lin Ming

Quoted 5 lines · Matt Fleming

I had a quick look over the patches and Peter is right - the group events stuff would probably fit quite well here. Unfortunately, due to holidays and things, I haven't been able to get them finished yet. I'll get on that ASAP.

Quoted 7 lines

Hi, Matt

What's the "group events stuff"?
Is there some discussion on LKML or elsewhere I can have a look at?

Thanks,
Lin Ming

The relevant information can be found here in this thread,
http://lkml.org/lkml/2010/8/4/174. I'm working on some patches for this but they're not finished yet. I can probably get something to show by next week.

The discussion started because the performance counters on SH do not generate an interrupt on overflow, so we need to periodically sample them. Am I correct in thinking that the energy counters also do not generate an interrupt on overflow and that's why you wrote the event as a software event?

Quoted 21 lines · Matt Fleming

Quoted 7 lines · Lin Ming

Quoted 5 lines · Matt Fleming

I had a quick look over the patches and Peter is right - the group events stuff would probably fit quite well here. Unfortunately, due to holidays and things, I haven't been able to get them finished yet. I'll get on that ASAP.

Quoted 7 lines

Hi, Matt

What's the "group events stuff"?
Is there some discussion on LKML or elsewhere I can have a look at?

Thanks,
Lin Ming

The relevant information can be found here in this thread,
http://lkml.org/lkml/2010/8/4/174. I'm working on some patches for this but they're not finished yet. I can probably get something to show by next week.

Quoted 5 lines

The discussion started because the performance counters on SH do not generate an interrupt on overflow, so we need to periodically sample them. Am I correct in thinking that the energy counters also do not generate an interrupt on overflow and that's why you wrote the event as a software event?

right.

BTW, I'm not quite familiar with perf tool, and now I'm wondering if the periodically sample is needed.
because IMO, .start is invoked every time the process is scheduled in, and .stop is invoked when it's scheduled out. It seems that we just need to read the energy consumed in .start and .stop, and update the counter in .stop, right?

How big is the hardware counter? The problem comes when the process is scheduled in and runs for a long time, e.g. so long that the energy hardware counter wraps. This is why it's necessary to periodically sample the counter.

Quoted 7 lines · Matt Fleming

How big is the hardware counter? The problem comes when the process is scheduled in and runs for a long time, e.g. so long that the energy hardware counter wraps. This is why it's necessary to periodically sample the counter.

Long running processes aren't the only case, you could associate an event with a CPU.

Right, short counters (like SH when not chained) need something to accumulate deltas into the larger u64. You can indeed use timers for that, hr or otherwise, but you don't need the swcounter hrtimer infrastructure for that.

Quoted 4 lines

Right, short counters (like SH when not chained) need something to accumulate deltas into the larger u64. You can indeed use timers for that, hr or otherwise, but you don't need the swcounter hrtimer infrastructure for that.

So what is the point in simulating a PMI using an hrtimer? It won't be based on periods on the interesting counter but on time periods. This is not how we want the samples. If we want timer based samples, we can just launch a seperate software timer based event.

Quoted 8 lines · Peter Zijlstra

Quoted 7 lines · Matt Fleming

How big is the hardware counter? The problem comes when the process is scheduled in and runs for a long time, e.g. so long that the energy hardware counter wraps. This is why it's necessary to periodically sample the counter.

Quoted 2 lines

Long running processes aren't the only case, you could associate an event with a CPU.

I don't understand what you mean.

Quoted 4 lines

Right, short counters (like SH when not chained) need something to accumulate deltas into the larger u64. You can indeed use timers for that, hr or otherwise, but you don't need the swcounter hrtimer infrastructure for that.

So what is the point in simulating a PMI using an hrtimer? It won't be based on periods on the interesting counter but on time periods. This is not how we want the samples. If we want timer based samples, we can just launch a seperate software timer based event.

In the case of SH where we need to flush to avoid wraps, I understand, but oterwise?

Quoted 47 lines · Matt Fleming

Quoted 37 lines · Peter Zijlstra

Quoted 27 lines · Zhang Rui

Hi, all,

RAPL(running average power limit) is a new feature which provides mechanisms to enforce power consumption limit, on some new processors.

Generally speaking, by using RAPL, OS can set a power budget in a certain time window, and let Hardware to throttle the processor P/T-state to meet this energy limitation.

RAPL also provides a new MSR, i.e. MSR_PKG_ENERGY_STATUS, which reports the total amount of energy consumed by the package.

I'm not sure if to support RAPL or not, but anyway, it sounds like a good idea to export the energy status in perf.

So a new perf pmu and event to show the package energy consumed is introduced in this patch.

Here is what I get after applying the three patches,

#./perf stat -e energy test
Performance counter stats for 'test':

202 Joules cost by package
7.926001238 seconds time elapsed

Quoted 5 lines

Note that this patch set is made based on Peter's perf-pmu branch,
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/linux-2.6-perf.git
which provides better interfaces to register/unregister a new pmu.

any comment are welcome. :)

Nice,.. however:

Quoted 8 lines

- if it is a pure read-only counter without sampling support, expose it as such, don't fudge in the hrtimer stuff. Simply fail to create a sampling event.

SH has the same problem for its 'normal' PMU, the solution is to use event groups, Matt was looking at adding support to perf-record for that, if creating a sampling event fails, fall back to {hrtimer, $event} groups.

Quoted 4 lines

I had a quick look over the patches and Peter is right - the group events stuff would probably fit quite well here. Unfortunately, due to holidays and things, I haven't been able to get them finished yet. I'll get on that ASAP.

Hi, Matt

What's the "group events stuff"?
Is there some discussion on LKML or elsewhere I can have a look at?

Quoted 15 lines

Quoted 10 lines

Quoted 9 lines

- if it is a pure read-only counter without sampling support, expose it as such, don't fudge in the hrtimer stuff. Simply fail to create a sampling event.

SH has the same problem for its 'normal' PMU, the solution is to use event groups, Matt was looking at adding support to perf-record for that, if creating a sampling event fails, fall back to {hrtimer, $event} groups.

Quoted 4 lines

I had a quick look over the patches and Peter is right - the group events stuff would probably fit quite well here. Unfortunately, due to holidays and things, I haven't been able to get them finished yet. I'll get on that ASAP.

Quoted 4 lines

Hi, Matt

What's the "group events stuff"?
Is there some discussion on LKML or elsewhere I can have a look at?

its some obscure perf feature:

leader = sys_perf_event_open(&hrtimer_attr, pid, cpu, 0, 0); sibling = sys_perf_event_open(&rapl_attr, pid, cpu, leader, 0);

will create an even group (which means that both events require to be co-scheduled). If you then provided:

hrtimer_attr.read_format |= PERF_FORMAT_GROUP;
hrtimer_attr.sample_type |= PERF_SAMPLE_READ;

the samples from the hrtimer will contain a field like:

 *      { u64           nr;
 *        { u64         time_enabled; } && PERF_FORMAT_ENABLED
 *        { u64         time_running; } && PERF_FORMAT_RUNNING
 *        { u64         value;
 *          { u64       id;           } && PERF_FORMAT_ID
 *        }             cntr[nr];

* } && PERF_FORMAT_GROUP

Which contains both the hrtimer count (ns) and the RAPL count (watts).

Using that you can compute the RAPL delta between consecutive samples and use that to weight the sample.

For perf-stat non of this is needed, since it doesn't use sampling counters anyway ;-).

Quoted 5 lines · Peter Zijlstra

its some obscure perf feature:

leader = sys_perf_event_open(&hrtimer_attr, pid, cpu, 0, 0); sibling = sys_perf_event_open(&rapl_attr, pid, cpu, leader, 0);

Quoted 6 lines

will create an even group (which means that both events require to be co-scheduled). If you then provided:

hrtimer_attr.read_format |= PERF_FORMAT_GROUP;
hrtimer_attr.sample_type |= PERF_SAMPLE_READ;

hrtimer_attr is only shared in an event group, and rapl needs its owen event group, right?

so what do you think the rapl counter should look like in userspace? showing it in perf-stat looks nice, right? :)

Quoted 7 lines · Zhang Rui

Quoted 5 lines · Peter Zijlstra

its some obscure perf feature:

leader = sys_perf_event_open(&hrtimer_attr, pid, cpu, 0, 0); sibling = sys_perf_event_open(&rapl_attr, pid, cpu, leader, 0);

Quoted 8 lines

Quoted 6 lines

will create an even group (which means that both events require to be co-scheduled). If you then provided:

hrtimer_attr.read_format |= PERF_FORMAT_GROUP;
hrtimer_attr.sample_type |= PERF_SAMPLE_READ;

hrtimer_attr is only shared in an event group, and rapl needs its owen event group, right?

Uhm, no. The idea is to group the hrtimer and rapl event in order to obtain rapl 'samples'.

That is, you get hrtimer samples which include the rapl count. For this we use the grouping construct where group siblings are always co-scheduled and can report on each others count.

Quoted 2 lines

so what do you think the rapl counter should look like in userspace? showing it in perf-stat looks nice, right? :)

Right, so the userspace interface would be using Lin's sysfs bits, which I still need to read up on. But the general idea is that each PMU gets a sysfs representation somewhere in the system topology reflecting its actual site (RAPL would be CPU local), this sysfs representation would then also allow you to discover all events it provides.

perf list will then use sysfs to discover all available events, and you can still use perf stat -e $foo to select it, where foo is some to be determined string that identifies the thing, maybe something like: rapl:watts or somesuch (with rapl identifying the pmu and watts the actual event for that pmu).

Btw., some 'perf list' thoughts. We could do a:

   perf list --help rapl:watts

Which gives the user some idea what an event does. Also, short descriptive line in perf list output would be nice:

$ perf list

List of pre-defined events (to be used in -e):

  cpu-cycles OR cycles                       [Hardware event]   # CPU cycles
  instructions                               [Hardware event]   # instructions executed

...

rapl:watts [Tracepoint] # watts usage

or something like that. Perhaps even a TUI for perf list, to browse between event types? (in that case it would probably be useful to make them collapse along natural grouping)

We want users/developers to discover new events, see and understand their purpose and combine them in not-seen-before ways.

Quoted 27 lines · Zhang Rui

Hi, all,

RAPL(running average power limit) is a new feature which provides mechanisms to enforce power consumption limit, on some new processors.

Generally speaking, by using RAPL, OS can set a power budget in a certain time window, and let Hardware to throttle the processor P/T-state to meet this energy limitation.

RAPL also provides a new MSR, i.e. MSR_PKG_ENERGY_STATUS, which reports the total amount of energy consumed by the package.

I'm not sure if to support RAPL or not, but anyway, it sounds like a good idea to export the energy status in perf.

So a new perf pmu and event to show the package energy consumed is introduced in this patch.

Here is what I get after applying the three patches,

#./perf stat -e energy test
Performance counter stats for 'test':

202 Joules cost by package
7.926001238 seconds time elapsed

Quoted 5 lines

Note that this patch set is made based on Peter's perf-pmu branch,
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/linux-2.6-perf.git
which provides better interfaces to register/unregister a new pmu.

any comment are welcome. :)

Nice,.. however:

- if it is a pure read-only counter without sampling support, expose it as such, don't fudge in the hrtimer stuff. Simply fail to create a sampling event.

SH has the same problem for its 'normal' PMU, the solution is to use event groups, Matt was looking at adding support to perf-record for that, if creating a sampling event fails, fall back to {hrtimer, $event} groups.

- since its a free-running, non-configurable counter, you can indeed act like its a 'software' event in that you can schedule consumers without constraints, however I don't think the PERF_COUNT_SW_* space is the right way to expose this counter.

Better would be to use the sysfs stuff Lin has been working on (for

which I still need to catch up on the latest discussions), it would then be tied to the pmu instance and appear/disappear when you load/ unload the module.

However for testing purposes I see why you'd want to have _a_ interface :-)

- it would be nice if you'd write the cpu detection a bit more readable, also, it looks like you forgot to check x86_vendor == X86_VENDOR_INTEL.

Quoted 7 lines

(quoted diff) +7 -0

+static int __init intel_rapl_init(void)
+{
+	/*
+	 * RAPL features are only supported on processors have a CPUID
+	 * signature with DisplayFamily_DisplayModel of 06_2AH, 06_2DH
+	 */
+	if (boot_cpu_data.x86 != 0x06 ||

Review

Quoted 10 lines

(quoted diff) +10 -0

+	    (boot_cpu_data.x86_model != 0x2A &&
+	    boot_cpu_data.x86_model != 0x2D))
+		return -ENODEV;
+
+	if (rapl_check_unit())
+		return -ENODEV;
+
+	perf_pmu_register(&rapl_pmu);
+	return 0;
+}

Maybe something like (see intel_pmu_init() for example):

if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)

    return -ENODEV;

if (boot_cpu_data.x86 != 0x06)

    return -ENODEV;

  switch (boot_cpu_data.x86_model) {
  case 0x2A: /* sandybridge ?! 32nm */
  case 0x2D: /* othermodel 32nm */
    break;

  default:
    return -ENODEV;
  }

Which again reminds me to ask of Intel, a comprehensive x86_model list, please?

Alternatively, you can create a X86_FEATURE_RAPL and simply use boot_cpu_has(X86_FEATURE_RAPL) (much like intel_ds_init() has).

Sorry that I have no good idea how to export the various tracepoints events automatically, so this work will take time.

Quoted 3 lines · Lin Ming

Sorry that I have no good idea how to export the various tracepoints events automatically, so this work will take time.

Well, we could start with just he hardware bits and leave the tracepoint bits for later, right?