Skip to contentSkip to conversation

perf/x86: Add Intel power cstate PMUs support

PATCH Superseded [email protected] 3 participants 18 messages Jul 27, 2015 – Aug 28, 2015

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 perf/x86: Add Intel power cstate PMUs support

From: Kan Liang <[email protected]>

This patch adds new PMUs to support power cstate related free running (read-only) counters. These counters may be used simultaneously by other tools, such as turbostat. However, it still make sense to implement them in perf. Because we can conveniently collect them together with other events, and allow to use them from tools without special MSR access code.

These counters include CORE_C*_RESIDENCY and PKG_C*_RESIDENCY. According to counters' scope and category, two PMUs are registered with the perf_event core subsystem.
- 'cstate_core': The counter is available for each processor core. The counters include CORE_C*_RESIDENCY.
- 'cstate_pkg': The counter is available for each physical package. The counters include PKG_C*_RESIDENCY.

The events are exposed in sysfs for use by perf stat and other tools. The files are:
/sys/devices/cstate_core/events/c*-residency /sys/devices/cstate_pkg/events/c*-residency

These events only support system-wide mode counting.
The /sys/devices/cstate_*/cpumask file can be used by tools to figure out which CPUs to monitor by default.

The PMU type (attr->type) is dynamically allocated and is available from /sys/devices/core_misc/type and /sys/device/cstate_*/type.

Sampling is not supported.

Here is an example.

- To caculate the fraction of time when the core is running in C6 state CORE_C6_time% = CORE_C6_RESIDENCY / TSC

 $ perf stat -x, -e"cstate_core/c6-residency/,msr/tsc/" -C0
   -- taskset -c 0 sleep 5
  11453299029,,cstate_core/c6-residency/,4998723500,100.00
  11476922553,,msr/tsc/,4998738042,100.00
 For sleep, 99.8% of time run in C6 state.
 $ perf stat -x, -e"cstate_core/c6-residency/,msr/tsc/" -C0
   -- taskset -c 0 busyloop
 1253316,,cstate_core/c6-residency/,4360969154,100.00
 10012635248,,msr/tsc/,4360972366,100.00
 For busyloop, 0.01% of time run in C6 state.

---

Changes since V1
- Kernel take care of mapping pkg c6 to correct msr

 arch/x86/kernel/cpu/Makefile                  |   1 +
 arch/x86/kernel/cpu/perf_event_intel_cstate.c | 750 ++++++++++++++++++++++++++
 2 files changed, 751 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_cstate.c
arch/x86/kernel/cpu/Makefile +1 -0
index 9bff687..4cdb270 100644
@@ -41,6 +41,7 @@ obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_p6.o perf_event_knc.o perf_event_p4.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_lbr.o perf_event_intel_ds.o perf_event_intel.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_rapl.o perf_event_intel_cqm.o
 obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_pt.o perf_event_intel_bts.o
+obj-$(CONFIG_CPU_SUP_INTEL)		+= perf_event_intel_cstate.o
 
 obj-$(CONFIG_PERF_EVENTS_INTEL_UNCORE)	+= perf_event_intel_uncore.o \
 					   perf_event_intel_uncore_snb.o \
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +750 -0

750 lines not on this page · raw message

Signature · 1 line
1.8.3.1
maintainer
#1 lore raw

On Mon, Jul 27, 2015 at 04:46:16AM -0400, [email protected] wrote:

As a general comment; this thing is unreadable. Far too much macro foo to instantiate the different PMUs.

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 111

Quoted 13 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +13 -0
+struct perf_power_cstate_event_msr {
+	int	id;
+	u64	msr;
+};
+
+enum perf_power_cstate_id {
+	/*
+	 * power_cstate events, generalized by the kernel:
+	 */
+	PERF_POWER_CORE_C1_RES = 0,
+	PERF_POWER_CORE_C3_RES,
+	PERF_POWER_CORE_C6_RES,
+	PERF_POWER_CORE_C7_RES,

These are two different PMUs, why are they in the same enum space?

Quoted 23 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +23 -0
+	PERF_POWER_PKG_C2_RES,
+	PERF_POWER_PKG_C3_RES,
+	PERF_POWER_PKG_C6_RES,
+	PERF_POWER_PKG_C7_RES,
+	PERF_POWER_PKG_C8_RES,
+	PERF_POWER_PKG_C9_RES,
+	PERF_POWER_PKG_C10_RES,
+
+	PERF_POWER_CSTATE_EVENT_MAX,       /* non-ABI */
+};
+
+struct power_cstate_pmu {
+	struct intel_power_cstate_type	*power_cstate_type;
+	struct pmu			*pmu;
+};
+
+
+static struct intel_power_cstate_type *empty_power_cstate[] = { NULL, };
+struct intel_power_cstate_type **power_cstate = empty_power_cstate;
+
+static struct perf_power_cstate_event_msr power_cstate_events[] = {
+	{ PERF_POWER_CORE_C1_RES, MSR_CORE_C1_RES },
+	{ PERF_POWER_CORE_C3_RES, MSR_CORE_C3_RESIDENCY },

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 144

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +10 -0
+	{ PERF_POWER_CORE_C6_RES, MSR_CORE_C6_RESIDENCY },
+	{ PERF_POWER_CORE_C7_RES, MSR_CORE_C7_RESIDENCY },
+	{ PERF_POWER_PKG_C2_RES, MSR_PKG_C2_RESIDENCY },
+	{ PERF_POWER_PKG_C3_RES, MSR_PKG_C3_RESIDENCY },
+	{ PERF_POWER_PKG_C6_RES, MSR_PKG_C6_RESIDENCY },
+	{ PERF_POWER_PKG_C7_RES, MSR_PKG_C7_RESIDENCY },
+	{ PERF_POWER_PKG_C8_RES, MSR_PKG_C8_RESIDENCY },
+	{ PERF_POWER_PKG_C9_RES, MSR_PKG_C9_RESIDENCY },
+	{ PERF_POWER_PKG_C10_RES, MSR_PKG_C10_RESIDENCY },
+};

What's the point of the first entry? You already know which index it is, no point in storing that again.

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 150

Quoted 7 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +6 -0
+
+EVENT_ATTR_STR(c1-residency, power_core_c1_res, "event=0x00");
+EVENT_ATTR_STR(c3-residency, power_core_c3_res, "event=0x01");
+EVENT_ATTR_STR(c6-residency, power_core_c6_res, "event=0x02");
+EVENT_ATTR_STR(c7-residency, power_core_c7_res, "event=0x03");
 
+EVENT_ATTR_STR(c2-residency, power_pkg_c2_res, "event=0x04");

I would very much expect this thing to start counting at 0 again.

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 158

Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +8 -0
+EVENT_ATTR_STR(c3-residency, power_pkg_c3_res, "event=0x05");
+EVENT_ATTR_STR(c6-residency, power_pkg_c6_res, "event=0x06");
+EVENT_ATTR_STR(c7-residency, power_pkg_c7_res, "event=0x07");
+EVENT_ATTR_STR(c8-residency, power_pkg_c8_res, "event=0x08");
+EVENT_ATTR_STR(c9-residency, power_pkg_c9_res, "event=0x09");
+EVENT_ATTR_STR(c10-residency, power_pkg_c10_res, "event=0x0a");
+
+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

Quoted 15 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +14 -0
+static cpumask_t power_cstate_pkg_cpu_mask;
 
+static int power_cstate_pmu_event_init(struct perf_event *event)
+{
+	u64 cfg = event->attr.config;
+	int ret = 0;
+
+	if (event->attr.type != event->pmu->type)
+		return -ENOENT;
+
+	/*
+	 * check event is known (determines counter)
+	 */
+	if (cfg >= PERF_POWER_CSTATE_EVENT_MAX)
+		return -EINVAL;

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 188

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +10 -0
+
+	/* unsupported modes and filters */
+	if (event->attr.exclude_user   ||
+	    event->attr.exclude_kernel ||
+	    event->attr.exclude_hv     ||
+	    event->attr.exclude_idle   ||
+	    event->attr.exclude_host   ||
+	    event->attr.exclude_guest  ||
+	    event->attr.sample_period) /* no sampling */
+		return -EINVAL;

Same as with the MSR thing, but worse. What happens if we hand a config value for the 'wrong' PMU type?

What happens if we hand a pkg c10 config to one that doesn't have c10?

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 202

Quoted 13 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +13 -0
+	/* must be done before validate_group */
+	event->hw.event_base = power_cstate_events[cfg].msr;
+	event->hw.config = cfg;
+	event->hw.idx = -1;
+
+	return ret;
+}
+
+static inline u64 power_cstate_pmu_read_counter(struct perf_event *event)
+{
+	u64 val;
+
+	rdmsrl_safe(event->hw.event_base, &val);

No, at this point you had better know if that MSR exists or not. If it does not the event should not exist.

Quoted 2 lines
(quoted diff) +2 -0
+	return val;
+}
#2 lore raw
Quoted 58 lines · Peter Zijlstra

On Mon, Jul 27, 2015 at 04:46:16AM -0400, [email protected] wrote:

As a general comment; this thing is unreadable. Far too much macro foo to instantiate the different PMUs.

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 111

Quoted 13 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +13 -0
+struct perf_power_cstate_event_msr {
+     int     id;
+     u64     msr;
+};
+
+enum perf_power_cstate_id {
+     /*
+      * power_cstate events, generalized by the kernel:
+      */
+     PERF_POWER_CORE_C1_RES = 0,
+     PERF_POWER_CORE_C3_RES,
+     PERF_POWER_CORE_C6_RES,
+     PERF_POWER_CORE_C7_RES,

These are two different PMUs, why are they in the same enum space?

Quoted 23 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +23 -0
+     PERF_POWER_PKG_C2_RES,
+     PERF_POWER_PKG_C3_RES,
+     PERF_POWER_PKG_C6_RES,
+     PERF_POWER_PKG_C7_RES,
+     PERF_POWER_PKG_C8_RES,
+     PERF_POWER_PKG_C9_RES,
+     PERF_POWER_PKG_C10_RES,
+
+     PERF_POWER_CSTATE_EVENT_MAX,       /* non-ABI */
+};
+
+struct power_cstate_pmu {
+     struct intel_power_cstate_type  *power_cstate_type;
+     struct pmu                      *pmu;
+};
+
+
+static struct intel_power_cstate_type *empty_power_cstate[] = { NULL, };
+struct intel_power_cstate_type **power_cstate = empty_power_cstate;
+
+static struct perf_power_cstate_event_msr power_cstate_events[] = {
+     { PERF_POWER_CORE_C1_RES, MSR_CORE_C1_RES },
+     { PERF_POWER_CORE_C3_RES, MSR_CORE_C3_RESIDENCY },

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 144

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +10 -0
+     { PERF_POWER_CORE_C6_RES, MSR_CORE_C6_RESIDENCY },
+     { PERF_POWER_CORE_C7_RES, MSR_CORE_C7_RESIDENCY },
+     { PERF_POWER_PKG_C2_RES, MSR_PKG_C2_RESIDENCY },
+     { PERF_POWER_PKG_C3_RES, MSR_PKG_C3_RESIDENCY },
+     { PERF_POWER_PKG_C6_RES, MSR_PKG_C6_RESIDENCY },
+     { PERF_POWER_PKG_C7_RES, MSR_PKG_C7_RESIDENCY },
+     { PERF_POWER_PKG_C8_RES, MSR_PKG_C8_RESIDENCY },
+     { PERF_POWER_PKG_C9_RES, MSR_PKG_C9_RESIDENCY },
+     { PERF_POWER_PKG_C10_RES, MSR_PKG_C10_RESIDENCY },
+};

What's the point of the first entry? You already know which index it is, no point in storing that again.

Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+EVENT_ATTR_STR(c1-residency, power_core_c1_res, "event=0x00");

I would not use an event code of 0. Start at 0x1

Quoted 8 lines

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 150

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +4 -0
+EVENT_ATTR_STR(c3-residency, power_core_c3_res, "event=0x01");
+EVENT_ATTR_STR(c6-residency, power_core_c6_res, "event=0x02");
+EVENT_ATTR_STR(c7-residency, power_core_c7_res, "event=0x03");
 
+EVENT_ATTR_STR(c2-residency, power_pkg_c2_res, "event=0x04");

I would very much expect this thing to start counting at 0 again.

Quoted 11 lines

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 158

Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +8 -0
+EVENT_ATTR_STR(c3-residency, power_pkg_c3_res, "event=0x05");
+EVENT_ATTR_STR(c6-residency, power_pkg_c6_res, "event=0x06");
+EVENT_ATTR_STR(c7-residency, power_pkg_c7_res, "event=0x07");
+EVENT_ATTR_STR(c8-residency, power_pkg_c8_res, "event=0x08");
+EVENT_ATTR_STR(c9-residency, power_pkg_c9_res, "event=0x09");
+EVENT_ATTR_STR(c10-residency, power_pkg_c10_res, "event=0x0a");
+
+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my understanding.

Quoted 20 lines
Quoted 20 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +19 -0
+static cpumask_t power_cstate_pkg_cpu_mask;
 
+static int power_cstate_pmu_event_init(struct perf_event *event)
+{
+     u64 cfg = event->attr.config;
+     int ret = 0;
+
+     if (event->attr.type != event->pmu->type)
+             return -ENOENT;
+
+     /*
+      * check event is known (determines counter)
+      */
+     if (cfg >= PERF_POWER_CSTATE_EVENT_MAX)
+             return -EINVAL;
+
+     /* unsupported modes and filters */
+     if (event->attr.exclude_user   ||
+         event->attr.exclude_kernel ||
+         event->attr.exclude_hv     ||
Quoted 9 lines

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 188

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +5 -0
+         event->attr.exclude_idle   ||
+         event->attr.exclude_host   ||
+         event->attr.exclude_guest  ||
+         event->attr.sample_period) /* no sampling */
+             return -EINVAL;

Same as with the MSR thing, but worse. What happens if we hand a config value for the 'wrong' PMU type?

I think that's taken care of by the first test in the function.

Quoted 23 lines

What happens if we hand a pkg c10 config to one that doesn't have c10?

arch/x86/kernel/cpu/perf_event_intel_cstate.c line 202

Quoted 13 lines
arch/x86/kernel/cpu/perf_event_intel_cstate.c new file +13 -0
+     /* must be done before validate_group */
+     event->hw.event_base = power_cstate_events[cfg].msr;
+     event->hw.config = cfg;
+     event->hw.idx = -1;
+
+     return ret;
+}
+
+static inline u64 power_cstate_pmu_read_counter(struct perf_event *event)
+{
+     u64 val;
+
+     rdmsrl_safe(event->hw.event_base, &val);

No, at this point you had better know if that MSR exists or not. If it does not the event should not exist.

Quoted 2 lines
(quoted diff) +2 -0
+     return val;
+}

I understand that these metrics are useful and needed however if I look at the broader
picture I see many PMUs doing similar things or appearing different when they are actually
very close. It would be nice to have a more unified approach. You have RAPL (client, server)
which appears as the power PMU. You have the PCU uncore on servers which also provides
C-state residency info. Yet, all these appear differently and expose events with different names.
I think we could benefit from a more unifie approach here such that you would be able to do

$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

author
#3 lore raw
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my understanding.

CORE_C*_RESIDENCY are available for physical processor core.
So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Quoted 11 lines
Quoted 1 line

I understand that these metrics are useful and needed however if I look at the broader picture I see many PMUs doing similar things or appearing different when they are actually very close. It would be nice to have a more unified approach. You have RAPL (client, server) which appears as the power PMU. You have the PCU uncore on servers which also provides C-state residency info. Yet, all these appear differently and expose events with different names.
I think we could benefit from a more unifie approach here such that you would be able to do

Quoted 4 lines
$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

Yes, I agree. I'll think about it.

Thanks,
Kan ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#4 lore raw
Quoted 10 lines · Kan
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my understanding.

Quoted 6 lines

CORE_C*_RESIDENCY are available for physical processor core.
So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Did you restrict these events to system-wide mode only?

Quoted 20 lines
Quoted 11 lines
Quoted 1 line

I understand that these metrics are useful and needed however if I look at the broader picture I see many PMUs doing similar things or appearing different when they are actually very close. It would be nice to have a more unified approach. You have RAPL (client, server) which appears as the power PMU. You have the PCU uncore on servers which also provides C-state residency info. Yet, all these appear differently and expose events with different names.
I think we could benefit from a more unifie approach here such that you would be able to do

Quoted 4 lines
$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

Yes, I agree. I'll think about it.

Thanks,
Kan

author
#5 lore raw
Quoted 13 lines · Kan
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my understanding.

Quoted 8 lines
Quoted 6 lines

CORE_C*_RESIDENCY are available for physical processor core. So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Did you restrict these events to system-wide mode only?

Yes ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#6 lore raw
Quoted 13 lines · Kan
Quoted 13 lines · Kan
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my understanding.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

CORE_C*_RESIDENCY are available for physical processor core. So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Did you restrict these events to system-wide mode only?

Ok, so that means that your cpumask includes one HT per physical core. But then, the result is not the simple aggregation of all the N/2 CPUs.

author
#7 lore raw
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my

understanding.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

CORE_C*_RESIDENCY are available for physical processor core. So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

Did you restrict these events to system-wide mode only?

Ok, so that means that your cpumask includes one HT per physical core. But then, the result is not the simple aggregation of all the N/2 CPUs.

The counter counts per physical core. The result is the aggregation of
all HT cpus in same physical core.
It's similar as per socket counter. But the scope is physical core now. ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#8 lore raw
Quoted 16 lines · Kan
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my

understanding.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

CORE_C*_RESIDENCY are available for physical processor core. So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

Did you restrict these events to system-wide mode only?

Ok, so that means that your cpumask includes one HT per physical core. But then, the result is not the simple aggregation of all the N/2 CPUs.

The counter counts per physical core. The result is the aggregation of all HT cpus in same physical core.

But then don't you need to divide by 2 to get a meaningful result?

author
#9 lore raw
Quoted 30 lines

-----Original Message-----
From: Stephane Eranian [mailto:[email protected]]
Sent: Thursday, August 06, 2015 4:38 PM
To: Liang, Kan
Cc: Peter Zijlstra; [email protected]; Arnaldo Carvalho de Melo; [email protected]; LKML
Subject: Re: [PATCH V2 1/1] perf/x86: Add Intel power cstate PMUs support

Quoted 18 lines · Kan
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my

understanding.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 1 line

CORE_C*_RESIDENCY are available for physical processor core. So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Did you restrict these events to system-wide mode only?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Ok, so that means that your cpumask includes one HT per physical core. But then, the result is not the simple aggregation of all the N/2 CPUs.

The counter counts per physical core. The result is the aggregation of all HT cpus in same physical core.

But then don't you need to divide by 2 to get a meaningful result?

Rethink of it. I think I was unclear about the aggregation of all HT cpus in same physical core.

physical core Cstate should equal to min(logical core C-state). So only all logical core enters C6-state, the physical core enters C6-state, then CORE_C6_RESIDENCY counts.

So if we only count on one logical core/HT for CORE_C6_RESIDENCY. We don't need to divide by 2. The count result is the residency when all logical core in C6 (some may deeper).

Thanks,
Kan

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#10 lore raw
Quoted 41 lines · Kan
Quoted 30 lines

-----Original Message-----
From: Stephane Eranian [mailto:[email protected]]
Sent: Thursday, August 06, 2015 4:38 PM
To: Liang, Kan
Cc: Peter Zijlstra; [email protected]; Arnaldo Carvalho de Melo; [email protected]; LKML
Subject: Re: [PATCH V2 1/1] perf/x86: Add Intel power cstate PMUs support

Quoted 18 lines · Kan
Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my

understanding.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 1 line

CORE_C*_RESIDENCY are available for physical processor core. So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Did you restrict these events to system-wide mode only?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Ok, so that means that your cpumask includes one HT per physical core. But then, the result is not the simple aggregation of all the N/2 CPUs.

The counter counts per physical core. The result is the aggregation of all HT cpus in same physical core.

But then don't you need to divide by 2 to get a meaningful result?

Rethink of it. I think I was unclear about the aggregation of all HT cpus in same physical core.

Quoted 8 lines

physical core Cstate should equal to min(logical core C-state). So only all logical core enters C6-state, the physical core enters C6-state, then CORE_C6_RESIDENCY counts.

So if we only count on one logical core/HT for CORE_C6_RESIDENCY. We don't need to divide by 2. The count result is the residency when all logical core in C6 (some may deeper).

Ok and here you are assuming you are only measuring one logical CPU per physical core. If this is the case, then I think you are alright. But I wonder what
you'd get when perf stat -a aggregates across all measured CPUs, i.e., one CPU per core.

author
#11 lore raw
Quoted 38 lines
Quoted 20 lines
Quoted 20 lines · Kan
Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my

understanding.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

CORE_C*_RESIDENCY are available for physical processor core. So logical processor in same physical processor core share the same counter.
I think we need the cpumask to identify the default logical processor which do counting.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

Did you restrict these events to system-wide mode only?

Ok, so that means that your cpumask includes one HT per physical

core.

Quoted 14 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

But then, the result is not the simple aggregation of all the N/2 CPUs.

The counter counts per physical core. The result is the aggregation of all HT cpus in same physical core.

But then don't you need to divide by 2 to get a meaningful result?

Rethink of it. I think I was unclear about the aggregation of all HT cpus in same physical core.

physical core Cstate should equal to min(logical core C-state). So only all logical core enters C6-state, the physical core enters C6-state, then CORE_C6_RESIDENCY counts.

Quoted 8 lines
Quoted 4 lines

So if we only count on one logical core/HT for CORE_C6_RESIDENCY. We don't need to divide by 2. The count result is the residency when all logical core in C6 (some may deeper).

Ok and here you are assuming you are only measuring one logical CPU per physical core. If this is the case, then I think you are alright. But I wonder what you'd get when perf stat -a aggregates across all measured CPUs, i.e., one CPU per core.

Just add them all together.
I think we do the same thing for other PMUs as well.
For uncore or rapl, we get meaningful result by applying --per-socket. Here we can use --per-core.

Thanks,
Kan

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

author
#12 lore raw
In reply to Stephane Eranian
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 2 lines

I understand that these metrics are useful and needed however if I look at the broader picture I see many PMUs doing similar things or appearing different when they are actually very close. It would be nice to have a more unified approach. You have RAPL (client, server) which appears as the power PMU. You have the PCU uncore on servers which also provides C-state residency info. Yet, all these appear differently and expose events with different names.
I think we could benefit from a more unifie approach here such that you would be able to do

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

Yes, I agree. I'll think about it.

Hi Stephane,

I thought more about your suggestion regarding to create a unified power PMU for all related events include RAPL and residency.
It looks we can benefit from a simple unified name, but it also brings too much confusion.
- cstate residency is the time of the core/socket in specific cstate. While RAPL event is the power core/socket which consumed. They have different concepts.
- cstate residency includes both per-core and per-socket events. RAPL events is only per-socket. So the CPU mask is different. It's very confused that the events in same PMU has different CPU mask.

So I think it should be better to use different PMUs for RAPL and residency.

What do you think?

Thanks,
Kan

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#13 lore raw
Quoted 33 lines · Kan
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 2 lines

I understand that these metrics are useful and needed however if I look at the broader picture I see many PMUs doing similar things or appearing different when they are actually very close. It would be nice to have a more unified approach. You have RAPL (client, server) which appears as the power PMU. You have the PCU uncore on servers which also provides C-state residency info. Yet, all these appear differently and expose events with different names.
I think we could benefit from a more unifie approach here such that you would be able to do

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

Yes, I agree. I'll think about it.

Hi Stephane,

I thought more about your suggestion regarding to create a unified power PMU for all related events include RAPL and residency.
It looks we can benefit from a simple unified name, but it also brings too much confusion.
- cstate residency is the time of the core/socket in specific cstate. While RAPL event is the power core/socket which consumed. They have different concepts.
- cstate residency includes both per-core and per-socket events. RAPL events is only per-socket. So the CPU mask is different. It's very confused that the events in same PMU has different CPU mask.

Quoted 4 lines

So I think it should be better to use different PMUs for RAPL and residency.

What do you think?

Well, you are maybe confusing events with PMU. If you look at the core PMU, it cover many events measuring vastly different aspects of the core. Some events are per-thread, others are per-core.

Here, I was thinking it would be good to have some power// PMU with many events covering cstate residency, energy consumption. And yes, some events would be per-socket, others per-core.

author
#14 lore raw
Quoted 43 lines · Kan
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

I understand that these metrics are useful and needed however if I look at the broader picture I see many PMUs doing similar things or appearing different when they are actually very close. It would be nice to have a more unified approach. You have RAPL (client, server) which appears as the power PMU. You have the PCU uncore

on

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

servers which also provides C-state residency info. Yet, all these appear differently and expose events with different names. I think we could benefit from a more unifie approach here such that you would be able to do

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

Yes, I agree. I'll think about it.

Hi Stephane,

Quoted 10 lines

I thought more about your suggestion regarding to create a unified power PMU for all related events include RAPL and residency. It looks we can benefit from a simple unified name, but it also brings too much confusion.
- cstate residency is the time of the core/socket in specific cstate. While RAPL event is the power core/socket which consumed. They have different concepts.
- cstate residency includes both per-core and per-socket events. RAPL events is only per-socket. So the CPU mask is different. It's very confused that the events in same PMU has different CPU

mask.

Quoted 2 lines

So I think it should be better to use different PMUs for RAPL and

residency.

Quoted 3 lines

What do you think?

Quoted 7 lines

Well, you are maybe confusing events with PMU. If you look at the core PMU, it cover many events measuring vastly different aspects of the core. Some events are per-thread, others are per-core.

Here, I was thinking it would be good to have some power// PMU with many events covering cstate residency, energy consumption. And yes, some events would be per-socket, others per-core.

So you agree to create two new cstate PMUs (per-core and per-socket) to cover cstate residency?
If so, I will start to implement the V3 version for two new PMUs.

Thanks,
Kan

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#15 lore raw
Quoted 47 lines · Kan
Quoted 47 lines · Kan
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

I understand that these metrics are useful and needed however if I look at the broader picture I see many PMUs doing similar things or appearing different when they are actually very close. It would be nice to have a more unified approach. You have RAPL (client, server) which appears as the power PMU. You have the PCU uncore

on

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

servers which also provides C-state residency info. Yet, all these appear differently and expose events with different names. I think we could benefit from a more unifie approach here such that you would be able to do

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

Yes, I agree. I'll think about it.

Hi Stephane,

Quoted 10 lines

I thought more about your suggestion regarding to create a unified power PMU for all related events include RAPL and residency. It looks we can benefit from a simple unified name, but it also brings too much confusion.
- cstate residency is the time of the core/socket in specific cstate. While RAPL event is the power core/socket which consumed. They have different concepts.
- cstate residency includes both per-core and per-socket events. RAPL events is only per-socket. So the CPU mask is different. It's very confused that the events in same PMU has different CPU

mask.

Quoted 2 lines

So I think it should be better to use different PMUs for RAPL and

residency.

Quoted 3 lines

What do you think?

Well, you are maybe confusing events with PMU. If you look at the core PMU, it cover many events measuring vastly different aspects of the core. Some events are per-thread, others are per-core.

Quoted 8 lines
Quoted 3 lines

Here, I was thinking it would be good to have some power// PMU with many events covering cstate residency, energy consumption. And yes, some events would be per-socket, others per-core.

So you agree to create two new cstate PMUs (per-core and per-socket) to cover cstate residency?
If so, I will start to implement the V3 version for two new PMUs.

I did not say that. Instead I said there is some benefits in having everything under a power// PMU, including possibly portability to other non x86 architectures.

author
#16 lore raw
Quoted 59 lines · Kan
Quoted 3 lines
Quoted 3 lines

On Tue, Aug 25, 2015 at 1:15 PM, Liang, Kan <[email protected]>

Quoted 8 lines · wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

I understand that these metrics are useful and needed however if I look at the broader picture I see many PMUs doing similar things or appearing different when they are actually very close. It would be nice to have a more unified approach. You have RAPL (client,
server) which appears as the power PMU. You have the PCU

uncore

Quoted 29 lines
Quoted 29 lines

on

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

servers which also provides C-state residency info. Yet, all these appear differently and expose events with different names. I think we could benefit from a more unifie approach here such that you would be able to do

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
$    perf stat -a -e power/c6-residency/, power/energy-pkg/

on client and server without having to change the pmu name of the event names.

Yes, I agree. I'll think about it.

Hi Stephane,

Quoted 11 lines

I thought more about your suggestion regarding to create a unified power PMU for all related events include RAPL and residency. It looks we can benefit from a simple unified name, but it also brings too much confusion.
- cstate residency is the time of the core/socket in specific cstate. While RAPL event is the power core/socket which consumed. They have different concepts.
- cstate residency includes both per-core and per-socket events. RAPL events is only per-socket. So the CPU mask is different. It's very confused that the events in same PMU has different
CPU

mask.

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

So I think it should be better to use different PMUs for RAPL and

residency.

Quoted 3 lines

What do you think?

Well, you are maybe confusing events with PMU. If you look at the core PMU, it cover many events measuring vastly different aspects of

the core.

Quoted 6 lines
Quoted 5 lines

Some events are per-thread, others are per-core.

Here, I was thinking it would be good to have some power// PMU with many events covering cstate residency, energy consumption. And yes, some events would be per-socket, others per-core.

Quoted 7 lines
Quoted 4 lines

So you agree to create two new cstate PMUs (per-core and per-socket) to cover cstate residency?
If so, I will start to implement the V3 version for two new PMUs.

I did not say that. Instead I said there is some benefits in having everything under a power// PMU, including possibly portability to other non x86 architectures.

But the events in power// PMU will be per-socket or per-core. How should we handle it?
What should we show in cpumask?

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

maintainer
#17 lore raw
In reply to Stephane Eranian
Quoted 7 lines · Stephane Eranian
Quoted 4 lines · Peter Zijlstra
Quoted 1 line

+static cpumask_t power_cstate_core_cpu_mask;

That one typically does not need a cpumask.

You need to pick one CPU out of the multi-core. But it is for client parts thus there is only one socket. At least this is my understanding.

Oh crud, core as in a group of SMT. There's so much core as in logical cpu usage around these days I got confused :/