Skip to contentSkip to conversation

perf/x86: simplify PEBS event constraint management

PATCH Superseded Stephane Eranian 1 participant 5 messages Aug 8, 2014

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover perf/x86: simplify PEBS event constraint management

This patchkit revamps the PEBS tables on most Intel CPUs, both simplifying and fixing a couple of problems on Haswell processors:
- All PEBS event supported
- data address and data source is now reported correctly for all events

We simplify the tables by using the fact that non-PEBS event when used in PEBS mode do not generate any PEBS records , thus they are safe to use. This helps simplify the PEBS event constraints because they do not need to list all suported PEBS events anymore. They only need to list the special PEBS events, such as precise store and load latency. This avoid some issues where the kernel is missing some PEBS events.

Andi Kleen (2):
perf/x86: Revamp PEBS event selection
perf/x86: Don't mark DataLA addresses as store

Stephane Eranian (2):
perf/x86: fix data source encoding issues for load latency/precise
store
perf/x86: code cleanups for __intel_pmu_pebs_event()

 arch/x86/include/asm/perf_event.h         |    8 ++
 arch/x86/kernel/cpu/perf_event.h          |   48 +++++++-
 arch/x86/kernel/cpu/perf_event_intel_ds.c |  169 +++++++++++------------------
 include/linux/perf_event.h                |    9 +-
 4 files changed, 124 insertions(+), 110 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH 1/4 perf/x86: Revamp PEBS event selection

From: Andi Kleen <[email protected]>

The basic idea is that it does not make sense to list all PEBS events individually. The list is very long, sometimes outdated and the hardware doesn't need it. If an event does not support PEBS it will just not count, there is no security issue.

We need to only list events that something special, like supporting load or store addresses.

This vastly simplifies the PEBS event selection. It also speeds up the scheduling because the scheduler doesn't have to walk as many constraints.

Bugs fixed:
- We do not allow setting forbidden flags with PEBS anymore (SDM 18.9.4), except for the special cycle event.
This is done using a new constraint macro that also
matches on the event flags.
- Correct DataLA and load/store/na flags reporting on Haswell [Requires a followon patch]
- We did not allow all PEBS events on Haswell:
We were missing some valid subevents in d1-d2 (MEM_LOAD_UOPS_RETIRED.*, MEM_LOAD_UOPS_RETIRED_L3_HIT_RETIRED.*)

This includes the changes proposed by Stephane earlier and obsoletes his patchkit (except for some changes on pre Sandy Bridge/Silvermont CPUs)

I only did Sandy Bridge and Silvermont and later so far, mostly because these are the parts I could directly confirm the hardware behavior with hardware architects. Also I do not believe the older CPUs have any
missing events in their PEBS list, so there's no pressing
need to change them.

I did not implement the flag proposed by Peter to allow setting forbidden flags. If really needed this could be implemented on to of this patch.

v2: Fix broken store events on SNB/IVB (Stephane Eranian)
v3: More fixes. Rename some arguments (Stephane Eranian)
v4: List most Haswell events individually again to report memory operation type correctly.
Add new flags to describe load/store/na for datala. Update description.

---
 arch/x86/include/asm/perf_event.h         |    8 +++
 arch/x86/kernel/cpu/perf_event.h          |   48 +++++++++++--
 arch/x86/kernel/cpu/perf_event_intel_ds.c |  107 ++++++++++-------------------
 3 files changed, 85 insertions(+), 78 deletions(-)
arch/x86/include/asm/perf_event.h +8 -0
index 8249df4..8dfc9fd 100644
@@ -51,6 +51,14 @@
 	 ARCH_PERFMON_EVENTSEL_EDGE  |	\
 	 ARCH_PERFMON_EVENTSEL_INV   |	\
 	 ARCH_PERFMON_EVENTSEL_CMASK)
+#define X86_ALL_EVENT_FLAGS  			\
+	(ARCH_PERFMON_EVENTSEL_EDGE |  		\
+	 ARCH_PERFMON_EVENTSEL_INV | 		\
+	 ARCH_PERFMON_EVENTSEL_CMASK | 		\
+	 ARCH_PERFMON_EVENTSEL_ANY | 		\
+	 ARCH_PERFMON_EVENTSEL_PIN_CONTROL | 	\
+	 HSW_IN_TX | 				\
+	 HSW_IN_TX_CHECKPOINTED)
 #define AMD64_RAW_EVENT_MASK		\
 	(X86_RAW_EVENT_MASK          |  \
 	 AMD64_EVENTSEL_EVENT)
arch/x86/kernel/cpu/perf_event.h +42 -6

68 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event_intel_ds.c +35 -72

145 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#2 lore raw

PATCH 2/4 perf/x86: Don't mark DataLA addresses as store

In reply to Stephane Eranian

From: Andi Kleen <[email protected]>

Haswell supports reporting the data address for a range of PEBS events, including:

UOPS_RETIRED.ALL
MEM_UOPS_RETIRED.STLB_MISS_LOADS
MEM_UOPS_RETIRED.STLB_MISS_STORES
MEM_UOPS_RETIRED.LOCK_LOADS
MEM_UOPS_RETIRED.SPLIT_LOADS
MEM_UOPS_RETIRED.SPLIT_STORES
MEM_UOPS_RETIRED.ALL_LOADS
MEM_UOPS_RETIRED.ALL_STORES
MEM_LOAD_UOPS_RETIRED.L1_HIT
MEM_LOAD_UOPS_RETIRED.L2_HIT
MEM_LOAD_UOPS_RETIRED.L3_HIT
MEM_LOAD_UOPS_RETIRED.L1_MISS
MEM_LOAD_UOPS_RETIRED.L2_MISS
MEM_LOAD_UOPS_RETIRED.L3_MISS
MEM_LOAD_UOPS_RETIRED.HIT_LFB
MEM_LOAD_UOPS_L3_HIT_RETIRED.XSNP_MISS
MEM_LOAD_UOPS_L3_HIT_RETIRED.XSNP_HIT
MEM_LOAD_UOPS_L3_HIT_RETIRED.XSNP_HITM
MEM_LOAD_UOPS_L3_HIT_RETIRED.XSNP_NONE
MEM_LOAD_UOPS_L3_MISS_RETIRED.LOCAL_DRAM

This facility was already enabled earlier with the original Haswell perf changes.

However these addresses were always reports as stores by perf, which is wrong, as they could be loads too. The hardware does not distinguish loads and stores for these instructions, so there's no (cheap) way for the profiler to find out.

Change the type to PERF_MEM_OP_NA instead.

---
 arch/x86/kernel/cpu/perf_event_intel_ds.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/perf_event_intel_ds.c +1 -1
index aca77e9..a9b60f3 100644
@@ -114,7 +114,7 @@ static u64 precise_store_data_hsw(struct perf_event *event, u64 status)
 	u64 cfg = event->hw.config & INTEL_ARCH_EVENT_MASK;
 
 	dse.val = 0;
-	dse.mem_op = PERF_MEM_OP_STORE;
+	dse.mem_op = PERF_MEM_OP_NA;
 	dse.mem_lvl = PERF_MEM_LVL_NA;
 
 	/*
Signature · 1 line
1.7.9.5
#3 lore raw

PATCH 3/4 perf/x86: fix data source encoding issues for load latency/precise store

In reply to Stephane Eranian

This patch fixes issues introuduce by Andi's previous patch 'Revamp PEBS' series.

This patch fixes the following:
- 0 is not a valid init value for data source. Define PERF_MEM_NA as the default value

- precise_store_data_hsw() encode the mem op type whenever we can
- precise_store_data_hsw set the default data source correctly

---
 arch/x86/kernel/cpu/perf_event_intel_ds.c |   11 +++++++----
 include/linux/perf_event.h                |    9 ++++++++-
 2 files changed, 15 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/perf_event_intel_ds.c +7 -4
index a9b60f3..67919ce 100644
@@ -113,9 +113,12 @@ static u64 precise_store_data_hsw(struct perf_event *event, u64 status)
 	union perf_mem_data_src dse;
 	u64 cfg = event->hw.config & INTEL_ARCH_EVENT_MASK;
 
-	dse.val = 0;
-	dse.mem_op = PERF_MEM_OP_NA;
-	dse.mem_lvl = PERF_MEM_LVL_NA;
+	dse.val = PERF_MEM_NA;
+
+	if (event->hw.flags & PERF_X86_EVENT_PEBS_ST_HSW)
+		dse.mem_op = PERF_MEM_OP_STORE;
+	else if (event->hw.flags & PERF_X86_EVENT_PEBS_LD_HSW)
+		dse.mem_op = PERF_MEM_OP_LOAD;
 
 	/*
 	 * L1 info only valid for following events:
@@ -126,7 +129,7 @@ static u64 precise_store_data_hsw(struct perf_event *event, u64 status)
 	 * MEM_UOPS_RETIRED.ALL_STORES
 	 */
 	if (cfg != 0x12d0 && cfg != 0x22d0 && cfg != 0x42d0 && cfg != 0x82d0)
-		return dse.mem_lvl;
+		return dse.val;
 
 	if (status & 1)
 		dse.mem_lvl = PERF_MEM_LVL_L1 | PERF_MEM_LVL_HIT;
include/linux/perf_event.h +8 -1
index 707617a..8b206aa 100644
@@ -604,6 +604,13 @@ struct perf_sample_data {
 	u64				txn;
 };
 
+/* default value for data source */
+#define PERF_MEM_NA (PERF_MEM_S(OP, NA)   |\
+		    PERF_MEM_S(LVL, NA)   |\
+		    PERF_MEM_S(SNOOP, NA) |\
+		    PERF_MEM_S(LOCK, NA)  |\
+		    PERF_MEM_S(TLB, NA))
+
 static inline void perf_sample_data_init(struct perf_sample_data *data,
 					 u64 addr, u64 period)
 {
@@ -616,7 +623,7 @@ static inline void perf_sample_data_init(struct perf_sample_data *data,
 	data->regs_user.regs = NULL;
 	data->stack_user_size = 0;
 	data->weight = 0;
-	data->data_src.val = 0;
+	data->data_src.val = PERF_MEM_NA;
 	data->txn = 0;
 }
 
Signature · 1 line
1.7.9.5
#4 lore raw

PATCH 4/4 perf/x86: code cleanups for __intel_pmu_pebs_event()

In reply to Stephane Eranian

This patch makes the code more readable. It also renames precise_store_data_hsw() to precise_datala_hsw() because the function is called for both loads and stores on HSW.

---
 arch/x86/kernel/cpu/perf_event_intel_ds.c |   65 ++++++++++++++---------------
 1 file changed, 31 insertions(+), 34 deletions(-)
arch/x86/kernel/cpu/perf_event_intel_ds.c +31 -34
index 67919ce..97890c2 100644
@@ -108,7 +108,7 @@ static u64 precise_store_data(u64 status)
 	return val;
 }
 
-static u64 precise_store_data_hsw(struct perf_event *event, u64 status)
+static u64 precise_datala_hsw(struct perf_event *event, u64 status)
 {
 	union perf_mem_data_src dse;
 	u64 cfg = event->hw.config & INTEL_ARCH_EVENT_MASK;
@@ -825,6 +825,10 @@ static inline u64 intel_hsw_transaction(struct pebs_record_hsw *pebs)
 static void __intel_pmu_pebs_event(struct perf_event *event,
 				   struct pt_regs *iregs, void *__pebs)
 {
+#define PERF_X86_EVENT_PEBS_HSW_PREC \
+		(PERF_X86_EVENT_PEBS_ST_HSW | \
+		 PERF_X86_EVENT_PEBS_LD_HSW | \
+		 PERF_X86_EVENT_PEBS_NA_HSW)
 	/*
 	 * We cast to the biggest pebs_record but are careful not to
 	 * unconditionally access the 'extra' entries.
@@ -834,47 +838,40 @@ static void __intel_pmu_pebs_event(struct perf_event *event,
 	struct perf_sample_data data;
 	struct pt_regs regs;
 	u64 sample_type;
-	int fll, fst;
+	int fll, fst, dsrc;
+	int fl = event->hw.flags;
 
 	if (!intel_pmu_save_and_restart(event))
 		return;
 
-	fll = event->hw.flags & PERF_X86_EVENT_PEBS_LDLAT;
-	fst = event->hw.flags & (PERF_X86_EVENT_PEBS_ST |
-				 PERF_X86_EVENT_PEBS_ST_HSW |
-				 PERF_X86_EVENT_PEBS_LD_HSW |
-				 PERF_X86_EVENT_PEBS_NA_HSW);
+	sample_type = event->attr.sample_type;
+	dsrc = sample_type & PERF_SAMPLE_DATA_SRC;
+
+	fll = fl & PERF_X86_EVENT_PEBS_LDLAT;
+	fst = fl & (PERF_X86_EVENT_PEBS_ST | PERF_X86_EVENT_PEBS_HSW_PREC);
 
 	perf_sample_data_init(&data, 0, event->hw.last_period);
 
 	data.period = event->hw.last_period;
-	sample_type = event->attr.sample_type;
 
 	/*
-	 * if PEBS-LL or PreciseStore
+	 * Use latency for weight (only avail with PEBS-LL)
 	 */
-	if (fll || fst) {
-		/*
-		 * Use latency for weight (only avail with PEBS-LL)
-		 */
-		if (fll && (sample_type & PERF_SAMPLE_WEIGHT))
-			data.weight = pebs->lat;
-
-		/*
-		 * data.data_src encodes the data source
-		 */
-		if (sample_type & PERF_SAMPLE_DATA_SRC) {
-			if (fll)
-				data.data_src.val = load_latency_data(pebs->dse);
-			else if (event->hw.flags &
-					(PERF_X86_EVENT_PEBS_ST_HSW|
-					 PERF_X86_EVENT_PEBS_LD_HSW|
-					 PERF_X86_EVENT_PEBS_NA_HSW))
-				data.data_src.val =
-					precise_store_data_hsw(event, pebs->dse);
-			else
-				data.data_src.val = precise_store_data(pebs->dse);
-		}
+	if (fll && (sample_type & PERF_SAMPLE_WEIGHT))
+		data.weight = pebs->lat;
+
+	/*
+	 * data.data_src encodes the data source
+	 */
+	if (dsrc) {
+		u64 val = PERF_MEM_NA;
+		if (fll)
+			val = load_latency_data(pebs->dse);
+		else if (fst && (fl & PERF_X86_EVENT_PEBS_HSW_PREC))
+			val = precise_datala_hsw(event, pebs->dse);
+		else if (fst)
+			val = precise_store_data(pebs->dse);
+		data.data_src.val = val;
 	}
 
 	/*
@@ -901,16 +898,16 @@ static void __intel_pmu_pebs_event(struct perf_event *event,
 	else
 		regs.flags &= ~PERF_EFLAGS_EXACT;
 
-	if ((event->attr.sample_type & PERF_SAMPLE_ADDR) &&
+	if ((sample_type & PERF_SAMPLE_ADDR) &&
 	    x86_pmu.intel_cap.pebs_format >= 1)
 		data.addr = pebs->dla;
 
 	if (x86_pmu.intel_cap.pebs_format >= 2) {
 		/* Only set the TSX weight when no memory weight. */
-		if ((event->attr.sample_type & PERF_SAMPLE_WEIGHT) && !fll)
+		if ((sample_type & PERF_SAMPLE_WEIGHT) && !fll)
 			data.weight = intel_hsw_weight(pebs);
 
-		if (event->attr.sample_type & PERF_SAMPLE_TRANSACTION)
+		if (sample_type & PERF_SAMPLE_TRANSACTION)
 			data.txn = intel_hsw_transaction(pebs);
 	}
 
Signature · 1 line
1.7.9.5