Skip to contentSkip to conversation

perf/x86/amd: Add AMD Fam19h Branch Sampling support

PATCH Superseded Stephane Eranian 4 participants 28 messages Nov 11, 2021 – Nov 29, 2021

Subsystems (3)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v2 cover perf/x86/amd: Add AMD Fam19h Branch Sampling support

This patch series adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1 section 2.1.13. This is a model specific extension. It is not an architected AMD feature.

The Branch Sampling Feature (BRS) provides the statistical taken branch information necessary to enable autoFDO-style optimization by compilers, i.e., basic block execution counts.

BRS operates with a 16-deep saturating buffer in MSR registers. There is no hardware branch type filtering. All control flow changes are captured. BRS relies on specific programming of the core PMU of Fam19h. In particular, the following requirements must be met:
- the sampling period be greater than 16 (BRS depth)
- the sampling period must use fixed and not frequency mode

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the NMI interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel. Between the overflow and the time BRS activates more branches may be executed skewing the period. All along, the sampling event keeps counting. The skid may be attenuated by reducing the sampling period by 16.

BRS is integrated into perf_events seamlessly via the same PERF_RECORD_BRANCH_STACK sample format. BRS generates branch perf_branch_entry records in the sampling buffer. There is no prediction or latency information supported. The branches are stored in reverse order of execution. The most recent branch is the first entry in each record.

Because BRS must be stopped when a CPU goes into low power mode, the series includes patches to add callbacks on ACPI low power entry and exit which is used on AMD processors.

Given that there is no privilege filterting with BRS, the kernel implements filtering on privlege level.

This version adds a few simple modifications to perf record and report.
1. add the branch-brs event as a builtin such as it can used directly:
perf record -e branch-brs ...
2. improve error handling for AMD IBS and is contributed by Kim Phillips.
3. use the better error handling to improve error handling for BRS.
4. add two new sort dimensions to help display the branch sampling information. Because there is no latency information associated with the branch sampling feature perf report would collapse all samples within a function into a single histogram entry. This is expected because the default sort mode for PERF_SAMPLE_BRANCH_STACK is symbol_from/symbol_to. This propagates to the annotation.

For more detailed view of the branch samples, the new sort dimensions addr_from,addr_to can be used instead as follows:

$ perf report --sort=overhead,comm,dso,addr_from,addr_to 
# Overhead  Command    Shared Object     Source Address   Target Address
# ........  .......... ..............  ..............     ..............
#
     4.21%  test_prg   test_prg       [.] test_threa+0x3c [.] test_threa+0x4
     4.14%  test_prg   test_prg       [.] test_threa+0x3e [.] test_threa+0x2
     4.10%  test_prg   test_prg       [.] test_threa+0x4  [.] test_threa+0x3a
     4.07%  test_prg   test_prg       [.] test_threa+0x2  [.] test_threa+0x3c

Versus the default output:

$ perf report 
# Overhead  Command   Source Shared Object Source Symbol    Target Symbol      Basic Block Cycles
# ........  ......... .................... ................ .................  ..................
#
    99.52%  test_prg  test_prg             [.] test_thread  [.] test_thread    -                 

BRS can be used with any sampling event. However, it is recommended to use the RETIRED_BRANCH event because it matches what the BRS captures. For convenience, a pseudo event matching the branches captured by BRS is exported by the kernel (branch-brs):

$ perf record -b -e cpu/branch-brs/ -c 1000037 test
$ perf report -D
56531696056126 0x193c000 [0x1a8]: PERF_RECORD_SAMPLE(IP, 0x2): 18122/18230: 0x401d24 period: 1000037 addr: 0
... branch stack: nr:16
.....  0: 0000000000401d24 -> 0000000000401d5a 0 cycles      0
.....  1: 0000000000401d5c -> 0000000000401d24 0 cycles      0
.....  2: 0000000000401d22 -> 0000000000401d5c 0 cycles      0
.....  3: 0000000000401d5e -> 0000000000401d22 0 cycles      0
.....  4: 0000000000401d20 -> 0000000000401d5e 0 cycles      0
.....  5: 0000000000401d3e -> 0000000000401d20 0 cycles      0
.....  6: 0000000000401d42 -> 0000000000401d3e 0 cycles      0
.....  7: 0000000000401d3c -> 0000000000401d42 0 cycles      0
.....  8: 0000000000401d44 -> 0000000000401d3c 0 cycles      0
.....  9: 0000000000401d3a -> 0000000000401d44 0 cycles      0
..... 10: 0000000000401d46 -> 0000000000401d3a 0 cycles      0
..... 11: 0000000000401d38 -> 0000000000401d46 0 cycles      0
..... 12: 0000000000401d48 -> 0000000000401d38 0 cycles      0
..... 13: 0000000000401d36 -> 0000000000401d48 0 cycles      0
..... 14: 0000000000401d4a -> 0000000000401d36 0 cycles      0
..... 15: 0000000000401d34 -> 0000000000401d4a 0 cycles      0
 ... thread: test:18230
 ...... dso: test

Special thanks to Kim Phillips @ AMD for the testing, reviews and contributions.

V2 makes the following changes:
- the low power callback code has be reworked completly. It is not impacting the generic perf_events code anymore. This is all handled via x86 code and only for ACPI low power driver which seems to be the default on AMD. The change in acpi_pad.c and processor_idle.c has no impact on non x86 architectures, on Intel x86 or AMD without BRS, a jump label is used to void the code unless necessary

- BRS is an opt-in compile time option for the kernel

- branch_stack bit clearing helper is introduced

- As for the fact that BRS holds the NMI and that it may conflict with other sampling events and introduced skid, this is not really a problem because AMD PMI skid is already very large prompting special handling in amd_pmu_wait_on_overflow(), so adding a few cycles while the CPU executes at most 16 taken branches is not a problem.

Kim Phillips (1):
perf tools: improve IBS error handling

Stephane Eranian (12):
perf/core: add perf_clear_branch_entry_bitfields() helper x86/cpufeatures: add AMD Fam19h Branch Sampling feature perf/x86/amd: add AMD Fam19h Branch Sampling support perf/x86/amd: add branch-brs helper event for Fam19h BRS perf/x86/amd: enable branch sampling priv level filtering perf/x86/amd: add AMD branch sampling period adjustment perf/x86/amd: make Zen3 branch sampling opt-in
ACPI: add perf low power callback
perf/x86/amd: add idle hooks for branch sampling perf tools: add branch-brs as a new event
perf tools: improve error handling of AMD Branch Sampling perf report: add addr_from/addr_to sort dimensions

 arch/x86/events/Kconfig            |   8 +                                   
 arch/x86/events/amd/Makefile       |   1 +                                   
 arch/x86/events/amd/brs.c          | 349 +++++++++++++++++++++++++++++       
 arch/x86/events/amd/core.c         | 221 +++++++++++++++++-                  
 arch/x86/events/core.c             |  22 +-                                  
 arch/x86/events/intel/lbr.c        |  36 ++-                                 
 arch/x86/events/perf_event.h       | 129 +++++++++--                         
 arch/x86/include/asm/cpufeatures.h |   1 +
 arch/x86/include/asm/msr-index.h   |   4 +
 arch/x86/include/asm/perf_event.h  |  21 ++
 drivers/acpi/acpi_pad.c            |   6 +
 drivers/acpi/processor_idle.c      |   5 +
 include/linux/perf_event.h         |  22 ++
 tools/perf/util/evsel.c            |  50 +++++
 tools/perf/util/hist.c             |   2 +
 tools/perf/util/hist.h             |   2 +
 tools/perf/util/parse-events.l     |   1 +
 tools/perf/util/sort.c             | 128 +++++++++++
 tools/perf/util/sort.h             |   2 +
 19 files changed, 970 insertions(+), 40 deletions(-)
 create mode 100644 arch/x86/events/amd/brs.c
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#1 lore raw

PATCH v2 1/13 perf/core: add perf_clear_branch_entry_bitfields() helper

To make it simpler to reset all the info fields on the perf_branch_entry, this patch adds a helper inline function. The goal is to centralize the initialization to avoid missing a field in case more are added.

---
 arch/x86/events/intel/lbr.c | 36 +++++++++++++++++-------------------
 include/linux/perf_event.h  | 16 ++++++++++++++++
 2 files changed, 33 insertions(+), 19 deletions(-)
arch/x86/events/intel/lbr.c +17 -19
index 6b72e9b55c69..78bf4636831a 100644
@@ -776,6 +776,7 @@ void intel_pmu_lbr_disable_all(void)
 void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
 {
 	unsigned long mask = x86_pmu.lbr_nr - 1;
+	struct perf_branch_entry *br = cpuc->lbr_entries;
 	u64 tos = intel_pmu_lbr_tos();
 	int i;
 
@@ -791,15 +792,11 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
 
 		rdmsrl(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr);
 
-		cpuc->lbr_entries[i].from	= msr_lastbranch.from;
-		cpuc->lbr_entries[i].to		= msr_lastbranch.to;
-		cpuc->lbr_entries[i].mispred	= 0;
-		cpuc->lbr_entries[i].predicted	= 0;
-		cpuc->lbr_entries[i].in_tx	= 0;
-		cpuc->lbr_entries[i].abort	= 0;
-		cpuc->lbr_entries[i].cycles	= 0;
-		cpuc->lbr_entries[i].type	= 0;
-		cpuc->lbr_entries[i].reserved	= 0;
+		perf_clear_branch_entry_bitfields(br);
+
+		br->from	= msr_lastbranch.from;
+		br->to		= msr_lastbranch.to;
+		br++;
 	}
 	cpuc->lbr_stack.nr = i;
 	cpuc->lbr_stack.hw_idx = tos;
@@ -814,6 +811,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 {
 	bool need_info = false, call_stack = false;
 	unsigned long mask = x86_pmu.lbr_nr - 1;
+	struct perf_branch_entry *br = cpuc->lbr_entries;
 	int lbr_format = x86_pmu.intel_cap.lbr_format;
 	u64 tos = intel_pmu_lbr_tos();
 	int i;
@@ -886,15 +884,14 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
 		if (abort && x86_pmu.lbr_double_abort && out > 0)
 			out--;
 
-		cpuc->lbr_entries[out].from	 = from;
-		cpuc->lbr_entries[out].to	 = to;
-		cpuc->lbr_entries[out].mispred	 = mis;
-		cpuc->lbr_entries[out].predicted = pred;
-		cpuc->lbr_entries[out].in_tx	 = in_tx;
-		cpuc->lbr_entries[out].abort	 = abort;
-		cpuc->lbr_entries[out].cycles	 = cycles;
-		cpuc->lbr_entries[out].type	 = 0;
-		cpuc->lbr_entries[out].reserved	 = 0;
+		perf_clear_branch_entry_bitfields(br+out);
+		br[out].from	 = from;
+		br[out].to	 = to;
+		br[out].mispred	 = mis;
+		br[out].predicted = pred;
+		br[out].in_tx	 = in_tx;
+		br[out].abort	 = abort;
+		br[out].cycles	 = cycles;
 		out++;
 	}
 	cpuc->lbr_stack.nr = out;
@@ -956,6 +953,8 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 		to = rdlbr_to(i, lbr);
 		info = rdlbr_info(i, lbr);
 
+		perf_clear_branch_entry_bitfields(e);
+
 		e->from		= from;
 		e->to		= to;
 		e->mispred	= get_lbr_mispred(info);
@@ -964,7 +963,6 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *cpuc,
 		e->abort	= !!(info & LBR_INFO_ABORT);
 		e->cycles	= get_lbr_cycles(info);
 		e->type		= get_lbr_br_type(info);
-		e->reserved	= 0;
 	}
 
 	cpuc->lbr_stack.nr = i;
include/linux/perf_event.h +16 -0
index 0dcfd265beed..0c96448059be 100644
@@ -1059,6 +1059,22 @@ static inline void perf_sample_data_init(struct perf_sample_data *data,
 	data->txn = 0;
 }
 
+/*
+ * Clear all bitfields in the perf_branch_entry.
+ * The to and from fields are not cleared because they are
+ * systematically modified by caller.
+ */
+static inline void perf_clear_branch_entry_bitfields(struct perf_branch_entry *br)
+{
+	br->mispred = 0;
+	br->predicted = 0;
+	br->in_tx = 0;
+	br->abort = 0;
+	br->cycles = 0;
+	br->type = 0;
+	br->reserved = 0;
+}
+
 extern void perf_output_sample(struct perf_output_handle *handle,
 			       struct perf_event_header *header,
 			       struct perf_sample_data *data,
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#2 lore raw

PATCH v2 2/13 x86/cpufeatures: add AMD Fam19h Branch Sampling feature

In reply to Stephane Eranian

This patch adds a cpu feature for AMD Fam19h Branch Sampling feature as bit 31 of EBX on CPUID leaf function 0x80000008.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index d5b5f2ab87a0..e71443f93f04 100644
@@ -315,6 +315,7 @@
 #define X86_FEATURE_AMD_SSBD		(13*32+24) /* "" Speculative Store Bypass Disable */
 #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* Virtualized Speculative Store Bypass Disable */
 #define X86_FEATURE_AMD_SSB_NO		(13*32+26) /* "" Speculative Store Bypass is fixed in hardware. */
+#define X86_FEATURE_AMD_BRS		(13*32+31) /* Branch Sampling available */
 
 /* Thermal and Power Management Leaf, CPUID level 0x00000006 (EAX), word 14 */
 #define X86_FEATURE_DTHERM		(14*32+ 0) /* Digital Thermal Sensor */
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
maintainer
#3 lore raw

Re: [PATCH v2 02/13] x86/cpufeatures: add AMD Fam19h Branch Sampling feature

Quoted 1 line · Stephane Eranian

This patch adds a cpu feature for AMD Fam19h Branch Sampling feature as bit

For all your commit messages:

s/This patch adds/Add/

"This patch" in a commit message is a tautology.

Quoted 12 lines

31 of EBX on CPUID leaf function 0x80000008.

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +0 -0
index d5b5f2ab87a0..e71443f93f04 100644

arch/x86/include/asm/cpufeatures.h line 318

Quoted 4 lines
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -315,6 +315,7 @@
 #define X86_FEATURE_AMD_SSBD		(13*32+24) /* "" Speculative Store Bypass Disable */
 #define X86_FEATURE_VIRT_SSBD		(13*32+25) /* Virtualized Speculative Store Bypass Disable */
 #define X86_FEATURE_AMD_SSB_NO		(13*32+26) /* "" Speculative Store Bypass is fixed in hardware. */
+#define X86_FEATURE_AMD_BRS		(13*32+31) /* Branch Sampling available */
	X86_FEATURE_BRS

is perfectly fine - there's no need to say "amd_brs" in /proc/cpuinfo on AMD machines as this is where this flag will be only set.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#4 lore raw

PATCH v2 3/13 perf/x86/amd: add AMD Fam19h Branch Sampling support

In reply to Stephane Eranian

This patch adds support for the AMD Fam19h 16-deep branch sampling feature as described in the AMD PPR Fam19h Model 01h Revision B1. This is a model specific extension. It is not an architected AMD feature.

The Branch Sampling (BRS) operates with a 16-deep saturating buffer in MSR registers. There is no branch type filtering. All control flow changes are captured. BRS relies on specific programming of the core PMU of Fam19h. In particular, the following requirements must be met:
- the sampling period be greater than 16 (BRS depth)
- the sampling period must use a fixed and not frequency mode

BRS interacts with the NMI interrupt as well. Because enabling BRS is expensive, it is only activated after P event occurrences, where P is the desired sampling period. At P occurrences of the event, the counter overflows, the CPU catches the interrupt, activates BRS for 16 branches until it saturates, and then delivers the NMI to the kernel. Between the overflow and the time BRS activates more branches may be executed skewing the period. All along, the sampling event keeps counting. The skid may be attenuated by reducing the sampling period by 16 (subsequent patch).

BRS is integrated into perf_events seamlessly via the same PERF_RECORD_BRANCH_STACK sample format. BRS generates perf_branch_entry records in the sampling buffer. No prediction information is supported. The branches are stored in reverse order of execution. The most recent branch is the first entry in each record.

No modification to the perf tool is necessary.

BRS can be used with any sampling event. However, it is recommended to use the RETIRED_BRANCH_INSTRUCTIONS event because it matches what the BRS captures.

$ perf record -b -c 1000037 -e cpu/event=0xc2,name=ret_br_instructions/ test
$ perf report -D
56531696056126 0x193c000 [0x1a8]: PERF_RECORD_SAMPLE(IP, 0x2): 18122/18230: 0x401d24 period: 1000037 addr: 0
... branch stack: nr:16
.....  0: 0000000000401d24 -> 0000000000401d5a 0 cycles      0
.....  1: 0000000000401d5c -> 0000000000401d24 0 cycles      0
.....  2: 0000000000401d22 -> 0000000000401d5c 0 cycles      0
.....  3: 0000000000401d5e -> 0000000000401d22 0 cycles      0
.....  4: 0000000000401d20 -> 0000000000401d5e 0 cycles      0
.....  5: 0000000000401d3e -> 0000000000401d20 0 cycles      0
.....  6: 0000000000401d42 -> 0000000000401d3e 0 cycles      0
.....  7: 0000000000401d3c -> 0000000000401d42 0 cycles      0
.....  8: 0000000000401d44 -> 0000000000401d3c 0 cycles      0
.....  9: 0000000000401d3a -> 0000000000401d44 0 cycles      0
..... 10: 0000000000401d46 -> 0000000000401d3a 0 cycles      0
..... 11: 0000000000401d38 -> 0000000000401d46 0 cycles      0
..... 12: 0000000000401d48 -> 0000000000401d38 0 cycles      0
..... 13: 0000000000401d36 -> 0000000000401d48 0 cycles      0
..... 14: 0000000000401d4a -> 0000000000401d36 0 cycles      0
..... 15: 0000000000401d34 -> 0000000000401d4a 0 cycles      0
 ... thread: test:18230
 ...... dso: test
---
 arch/x86/events/amd/Makefile     |   2 +-
 arch/x86/events/amd/brs.c        | 303 +++++++++++++++++++++++++++++++
 arch/x86/events/amd/core.c       | 201 +++++++++++++++++++-
 arch/x86/events/core.c           |  15 +-
 arch/x86/events/perf_event.h     |  92 ++++++++--
 arch/x86/include/asm/msr-index.h |   4 +
 6 files changed, 595 insertions(+), 22 deletions(-)
 create mode 100644 arch/x86/events/amd/brs.c
arch/x86/events/amd/Makefile +1 -1
index 6cbe38d5fd9d..cf323ffab5cd 100644
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_CPU_SUP_AMD)		+= core.o
+obj-$(CONFIG_CPU_SUP_AMD)		+= core.o brs.o
 obj-$(CONFIG_PERF_EVENTS_AMD_POWER)	+= power.o
 obj-$(CONFIG_X86_LOCAL_APIC)		+= ibs.o
 obj-$(CONFIG_PERF_EVENTS_AMD_UNCORE)	+= amd-uncore.o
arch/x86/events/amd/brs.c new file +303 -0

303 lines not on this page · raw message

arch/x86/events/amd/core.c +197 -4

284 lines not on this page · raw message

arch/x86/events/core.c +14 -1
index 38b2c779146f..28559557f966 100644
@@ -683,11 +683,16 @@ void x86_pmu_disable_all(void)
 
 		if (!test_bit(idx, cpuc->active_mask))
 			continue;
+
+		if (is_amd_brs(hwc))
+			amd_brs_disable();
+
 		rdmsrl(x86_pmu_config_addr(idx), val);
 		if (!(val & ARCH_PERFMON_EVENTSEL_ENABLE))
 			continue;
 		val &= ~ARCH_PERFMON_EVENTSEL_ENABLE;
 		wrmsrl(x86_pmu_config_addr(idx), val);
+
 		if (is_counter_pair(hwc))
 			wrmsrl(x86_pmu_config_addr(idx + 1), 0);
 	}
@@ -1338,6 +1343,10 @@ static void x86_pmu_enable(struct pmu *pmu)
 			if (hwc->state & PERF_HES_ARCH)
 				continue;
 
+			/*
+			 * if cpuc->enabled = 0, then no wrmsr as
+			 * per x86_pmu_enable_event()
+			 */
 			x86_pmu_start(event, PERF_EF_RELOAD);
 		}
 		cpuc->n_added = 0;
@@ -1704,11 +1713,15 @@ int x86_pmu_handle_irq(struct pt_regs *regs)
 		 * event overflow
 		 */
 		handled++;
-		perf_sample_data_init(&data, 0, event->hw.last_period);
 
 		if (!x86_perf_event_set_period(event))
 			continue;
 
+		perf_sample_data_init(&data, 0, event->hw.last_period);
+
+		if (has_branch_stack(event))
+			data.br_stack = &cpuc->lbr_stack;
+
 		if (perf_event_overflow(event, &data, regs))
 			x86_pmu_stop(event, 0);
 	}
arch/x86/events/perf_event.h +76 -16

122 lines not on this page · raw message

arch/x86/include/asm/msr-index.h +4 -0
index 01e2650b9585..bc7481fd8513 100644
@@ -650,6 +650,10 @@
 #define MSR_IA32_PERF_CTL		0x00000199
 #define INTEL_PERF_CTL_MASK		0xffff
 
+/* AMD Branch Sampling configuration */
+#define MSR_AMD_DBG_EXTN_CFG		0xc000010f
+#define MSR_AMD_SAMP_BR_FROM		0xc0010300
+
 #define MSR_IA32_MPERF			0x000000e7
 #define MSR_IA32_APERF			0x000000e8
 
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
maintainer
#5 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Quoted 11 lines · Stephane Eranian
arch/x86/events/core.c +3 -0
index 38b2c779146f..28559557f966 100644
@@ -683,11 +683,16 @@ void x86_pmu_disable_all(void)
 
 		if (!test_bit(idx, cpuc->active_mask))
 			continue;
+
+		if (is_amd_brs(hwc))
+			amd_brs_disable();

arch/x86/events/core.c void x86_pmu_disable_all(void) · line 698

Quoted 10 lines
arch/x86/events/core.c +2 -0
+
 		rdmsrl(x86_pmu_config_addr(idx), val);
 		if (!(val & ARCH_PERFMON_EVENTSEL_ENABLE))
 			continue;
 		val &= ~ARCH_PERFMON_EVENTSEL_ENABLE;
 		wrmsrl(x86_pmu_config_addr(idx), val);
+
 		if (is_counter_pair(hwc))
 			wrmsrl(x86_pmu_config_addr(idx + 1), 0);
 	}

Please, stick that in amd_pmu_disable_all().

#6 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Quoted 17 lines · Peter Zijlstra
Quoted 15 lines · Stephane Eranian
arch/x86/events/core.c +4 -0
index 38b2c779146f..28559557f966 100644
@@ -683,11 +683,16 @@ void x86_pmu_disable_all(void)
 
              if (!test_bit(idx, cpuc->active_mask))
                      continue;
+
+             if (is_amd_brs(hwc))
+                     amd_brs_disable();
+
              rdmsrl(x86_pmu_config_addr(idx), val);
              if (!(val & ARCH_PERFMON_EVENTSEL_ENABLE))
                      continue;
Quoted 8 lines

arch/x86/events/core.c void x86_pmu_disable_all(void) · line 698

Quoted 6 lines
arch/x86/events/core.c +1 -0
              val &= ~ARCH_PERFMON_EVENTSEL_ENABLE;
              wrmsrl(x86_pmu_config_addr(idx), val);
+
              if (is_counter_pair(hwc))
                      wrmsrl(x86_pmu_config_addr(idx + 1), 0);
      }

Please, stick that in amd_pmu_disable_all().

If I do that, I need to add a for_each_counter() loop to amd_pmu_disable_all() but it does not have one call systematically. If you are okay with it, then I am fine as well.

maintainer
#7 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Quoted 28 lines · Stephane Eranian
Quoted 17 lines · Peter Zijlstra
Quoted 15 lines · Stephane Eranian
arch/x86/events/core.c +4 -0
index 38b2c779146f..28559557f966 100644
@@ -683,11 +683,16 @@ void x86_pmu_disable_all(void)
 
              if (!test_bit(idx, cpuc->active_mask))
                      continue;
+
+             if (is_amd_brs(hwc))
+                     amd_brs_disable();
+
              rdmsrl(x86_pmu_config_addr(idx), val);
              if (!(val & ARCH_PERFMON_EVENTSEL_ENABLE))
                      continue;
Quoted 8 lines

arch/x86/events/core.c void x86_pmu_disable_all(void) · line 698

Quoted 6 lines
arch/x86/events/core.c +1 -0
              val &= ~ARCH_PERFMON_EVENTSEL_ENABLE;
              wrmsrl(x86_pmu_config_addr(idx), val);
+
              if (is_counter_pair(hwc))
                      wrmsrl(x86_pmu_config_addr(idx + 1), 0);
      }

Please, stick that in amd_pmu_disable_all().

Quoted 3 lines

If I do that, I need to add a for_each_counter() loop to amd_pmu_disable_all() but it does not have one call systematically. If you are okay with it, then I am fine as well.

Why ? There is only one BRS you can disable it first, and then do the loop. Same on the enable_all side, enable it once, not per event. ->add/->del can keep a per-event count for sharing purposes, but you don't need that for {en,dis}able_all, right?

#8 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Quoted 30 lines · Peter Zijlstra
Quoted 28 lines · Stephane Eranian
Quoted 25 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
arch/x86/events/core.c +3 -0
index 38b2c779146f..28559557f966 100644
@@ -683,11 +683,16 @@ void x86_pmu_disable_all(void)
 
              if (!test_bit(idx, cpuc->active_mask))
                      continue;
+
+             if (is_amd_brs(hwc))
+                     amd_brs_disable();

arch/x86/events/core.c void x86_pmu_disable_all(void) · line 698

Quoted 10 lines
arch/x86/events/core.c +2 -0
+
              rdmsrl(x86_pmu_config_addr(idx), val);
              if (!(val & ARCH_PERFMON_EVENTSEL_ENABLE))
                      continue;
              val &= ~ARCH_PERFMON_EVENTSEL_ENABLE;
              wrmsrl(x86_pmu_config_addr(idx), val);
+
              if (is_counter_pair(hwc))
                      wrmsrl(x86_pmu_config_addr(idx + 1), 0);
      }

Please, stick that in amd_pmu_disable_all().

Quoted 8 lines
Quoted 3 lines

If I do that, I need to add a for_each_counter() loop to amd_pmu_disable_all() but it does not have one call systematically. If you are okay with it, then I am fine as well.

Why ? There is only one BRS you can disable it first, and then do the loop. Same on the enable_all side, enable it once, not per event. ->add/->del can keep a per-event count for sharing purposes, but you don't need that for {en,dis}able_all, right?

Ok, I made the changes you suggested. It looks closer to the way LBR is handled. However, this means that there is no path by which you can get to amd_pmu_disable_event()
without having gone through amd_pmu_disable_all(). Is that always the case? And same thing
on the enable side.

maintainer
#9 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Quoted 40 lines · Stephane Eranian
Quoted 30 lines · Peter Zijlstra
Quoted 28 lines · Stephane Eranian
Quoted 25 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
arch/x86/events/core.c +3 -0
index 38b2c779146f..28559557f966 100644
@@ -683,11 +683,16 @@ void x86_pmu_disable_all(void)
 
              if (!test_bit(idx, cpuc->active_mask))
                      continue;
+
+             if (is_amd_brs(hwc))
+                     amd_brs_disable();

arch/x86/events/core.c void x86_pmu_disable_all(void) · line 698

Quoted 10 lines
arch/x86/events/core.c +2 -0
+
              rdmsrl(x86_pmu_config_addr(idx), val);
              if (!(val & ARCH_PERFMON_EVENTSEL_ENABLE))
                      continue;
              val &= ~ARCH_PERFMON_EVENTSEL_ENABLE;
              wrmsrl(x86_pmu_config_addr(idx), val);
+
              if (is_counter_pair(hwc))
                      wrmsrl(x86_pmu_config_addr(idx + 1), 0);
      }

Please, stick that in amd_pmu_disable_all().

Quoted 8 lines
Quoted 3 lines

If I do that, I need to add a for_each_counter() loop to amd_pmu_disable_all() but it does not have one call systematically. If you are okay with it, then I am fine as well.

Why ? There is only one BRS you can disable it first, and then do the loop. Same on the enable_all side, enable it once, not per event. ->add/->del can keep a per-event count for sharing purposes, but you don't need that for {en,dis}able_all, right?

Quoted 6 lines

Ok, I made the changes you suggested. It looks closer to the way LBR is handled. However, this means that there is no path by which you can get to amd_pmu_disable_event()
without having gone through amd_pmu_disable_all(). Is that always the case? And same thing
on the enable side.

So that's true for ->add() and ->del(), those cannot be called without being wrapped in ->pmu_disable(), ->pmu_enable().

There is however the ->stop() and ->start() usage for throttling, which can stop an individual event (while leaving the event scheduled on the PMU). Now, I think the ->stop() gets called with the PMU enabled, but the ->start() is with it disabled again.

The ramification would be that we'd stop the event, but leave BRS enabled for a throttled event. Which should be harmless, no?

maintainer
#10 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Stephane Eranian

Ok, I made the changes you suggested. It looks closer to the way LBR is handled. However, this means that there is no path by which you can get to amd_pmu_disable_event()
without having gone through amd_pmu_disable_all(). Is that always the case? And same thing
on the enable side.

Quoted 7 lines

So that's true for ->add() and ->del(), those cannot be called without being wrapped in ->pmu_disable(), ->pmu_enable().

There is however the ->stop() and ->start() usage for throttling, which can stop an individual event (while leaving the event scheduled on the PMU). Now, I think the ->stop() gets called with the PMU enabled, but the ->start() is with it disabled again.

I just looked, and the throttling depends on the PMU's PMI handler implementation, for Intel it will have the PMU disabled, for generic and AMD it has it enabled (see x86_pmu_handle_irq -- also these are really NMIs but lets not do a mass rename just now).

Quoted 2 lines

The ramification would be that we'd stop the event, but leave BRS enabled for a throttled event. Which should be harmless, no?

#11 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

Peter,

Back from a vacation break. Comments below.

Quoted 19 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Stephane Eranian

Ok, I made the changes you suggested. It looks closer to the way LBR is handled. However, this means that there is no path by which you can get to amd_pmu_disable_event()
without having gone through amd_pmu_disable_all(). Is that always the case? And same thing
on the enable side.

Quoted 7 lines

So that's true for ->add() and ->del(), those cannot be called without being wrapped in ->pmu_disable(), ->pmu_enable().

There is however the ->stop() and ->start() usage for throttling, which can stop an individual event (while leaving the event scheduled on the PMU). Now, I think the ->stop() gets called with the PMU enabled, but the ->start() is with it disabled again.

Quoted 5 lines

I just looked, and the throttling depends on the PMU's PMI handler implementation, for Intel it will have the PMU disabled, for generic and AMD it has it enabled (see x86_pmu_handle_irq -- also these are really NMIs but lets not do a mass rename just now).

Yes, I see that too. It has to do with the __perf_event_overflow() -> __perf_event_account_interrupt() code path which does not call perf_pmu_disable().
And that's because it knows it is called from PMI and let's the PMI code decide the state
of the PMU. Unfortunately, on AMD, the PMU is not stopped fully on PMI and that is because
there is no centralized way of doing this, so you'd have 6 wrmsr x 2 to disable/enable. OTOH,
I don't see the point of monitoring in the PMI code. So there is a discrepancy between Intel and
AMD here. I think we should fix it.

Quoted 2 lines
Quoted 2 lines

The ramification would be that we'd stop the event, but leave BRS enabled for a throttled event. Which should be harmless, no?

Well, the risk here is that if BRS is left enabled, it may hold the NMI for up to 16 taken branches.
If the associated event is disabled, then cpuc->events[idx] = NULL. The BRS drain function checks
for that and does not capture any sample. The brs_drain() function is called from the PMI handler if
cpuc->lbr_users > 0 which would be the case on the stop() path. I think this is the only risk we have
on the throttling code path.

There would be no problem if we were to stop/start the PMU in the AMD PMI handler.

Thanks.

maintainer
#12 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

In reply to Stephane Eranian

arch/x86/events/amd/core.c static void amd_pmu_disable_all(void) · line 673

Quoted 8 lines · Stephane Eranian
arch/x86/events/amd/core.c +8 -0
+static void amd_pmu_enable_event(struct perf_event *event)
+{
+	x86_pmu_enable_event(event);
+
+	if (__this_cpu_read(cpu_hw_events.enabled)) {
+		if (is_amd_brs(&event->hw))
+			amd_brs_enable();
+	}

Does this path actually trigger? I thought we always use the enable_all path these days. Could easily be I misremember though.

Quoted 8 lines
arch/x86/events/amd/core.c +8 -0
+}
+
+static void amd_pmu_enable_all(int added)
+{
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+	struct hw_perf_event *hwc;
+	int idx;
+

arch/x86/events/amd/core.c static void amd_pmu_disable_all(void) · line 691

Quoted 10 lines
arch/x86/events/amd/core.c +10 -0
+	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
+		hwc = &cpuc->events[idx]->hw;
+
+		/* only activate events which are marked as active */
+		if (!test_bit(idx, cpuc->active_mask))
+			continue;
+
+		amd_pmu_enable_event(cpuc->events[idx]);
+	}
+}

The thing is, since te BRS is a 'global' state, it doesn't really make sense to enable it per-event.

Anyway, I think the above actually works, just wondering if we can perhaps simplify it.

maintainer
#13 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

In reply to Stephane Eranian

arch/x86/events/amd/core.c static int __init amd_core_pmu_init(void) · line 1172

Quoted 9 lines · Stephane Eranian
arch/x86/events/amd/core.c +5 -0
@@ -989,6 +1165,23 @@ static int __init amd_core_pmu_init(void)
 		x86_pmu.flags |= PMU_FL_PAIR;
 	}
 
+	if (boot_cpu_data.x86 >= 0x19) {
+		/*
+		 * On AMD, invoking pmu_disable_all() is very expensive and the function is
+		 * invoked on context-switch in via sched_task_in(), so enable only when necessary
+		 */

Well, it only gets called when brs is in use, so afaict the above comment is a bit misleading. Still no point in installing that function when there it no point for it.

Quoted 15 lines
arch/x86/events/amd/core.c +12 -0
+		if (!amd_brs_init()) {
+			x86_pmu.get_event_constraints = amd_get_event_constraints_f19h;
+			x86_pmu.sched_task = amd_pmu_sched_task;
+			/*
+			 * The put_event_constraints callback is shared with
+			 * Fam17h, set above
+			 */
+		}
+	}
+
+	x86_pmu.attr_update = amd_attr_update;
+
 	pr_cont("core perfctr, ");
 	return 0;
 }
maintainer
#14 lore raw

Re: [PATCH v2 03/13] perf/x86/amd: add AMD Fam19h Branch Sampling support

In reply to Stephane Eranian
Quoted 33 lines · Stephane Eranian
arch/x86/events/amd/core.c +33 -0
+/*
+ * Because of the way BRS operates with an inactive and active phases, and
+ * the link to one counter, it is not possible to have two events using BRS
+ * scheduled at the same time. There would be an issue with enforcing the
+ * period of each one and given that the BRS saturates, it would not be possible
+ * to guarantee correlated content for all events. Therefore, in situations
+ * where multiple events want to use BRS, the kernel enforces mutual exclusion.
+ * Exclusion is enforced by chosing only one counter for events using BRS.
+ * The event scheduling logic will then automatically multiplex the
+ * events and ensure that at most one event is actively using BRS.
+ *
+ * The BRS counter could be any counter, but there is no constraint on Fam19h,
+ * therefore all counters are equal and thus we pick the first one: PMC0
+ */
+static struct event_constraint amd_fam19h_brs_cntr0_constraint =
+	EVENT_CONSTRAINT(0, 0x1, AMD64_RAW_EVENT_MASK);
+
+static struct event_constraint amd_fam19h_brs_pair_cntr0_constraint =
+	__EVENT_CONSTRAINT(0, 0x1, AMD64_RAW_EVENT_MASK, 1, 0, PERF_X86_EVENT_PAIR);
+
+static struct event_constraint *
+amd_get_event_constraints_f19h(struct cpu_hw_events *cpuc, int idx,
+			  struct perf_event *event)
+{
+	struct hw_perf_event *hwc = &event->hw;
+	bool has_brs = is_amd_brs(hwc);
+
+	/*
+	 * In case BRS is used with an event requiring a counter pair,
+	 * the kernel allows it but only on counter 0 & 1 to enforce
+	 * multiplexing requiring to protect BRS in case of multiple
+	 * BRS users
+	 */

arch/x86/events/amd/core.c static void amd_put_event_constraints_f17h(struct cpu_hw_events *cpuc, · line 1033

Quoted 10 lines
arch/x86/events/amd/core.c +10 -0
+	if (amd_is_pair_event_code(hwc)) {
+		return has_brs ? &amd_fam19h_brs_pair_cntr0_constraint
+			       : &pair_constraint;
+	}
+
+	if (has_brs)
+		return &amd_fam19h_brs_cntr0_constraint;
+
+	return &unconstrained;
+}

That still allows BRS to be used together with unrelated counters and it *will* destroy their utility by delaying the NMI.

BRS could perhaps share the PMU with !sampling counters, but sharing with sampling counters is just asking for trouble.

#15 lore raw

PATCH v2 4/13 perf/x86/amd: add branch-brs helper event for Fam19h BRS

In reply to Stephane Eranian

This patch adds a pseudo event called branch-brs to help use the FAM Fam19h Branch Sampling feature (BRS). BRS samples taken branches, so it is best used when sampling on a retired taken branch event (0xc4) which is what BRS captures. Instead of trying to remember the event code or actual event name, users can simply do:

$ perf record -b -e cpu/branch-brs/ -c 1000037 .....
---
 arch/x86/events/amd/core.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/x86/events/amd/core.c +16 -0
index 8031b9400aaa..d47cc2c6d85c 100644
@@ -1109,8 +1109,24 @@ static struct attribute_group group_caps_amd_brs = {
 	.is_visible = amd_brs_is_visible,
 };
 
+#define AMD_FAM19H_BRS_EVENT 0xc4 /* Fam19h RETIRED_TAKEN_BRANCH_INSTRUCTIONS */
+EVENT_ATTR_STR(branch-brs, amd_branch_brs,
+	       "event=" __stringify(AMD_FAM19H_BRS_EVENT)"\n");
+
+static struct attribute *amd_brs_events_attrs[] = {
+	EVENT_PTR(amd_branch_brs),
+	NULL,
+};
+
+static struct attribute_group group_events_amd_brs = {
+	.name       = "events",
+	.attrs      = amd_brs_events_attrs,
+	.is_visible = amd_brs_is_visible,
+};
+
 static const struct attribute_group *amd_attr_update[] = {
 	&group_caps_amd_brs,
+	&group_events_amd_brs,
 	NULL,
 };
 
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#16 lore raw

PATCH v2 5/13 perf/x86/amd: enable branch sampling priv level filtering

In reply to Stephane Eranian

The AMD Branch Sampling features does not provide hardware filtering by privilege level. The associated PMU counter does but not the branch sampling by itself. Given how BRS operates there is a possibility that BRS captures kernel level branches even though the event is programmed to count only at the user level. This patch implements a workaround in software by removing the branches which belong to the wrong privilege level. The privilege level is evaluated on the target of the branch and not the source so as to be compatible with other architectures. As a consequence of this patch, the number of entries in the PERF_RECORD_BRANCH_STACK buffer may be less than the maximum (16). It could even be zero. Another consequence is that consecutive entries in the branch stack may not reflect actual code path and may have discontinuities, in case kernel branches were suppressed. But this is no different than what happens on other architectures.

---
 arch/x86/events/amd/brs.c | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
arch/x86/events/amd/brs.c +20 -6
index f0d33511000d..771c4248979e 100644
@@ -92,10 +92,6 @@ int amd_brs_setup_filter(struct perf_event *event)
 	if ((type & ~PERF_SAMPLE_BRANCH_PLM_ALL) != PERF_SAMPLE_BRANCH_ANY)
 		return -EINVAL;
 
-	/* can only capture at all priv levels due to the way BRS works */
-	if ((type & PERF_SAMPLE_BRANCH_PLM_ALL) != PERF_SAMPLE_BRANCH_PLM_ALL)
-		return -EINVAL;
-
 	return 0;
 }
 
@@ -181,6 +177,21 @@ void amd_brs_disable(void)
 	}
 }
 
+static bool amd_brs_match_plm(struct perf_event *event, u64 to)
+{
+	int type = event->attr.branch_sample_type;
+	int plm_k = PERF_SAMPLE_BRANCH_KERNEL | PERF_SAMPLE_BRANCH_HV;
+	int plm_u = PERF_SAMPLE_BRANCH_USER;
+
+	if (!(type & plm_k) && kernel_ip(to))
+		return 0;
+
+	if (!(type & plm_u) && !kernel_ip(to))
+		return 0;
+
+	return 1;
+}
+
 /*
  * Caller must ensure amd_brs_inuse() is true before calling
  * return:
@@ -238,8 +249,6 @@ void amd_brs_drain(void)
 		if (to == BRS_POISON)
 			break;
 
-		rdmsrl(brs_from(brs_idx), from);
-
 		/*
 		 * Sign-extend SAMP_BR_TO to 64 bits, bits 61-63 are reserved.
 		 * Necessary to generate proper virtual addresses suitable for
@@ -247,6 +256,11 @@ void amd_brs_drain(void)
 		 */
 		to = (u64)(((s64)to << shift) >> shift);
 
+		if (!amd_brs_match_plm(event, to))
+			continue;
+
+		rdmsrl(brs_from(brs_idx), from);
+
 		perf_clear_branch_entry_bitfields(br+nr);
 
 		br[nr].from = from;
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#17 lore raw

PATCH v2 6/13 perf/x86/amd: add AMD branch sampling period adjustment

In reply to Stephane Eranian

This supplemental patch adds the code to adjust the sampling event period when used with the Branch Sampling feature (BRS). Given the depth of the BRS (16), the period is reduced by that depth such that in the best case scenario, BRS saturates at the desired sampling period. In practice, though, the processor may execute more branches. Given a desired period P and a depth D, the kernel programs the actual period at P - D. After P occurrences of the sampling event, the counter overflows. It then may take X branches (skid) before the NMI is caught and held by the hardware and BRS activates. Then, after D branches, BRS saturates and the NMI is delivered. With no skid, the effective period would be (P - D) + D = P. In practice, however, it will likely be (P - D) + X + D. There is no way to eliminate X or predict X.

---
 arch/x86/events/core.c       |  7 +++++++
 arch/x86/events/perf_event.h | 12 ++++++++++++
 2 files changed, 19 insertions(+)
arch/x86/events/core.c +7 -0
index 28559557f966..996212a02bfa 100644
@@ -1379,6 +1379,13 @@ int x86_perf_event_set_period(struct perf_event *event)
 	    x86_pmu.set_topdown_event_period)
 		return x86_pmu.set_topdown_event_period(event);
 
+	/*
+	 * decrease period by the depth of the BRS feature to get
+	 * the last N taken branches and approximate the desired period
+	 */
+	if (has_branch_stack(event))
+		period = amd_brs_adjust_period(period);
+
 	/*
 	 * If we are way outside a reasonable range then just skip forward:
 	 */
arch/x86/events/perf_event.h +12 -0
index c8aa9bb1e61b..f0026266a9d7 100644
@@ -1256,6 +1256,14 @@ static inline bool amd_brs_active(void)
 	return cpuc->brs_active;
 }
 
+static inline s64 amd_brs_adjust_period(s64 period)
+{
+	if (period > x86_pmu.lbr_nr)
+		return period - x86_pmu.lbr_nr;
+
+	return period;
+}
+
 #else /* CONFIG_CPU_SUP_AMD */
 
 static inline int amd_pmu_init(void)
@@ -1273,6 +1281,10 @@ static inline void amd_brs_drain(void)
 	return 0;
 }
 
+static inline s64 amd_brs_adjust_period(s64 period)
+{
+	return period;
+}
 #endif /* CONFIG_CPU_SUP_AMD */
 
 static inline int is_pebs_pt(struct perf_event *event)
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#18 lore raw

PATCH v2 7/13 perf/x86/amd: make Zen3 branch sampling opt-in

In reply to Stephane Eranian

This patch adds a kernel config option to make support for AMD Zen3 Branch Sampling (BRS) an opt-in compile time option.

---
 arch/x86/events/Kconfig      |  8 +++++++
 arch/x86/events/amd/Makefile |  3 ++-
 arch/x86/events/perf_event.h | 44 ++++++++++++++++++++++++++++--------
 3 files changed, 44 insertions(+), 11 deletions(-)
arch/x86/events/Kconfig +8 -0
index d6cdfe631674..1dc002ef66da 100644
@@ -44,4 +44,12 @@ config PERF_EVENTS_AMD_UNCORE
 
 	  To compile this driver as a module, choose M here: the
 	  module will be called 'amd-uncore'.
+
+config PERF_EVENTS_AMD_BRS
+	depends on PERF_EVENTS && CPU_SUP_AMD
+	tristate "AMD Zen3 Branch Sampling support"
+	help
+	  Enable AMD Zen3 branch sampling support (BRS) which samples up to
+	  16 consecutive taken branches in registers.
+
 endmenu
arch/x86/events/amd/Makefile +2 -1
index cf323ffab5cd..b9f5d4610256 100644
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_CPU_SUP_AMD)		+= core.o brs.o
+obj-$(CONFIG_CPU_SUP_AMD)		+= core.o
+obj-$(CONFIG_PERF_EVENTS_AMD_BRS)	+= brs.o
 obj-$(CONFIG_PERF_EVENTS_AMD_POWER)	+= power.o
 obj-$(CONFIG_X86_LOCAL_APIC)		+= ibs.o
 obj-$(CONFIG_PERF_EVENTS_AMD_UNCORE)	+= amd-uncore.o
arch/x86/events/perf_event.h +34 -10

64 lines not on this page · raw message

Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#19 lore raw

PATCH v2 8/13 ACPI: add perf low power callback

In reply to Stephane Eranian

This patch add an optional callback needed by some PMU features, e.g., AMD BRS, to give a chance to the perf_events code to change its state before going to low power and after coming back.

The callback is void when the PERF_NEEDS_LOPWR_CB flag is not set. This flag must be set in arch specific perf_event.h header whenever needed. When not set, there is no impact of the ACPI code.

---
 drivers/acpi/acpi_pad.c       | 6 ++++++
 drivers/acpi/processor_idle.c | 5 +++++
 include/linux/perf_event.h    | 6 ++++++
 3 files changed, 17 insertions(+)
drivers/acpi/acpi_pad.c +6 -0
index f45979aa2d64..a306a07a60b5 100644
@@ -164,6 +164,9 @@ static int power_saving_thread(void *data)
 				tsc_marked_unstable = 1;
 			}
 			local_irq_disable();
+
+			perf_lopwr_cb(true);
+
 			tick_broadcast_enable();
 			tick_broadcast_enter();
 			stop_critical_timings();
@@ -172,6 +175,9 @@ static int power_saving_thread(void *data)
 
 			start_critical_timings();
 			tick_broadcast_exit();
+
+			perf_lopwr_cb(false);
+
 			local_irq_enable();
 
 			if (time_before(expire_time, jiffies)) {
drivers/acpi/processor_idle.c +5 -0
index f37fba9e5ba0..d1171c6420d0 100644
@@ -20,6 +20,7 @@
 #include <linux/tick.h>
 #include <linux/cpuidle.h>
 #include <linux/cpu.h>
+#include <linux/perf_event.h>
 #include <acpi/processor.h>
 
 /*
@@ -546,6 +547,8 @@ static void wait_for_freeze(void)
  */
 static void __cpuidle acpi_idle_do_entry(struct acpi_processor_cx *cx)
 {
+	perf_lopwr_cb(true);
+
 	if (cx->entry_method == ACPI_CSTATE_FFH) {
 		/* Call into architectural FFH based C-state */
 		acpi_processor_ffh_cstate_enter(cx);
@@ -556,6 +559,8 @@ static void __cpuidle acpi_idle_do_entry(struct acpi_processor_cx *cx)
 		inb(cx->address);
 		wait_for_freeze();
 	}
+
+	perf_lopwr_cb(false);
 }
 
 /**
include/linux/perf_event.h +6 -0
index 0c96448059be..ebef56b552ba 100644
@@ -1654,4 +1654,10 @@ typedef int (perf_snapshot_branch_stack_t)(struct perf_branch_entry *entries,
 					   unsigned int cnt);
 DECLARE_STATIC_CALL(perf_snapshot_branch_stack, perf_snapshot_branch_stack_t);
 
+#ifndef PERF_NEEDS_LOPWR_CB
+static inline void perf_lopwr_cb(bool mode)
+{
+}
+#endif
+
 #endif /* _LINUX_PERF_EVENT_H */
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#20 lore raw

PATCH v2 9/13 perf/x86/amd: add idle hooks for branch sampling

In reply to Stephane Eranian

On AMD Fam19h Zen3, the branch sampling (BRS) feature must be disabled before entering low power and re-enabled (if was active) when returning from low power. Otherwise, the NMI interrupt may be held up for too long and cause problems. Stopping BRS will cause the NMI to be delivered if it was held up.

The patch defines a perf_amd_brs_lopwr_cb() callback to stop/restart BRS. The callback is protected by a jump label which is enabled only when AMD BRS is detected. In all other cases, the callback is never called.

---
 arch/x86/events/amd/brs.c         | 32 +++++++++++++++++++++++++++++++
 arch/x86/events/amd/core.c        |  4 ++++
 arch/x86/events/perf_event.h      |  1 +
 arch/x86/include/asm/perf_event.h | 21 ++++++++++++++++++++
 4 files changed, 58 insertions(+)
arch/x86/events/amd/brs.c +32 -0
index 771c4248979e..09697e0f6dc0 100644
@@ -7,6 +7,7 @@
  * Contributed by Stephane Eranian <[email protected]>
  */
 #include <linux/kernel.h>
+#include <linux/jump_label.h>
 #include <asm/msr.h>
 #include <asm/cpufeature.h>
 
@@ -315,3 +316,34 @@ void amd_pmu_brs_sched_task(struct perf_event_context *ctx, bool sched_in)
 	if (sched_in)
 		amd_brs_poison_buffer();
 }
+
+DEFINE_STATIC_KEY_FALSE(perf_lopwr_needed);
+
+/*
+ * called from ACPI processor_idle.c or acpi_pad.c
+ * with interrupts disabled
+ */
+void perf_amd_brs_lopwr_cb(bool lopwr_in)
+{
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+	union amd_debug_extn_cfg cfg;
+
+	/*
+	 * on mwait in, we may end up in non C0 state.
+	 * we must disable branch sampling to avoid holding the NMI
+	 * for too long. We disable it in hardware but we
+	 * keep the state in cpuc, so we can re-enable.
+	 *
+	 * The hardware will deliver the NMI if needed when brsmen cleared
+	 */
+	if (cpuc->brs_active) {
+		cfg.val = get_debug_extn_cfg();
+		cfg.brsmen = !lopwr_in;
+		set_debug_extn_cfg(cfg.val);
+	}
+}
+
+void __init amd_brs_lopwr_init(void)
+{
+	static_branch_enable(&perf_lopwr_needed);
+}
arch/x86/events/amd/core.c +4 -0
index d47cc2c6d85c..8f2bf25d34e2 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 #include <linux/perf_event.h>
+#include <linux/jump_label.h>
 #include <linux/export.h>
 #include <linux/types.h>
 #include <linux/init.h>
@@ -1193,6 +1194,9 @@ static int __init amd_core_pmu_init(void)
 			 * The put_event_constraints callback is shared with
 			 * Fam17h, set above
 			 */
+
+			/* branch sampling must be stopped when entering low power */
+			amd_brs_lopwr_init();
 		}
 	}
 
arch/x86/events/perf_event.h +1 -0
index f8dc8faed584..c129aaacfb3d 100644
@@ -1219,6 +1219,7 @@ int amd_brs_init(void);
 void amd_brs_disable(void);
 void amd_brs_enable(void);
 void amd_brs_drain(void);
+void amd_brs_lopwr_init(void);
 void amd_brs_disable_all(void);
 int amd_brs_setup_filter(struct perf_event *event);
 void amd_brs_reset(void);
arch/x86/include/asm/perf_event.h +21 -0
index 8fc1b5003713..e405847ccb56 100644
@@ -513,6 +513,27 @@ static inline void intel_pt_handle_vmx(int on)
 #if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_CPU_SUP_AMD)
  extern void amd_pmu_enable_virt(void);
  extern void amd_pmu_disable_virt(void);
+
+#if defined(CONFIG_PERF_EVENTS_AMD_BRS)
+
+#define PERF_NEEDS_LOPWR_CB 1
+
+/*
+ * architectural low power callback impacts
+ * drivers/acpi/processor_idle.c
+ * drivers/acpi/acpi_pad.c
+ */
+extern void perf_amd_brs_lopwr_cb(bool lopwr_in);
+DECLARE_STATIC_KEY_FALSE(perf_lopwr_needed);
+
+static inline void perf_lopwr_cb(bool mode)
+{
+	/* key enabled only when BRS is available */
+	if (static_branch_unlikely(&perf_lopwr_needed))
+		perf_amd_brs_lopwr_cb(mode);
+}
+#endif /* PERF_NEEDS_LOPWR_CB */
+
 #else
  static inline void amd_pmu_enable_virt(void) { }
  static inline void amd_pmu_disable_virt(void) { }
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#21 lore raw

PATCH v2 10/13 perf tools: add branch-brs as a new event

In reply to Stephane Eranian

Although branch-brs is specific to AMD Zen3, we provide it in the list of events recognized by perf as a convenience. It allows: $ perf stat -e branch-brs $ perf list

and enable some test suite to pass.

---
 tools/perf/util/parse-events.l | 1 +
 1 file changed, 1 insertion(+)
tools/perf/util/parse-events.l +1 -0
index 923849024b15..ae229f5c4758 100644
@@ -361,6 +361,7 @@ mem-loads-aux				|
 mem-stores				|
 topdown-[a-z-]+				|
 tx-capacity-[a-z-]+			|
+branch-brs				|
 el-capacity-[a-z-]+			{ return str(yyscanner, PE_KERNEL_PMU_EVENT); }
 
 L1-dcache|l1-d|l1d|L1-data		|
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#22 lore raw

PATCH v2 11/13 perf tools: improve IBS error handling

In reply to Stephane Eranian

From: Kim Phillips <[email protected]>

This patch improves the error message returned on failed perf_event_open() on AMD when using IBS.

---
 tools/perf/util/evsel.c | 42 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
tools/perf/util/evsel.c +42 -0
index dbfeceb2546c..1b24b3000d66 100644
@@ -2743,12 +2743,52 @@ static bool find_process(const char *name)
 	return ret ? false : true;
 }
 
+static bool is_amd;
+
+static char *fgrep(FILE *inf, const char *str)
+{
+	char line[256];
+	int slen = strlen(str);
+
+	while (!feof(inf)) {
+		if (!fgets(line, 256, inf))
+			break;
+		if (strncmp(line, str, slen))
+			continue;
+
+		return strdup(line);
+	}
+
+	return NULL;
+}
+
+static void detect_amd(void)
+{
+	FILE *inf = fopen("/proc/cpuinfo", "r");
+	char *res;
+
+	if (!inf)
+		return;
+
+	res = fgrep(inf, "vendor_id");
+
+	if (res) {
+		char *s = strchr(res, ':');
+
+		is_amd = s && !strcmp(s, ": AuthenticAMD\n");
+		free(res);
+	}
+	fclose(inf);
+}
+
 int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			 int err, char *msg, size_t size)
 {
 	char sbuf[STRERR_BUFSIZE];
 	int printed = 0, enforced = 0;
 
+	detect_amd();
+
 	switch (err) {
 	case EPERM:
 	case EACCES:
@@ -2841,6 +2881,8 @@ int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			return scnprintf(msg, size, "wrong clockid (%d).", clockid);
 		if (perf_missing_features.aux_output)
 			return scnprintf(msg, size, "The 'aux_output' feature is not supported, update the kernel.");
+		if (is_amd && (evsel->core.attr.precise_ip || !strncmp(evsel->pmu_name, "ibs", 3)) && (evsel->core.attr.exclude_kernel))
+			return scnprintf(msg, size, "AMD IBS can't exclude kernel events.  Try running at a higher privilege level.");
 		break;
 	case ENODATA:
 		return scnprintf(msg, size, "Cannot collect data source with the load latency event alone. "
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#23 lore raw

Re: [PATCH v2 11/13] perf tools: improve IBS error handling

Quoted 7 lines · Stephane Eranian

From: Kim Phillips <[email protected]>

This patch improves the error message returned on failed perf_event_open() on AMD when using IBS.

---

Hi, AFAICT this v2 == v1, i.e., this patch doesn't address the comments raised in v1:

https://lore.kernel.org/lkml/[email protected]/

Arnaldo, orthogonal to this series, it'd be great if you could apply these two patches:

https://lore.kernel.org/lkml/[email protected]/T/#mc4c9c582e3816ab31af6d0187e6803de1a98ac84

Then, Stephane, you could drop this patch and in your next version, patch 12/13 of this series would be replaced with what's at the end of this email:

https://lore.kernel.org/lkml/[email protected]/T/#m885ead1f280445357c91cce53c01217913bd017b

Thanks,

Kim

#24 lore raw

Re: [PATCH v2 11/13] perf tools: improve IBS error handling

Kim,

Quoted 13 lines · Kim Phillips
Quoted 7 lines · Stephane Eranian

From: Kim Phillips <[email protected]>

This patch improves the error message returned on failed perf_event_open() on AMD when using IBS.

---

Hi, AFAICT this v2 == v1, i.e., this patch doesn't address the comments raised in v1:

Quoted 7 lines

https://lore.kernel.org/lkml/[email protected]/

Arnaldo, orthogonal to this series, it'd be great if you could apply these two patches:

https://lore.kernel.org/lkml/[email protected]/T/#mc4c9c582e3816ab31af6d0187e6803de1a98ac84

I don't understand why this patch is related to the IBS error handling. Patch 11/13 does not modify the evsel__env() function. I have included your change in the new version of the patch. Thanks.

Quoted 9 lines

Then, Stephane, you could drop this patch and in your next version, patch 12/13 of this series would be replaced with what's at the end of this email:

https://lore.kernel.org/lkml/[email protected]/T/#m885ead1f280445357c91cce53c01217913bd017b

Thanks,

Kim

#25 lore raw

Re: [PATCH v2 11/13] perf tools: improve IBS error handling

Quoted 18 lines · Stephane Eranian

Kim,

Quoted 15 lines · Kim Phillips
Quoted 7 lines · Stephane Eranian

From: Kim Phillips <[email protected]>

This patch improves the error message returned on failed perf_event_open() on AMD when using IBS.

---

Hi, AFAICT this v2 == v1, i.e., this patch doesn't address the comments raised in v1:

https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
Quoted 5 lines

Arnaldo, orthogonal to this series, it'd be great if you could apply these two patches:

https://lore.kernel.org/lkml/[email protected]/T/#mc4c9c582e3816ab31af6d0187e6803de1a98ac84

I don't understand why this patch is related to the IBS error handling. Patch 11/13 does not modify the evsel__env() function.

Without it, after applying patch 2/2 of that series and executing the commands in 2/2's description will cause perf to sigsegv.

Quoted 2 lines

I have included your change in the new version of the patch. Thanks.

You mean this one?:

Quoted 5 lines
Quoted 5 lines

Then, Stephane, you could drop this patch and in your next version, patch 12/13 of this series would be replaced with what's at the end of this email:

https://lore.kernel.org/lkml/[email protected]/T/#m885ead1f280445357c91cce53c01217913bd017b

If so, great, but patch 1/2 in the two-series set mentioned above will need to be carried along with this series, unless acme applies those two directly.

Thanks,

Kim

#26 lore raw

PATCH v2 12/13 perf tools: improve error handling of AMD Branch Sampling

In reply to Stephane Eranian

This patch improves the error message printed by perf when perf_event_open() fails on AMD Zen3 when using the branch sampling feature. In the case of EINVAL, there are two main reasons: frequency mode or period is smaller than the depth of the branch sampling buffer (16). The patch checks the parameters of the call and tries to print a relevant message to explain the error:

$ perf record -b -e cpu/branch-brs/ -c 10 ls Error: AMD Branch Sampling does
not support sampling period smaller than what is reported in
/sys/devices/cpu/caps/branches.
$ perf record -b -e cpu/branch-brs/ ls Error: AMD Branch Sampling does not
support frequency mode sampling, must pass a fixed sampling period via -c
option or cpu/branch-brs,period=xxxx/.
---
 tools/perf/util/evsel.c | 8 ++++++++
 1 file changed, 8 insertions(+)
tools/perf/util/evsel.c +8 -0
index 1b24b3000d66..11aa6c28a9e4 100644
@@ -2883,6 +2883,14 @@ int evsel__open_strerror(struct evsel *evsel, struct target *target,
 			return scnprintf(msg, size, "The 'aux_output' feature is not supported, update the kernel.");
 		if (is_amd && (evsel->core.attr.precise_ip || !strncmp(evsel->pmu_name, "ibs", 3)) && (evsel->core.attr.exclude_kernel))
 			return scnprintf(msg, size, "AMD IBS can't exclude kernel events.  Try running at a higher privilege level.");
+
+		if (is_amd && ((evsel->core.attr.config & 0xff) == 0xc4) && (evsel->core.attr.sample_type & PERF_SAMPLE_BRANCH_STACK)) {
+			if (evsel->core.attr.freq)
+				return scnprintf(msg, size, "AMD Branch Sampling does not support frequency mode sampling, must pass a fixed sampling period via -c option or cpu/branch-brs,period=xxxx/.");
+			/* another reason is that the period is too small */
+			return scnprintf(msg, size, "AMD Branch Sampling does not support sampling period smaller than what is reported in /sys/devices/cpu/caps/branches.");
+		}
+
 		break;
 	case ENODATA:
 		return scnprintf(msg, size, "Cannot collect data source with the load latency event alone. "
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog
#27 lore raw

PATCH v2 13/13 perf report: add addr_from/addr_to sort dimensions

In reply to Stephane Eranian

This patch adds a new pair of sorting dimensions used with branch sampling. With the existing symbol_from/symbol_to, branches captured in the same function would be collapsed into a single function if the latency (cycles) was all the same. That is the case on Intel Broadwell for instance. Since Intel Skylake, the latency is captured by hardware and therefore is used to disambiguate branches.

This patch adds addr_from/addr_to sort dimensions that will sort branches based on their addresses and not the function there are in. The output is still the function name but the offset within the function is provided to uniquely identify each branch. These new sort dimensions also help with annotate because they create different entries in the histogram which, in turn, generates proper branch annotation.

Here is an example using AMD's branch sampling:

$ perf record -a -b -c 1000037 -e cpu/branch-brs/ test_prg
$ perf report
Samples: 6M of event 'cpu/branch-brs/', Event count (approx.): 6901276
Overhead  Command          Source Shared Object  Source Symbol                                   Target Symbol                                   Basic Block Cycle
  99.65%  test_prg	   test_prg              [.] test_thread                                 [.] test_thread                                 -
   0.02%  test_prg         [kernel.vmlinux]      [k] asm_sysvec_apic_timer_interrupt             [k] error_entry                                 -
$ perf report -F overhead,comm,dso,addr_from,addr_to
Samples: 6M of event 'cpu/branch-brs/', Event count (approx.): 6901276
Overhead  Command          Shared Object     Source Address          Target Address
   4.22%  test_prg         test_prg          [.] test_thread+0x3c    [.] test_thread+0x4
   4.13%  test_prg         test_prg          [.] test_thread+0x4     [.] test_thread+0x3a
   4.09%  test_prg         test_prg          [.] test_thread+0x3a    [.] test_thread+0x6
   4.08%  test_prg         test_prg          [.] test_thread+0x2     [.] test_thread+0x3c
   4.06%  test_prg         test_prg          [.] test_thread+0x3e    [.] test_thread+0x2
   3.87%  test_prg         test_prg          [.] test_thread+0x6     [.] test_thread+0x38
   3.84%  test_prg         test_prg          [.] test_thread         [.] test_thread+0x3e
   3.76%  test_prg         test_prg          [.] test_thread+0x1e    [.] test_thread
   3.76%  test_prg         test_prg          [.] test_thread+0x38    [.] test_thread+0x8
   3.56%  test_prg         test_prg          [.] test_thread+0x22    [.] test_thread+0x1e
   3.54%  test_prg         test_prg          [.] test_thread+0x8     [.] test_thread+0x36
   3.47%  test_prg         test_prg          [.] test_thread+0x1c    [.] test_thread+0x22
   3.45%  test_prg         test_prg          [.] test_thread+0x36    [.] test_thread+0xa
   3.28%  test_prg         test_prg          [.] test_thread+0x24    [.] test_thread+0x1c
   3.25%  test_prg         test_prg          [.] test_thread+0xa     [.] test_thread+0x34
   3.24%  test_prg         test_prg          [.] test_thread+0x1a    [.] test_thread+0x24
   3.20%  test_prg         test_prg          [.] test_thread+0x34    [.] test_thread+0xc
   3.04%  test_prg         test_prg          [.] test_thread+0x26    [.] test_thread+0x1a
   3.01%  test_prg         test_prg          [.] test_thread+0xc     [.] test_thread+0x32
   2.98%  test_prg         test_prg          [.] test_thread+0x18    [.] test_thread+0x26
   2.94%  test_prg         test_prg          [.] test_thread+0x32    [.] test_thread+0xe
   2.76%  test_prg         test_prg          [.] test_thread+0x28    [.] test_thread+0x18
   2.73%  test_prg         test_prg          [.] test_thread+0xe     [.] test_thread+0x30
   2.67%  test_prg         test_prg          [.] test_thread+0x30    [.] test_thread+0x10
   2.67%  test_prg         test_prg          [.] test_thread+0x16    [.] test_thread+0x28
   2.46%  test_prg         test_prg          [.] test_thread+0x10    [.] test_thread+0x2e
   2.44%  test_prg         test_prg          [.] test_thread+0x2a    [.] test_thread+0x16
   2.38%  test_prg         test_prg          [.] test_thread+0x14    [.] test_thread+0x2a
   2.32%  test_prg         test_prg          [.] test_thread+0x2e    [.] test_thread+0x12
   2.28%  test_prg         test_prg          [.] test_thread+0x12    [.] test_thread+0x2c
   2.16%  test_prg         test_prg          [.] test_thread+0x2c    [.] test_thread+0x14
   0.02%  test_prg         [kernel.vmlinux]  [k] asm_sysvec_apic_ti+0x5  [k] error_entry
---
 tools/perf/util/hist.c |   2 +
 tools/perf/util/hist.h |   2 +
 tools/perf/util/sort.c | 128 +++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/sort.h |   2 +
 4 files changed, 134 insertions(+)
tools/perf/util/hist.c +2 -0
index 65fe65ba03c2..39924202568f 100644
@@ -124,6 +124,7 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)
 		} else {
 			symlen = unresolved_col_width + 4 + 2;
 			hists__new_col_len(hists, HISTC_SYMBOL_FROM, symlen);
+			hists__new_col_len(hists, HISTC_ADDR_FROM, symlen);
 			hists__set_unres_dso_col_len(hists, HISTC_DSO_FROM);
 		}
 
@@ -138,6 +139,7 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)
 		} else {
 			symlen = unresolved_col_width + 4 + 2;
 			hists__new_col_len(hists, HISTC_SYMBOL_TO, symlen);
+			hists__new_col_len(hists, HISTC_ADDR_TO, symlen);
 			hists__set_unres_dso_col_len(hists, HISTC_DSO_TO);
 		}
 
tools/perf/util/hist.h +2 -0
index 5343b62476e6..9592853fab73 100644
@@ -76,6 +76,8 @@ enum hist_column {
 	HISTC_LOCAL_INS_LAT,
 	HISTC_GLOBAL_INS_LAT,
 	HISTC_P_STAGE_CYC,
+	HISTC_ADDR_FROM,
+	HISTC_ADDR_TO,
 	HISTC_NR_COLS, /* Last entry */
 };
 
tools/perf/util/sort.c +128 -0

146 lines not on this page · raw message

tools/perf/util/sort.h +2 -0
index b67c469aba79..571f6745dced 100644
@@ -250,6 +250,8 @@ enum sort_type {
 	SORT_SRCLINE_FROM,
 	SORT_SRCLINE_TO,
 	SORT_SYM_IPC,
+	SORT_ADDR_FROM,
+	SORT_ADDR_TO,
 
 	/* memory mode specific sort keys */
 	__SORT_MEMORY_MODE,
Signature · 1 line
2.34.0.rc0.344.g81b53c2807-goog