Skip to contentSkip to conversation

perf/x86/amd/core: Add AMD PerfMonV2 support

PATCH In review Sandipan Das 6 participants 21 messages Apr 27, 2022 – May 10, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover perf/x86/amd/core: Add AMD PerfMonV2 support

Add support for using AMD Performance Monitoring Version 2 (PerfMonV2) features on upcoming processors. New CPU features are introduced for PerfMonV2 detection. New MSR definitions are added to make use of an alternative PMC management scheme based on the new PMC global control and status registers.

The global control register provides the ability to start and stop multiple PMCs at the same time. This makes it possible to enable or disable all counters with a single MSR write instead of writing to the individual PMC control registers iteratively under x86_pmu_{enable,disable}(). The effects can be seen when counting the same events across multiple PMCs.

E.g.

  $ sudo perf stat -e "{cycles,instructions,cycles,instructions}" sleep 1

Before:

Performance counter stats for 'sleep 1':

             1013281      cycles
             1452859      instructions              #    1.43  insn per cycle
             1023462      cycles
             1461724      instructions              #    1.43  insn per cycle
         1.001644276 seconds time elapsed
  
         0.001948000 seconds user
         0.000000000 seconds sys

After:

Performance counter stats for 'sleep 1':

              999165      cycles
             1440456      instructions              #    1.44  insn per cycle
              999165      cycles
             1440456      instructions              #    1.44  insn per cycle
         1.001879504 seconds time elapsed
  
         0.001817000 seconds user
         0.000000000 seconds sys

No additional failures are seen upon running the following:
* perf built-in test suite
* perf_event_tests suite
* rr test suite

Previous versions can be found at:
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Changes in v4:
- Use ARCH_PERFMON feature bit instead of vendor info to determine the availability of CPUID leaf 0xA.

Changes in v3:
- Remove unused parameter from amd_pmu_cpu_reset().
- Add Hygon as a vendor that does not support CPUID leaf 0xA.

Changes in v2:
- Sort PerfCntrGlobal* register definitions based on MSR index.
- Use wrmsrl() in cpu_{starting,dead}().
- Add enum to extract bitfields from CPUID leaf 0x80000022.
- Remove static calls for counter management functions.
- Stop counters before inspecting overflow status in NMI handler.
- Save and restore PMU enabled state in NMI handler.
- Remove unused variable in NMI handler.
- Remove redundant write to APIC_LVTPC in NMI handler.
- Add comment on APIC_LVTPC mask bit behaviour during counter overflow.

Sandipan Das (7):
x86/cpufeatures: Add PerfMonV2 feature bit x86/msr: Add PerfCntrGlobal* registers perf/x86/amd/core: Detect PerfMonV2 support perf/x86/amd/core: Detect available counters perf/x86/amd/core: Add PerfMonV2 counter control perf/x86/amd/core: Add PerfMonV2 overflow handling
kvm: x86/cpuid: Fix CPUID leaf 0xA

 arch/x86/events/amd/core.c         | 227 +++++++++++++++++++++++++++--
 arch/x86/include/asm/cpufeatures.h |   2 +-
 arch/x86/include/asm/msr-index.h   |   5 +
 arch/x86/include/asm/perf_event.h  |  17 +++
 arch/x86/kernel/cpu/scattered.c    |   1 +
 arch/x86/kvm/cpuid.c               |   5 +
 6 files changed, 240 insertions(+), 17 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH v4 1/7 x86/cpufeatures: Add PerfMonV2 feature bit

CPUID leaf 0x80000022 i.e. ExtPerfMonAndDbg advertises some new performance monitoring features for AMD processors.

Bit 0 of EAX indicates support for Performance Monitoring Version 2 (PerfMonV2) features. If found to be set during PMU initialization, the EBX bits of the same CPUID function can be used to determine the number of available PMCs for different PMU types. Additionally, Core PMCs can be managed using new global control and status registers.

For better utilization of feature words, PerfMonV2 is added as a scattered feature bit.

---
 arch/x86/include/asm/cpufeatures.h | 2 +-
 arch/x86/kernel/cpu/scattered.c    | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +1 -1
index 30da1341f226..9b8ad6f4d310 100644
@@ -201,7 +201,7 @@
 #define X86_FEATURE_INVPCID_SINGLE	( 7*32+ 7) /* Effectively INVPCID && CR4.PCIDE=1 */
 #define X86_FEATURE_HW_PSTATE		( 7*32+ 8) /* AMD HW-PState */
 #define X86_FEATURE_PROC_FEEDBACK	( 7*32+ 9) /* AMD ProcFeedbackInterface */
-/* FREE!                                ( 7*32+10) */
+#define X86_FEATURE_PERFMON_V2		( 7*32+10) /* AMD Performance Monitoring Version 2 */
 #define X86_FEATURE_PTI			( 7*32+11) /* Kernel Page Table Isolation enabled */
 #define X86_FEATURE_RETPOLINE		( 7*32+12) /* "" Generic Retpoline mitigation for Spectre variant 2 */
 #define X86_FEATURE_RETPOLINE_LFENCE	( 7*32+13) /* "" Use LFENCE for Spectre variant 2 */
arch/x86/kernel/cpu/scattered.c +1 -0
index 4143b1e4c5c6..dbaa8326d6f2 100644
@@ -43,6 +43,7 @@ static const struct cpuid_bit cpuid_bits[] = {
 	{ X86_FEATURE_CPB,		CPUID_EDX,  9, 0x80000007, 0 },
 	{ X86_FEATURE_PROC_FEEDBACK,    CPUID_EDX, 11, 0x80000007, 0 },
 	{ X86_FEATURE_MBA,		CPUID_EBX,  6, 0x80000008, 0 },
+	{ X86_FEATURE_PERFMON_V2,	CPUID_EAX,  0, 0x80000022, 0 },
 	{ 0, 0, 0, 0, 0 }
 };
 
Signature · 1 line
2.34.1
#2 lore raw

PATCH v4 2/7 x86/msr: Add PerfCntrGlobal* registers

In reply to Sandipan Das

Add MSR definitions that will be used to enable the new AMD Performance Monitoring Version 2 (PerfMonV2) features. These include:

* Performance Counter Global Control (PerfCntrGlobalCtl)
* Performance Counter Global Status (PerfCntrGlobalStatus)
* Performance Counter Global Status Clear (PerfCntrGlobalStatusClr)

The new Performance Counter Global Control and Status MSRs provide an interface for enabling or disabling multiple counters at the same time and for testing overflow without probing the individual registers for each PMC.

The availability of these registers is indicated through the PerfMonV2 feature bit of CPUID leaf 0x80000022 EAX.

---
 arch/x86/include/asm/msr-index.h | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/include/asm/msr-index.h +5 -0
index 9e2e7185fc1d..a040f4af93c9 100644
@@ -527,6 +527,11 @@
 #define AMD_CPPC_DES_PERF(x)		(((x) & 0xff) << 16)
 #define AMD_CPPC_ENERGY_PERF_PREF(x)	(((x) & 0xff) << 24)
 
+/* AMD Performance Counter Global Status and Control MSRs */
+#define MSR_AMD64_PERF_CNTR_GLOBAL_STATUS	0xc0000300
+#define MSR_AMD64_PERF_CNTR_GLOBAL_CTL		0xc0000301
+#define MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR	0xc0000302
+
 /* Fam 17h MSRs */
 #define MSR_F17H_IRPERF			0xc00000e9
 
Signature · 1 line
2.34.1
#3 lore raw

PATCH v4 3/7 perf/x86/amd/core: Detect PerfMonV2 support

In reply to Sandipan Das

AMD Performance Monitoring Version 2 (PerfMonV2) introduces some new Core PMU features such as detection of the number of available PMCs and managing PMCs using global registers namely, PerfCntrGlobalCtl and PerfCntrGlobalStatus.

Clearing PerfCntrGlobalCtl and PerfCntrGlobalStatus ensures that all PMCs are inactive and have no pending overflows when CPUs are onlined or offlined.

The PMU version (x86_pmu.version) now indicates PerfMonV2 support and will be used to bypass the new features on unsupported processors.

---
 arch/x86/events/amd/core.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
arch/x86/events/amd/core.c +27 -0
index 8e1e818f8195..4a61257fc58d 100644
@@ -19,6 +19,9 @@ static unsigned long perf_nmi_window;
 #define AMD_MERGE_EVENT ((0xFULL << 32) | 0xFFULL)
 #define AMD_MERGE_EVENT_ENABLE (AMD_MERGE_EVENT | ARCH_PERFMON_EVENTSEL_ENABLE)
 
+/* PMC Enable and Overflow bits for PerfCntrGlobal* registers */
+static u64 amd_pmu_global_cntr_mask __read_mostly;
+
 static __initconst const u64 amd_hw_cache_event_ids
 				[PERF_COUNT_HW_CACHE_MAX]
 				[PERF_COUNT_HW_CACHE_OP_MAX]
@@ -578,6 +581,18 @@ static struct amd_nb *amd_alloc_nb(int cpu)
 	return nb;
 }
 
+static void amd_pmu_cpu_reset(void)
+{
+	if (x86_pmu.version < 2)
+		return;
+
+	/* Clear enable bits i.e. PerfCntrGlobalCtl.PerfCntrEn */
+	wrmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_CTL, 0);
+
+	/* Clear overflow bits i.e. PerfCntrGLobalStatus.PerfCntrOvfl */
+	wrmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, amd_pmu_global_cntr_mask);
+}
+
 static int amd_pmu_cpu_prepare(int cpu)
 {
 	struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
@@ -625,6 +640,7 @@ static void amd_pmu_cpu_starting(int cpu)
 	cpuc->amd_nb->refcnt++;
 
 	amd_brs_reset();
+	amd_pmu_cpu_reset();
 }
 
 static void amd_pmu_cpu_dead(int cpu)
@@ -644,6 +660,8 @@ static void amd_pmu_cpu_dead(int cpu)
 
 		cpuhw->amd_nb = NULL;
 	}
+
+	amd_pmu_cpu_reset();
 }
 
 /*
@@ -1185,6 +1203,15 @@ static int __init amd_core_pmu_init(void)
 	x86_pmu.eventsel	= MSR_F15H_PERF_CTL;
 	x86_pmu.perfctr		= MSR_F15H_PERF_CTR;
 	x86_pmu.num_counters	= AMD64_NUM_COUNTERS_CORE;
+
+	/* Check for Performance Monitoring v2 support */
+	if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) {
+		/* Update PMU version for later usage */
+		x86_pmu.version = 2;
+
+		amd_pmu_global_cntr_mask = (1ULL << x86_pmu.num_counters) - 1;
+	}
+
 	/*
 	 * AMD Core perfctr has separate MSRs for the NB events, see
 	 * the amd/uncore.c driver.
Signature · 1 line
2.34.1
#8 lore raw

PATCH v4 4/7 perf/x86/amd/core: Detect available counters

In reply to Sandipan Das

If AMD Performance Monitoring Version 2 (PerfMonV2) is supported, use CPUID leaf 0x80000022 EBX to detect the number of Core PMCs. This offers more flexibility if the counts change in later processor families.

---
 arch/x86/events/amd/core.c        |  6 ++++++
 arch/x86/include/asm/perf_event.h | 17 +++++++++++++++++
 2 files changed, 23 insertions(+)
arch/x86/events/amd/core.c +6 -0
index 4a61257fc58d..61a2fce99aa1 100644
@@ -1186,6 +1186,7 @@ static const struct attribute_group *amd_attr_update[] = {
 
 static int __init amd_core_pmu_init(void)
 {
+	union cpuid_0x80000022_ebx ebx;
 	u64 even_ctr_mask = 0ULL;
 	int i;
 
@@ -1206,9 +1207,14 @@ static int __init amd_core_pmu_init(void)
 
 	/* Check for Performance Monitoring v2 support */
 	if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) {
+		ebx.full = cpuid_ebx(EXT_PERFMON_DEBUG_FEATURES);
+
 		/* Update PMU version for later usage */
 		x86_pmu.version = 2;
 
+		/* Find the number of available Core PMCs */
+		x86_pmu.num_counters = ebx.split.num_core_pmc;
+
 		amd_pmu_global_cntr_mask = (1ULL << x86_pmu.num_counters) - 1;
 	}
 
arch/x86/include/asm/perf_event.h +17 -0
index a5dea5da1b52..7aa1d420c779 100644
@@ -186,6 +186,18 @@ union cpuid28_ecx {
 	unsigned int            full;
 };
 
+/*
+ * AMD "Extended Performance Monitoring and Debug" CPUID
+ * detection/enumeration details:
+ */
+union cpuid_0x80000022_ebx {
+	struct {
+		/* Number of Core Performance Counters */
+		unsigned int	num_core_pmc:4;
+	} split;
+	unsigned int		full;
+};
+
 struct x86_pmu_capability {
 	int		version;
 	int		num_counters_gp;
@@ -372,6 +384,11 @@ struct pebs_xmm {
 	u64 xmm[16*2];	/* two entries for each register */
 };
 
+/*
+ * AMD Extended Performance Monitoring and Debug cpuid feature detection
+ */
+#define EXT_PERFMON_DEBUG_FEATURES		0x80000022
+
 /*
  * IBS cpuid feature detection
  */
Signature · 1 line
2.34.1
#9 lore raw

PATCH v4 5/7 perf/x86/amd/core: Add PerfMonV2 counter control

In reply to Sandipan Das

If AMD Performance Monitoring Version 2 (PerfMonV2) is supported, use a new scheme to manage the Core PMCs using the new global control and status registers. This will be bypassed on unsupported hardware (x86_pmu.version < 2).

Currently, all PMCs have dedicated control (PERF_CTL) and counter (PERF_CTR) registers. For a given PMC, the enable (En) bit of its PERF_CTL register is used to start or stop counting.

The Performance Counter Global Control (PerfCntrGlobalCtl) register has enable (PerfCntrEn) bits for each PMC. For a PMC to start counting, both PERF_CTL and PerfCntrGlobalCtl enable bits must be set. If either of those are cleared, the PMC stops counting.

In x86_pmu_{en,dis}able_all(), the PERF_CTL registers of all active PMCs are written to in a loop. Ideally, PMCs counting the same event that were started and stopped at the same time should record the same counts. Due to delays in between writes to the PERF_CTL registers across loop iterations, the PMCs cannot be enabled or disabled at the same instant and hence, record slightly different counts. This is fixed by enabling or disabling all active PMCs at the same time with a single write to the PerfCntrGlobalCtl register.

---
 arch/x86/events/amd/core.c | 50 ++++++++++++++++++++++++++++++++++----
 1 file changed, 45 insertions(+), 5 deletions(-)
arch/x86/events/amd/core.c +45 -5
index 61a2fce99aa1..5b100a5f8489 100644
@@ -664,6 +664,11 @@ static void amd_pmu_cpu_dead(int cpu)
 	amd_pmu_cpu_reset();
 }
 
+static inline void amd_pmu_set_global_ctl(u64 ctl)
+{
+	wrmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_CTL, ctl);
+}
+
 /*
  * When a PMC counter overflows, an NMI is used to process the event and
  * reset the counter. NMI latency can result in the counter being updated
@@ -693,15 +698,11 @@ static void amd_pmu_wait_on_overflow(int idx)
 	}
 }
 
-static void amd_pmu_disable_all(void)
+static void amd_pmu_check_overflow(void)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
 	int idx;
 
-	amd_brs_disable_all();
-
-	x86_pmu_disable_all();
-
 	/*
 	 * This shouldn't be called from NMI context, but add a safeguard here
 	 * to return, since if we're in NMI context we can't wait for an NMI
@@ -748,6 +749,26 @@ static void amd_pmu_enable_all(int added)
 	}
 }
 
+static void amd_pmu_v2_enable_event(struct perf_event *event)
+{
+	struct hw_perf_event *hwc = &event->hw;
+
+	/*
+	 * Testing cpu_hw_events.enabled should be skipped in this case unlike
+	 * in x86_pmu_enable_event().
+	 *
+	 * Since cpu_hw_events.enabled is set only after returning from
+	 * x86_pmu_start(), the PMCs must be programmed and kept ready.
+	 * Counting starts only after x86_pmu_enable_all() is called.
+	 */
+	__x86_pmu_enable_event(hwc, ARCH_PERFMON_EVENTSEL_ENABLE);
+}
+
+static void amd_pmu_v2_enable_all(int added)
+{
+	amd_pmu_set_global_ctl(amd_pmu_global_cntr_mask);
+}
+
 static void amd_pmu_disable_event(struct perf_event *event)
 {
 	x86_pmu_disable_event(event);
@@ -765,6 +786,20 @@ static void amd_pmu_disable_event(struct perf_event *event)
 	amd_pmu_wait_on_overflow(event->hw.idx);
 }
 
+static void amd_pmu_disable_all(void)
+{
+	amd_brs_disable_all();
+	x86_pmu_disable_all();
+	amd_pmu_check_overflow();
+}
+
+static void amd_pmu_v2_disable_all(void)
+{
+	/* Disable all PMCs */
+	amd_pmu_set_global_ctl(0);
+	amd_pmu_check_overflow();
+}
+
 static void amd_pmu_add_event(struct perf_event *event)
 {
 	if (needs_branch_stack(event))
@@ -1216,6 +1251,11 @@ static int __init amd_core_pmu_init(void)
 		x86_pmu.num_counters = ebx.split.num_core_pmc;
 
 		amd_pmu_global_cntr_mask = (1ULL << x86_pmu.num_counters) - 1;
+
+		/* Update PMC handling functions */
+		x86_pmu.enable_all = amd_pmu_v2_enable_all;
+		x86_pmu.disable_all = amd_pmu_v2_disable_all;
+		x86_pmu.enable = amd_pmu_v2_enable_event;
 	}
 
 	/*
Signature · 1 line
2.34.1
#13 lore raw

PATCH v4 6/7 perf/x86/amd/core: Add PerfMonV2 overflow handling

In reply to Sandipan Das

If AMD Performance Monitoring Version 2 (PerfMonV2) is supported, use a new scheme to process Core PMC overflows in the NMI handler using the new global control and status registers. This will be bypassed on unsupported hardware (x86_pmu.version < 2).

In x86_pmu_handle_irq(), overflows are detected by testing the contents of the PERF_CTR register for each active PMC in a loop. The new scheme instead inspects the overflow bits of the global status register.

The Performance Counter Global Status (PerfCntrGlobalStatus) register has overflow (PerfCntrOvfl) bits for each PMC. This is, however, a read-only MSR. To acknowledge that overflows have been processed, the NMI handler must clear the bits by writing to the PerfCntrGlobalStatusClr register.

In x86_pmu_handle_irq(), PMCs counting the same event that are started and stopped at the same time record slightly different counts due to delays in between reads from the PERF_CTR registers. This is fixed by stopping and starting the PMCs at the same before and with a single write to the Performance Counter Global Control (PerfCntrGlobalCtl) upon entering and before exiting the NMI handler.

---
 arch/x86/events/amd/core.c | 144 ++++++++++++++++++++++++++++++++++---
 1 file changed, 133 insertions(+), 11 deletions(-)
arch/x86/events/amd/core.c +133 -11
index 5b100a5f8489..5de2c833acf2 100644
@@ -8,6 +8,7 @@
 #include <linux/delay.h>
 #include <linux/jiffies.h>
 #include <asm/apicdef.h>
+#include <asm/apic.h>
 #include <asm/nmi.h>
 
 #include "../perf_event.h"
@@ -669,6 +670,45 @@ static inline void amd_pmu_set_global_ctl(u64 ctl)
 	wrmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_CTL, ctl);
 }
 
+static inline u64 amd_pmu_get_global_status(void)
+{
+	u64 status;
+
+	/* PerfCntrGlobalStatus is read-only */
+	rdmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_STATUS, status);
+
+	return status & amd_pmu_global_cntr_mask;
+}
+
+static inline void amd_pmu_ack_global_status(u64 status)
+{
+	/*
+	 * PerfCntrGlobalStatus is read-only but an overflow acknowledgment
+	 * mechanism exists; writing 1 to a bit in PerfCntrGlobalStatusClr
+	 * clears the same bit in PerfCntrGlobalStatus
+	 */
+
+	/* Only allow modifications to PerfCntrGlobalStatus.PerfCntrOvfl */
+	status &= amd_pmu_global_cntr_mask;
+	wrmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, status);
+}
+
+static bool amd_pmu_test_overflow_topbit(int idx)
+{
+	u64 counter;
+
+	rdmsrl(x86_pmu_event_addr(idx), counter);
+
+	return !(counter & BIT_ULL(x86_pmu.cntval_bits - 1));
+}
+
+static bool amd_pmu_test_overflow_status(int idx)
+{
+	return amd_pmu_get_global_status() & BIT_ULL(idx);
+}
+
+DEFINE_STATIC_CALL(amd_pmu_test_overflow, amd_pmu_test_overflow_topbit);
+
 /*
  * When a PMC counter overflows, an NMI is used to process the event and
  * reset the counter. NMI latency can result in the counter being updated
@@ -681,7 +721,6 @@ static inline void amd_pmu_set_global_ctl(u64 ctl)
 static void amd_pmu_wait_on_overflow(int idx)
 {
 	unsigned int i;
-	u64 counter;
 
 	/*
 	 * Wait for the counter to be reset if it has overflowed. This loop
@@ -689,8 +728,7 @@ static void amd_pmu_wait_on_overflow(int idx)
 	 * forever...
 	 */
 	for (i = 0; i < OVERFLOW_WAIT_COUNT; i++) {
-		rdmsrl(x86_pmu_event_addr(idx), counter);
-		if (counter & (1ULL << (x86_pmu.cntval_bits - 1)))
+		if (!static_call(amd_pmu_test_overflow)(idx))
 			break;
 
 		/* Might be in IRQ context, so can't sleep */
@@ -830,6 +868,24 @@ static void amd_pmu_del_event(struct perf_event *event)
  * handled a counter. When an un-handled NMI is received, it will be claimed
  * only if arriving within that window.
  */
+static inline int amd_pmu_adjust_nmi_window(int handled)
+{
+	/*
+	 * If a counter was handled, record a timestamp such that un-handled
+	 * NMIs will be claimed if arriving within that window.
+	 */
+	if (handled) {
+		this_cpu_write(perf_nmi_tstamp, jiffies + perf_nmi_window);
+
+		return handled;
+	}
+
+	if (time_after(jiffies, this_cpu_read(perf_nmi_tstamp)))
+		return NMI_DONE;
+
+	return NMI_HANDLED;
+}
+
 static int amd_pmu_handle_irq(struct pt_regs *regs)
 {
 	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
@@ -857,20 +913,84 @@ static int amd_pmu_handle_irq(struct pt_regs *regs)
 	if (pmu_enabled)
 		amd_pmu_enable_all(0);
 
+	return amd_pmu_adjust_nmi_window(handled);
+}
+
+static int amd_pmu_v2_handle_irq(struct pt_regs *regs)
+{
+	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+	struct perf_sample_data data;
+	struct hw_perf_event *hwc;
+	struct perf_event *event;
+	int handled = 0, idx;
+	u64 status, mask;
+	bool pmu_enabled;
+
 	/*
-	 * If a counter was handled, record a timestamp such that un-handled
-	 * NMIs will be claimed if arriving within that window.
+	 * Save the PMU state as it needs to be restored when leaving the
+	 * handler
 	 */
-	if (handled) {
-		this_cpu_write(perf_nmi_tstamp, jiffies + perf_nmi_window);
+	pmu_enabled = cpuc->enabled;
+	cpuc->enabled = 0;
 
-		return handled;
+	/* Stop counting */
+	amd_pmu_v2_disable_all();
+
+	status = amd_pmu_get_global_status();
+
+	/* Check if any overflows are pending */
+	if (!status)
+		goto done;
+
+	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
+		if (!test_bit(idx, cpuc->active_mask))
+			continue;
+
+		event = cpuc->events[idx];
+		hwc = &event->hw;
+		x86_perf_event_update(event);
+		mask = BIT_ULL(idx);
+
+		if (!(status & mask))
+			continue;
+
+		/* Event overflow */
+		handled++;
+		perf_sample_data_init(&data, 0, hwc->last_period);
+
+		if (!x86_perf_event_set_period(event))
+			continue;
+
+		if (perf_event_overflow(event, &data, regs))
+			x86_pmu_stop(event, 0);
+
+		status &= ~mask;
 	}
 
-	if (time_after(jiffies, this_cpu_read(perf_nmi_tstamp)))
-		return NMI_DONE;
+	/*
+	 * It should never be the case that some overflows are not handled as
+	 * the corresponding PMCs are expected to be inactive according to the
+	 * active_mask
+	 */
+	WARN_ON(status > 0);
 
-	return NMI_HANDLED;
+	/* Clear overflow bits */
+	amd_pmu_ack_global_status(~status);
+
+	/*
+	 * Unmasking the LVTPC is not required as the Mask (M) bit of the LVT
+	 * PMI entry is not set by the local APIC when a PMC overflow occurs
+	 */
+	inc_irq_stat(apic_perf_irqs);
+
+done:
+	cpuc->enabled = pmu_enabled;
+
+	/* Resume counting only if PMU is active */
+	if (pmu_enabled)
+		amd_pmu_v2_enable_all(0);
+
+	return amd_pmu_adjust_nmi_window(handled);
 }
 
 static struct event_constraint *
@@ -1256,6 +1376,8 @@ static int __init amd_core_pmu_init(void)
 		x86_pmu.enable_all = amd_pmu_v2_enable_all;
 		x86_pmu.disable_all = amd_pmu_v2_disable_all;
 		x86_pmu.enable = amd_pmu_v2_enable_event;
+		x86_pmu.handle_irq = amd_pmu_v2_handle_irq;
+		static_call_update(amd_pmu_test_overflow, amd_pmu_test_overflow_status);
 	}
 
 	/*
Signature · 1 line
2.34.1
#16 lore raw

PATCH v4 7/7 kvm: x86/cpuid: Fix CPUID leaf 0xA

In reply to Sandipan Das

On some x86 processors, CPUID leaf 0xA provides information on Architectural Performance Monitoring features. It advertises a PMU version which Qemu uses to determine the availability of additional MSRs to manage the PMCs.

Upon receiving a KVM_GET_SUPPORTED_CPUID ioctl request for the same, the kernel constructs return values based on the x86_pmu_capability irrespective of the vendor.

This leaf and the additional MSRs are not supported on AMD and Hygon processors. If AMD PerfMonV2 is detected, the PMU version is set to 2 and guest startup breaks because of an attempt to access a non-existent MSR. Return zeros to avoid this.

Fixes a6c06ed1a60a ("KVM: Expose the architectural performance monitoring CPUID leaf")
---
 arch/x86/kvm/cpuid.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/kvm/cpuid.c +5 -0
index 4b62d80bb22f..e66ebb747084 100644
@@ -872,6 +872,11 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		union cpuid10_eax eax;
 		union cpuid10_edx edx;
 
+		if (!static_cpu_has(X86_FEATURE_ARCH_PERFMON)) {
+			entry->eax = entry->ebx = entry->ecx = entry->edx = 0;
+			break;
+		}
+
 		perf_get_x86_pmu_capability(&cap);
 
 		/*
Signature · 1 line
2.34.1
maintainer
#20 lore raw
In reply to Sandipan Das
Quoted 9 lines · Sandipan Das

Sandipan Das (7):
x86/cpufeatures: Add PerfMonV2 feature bit x86/msr: Add PerfCntrGlobal* registers perf/x86/amd/core: Detect PerfMonV2 support perf/x86/amd/core: Detect available counters perf/x86/amd/core: Add PerfMonV2 counter control perf/x86/amd/core: Add PerfMonV2 overflow handling
kvm: x86/cpuid: Fix CPUID leaf 0xA

Quoted 7 lines
 arch/x86/events/amd/core.c         | 227 +++++++++++++++++++++++++++--
 arch/x86/include/asm/cpufeatures.h |   2 +-
 arch/x86/include/asm/msr-index.h   |   5 +
 arch/x86/include/asm/perf_event.h  |  17 +++
 arch/x86/kernel/cpu/scattered.c    |   1 +
 arch/x86/kvm/cpuid.c               |   5 +
 6 files changed, 240 insertions(+), 17 deletions(-)

From: Documentation/process/submitting-patches.rst

Don't get discouraged - or impatient
------------------------------------

After you have submitted your change, be patient and wait. Reviewers are busy people and may not get to your patch right away.

Once upon a time, patches used to disappear into the void without comment, but the development process works more smoothly than that now. You should receive comments within a week or so; if that does not happen, make sure that you have sent your patches to the right place. Wait for a minimum of

						     ^^^^^^^^^^^^^^^^^^^^^

one week before resubmitting or pinging reviewers - possibly longer during ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
busy times like merge windows.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#14 lore raw

Re: [PATCH v4 6/7] perf/x86/amd/core: Add PerfMonV2 overflow handling

In reply to Sandipan Das

arch/x86/events/amd/core.c static inline void amd_pmu_set_global_ctl(u64 ctl) · line 686

Quoted 4 lines · Sandipan Das
arch/x86/events/amd/core.c +4 -0
+static inline void amd_pmu_ack_global_status(u64 status)
+{
+	/*
+	 * PerfCntrGlobalStatus is read-only but an overflow acknowledgment

If wrmsrl PerfCntrGlobalStatus, remain silent or report #GP ?

arch/x86/events/amd/core.c static inline void amd_pmu_set_global_ctl(u64 ctl) · line 693

Quoted 7 lines
arch/x86/events/amd/core.c +7 -0
+	 * mechanism exists; writing 1 to a bit in PerfCntrGlobalStatusClr
+	 * clears the same bit in PerfCntrGlobalStatus
+	 */
+
+	/* Only allow modifications to PerfCntrGlobalStatus.PerfCntrOvfl */
+	status &= amd_pmu_global_cntr_mask;
+	wrmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, status);

If rdmsrl PerfCntrGlobalStatusClr, does it return 0 or the value of PerfCntrGlobalStatus ?

Quoted 1 line

+}

#15 lore raw

Re: [PATCH v4 6/7] perf/x86/amd/core: Add PerfMonV2 overflow handling

Quoted 8 lines · Like Xu

arch/x86/events/amd/core.c static inline void amd_pmu_set_global_ctl(u64 ctl) · line 686

Quoted 4 lines · Sandipan Das
arch/x86/events/amd/core.c +4 -0
+static inline void amd_pmu_ack_global_status(u64 status)
+{
+    /*
+     * PerfCntrGlobalStatus is read-only but an overflow acknowledgment

If wrmsrl PerfCntrGlobalStatus, remain silent or report #GP ?

This is a read-only MSR and writes are silently ignored.

Quoted 10 lines
Quoted 7 lines

+     * mechanism exists; writing 1 to a bit in PerfCntrGlobalStatusClr +     * clears the same bit in PerfCntrGlobalStatus
+     */
+
+    /* Only allow modifications to PerfCntrGlobalStatus.PerfCntrOvfl */ +    status &= amd_pmu_global_cntr_mask;
+    wrmsrl(MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, status);

If rdmsrl PerfCntrGlobalStatusClr, does it return 0 or the value of PerfCntrGlobalStatus ?

This is a write-only MSR and reads are undefined.

The "Field Access Type" section from the AMD Processor Programming Reference (PPR) has these details.

- Sandipan

#17 lore raw

Re: [PATCH v4 7/7] kvm: x86/cpuid: Fix CPUID leaf 0xA

In reply to Sandipan Das
Quoted 28 lines · Sandipan Das

On some x86 processors, CPUID leaf 0xA provides information on Architectural Performance Monitoring features. It advertises a PMU version which Qemu uses to determine the availability of additional MSRs to manage the PMCs.

Upon receiving a KVM_GET_SUPPORTED_CPUID ioctl request for the same, the kernel constructs return values based on the x86_pmu_capability irrespective of the vendor.

This leaf and the additional MSRs are not supported on AMD and Hygon processors. If AMD PerfMonV2 is detected, the PMU version is set to 2 and guest startup breaks because of an attempt to access a non-existent MSR. Return zeros to avoid this.

Fixes a6c06ed1a60a ("KVM: Expose the architectural performance monitoring CPUID leaf")
---
 arch/x86/kvm/cpuid.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/kvm/cpuid.c +0 -0
index 4b62d80bb22f..e66ebb747084 100644

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 875

Quoted 4 lines
arch/x86/kvm/cpuid.c +1 -0
@@ -872,6 +872,11 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
                union cpuid10_eax eax;
                union cpuid10_edx edx;
 
+               if (!static_cpu_has(X86_FEATURE_ARCH_PERFMON)) {

Should this be checking kvm_cpu_cap_has(X86_FEATURE_ARCH_PERFMON) instead?

Quoted 10 lines
arch/x86/kvm/cpuid.c +4 -1
+                       entry->eax = entry->ebx = entry->ecx = entry->edx = 0;
+                       break;
+               }
+
                perf_get_x86_pmu_capability(&cap);
 
                /*
--

2.34.1

maintainer
#18 lore raw

Re: [PATCH v4 7/7] kvm: x86/cpuid: Fix CPUID leaf 0xA

Quoted 32 lines · Jim Mattson
Quoted 31 lines · Sandipan Das

On some x86 processors, CPUID leaf 0xA provides information on Architectural Performance Monitoring features. It advertises a PMU version which Qemu uses to determine the availability of additional MSRs to manage the PMCs.

Upon receiving a KVM_GET_SUPPORTED_CPUID ioctl request for the same, the kernel constructs return values based on the x86_pmu_capability irrespective of the vendor.

This leaf and the additional MSRs are not supported on AMD and Hygon processors. If AMD PerfMonV2 is detected, the PMU version is set to 2 and guest startup breaks because of an attempt to access a non-existent MSR. Return zeros to avoid this.

Fixes a6c06ed1a60a ("KVM: Expose the architectural performance monitoring CPUID leaf")
---
  arch/x86/kvm/cpuid.c | 5 +++++
  1 file changed, 5 insertions(+)
arch/x86/kvm/cpuid.c +0 -0
index 4b62d80bb22f..e66ebb747084 100644
@@ -872,6 +872,11 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
                 union cpuid10_eax eax;
                 union cpuid10_edx edx;
 
Quoted 3 lines

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 875

Quoted 1 line
arch/x86/kvm/cpuid.c +1 -0
+               if (!static_cpu_has(X86_FEATURE_ARCH_PERFMON)) {

Should this be checking kvm_cpu_cap_has(X86_FEATURE_ARCH_PERFMON) instead?

Yes, it should. I have queued this patch for 5.18.

Paolo

maintainer
#19 lore raw

Re: [PATCH v4 7/7] kvm: x86/cpuid: Fix CPUID leaf 0xA

In reply to Jim Mattson
Quoted 32 lines · Jim Mattson
Quoted 31 lines · Sandipan Das

On some x86 processors, CPUID leaf 0xA provides information on Architectural Performance Monitoring features. It advertises a PMU version which Qemu uses to determine the availability of additional MSRs to manage the PMCs.

Upon receiving a KVM_GET_SUPPORTED_CPUID ioctl request for the same, the kernel constructs return values based on the x86_pmu_capability irrespective of the vendor.

This leaf and the additional MSRs are not supported on AMD and Hygon processors. If AMD PerfMonV2 is detected, the PMU version is set to 2 and guest startup breaks because of an attempt to access a non-existent MSR. Return zeros to avoid this.

Fixes a6c06ed1a60a ("KVM: Expose the architectural performance monitoring CPUID leaf")
---
  arch/x86/kvm/cpuid.c | 5 +++++
  1 file changed, 5 insertions(+)
arch/x86/kvm/cpuid.c +0 -0
index 4b62d80bb22f..e66ebb747084 100644
@@ -872,6 +872,11 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
                 union cpuid10_eax eax;
                 union cpuid10_edx edx;
 
Quoted 3 lines

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 875

Quoted 1 line
arch/x86/kvm/cpuid.c +1 -0
+               if (!static_cpu_has(X86_FEATURE_ARCH_PERFMON)) {

Should this be checking kvm_cpu_cap_has(X86_FEATURE_ARCH_PERFMON) instead?

Ah, it cannot because X86_FEATURE_ARCH_PERFMON is a synthetic feature. kvm_cpu_cap_has only works with features that are backed by CPUID bits.

Paolo

#10 lore raw

Re: [PATCH v4 5/7] perf/x86/amd/core: Add PerfMonV2 counter control

In reply to Sandipan Das

Review static int __init amd_core_pmu_init(void) · line 1259

Quoted 13 lines · Sandipan Das
  static void amd_pmu_add_event(struct perf_event *event)
  {
  	if (needs_branch_stack(event))
(quoted diff) +5 -0
@@ -1216,6 +1251,11 @@ static int __init amd_core_pmu_init(void)
  		x86_pmu.num_counters = ebx.split.num_core_pmc;
 
  		amd_pmu_global_cntr_mask = (1ULL << x86_pmu.num_counters) - 1;
+
+		/* Update PMC handling functions */
+		x86_pmu.enable_all = amd_pmu_v2_enable_all;
+		x86_pmu.disable_all = amd_pmu_v2_disable_all;
+		x86_pmu.enable = amd_pmu_v2_enable_event;
  	}

Considering the below part of code also run on the PerfMonV2 host

whatever the guest PMU version, how about updating these two use cases as well in this patch ?

arch/x86/events/amd/core.c +0 -0
index 262e39a85031..2f7c62e46314 100644
@@ -1469,8 +1469,8 @@ void amd_pmu_enable_virt(void)

        cpuc->perf_ctr_virt_mask = 0;

        /* Reload all events */
-       amd_pmu_disable_all();
-       x86_pmu_enable_all(0);
+       amd_pmu.disable_all();
+       amd_pmu.enable_all(0);
 }
 EXPORT_SYMBOL_GPL(amd_pmu_enable_virt);

@@ -1487,7 +1487,7 @@ void amd_pmu_disable_virt(void) cpuc->perf_ctr_virt_mask = AMD64_EVENTSEL_HOSTONLY;

        /* Reload all events */
-       amd_pmu_disable_all();
-       x86_pmu_enable_all(0);
+       amd_pmu.disable_all();
+       amd_pmu.enable_all(0);
 }
 EXPORT_SYMBOL_GPL(amd_pmu_disable_virt);

#11 lore raw

Re: [PATCH v4 5/7] perf/x86/amd/core: Add PerfMonV2 counter control

Quoted 36 lines · Like Xu

Review static int __init amd_core_pmu_init(void) · line 1258

Quoted 12 lines · Sandipan Das

  static void amd_pmu_add_event(struct perf_event *event) {
      if (needs_branch_stack(event))

(quoted diff) +5 -0
@@ -1216,6 +1251,11 @@ static int __init amd_core_pmu_init(void)
           x86_pmu.num_counters = ebx.split.num_core_pmc;
             amd_pmu_global_cntr_mask = (1ULL << x86_pmu.num_counters) - 1;
+
+        /* Update PMC handling functions */
+        x86_pmu.enable_all = amd_pmu_v2_enable_all;
+        x86_pmu.disable_all = amd_pmu_v2_disable_all;
+        x86_pmu.enable = amd_pmu_v2_enable_event;
       }

Considering the below part of code also run on the PerfMonV2 host

whatever the guest PMU version, how about updating these two use cases as well in this patch ?

arch/x86/events/amd/core.c +2 -2
index 262e39a85031..2f7c62e46314 100644
@@ -1469,8 +1469,8 @@ void amd_pmu_enable_virt(void)
         cpuc->perf_ctr_virt_mask = 0;
 
         /* Reload all events */
-       amd_pmu_disable_all();
-       x86_pmu_enable_all(0);
+       amd_pmu.disable_all();
+       amd_pmu.enable_all(0);
  }
  EXPORT_SYMBOL_GPL(amd_pmu_enable_virt);
 

arch/x86/events/amd/core.c void amd_pmu_disable_virt(void) · line 1493

Quoted 9 lines
arch/x86/events/amd/core.c +2 -2
@@ -1487,7 +1487,7 @@ void amd_pmu_disable_virt(void)
         cpuc->perf_ctr_virt_mask = AMD64_EVENTSEL_HOSTONLY;
 
         /* Reload all events */
-       amd_pmu_disable_all();
-       x86_pmu_enable_all(0);
+       amd_pmu.disable_all();
+       amd_pmu.enable_all(0);
  }
  EXPORT_SYMBOL_GPL(amd_pmu_disable_virt);

Good point. I had made these changes as a part of the guest PerfMonV2 enablement.

Peter,

Since these patches are now in perf/core, should I send out a separate fix for this?

- Sandipan

maintainer
#12 lore raw

Re: [PATCH v4 5/7] perf/x86/amd/core: Add PerfMonV2 counter control

Quoted 3 lines · Sandipan Das

Since these patches are now in perf/core, should I send out a separate fix for this?

Yes please.

#4 lore raw

Re: [PATCH v4 3/7] perf/x86/amd/core: Detect PerfMonV2 support

In reply to Sandipan Das
Quoted 1 line · Sandipan Das
  	x86_pmu.num_counters	= AMD64_NUM_COUNTERS_CORE;

Thus boot_cpu_has(X86_FEATURE_PERFCTR_CORE) is true.

arch/x86/events/amd/core.c static int __init amd_core_pmu_init(void) · line 1210

Quoted 5 lines
arch/x86/events/amd/core.c +5 -0
+
+	/* Check for Performance Monitoring v2 support */
+	if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) {
+		/* Update PMU version for later usage */
+		x86_pmu.version = 2;

Is it safe to assume that once AMD CPU has the PERFMON_V2 (or further) bit,

it must also have the PERFCTR_CORE bit set ?

#5 lore raw

Re: [PATCH v4 3/7] perf/x86/amd/core: Detect PerfMonV2 support

Quoted 12 lines · Like Xu
Quoted 1 line · Sandipan Das

      x86_pmu.num_counters    = AMD64_NUM_COUNTERS_CORE;

Thus boot_cpu_has(X86_FEATURE_PERFCTR_CORE) is true.

Quoted 5 lines

+
+    /* Check for Performance Monitoring v2 support */ +    if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) { +        /* Update PMU version for later usage */ +        x86_pmu.version = 2;

Quoted 4 lines

Is it safe to assume that once AMD CPU has the PERFMON_V2 (or further) bit,

it must also have the PERFCTR_CORE bit set ?

Yes, always. There won't be a case where PERFCTR_CORE is absent but PERFMON_V2 is present.

- Sandipan

maintainer
#6 lore raw

Re: [PATCH v4 3/7] perf/x86/amd/core: Detect PerfMonV2 support

Quoted 14 lines · Sandipan Das
Quoted 12 lines · Like Xu
Quoted 1 line · Sandipan Das

      x86_pmu.num_counters    = AMD64_NUM_COUNTERS_CORE;

Thus boot_cpu_has(X86_FEATURE_PERFCTR_CORE) is true.

Quoted 5 lines

+
+    /* Check for Performance Monitoring v2 support */ +    if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) { +        /* Update PMU version for later usage */ +        x86_pmu.version = 2;

Quoted 7 lines
Quoted 4 lines

Is it safe to assume that once AMD CPU has the PERFMON_V2 (or further) bit,

it must also have the PERFCTR_CORE bit set ?

Yes, always. There won't be a case where PERFCTR_CORE is absent but PERFMON_V2 is present.

Let me introduce you to this dodgy virt stuff :-) Best put a sanity check on it.

#7 lore raw

Re: [PATCH v4 3/7] perf/x86/amd/core: Detect PerfMonV2 support

Hi Peter,

Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Sandipan Das
Quoted 14 lines · Like Xu
Quoted 1 line · Sandipan Das

      x86_pmu.num_counters    = AMD64_NUM_COUNTERS_CORE;

Thus boot_cpu_has(X86_FEATURE_PERFCTR_CORE) is true.

Quoted 5 lines

+
+    /* Check for Performance Monitoring v2 support */ +    if (boot_cpu_has(X86_FEATURE_PERFMON_V2)) { +        /* Update PMU version for later usage */ +        x86_pmu.version = 2;

Is it safe to assume that once AMD CPU has the PERFMON_V2 (or further) bit,

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

it must also have the PERFCTR_CORE bit set ?

Yes, always. There won't be a case where PERFCTR_CORE is absent but PERFMON_V2 is present.

Let me introduce you to this dodgy virt stuff :-) Best put a sanity check on it.

I see that amd_core_pmu_init() returns if X86_FEATURE_PERFCTR_CORE is not found right after entry. Is there anywhere else that you feel should have an additional sanity check?

I also noticed a bug in the X86_FEATURE_PERFCTR_CORE feature check where it should have returned something like -ENODEV instead of 0. Will send out a fix for it.

- Sandipan