Skip to contentSkip to conversation

Support Perf Extension on AMD KVM guests

PATCH Superseded Janakarajan Natarajan 6 participants 21 messages Nov 6, 2017 – Dec 6, 2017

Subsystems (3)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover Support Perf Extension on AMD KVM guests

This patchset adds support for Perf Extension on AMD KVM guests.

When perf runs on a guest with family = 15h || 17h, the MSRs that are accessed, when the Perf Extension flag is made available, differ from the existing K7 MSRs. The accesses are to the AMD Core Performance Extension counters which provide 2 extra counters and new MSRs for both the event select and counter registers.

Routines are introduced to choose the proper MSR based on the guest family. Since the new event select and counter MSRs are interleaved and K7 MSRs are contiguous, the logic to map them to the gp_counters[] is changed.

Additionally, a fix is provided for CPUID_8000_0001_ECX in reverse_cpuid[] to change the CPUID function from 0xc0000001 to 0x80000001.

This patchset has been tested with Family 17h and Opteron G1 guests.

v1->v2:
* Rearranged MSR #defines based on Boris's suggestion.

Janakarajan Natarajan (4):
x86/kvm/cpuid: Fix CPUID function for word 6 (80000001_ECX) Add AMD Core Perf Extension MSRs
Add support for AMD Core Perf Extension in guest Expose AMD Core Perf Extension flag to guests

 arch/x86/include/asm/msr-index.h |  14 +++++
 arch/x86/kvm/cpuid.c             |   8 ++-
 arch/x86/kvm/cpuid.h             |   2 +-
 arch/x86/kvm/pmu_amd.c           | 133 ++++++++++++++++++++++++++++++++++-----
 arch/x86/kvm/x86.c               |   1 +
 5 files changed, 142 insertions(+), 16 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/4 x86/kvm/cpuid: Fix CPUID function for word 6 (80000001_ECX)

The function for CPUID 80000001 ECX is set to 0xc0000001. Set it to 0x80000001.

---
 arch/x86/kvm/cpuid.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/cpuid.h +1 -1
index 0bc5c13..b21b1d2 100644
@@ -43,7 +43,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
 	[CPUID_8086_0001_EDX] = {0x80860001, 0, CPUID_EDX},
 	[CPUID_1_ECX]         = {         1, 0, CPUID_ECX},
 	[CPUID_C000_0001_EDX] = {0xc0000001, 0, CPUID_EDX},
-	[CPUID_8000_0001_ECX] = {0xc0000001, 0, CPUID_ECX},
+	[CPUID_8000_0001_ECX] = {0x80000001, 0, CPUID_ECX},
 	[CPUID_7_0_EBX]       = {         7, 0, CPUID_EBX},
 	[CPUID_D_1_EAX]       = {       0xd, 1, CPUID_EAX},
 	[CPUID_F_0_EDX]       = {       0xf, 0, CPUID_EDX},
Signature · 1 line
2.7.4
Reviewed-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] x86/kvm/cpuid: Fix CPUID function for word 6 (80000001_ECX)

Quoted 24 lines · Janakarajan Natarajan

The function for CPUID 80000001 ECX is set to 0xc0000001. Set it to 0x80000001.

---
 arch/x86/kvm/cpuid.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/cpuid.h +1 -2
index 0bc5c13..b21b1d2 100644
@@ -43,7 +43,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
        [CPUID_8086_0001_EDX] = {0x80860001, 0, CPUID_EDX},
        [CPUID_1_ECX]         = {         1, 0, CPUID_ECX},
        [CPUID_C000_0001_EDX] = {0xc0000001, 0, CPUID_EDX},
-       [CPUID_8000_0001_ECX] = {0xc0000001, 0, CPUID_ECX},
+       [CPUID_8000_0001_ECX] = {0x80000001, 0, CPUID_ECX},
        [CPUID_7_0_EBX]       = {         7, 0, CPUID_EBX},
        [CPUID_D_1_EAX]       = {       0xd, 1, CPUID_EAX},
        [CPUID_F_0_EDX]       = {       0xf, 0, CPUID_EDX},
--

2.7.4

Reviewed-by on v2 1/4
#3 lore raw

Re: [PATCH v2 1/4] x86/kvm/cpuid: Fix CPUID function for word 6 (80000001_ECX)

In reply to Janakarajan Natarajan
Quoted 13 lines · Janakarajan Natarajan

The function for CPUID 80000001 ECX is set to 0xc0000001. Set it to 0x80000001.

---
  arch/x86/kvm/cpuid.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/cpuid.h +0 -0
index 0bc5c13..b21b1d2 100644

arch/x86/kvm/cpuid.h static const struct cpuid_reg reverse_cpuid[] = { · line 49

Quoted 8 lines
arch/x86/kvm/cpuid.h +1 -1
@@ -43,7 +43,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
  	[CPUID_8086_0001_EDX] = {0x80860001, 0, CPUID_EDX},
  	[CPUID_1_ECX]         = {         1, 0, CPUID_ECX},
  	[CPUID_C000_0001_EDX] = {0xc0000001, 0, CPUID_EDX},
-	[CPUID_8000_0001_ECX] = {0xc0000001, 0, CPUID_ECX},
+	[CPUID_8000_0001_ECX] = {0x80000001, 0, CPUID_ECX},
  	[CPUID_7_0_EBX]       = {         7, 0, CPUID_EBX},
  	[CPUID_D_1_EAX]       = {       0xd, 1, CPUID_EAX},
  	[CPUID_F_0_EDX]       = {       0xf, 0, CPUID_EDX},
#4 lore raw

Re: [PATCH v2 1/4] x86/kvm/cpuid: Fix CPUID function for word 6 (80000001_ECX)

In reply to Janakarajan Natarajan

I forgot to add Boris's Reviewed-by Tag. If the patchset is acceptable, please let me know if I should send another version with the Tag or if the Tag can be added when it is merged.

Quoted 21 lines · Janakarajan Natarajan

The function for CPUID 80000001 ECX is set to 0xc0000001. Set it to 0x80000001.

---
  arch/x86/kvm/cpuid.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/cpuid.h +1 -1
index 0bc5c13..b21b1d2 100644
@@ -43,7 +43,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
  	[CPUID_8086_0001_EDX] = {0x80860001, 0, CPUID_EDX},
  	[CPUID_1_ECX]         = {         1, 0, CPUID_ECX},
  	[CPUID_C000_0001_EDX] = {0xc0000001, 0, CPUID_EDX},
-	[CPUID_8000_0001_ECX] = {0xc0000001, 0, CPUID_ECX},
+	[CPUID_8000_0001_ECX] = {0x80000001, 0, CPUID_ECX},
  	[CPUID_7_0_EBX]       = {         7, 0, CPUID_EBX},
  	[CPUID_D_1_EAX]       = {       0xd, 1, CPUID_EAX},
  	[CPUID_F_0_EDX]       = {       0xf, 0, CPUID_EDX},
#5 lore raw

Re: [PATCH v2 1/4] x86/kvm/cpuid: Fix CPUID function for word 6 (80000001_ECX)

2017-11-06 14:38-0600, Janakarajan Natarajan:

Quoted 3 lines

I forgot to add Boris's Reviewed-by Tag. If the patchset is acceptable, please let me know if I should send another version with the Tag or if the Tag can be added when it is merged.

No problem, I have added that while applying, thanks for mentioning it. I have applied only the first patch for now as there were comment for the latter patches.

#6 lore raw

PATCH v2 2/4 Add AMD Core Perf Extension MSRs

In reply to Janakarajan Natarajan

Add the EventSelect and Counter MSRs for AMD Core Perf Extension.

---
 arch/x86/include/asm/msr-index.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/include/asm/msr-index.h +14 -0
index 17f5c12..1dcbd29 100644
@@ -337,7 +337,21 @@
 
 /* Fam 15h MSRs */
 #define MSR_F15H_PERF_CTL		0xc0010200
+#define MSR_F15H_PERF_CTL0		MSR_F15H_PERF_CTL
+#define MSR_F15H_PERF_CTL1		(MSR_F15H_PERF_CTL + 2)
+#define MSR_F15H_PERF_CTL2		(MSR_F15H_PERF_CTL + 4)
+#define MSR_F15H_PERF_CTL3		(MSR_F15H_PERF_CTL + 6)
+#define MSR_F15H_PERF_CTL4		(MSR_F15H_PERF_CTL + 8)
+#define MSR_F15H_PERF_CTL5		(MSR_F15H_PERF_CTL + 10)
+
 #define MSR_F15H_PERF_CTR		0xc0010201
+#define MSR_F15H_PERF_CTR0		MSR_F15H_PERF_CTR
+#define MSR_F15H_PERF_CTR1		(MSR_F15H_PERF_CTR + 2)
+#define MSR_F15H_PERF_CTR2		(MSR_F15H_PERF_CTR + 4)
+#define MSR_F15H_PERF_CTR3		(MSR_F15H_PERF_CTR + 6)
+#define MSR_F15H_PERF_CTR4		(MSR_F15H_PERF_CTR + 8)
+#define MSR_F15H_PERF_CTR5		(MSR_F15H_PERF_CTR + 10)
+
 #define MSR_F15H_NB_PERF_CTL		0xc0010240
 #define MSR_F15H_NB_PERF_CTR		0xc0010241
 #define MSR_F15H_PTSC			0xc0010280
Signature · 1 line
2.7.4
Reviewed-by on v2 2/4
#7 lore raw

Re: [PATCH v2 2/4] Add AMD Core Perf Extension MSRs

Quoted 24 lines · Janakarajan Natarajan

Add the EventSelect and Counter MSRs for AMD Core Perf Extension.

---
 arch/x86/include/asm/msr-index.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/include/asm/msr-index.h +7 -0
index 17f5c12..1dcbd29 100644
@@ -337,7 +337,21 @@
 
 /* Fam 15h MSRs */
 #define MSR_F15H_PERF_CTL		0xc0010200
+#define MSR_F15H_PERF_CTL0		MSR_F15H_PERF_CTL
+#define MSR_F15H_PERF_CTL1		(MSR_F15H_PERF_CTL + 2)
+#define MSR_F15H_PERF_CTL2		(MSR_F15H_PERF_CTL + 4)
+#define MSR_F15H_PERF_CTL3		(MSR_F15H_PERF_CTL + 6)
+#define MSR_F15H_PERF_CTL4		(MSR_F15H_PERF_CTL + 8)
+#define MSR_F15H_PERF_CTL5		(MSR_F15H_PERF_CTL + 10)
+
 #define MSR_F15H_PERF_CTR		0xc0010201

arch/x86/include/asm/msr-index.h line 357

Quoted 10 lines
arch/x86/include/asm/msr-index.h +7 -0
+#define MSR_F15H_PERF_CTR0		MSR_F15H_PERF_CTR
+#define MSR_F15H_PERF_CTR1		(MSR_F15H_PERF_CTR + 2)
+#define MSR_F15H_PERF_CTR2		(MSR_F15H_PERF_CTR + 4)
+#define MSR_F15H_PERF_CTR3		(MSR_F15H_PERF_CTR + 6)
+#define MSR_F15H_PERF_CTR4		(MSR_F15H_PERF_CTR + 8)
+#define MSR_F15H_PERF_CTR5		(MSR_F15H_PERF_CTR + 10)
+
 #define MSR_F15H_NB_PERF_CTL		0xc0010240
 #define MSR_F15H_NB_PERF_CTR		0xc0010241
 #define MSR_F15H_PTSC			0xc0010280
Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#8 lore raw

PATCH v2 3/4 Add support for AMD Core Perf Extension in guest

In reply to Janakarajan Natarajan

This patch adds support for AMD Core Performance counters in the guest. The base event select and counter MSRs are changed. In addition, with the core extension, there are 2 extra counters available for performance measurements for a total of 6.

With the new MSRs, the logic to map them to the gp_counters[] is changed. New functions are introduced to get the right base MSRs and to check the validity of the get/set MSRs.

If the guest has vcpus of either family 16h or a generation < 15h, it falls back to using K7 MSRs and the number of counters the guest can access is set to 4.

---
 arch/x86/kvm/pmu_amd.c | 133 +++++++++++++++++++++++++++++++++++++++++++------
 arch/x86/kvm/x86.c     |   1 +
 2 files changed, 120 insertions(+), 14 deletions(-)
arch/x86/kvm/pmu_amd.c +119 -14
index cd94443..2c694446 100644
@@ -19,6 +19,11 @@
 #include "lapic.h"
 #include "pmu.h"
 
+enum pmu_type {
+	PMU_TYPE_COUNTER = 0,
+	PMU_TYPE_EVNTSEL,
+};
+
 /* duplicated from amd_perfmon_event_map, K7 and above should work. */
 static struct kvm_event_hw_type_mapping amd_event_mapping[] = {
 	[0] = { 0x76, 0x00, PERF_COUNT_HW_CPU_CYCLES },
@@ -31,6 +36,86 @@ static struct kvm_event_hw_type_mapping amd_event_mapping[] = {
 	[7] = { 0xd1, 0x00, PERF_COUNT_HW_STALLED_CYCLES_BACKEND },
 };
 
+static unsigned int get_msr_base(struct kvm_pmu *pmu, enum pmu_type type)
+{
+	struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu);
+	int family;
+	bool has_perf_ext;
+
+	family = guest_cpuid_family(vcpu);
+	has_perf_ext = guest_cpuid_has(vcpu, X86_FEATURE_PERFCTR_CORE);
+
+	switch (family) {
+	case 0x15:
+	case 0x17:
+		if (has_perf_ext) {
+			if (type == PMU_TYPE_COUNTER)
+				return MSR_F15H_PERF_CTR;
+			else
+				return MSR_F15H_PERF_CTL;
+			break;
+		}
+		/*
+		 * Fall-through because the K7 MSRs are
+		 * backwards compatible
+		 */
+	default:
+		if (type == PMU_TYPE_COUNTER)
+			return MSR_K7_PERFCTR0;
+		else
+			return MSR_K7_EVNTSEL0;
+	}
+}
+
+static inline struct kvm_pmc *get_gp_pmc_amd(struct kvm_pmu *pmu, u32 msr,
+					     enum pmu_type type)
+{
+	unsigned int base = get_msr_base(pmu, type);
+
+	if (base == MSR_F15H_PERF_CTL) {
+		switch (msr) {
+		case MSR_F15H_PERF_CTL0:
+		case MSR_F15H_PERF_CTL1:
+		case MSR_F15H_PERF_CTL2:
+		case MSR_F15H_PERF_CTL3:
+		case MSR_F15H_PERF_CTL4:
+		case MSR_F15H_PERF_CTL5:
+			/*
+			 * AMD Perf Extension MSRs are not continuous.
+			 *
+			 * E.g. MSR_F15H_PERF_CTR0 -> 0xc0010201
+			 *	MSR_F15H_PERF_CTR1 -> 0xc0010203
+			 *
+			 * These are mapped to work with gp_counters[].
+			 * The index into the array is calculated by
+			 * dividing the difference between the requested
+			 * msr and the msr base by 2.
+			 *
+			 * E.g. MSR_F15H_PERF_CTR1 uses
+			 *	->gp_counters[(0xc0010203-0xc0010201)/2]
+			 *	->gp_counters[1]
+			 */
+			return &pmu->gp_counters[(msr - base) >> 1];
+		default:
+			return NULL;
+		}
+	} else if (base == MSR_F15H_PERF_CTR) {
+		switch (msr) {
+		case MSR_F15H_PERF_CTR0:
+		case MSR_F15H_PERF_CTR1:
+		case MSR_F15H_PERF_CTR2:
+		case MSR_F15H_PERF_CTR3:
+		case MSR_F15H_PERF_CTR4:
+		case MSR_F15H_PERF_CTR5:
+			return &pmu->gp_counters[(msr - base) >> 1];
+		default:
+			return NULL;
+		}
+	} else {
+		return get_gp_pmc(pmu, msr, base);
+	}
+}
+
 static unsigned amd_find_arch_event(struct kvm_pmu *pmu,
 				    u8 event_select,
 				    u8 unit_mask)
@@ -64,7 +149,20 @@ static bool amd_pmc_is_enabled(struct kvm_pmc *pmc)
 
 static struct kvm_pmc *amd_pmc_idx_to_pmc(struct kvm_pmu *pmu, int pmc_idx)
 {
-	return get_gp_pmc(pmu, MSR_K7_EVNTSEL0 + pmc_idx, MSR_K7_EVNTSEL0);
+	unsigned int base = get_msr_base(pmu, PMU_TYPE_COUNTER);
+	unsigned int family;
+
+	family = guest_cpuid_family(pmu_to_vcpu(pmu));
+
+	if (family == 0x15 || family == 0x17) {
+		/*
+		 * The idx is contiguous. The MSRs are not. The counter MSRs
+		 * are interleaved with the event select MSRs.
+		 */
+		pmc_idx *= 2;
+	}
+
+	return get_gp_pmc_amd(pmu, base + pmc_idx, PMU_TYPE_COUNTER);
 }
 
 /* returns 0 if idx's corresponding MSR exists; otherwise returns 1. */
@@ -96,8 +194,8 @@ static bool amd_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 	int ret = false;
 
-	ret = get_gp_pmc(pmu, msr, MSR_K7_PERFCTR0) ||
-		get_gp_pmc(pmu, msr, MSR_K7_EVNTSEL0);
+	ret = get_gp_pmc_amd(pmu, msr, PMU_TYPE_COUNTER) ||
+		get_gp_pmc_amd(pmu, msr, PMU_TYPE_EVNTSEL);
 
 	return ret;
 }
@@ -107,14 +205,14 @@ static int amd_pmu_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *data)
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 	struct kvm_pmc *pmc;
 
-	/* MSR_K7_PERFCTRn */
-	pmc = get_gp_pmc(pmu, msr, MSR_K7_PERFCTR0);
+	/* MSR_PERFCTRn */
+	pmc = get_gp_pmc_amd(pmu, msr, PMU_TYPE_COUNTER);
 	if (pmc) {
 		*data = pmc_read_counter(pmc);
 		return 0;
 	}
-	/* MSR_K7_EVNTSELn */
-	pmc = get_gp_pmc(pmu, msr, MSR_K7_EVNTSEL0);
+	/* MSR_EVNTSELn */
+	pmc = get_gp_pmc_amd(pmu, msr, PMU_TYPE_EVNTSEL);
 	if (pmc) {
 		*data = pmc->eventsel;
 		return 0;
@@ -130,14 +228,14 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	u32 msr = msr_info->index;
 	u64 data = msr_info->data;
 
-	/* MSR_K7_PERFCTRn */
-	pmc = get_gp_pmc(pmu, msr, MSR_K7_PERFCTR0);
+	/* MSR_PERFCTRn */
+	pmc = get_gp_pmc_amd(pmu, msr, PMU_TYPE_COUNTER);
 	if (pmc) {
 		pmc->counter += data - pmc_read_counter(pmc);
 		return 0;
 	}
-	/* MSR_K7_EVNTSELn */
-	pmc = get_gp_pmc(pmu, msr, MSR_K7_EVNTSEL0);
+	/* MSR_EVNTSELn */
+	pmc = get_gp_pmc_amd(pmu, msr, PMU_TYPE_EVNTSEL);
 	if (pmc) {
 		if (data == pmc->eventsel)
 			return 0;
@@ -153,8 +251,15 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 static void amd_pmu_refresh(struct kvm_vcpu *vcpu)
 {
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
+	int family, nr_counters;
+
+	family = guest_cpuid_family(vcpu);
+	if (family == 0x15 || family == 0x17)
+		nr_counters = AMD64_NUM_COUNTERS_CORE;
+	else
+		nr_counters = AMD64_NUM_COUNTERS;
 
-	pmu->nr_arch_gp_counters = AMD64_NUM_COUNTERS;
+	pmu->nr_arch_gp_counters = nr_counters;
 	pmu->counter_bitmask[KVM_PMC_GP] = ((u64)1 << 48) - 1;
 	pmu->reserved_bits = 0xffffffff00200000ull;
 	/* not applicable to AMD; but clean them to prevent any fall out */
@@ -169,7 +274,7 @@ static void amd_pmu_init(struct kvm_vcpu *vcpu)
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 	int i;
 
-	for (i = 0; i < AMD64_NUM_COUNTERS ; i++) {
+	for (i = 0; i < AMD64_NUM_COUNTERS_CORE ; i++) {
 		pmu->gp_counters[i].type = KVM_PMC_GP;
 		pmu->gp_counters[i].vcpu = vcpu;
 		pmu->gp_counters[i].idx = i;
@@ -181,7 +286,7 @@ static void amd_pmu_reset(struct kvm_vcpu *vcpu)
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 	int i;
 
-	for (i = 0; i < AMD64_NUM_COUNTERS; i++) {
+	for (i = 0; i < AMD64_NUM_COUNTERS_CORE; i++) {
 		struct kvm_pmc *pmc = &pmu->gp_counters[i];
 
 		pmc_stop_counter(pmc);
arch/x86/kvm/x86.c +1 -0
index 03869eb..5a6ad6f 100644
@@ -2433,6 +2433,7 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_AMD64_DC_CFG:
 		msr_info->data = 0;
 		break;
+	case MSR_F15H_PERF_CTL0 ... MSR_F15H_PERF_CTR5:
 	case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3:
 	case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3:
 	case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1:
Signature · 1 line
2.7.4
#9 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 2 lines

Subject: Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Btw, your subjects need a prefix:

"x86/kvm: Add guest support for the AMD core performance counters"

for example.

Quoted 1 line · Janakarajan Natarajan

This patch adds support for AMD Core Performance counters in the guest.

Never say "This patch" in the commit message of a patch. It is tautologically useless.

Quoted 9 lines

The base event select and counter MSRs are changed. In addition, with the core extension, there are 2 extra counters available for performance measurements for a total of 6.

With the new MSRs, the logic to map them to the gp_counters[] is changed. New functions are introduced to get the right base MSRs and to check the validity of the get/set MSRs.

If the guest has vcpus of either family 16h or a generation < 15h, it

You're only talking about families here, the "generation" thing is confusing.

Quoted 8 lines

falls back to using K7 MSRs and the number of counters the guest can access is set to 4.

---
 arch/x86/kvm/pmu_amd.c | 133 +++++++++++++++++++++++++++++++++++++++++++------
 arch/x86/kvm/x86.c     |   1 +
 2 files changed, 120 insertions(+), 14 deletions(-)

...

Quoted 19 lines
arch/x86/kvm/pmu_amd.c +19 -0
+static inline struct kvm_pmc *get_gp_pmc_amd(struct kvm_pmu *pmu, u32 msr,
+					     enum pmu_type type)
+{
+	unsigned int base = get_msr_base(pmu, type);
+
+	if (base == MSR_F15H_PERF_CTL) {
+		switch (msr) {
+		case MSR_F15H_PERF_CTL0:
+		case MSR_F15H_PERF_CTL1:
+		case MSR_F15H_PERF_CTL2:
+		case MSR_F15H_PERF_CTL3:
+		case MSR_F15H_PERF_CTL4:
+		case MSR_F15H_PERF_CTL5:
+			/*
+			 * AMD Perf Extension MSRs are not continuous.
+			 *
+			 * E.g. MSR_F15H_PERF_CTR0 -> 0xc0010201
+			 *	MSR_F15H_PERF_CTR1 -> 0xc0010203
+			 *

arch/x86/kvm/pmu_amd.c static struct kvm_event_hw_type_mapping amd_event_mapping[] = { · line 98

Quoted 10 lines
arch/x86/kvm/pmu_amd.c +10 -0
+			 * These are mapped to work with gp_counters[].
+			 * The index into the array is calculated by
+			 * dividing the difference between the requested
+			 * msr and the msr base by 2.
+			 *
+			 * E.g. MSR_F15H_PERF_CTR1 uses
+			 *	->gp_counters[(0xc0010203-0xc0010201)/2]
+			 *	->gp_counters[1]
+			 */
+			return &pmu->gp_counters[(msr - base) >> 1];

Ok, it took me a bit of staring to understand what you're doing here. And frankly, this scheme is silly and fragile. You're relying on the fact that you can do math with the MSR numbers to get you the GP counter number. The moment that changes in future families, you are going to have to devise a new scheme for the new family.

And instead of doing that, you're much better off producing a simple MSR -> counter mapping for each family which is a simple switch-case. No need to do get_msr_base() and whatnot - you simply feed in the MSR number and the function spits out a gp_counters index. You only need to check the family of the vcpu.

Then lookers will be able to understand the code at a quick glance too.

...

Quoted 18 lines
arch/x86/kvm/pmu_amd.c +8 -1
@@ -153,8 +251,15 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 static void amd_pmu_refresh(struct kvm_vcpu *vcpu)
 {
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
+	int family, nr_counters;
+
+	family = guest_cpuid_family(vcpu);
+	if (family == 0x15 || family == 0x17)
+		nr_counters = AMD64_NUM_COUNTERS_CORE;
+	else
+		nr_counters = AMD64_NUM_COUNTERS;
 
-	pmu->nr_arch_gp_counters = AMD64_NUM_COUNTERS;
+	pmu->nr_arch_gp_counters = nr_counters;
 	pmu->counter_bitmask[KVM_PMC_GP] = ((u64)1 << 48) - 1;
 	pmu->reserved_bits = 0xffffffff00200000ull;
 	/* not applicable to AMD; but clean them to prevent any fall out */

arch/x86/kvm/pmu_amd.c static void amd_pmu_init(struct kvm_vcpu *vcpu) · line 277

Quoted 5 lines
arch/x86/kvm/pmu_amd.c +1 -1
@@ -169,7 +274,7 @@ static void amd_pmu_init(struct kvm_vcpu *vcpu)
 	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
 	int i;
 
-	for (i = 0; i < AMD64_NUM_COUNTERS ; i++) {
+	for (i = 0; i < AMD64_NUM_COUNTERS_CORE ; i++) {

This all works because INTEL_PMC_MAX_GENERIC is bigger than the AMD num counters but you need to check all that.

Also, the finding out of the nr_counters you do in amd_pmu_refresh() should happen here, in the init function so that you have pmu->nr_arch_gp_counters properly set and then when you iterate over counters in the remaining functions, you do:

	for (i = 0; i < pmu->nr_arch_gp_counters ; i++) {

instead of using those defines which are not always correct, depending on the family.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#10 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 6 lines · Borislav Petkov
Quoted 1 line

Subject: Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Btw, your subjects need a prefix:

"x86/kvm: Add guest support for the AMD core performance counters"

for example.

Okay.

Quoted 4 lines · Janakarajan Natarajan
Quoted 1 line

This patch adds support for AMD Core Performance counters in the guest.

Never say "This patch" in the commit message of a patch. It is tautologically useless.

Okay.

Quoted 11 lines
Quoted 9 lines

The base event select and counter MSRs are changed. In addition, with the core extension, there are 2 extra counters available for performance measurements for a total of 6.

With the new MSRs, the logic to map them to the gp_counters[] is changed. New functions are introduced to get the right base MSRs and to check the validity of the get/set MSRs.

If the guest has vcpus of either family 16h or a generation < 15h, it

You're only talking about families here, the "generation" thing is confusing.

I'll change that.

Quoted 51 lines
Quoted 8 lines

falls back to using K7 MSRs and the number of counters the guest can access is set to 4.

---
  arch/x86/kvm/pmu_amd.c | 133 +++++++++++++++++++++++++++++++++++++++++++------
  arch/x86/kvm/x86.c     |   1 +
  2 files changed, 120 insertions(+), 14 deletions(-)

...

Quoted 19 lines
arch/x86/kvm/pmu_amd.c +19 -0
+static inline struct kvm_pmc *get_gp_pmc_amd(struct kvm_pmu *pmu, u32 msr,
+					     enum pmu_type type)
+{
+	unsigned int base = get_msr_base(pmu, type);
+
+	if (base == MSR_F15H_PERF_CTL) {
+		switch (msr) {
+		case MSR_F15H_PERF_CTL0:
+		case MSR_F15H_PERF_CTL1:
+		case MSR_F15H_PERF_CTL2:
+		case MSR_F15H_PERF_CTL3:
+		case MSR_F15H_PERF_CTL4:
+		case MSR_F15H_PERF_CTL5:
+			/*
+			 * AMD Perf Extension MSRs are not continuous.
+			 *
+			 * E.g. MSR_F15H_PERF_CTR0 -> 0xc0010201
+			 *	MSR_F15H_PERF_CTR1 -> 0xc0010203
+			 *

arch/x86/kvm/pmu_amd.c static struct kvm_event_hw_type_mapping amd_event_mapping[] = { · line 98

Quoted 10 lines
arch/x86/kvm/pmu_amd.c +10 -0
+			 * These are mapped to work with gp_counters[].
+			 * The index into the array is calculated by
+			 * dividing the difference between the requested
+			 * msr and the msr base by 2.
+			 *
+			 * E.g. MSR_F15H_PERF_CTR1 uses
+			 *	->gp_counters[(0xc0010203-0xc0010201)/2]
+			 *	->gp_counters[1]
+			 */
+			return &pmu->gp_counters[(msr - base) >> 1];

Ok, it took me a bit of staring to understand what you're doing here. And frankly, this scheme is silly and fragile. You're relying on the fact that you can do math with the MSR numbers to get you the GP counter number. The moment that changes in future families, you are going to have to devise a new scheme for the new family.

And instead of doing that, you're much better off producing a simple MSR -> counter mapping for each family which is a simple switch-case. No need to do get_msr_base() and whatnot - you simply feed in the MSR number and the function spits out a gp_counters index. You only need to check the family of the vcpu.

Quoted 3 lines

Then lookers will be able to understand the code at a quick glance too.

...

I'll put out a v3 with those changes.

Quoted 31 lines
Quoted 18 lines
arch/x86/kvm/pmu_amd.c +8 -1
@@ -153,8 +251,15 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
  static void amd_pmu_refresh(struct kvm_vcpu *vcpu)
  {
  	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
+	int family, nr_counters;
+
+	family = guest_cpuid_family(vcpu);
+	if (family == 0x15 || family == 0x17)
+		nr_counters = AMD64_NUM_COUNTERS_CORE;
+	else
+		nr_counters = AMD64_NUM_COUNTERS;
 
-	pmu->nr_arch_gp_counters = AMD64_NUM_COUNTERS;
+	pmu->nr_arch_gp_counters = nr_counters;
  	pmu->counter_bitmask[KVM_PMC_GP] = ((u64)1 << 48) - 1;
  	pmu->reserved_bits = 0xffffffff00200000ull;
  	/* not applicable to AMD; but clean them to prevent any fall out */

arch/x86/kvm/pmu_amd.c static void amd_pmu_init(struct kvm_vcpu *vcpu) · line 277

Quoted 5 lines
arch/x86/kvm/pmu_amd.c +1 -1
@@ -169,7 +274,7 @@ static void amd_pmu_init(struct kvm_vcpu *vcpu)
  	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
  	int i;
 
-	for (i = 0; i < AMD64_NUM_COUNTERS ; i++) {
+	for (i = 0; i < AMD64_NUM_COUNTERS_CORE ; i++) {

This all works because INTEL_PMC_MAX_GENERIC is bigger than the AMD num counters but you need to check all that.

Also, the finding out of the nr_counters you do in amd_pmu_refresh() should happen here, in the init function so that you have pmu->nr_arch_gp_counters properly set and then when you iterate over counters in the remaining functions, you do:

Quoted 4 lines
	for (i = 0; i < pmu->nr_arch_gp_counters ; i++) {

instead of using those defines which are not always correct, depending on the family.

So, when the amd_pmu_init is called, a query to guest_cpuid_family() gives a value of -1. If this is the case where qemu sets the family details later on with a kvm ioctl, it would make sense to just initialize the maximum number of gp_counters that will be used and set the nr_arch_gp_counters based on the family during the amd_pmu_refresh().

Thanks,
Janakarajan

Quoted 1 line
#11 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 2 lines · Janakarajan

So, when the amd_pmu_init is called, a query to guest_cpuid_family() gives a value of -1.

And that is because...? And it can be fixed to give the proper guest family I presume ...?

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#12 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Janakarajan

So, when the amd_pmu_init is called, a query to guest_cpuid_family() gives a value of -1.

And that is because...? And it can be fixed to give the proper guest family I presume ...?

Qemu initially creates a vcpu with KVM_CREATE_VCPU which in kvm initializes the pmu. Later on the cpuid is set using KVM_SET_CPUID2 which sets all the cpuid entries in kvm and the pmu refresh is called.

Quoted 1 line
#13 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 11 lines · Janakarajan
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Janakarajan

So, when the amd_pmu_init is called, a query to guest_cpuid_family() gives a value of -1.

And that is because...? And it can be fixed to give the proper guest family I presume ...?

Qemu initially creates a vcpu with KVM_CREATE_VCPU which in kvm initializes the pmu. Later on
the cpuid is set using KVM_SET_CPUID2 which sets all the cpuid entries in kvm and the pmu refresh is called.

That doesn't answer my questions above.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#14 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 5 lines · Borislav Petkov
Quoted 4 lines · Janakarajan
Quoted 3 lines · Borislav Petkov
Quoted 2 lines · Janakarajan

So, when the amd_pmu_init is called, a query to guest_cpuid_family() gives a value of -1.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

And that is because...? And it can be fixed to give the proper guest family I presume ...?

Qemu initially creates a vcpu with KVM_CREATE_VCPU which in kvm initializes the pmu. Later on
the cpuid is set using KVM_SET_CPUID2 which sets all the cpuid entries in kvm and the pmu refresh is called.

That doesn't answer my questions above.

Ah my apologies. So when the pmu is initialized the cpuid entries aren't available then. We'll have to wait until the pmu refresh is called to get the family value. So there is no place to fix it. The setting of the nr_counters based on the family will have to be done in the amd_pmu_refresh() and not in the amd_pmu_init().

Quoted 1 line
#15 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 2 lines · Janakarajan

Ah my apologies. So when the pmu is initialized the cpuid entries aren't available then.

So let's see:

... kvm_arch_vcpu_create() ->
svm_create_vcpu() ->
kvm_vcpu_init() ->
kvm_arch_vcpu_init() ->

<--- HERE

kvm_pmu_init()

But at HERE in kvm_arch_vcpu_init() right before kvm_pmu_init() we do already query cpuid:

	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);

so it's not like we don't know about cpuid leafs at that point. Which would mean that the code can be made to set the CPU family earlier, before kvm_pmu_init() runs so that you have the proper CPU family and thus have this thing properly designed.

Maybe Paolo and Radim have a better suggestion here...

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 
#16 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 10 lines · Borislav Petkov
Quoted 2 lines · Janakarajan

Ah my apologies. So when the pmu is initialized the cpuid entries aren't available then.

So let's see:

... kvm_arch_vcpu_create() ->
svm_create_vcpu() ->
kvm_vcpu_init() ->
kvm_arch_vcpu_init() ->

Quoted 8 lines

<--- HERE

kvm_pmu_init()

But at HERE in kvm_arch_vcpu_init() right before kvm_pmu_init() we do already query cpuid:

	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);

cpuid_query_maxphyaddr first checks for for the 0x80000008 cpuid entry, and if it can't find it returns a default value of 36.

When invoked in the kvm_arch_vcpu_init(), the default is being returned. The vcpu->arch.maxphyaddr is later updated in the kvm_update_cpuid(), which in my case was setting the maxphyaddr to 40. I believe the cpuid entry is not available when invoked in the init call and a default value is being used as a placeholder until the entries are updated by qemu.

Quoted 7 lines

so it's not like we don't know about cpuid leafs at that point. Which would mean that the code can be made to set the CPU family earlier, before kvm_pmu_init() runs so that you have the proper CPU family and thus have this thing properly designed.

Maybe Paolo and Radim have a better suggestion here...

#17 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

In reply to Borislav Petkov
Quoted 17 lines · Borislav Petkov
Quoted 2 lines · Janakarajan

Ah my apologies. So when the pmu is initialized the cpuid entries aren't available then.

So let's see:

... kvm_arch_vcpu_create() ->
svm_create_vcpu() ->
kvm_vcpu_init() ->
kvm_arch_vcpu_init() ->

<--- HERE

kvm_pmu_init()

But at HERE in kvm_arch_vcpu_init() right before kvm_pmu_init() we do already query cpuid:

Quoted 8 lines
	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);

so it's not like we don't know about cpuid leafs at that point. Which would mean that the code can be made to set the CPU family earlier, before kvm_pmu_init() runs so that you have the proper CPU family and thus have this thing properly designed.

Maybe Paolo and Radim have a better suggestion here...

Paolo, Radim any suggestions about this. I feel that the number of counters initialized can be 6 and the subsequent code (kvm_pmu_refresh()) takes care of the number of counters to be used (4 or 6) based on the vcpu family.

Quoted 1 line
#18 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

2017-12-01 13:30-0600, Natarajan, Janakarajan:

Quoted 25 lines · Borislav Petkov
Quoted 24 lines · Janakarajan
Quoted 2 lines

Ah my apologies. So when the pmu is initialized the cpuid entries aren't available then.

So let's see:

... kvm_arch_vcpu_create() ->
svm_create_vcpu() ->
kvm_vcpu_init() ->
kvm_arch_vcpu_init() ->

<--- HERE

kvm_pmu_init()

But at HERE in kvm_arch_vcpu_init() right before kvm_pmu_init() we do already query cpuid:

	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);

so it's not like we don't know about cpuid leafs at that point. Which would mean that the code can be made to set the CPU family earlier, before kvm_pmu_init() runs so that you have the proper CPU family and thus have this thing properly designed.

Quoted 4 lines
Quoted 1 line

Maybe Paolo and Radim have a better suggestion here...

Paolo, Radim any suggestions about this. I feel that the number of counters initialized can be 6

This is the best solution with the current framework.

Quoted 3 lines

and the subsequent code (kvm_pmu_refresh()) takes care of the number of counters to be used (4 or 6)
based on the vcpu family.

Can't we look only at X86_FEATURE_PERFCTR_CORE and completely ignore the AMD family?
Using the family would bring problems with compatiblity.

Thanks.

#19 lore raw

Re: [PATCH v2 3/4] Add support for AMD Core Perf Extension in guest

Quoted 26 lines · Radim Krcmar

2017-12-01 13:30-0600, Natarajan, Janakarajan:

Quoted 25 lines · Borislav Petkov
Quoted 24 lines · Janakarajan
Quoted 2 lines

Ah my apologies. So when the pmu is initialized the cpuid entries aren't available then.

So let's see:

... kvm_arch_vcpu_create() ->
svm_create_vcpu() ->
kvm_vcpu_init() ->
kvm_arch_vcpu_init() ->

<--- HERE

kvm_pmu_init()

But at HERE in kvm_arch_vcpu_init() right before kvm_pmu_init() we do already query cpuid:

	vcpu->arch.maxphyaddr = cpuid_query_maxphyaddr(vcpu);

so it's not like we don't know about cpuid leafs at that point. Which would mean that the code can be made to set the CPU family earlier, before kvm_pmu_init() runs so that you have the proper CPU family and thus have this thing properly designed.

Quoted 4 lines
Quoted 3 lines
Quoted 1 line

Maybe Paolo and Radim have a better suggestion here...

Paolo, Radim any suggestions about this. I feel that the number of counters initialized can be 6

This is the best solution with the current framework.

Okay.

Quoted 7 lines
Quoted 4 lines

and the subsequent code (kvm_pmu_refresh()) takes care of the number of counters to be used (4 or 6)
based on the vcpu family.

Can't we look only at X86_FEATURE_PERFCTR_CORE and completely ignore the AMD family?
Using the family would bring problems with compatiblity.

Yeah. We can just use the X86_FEATURE_PERFCTR_CORE flag as a check. I'll send a v3 with the changes.

Thanks.

Quoted 2 lines

Thanks.

#20 lore raw

PATCH v2 4/4 Expose AMD Core Perf Extension flag to guests

In reply to Janakarajan Natarajan

Expose the AMD Core Perf Extension flag to the guests.

---
 arch/x86/kvm/cpuid.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/kvm/cpuid.c +7 -1
index 0099e10..8c95a7c 100644
@@ -55,6 +55,11 @@ bool kvm_mpx_supported(void)
 }
 EXPORT_SYMBOL_GPL(kvm_mpx_supported);
 
+bool perf_ext_supported(void)
+{
+	return boot_cpu_has(X86_FEATURE_PERFCTR_CORE);
+}
+
 u64 kvm_supported_xcr0(void)
 {
 	u64 xcr0 = KVM_SUPPORTED_XCR0 & host_xcr0;
@@ -327,6 +332,7 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 	unsigned f_invpcid = kvm_x86_ops->invpcid_supported() ? F(INVPCID) : 0;
 	unsigned f_mpx = kvm_mpx_supported() ? F(MPX) : 0;
 	unsigned f_xsaves = kvm_x86_ops->xsaves_supported() ? F(XSAVES) : 0;
+	unsigned f_perfext = perf_ext_supported() ? F(PERFCTR_CORE) : 0;
 
 	/* cpuid 1.edx */
 	const u32 kvm_cpuid_1_edx_x86_features =
@@ -365,7 +371,7 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 		F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ |
 		F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) |
 		F(3DNOWPREFETCH) | F(OSVW) | 0 /* IBS */ | F(XOP) |
-		0 /* SKINIT, WDT, LWP */ | F(FMA4) | F(TBM);
+		0 /* SKINIT, WDT, LWP */ | F(FMA4) | F(TBM) | f_perfext;
 
 	/* cpuid 0xC0000001.edx */
 	const u32 kvm_cpuid_C000_0001_edx_x86_features =
Signature · 1 line
2.7.4