Skip to contentSkip to conversation

KVM: arm64: PMU: Correct the handling of PMUSERENR_EL0

PATCH Applied Reiji Watanabe 2 participants 4 messages Jun 3, 2023 – Jun 4, 2023

Subsystems (5)
Arm (32-bit) KVM arm64 KVM arm64 +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v5 cover KVM: arm64: PMU: Correct the handling of PMUSERENR_EL0

This series will fix bugs in KVM's handling of PMUSERENR_EL0.

With PMU access support from EL0 [1], the perf subsystem would set CR and ER bits of PMUSERENR_EL0 as needed to allow EL0 to have a direct access to PMU counters. However, KVM appears to assume that the register value is always zero for the host EL0, and has the following two problems in handling the register.

[A] The host EL0 might lose the direct access to PMU counters, as KVM always clears PMUSERENR_EL0 before returning to userspace.

[B] With VHE, the guest EL0 access to PMU counters might be trapped to EL1 instead of to EL2 (even when PMUSERENR_EL0 for the guest indicates that the guest EL0 has an access to the counters). This is because, with VHE, KVM sets ER, CR, SW and EN bits of PMUSERENR_EL0 to 1 on vcpu_load() to ensure to trap PMU access from the guset EL0 to EL2, but those bits might be cleared by the perf subsystem after vcpu_load() (when PMU counters are programmed for the vPMU emulation).

Patch-1 will fix [A], and Patch-2 will fix [B] respectively. The series is based on 6.4-rc4.

v5:
- Move IRQ save/restore to {activate,deactivate}_traps_vhe_{load,put}().

v4: https://lore.kernel.org/all/[email protected]/
- Introduce NO_DEBUG_IRQFLAGS to exclude warn_bogus_irq_restore() from the nVHE hyp code. This is to address the issue [2] that was reported by kernel test robot <[email protected]>.

v3: https://lore.kernel.org/all/[email protected]/
- While vcpu_{put,load}() are manipulating PMUSERENR_EL0, disable IRQs to prevent a race condition between these processes and IPIs that updates PMUSERENR_EL0. [Mark]

v2: https://lore.kernel.org/all/[email protected]/
- Save the PMUSERENR_EL0 for the host in the sysreg array of kvm_host_data. [Marc]
- Don't let armv8pmu_start() overwrite PMUSERENR if the vCPU is loaded, instead have KVM update the saved shadow register value for the host. [Marc, Mark]

v1: https://lore.kernel.org/all/[email protected]/

[1] https://github.com/torvalds/linux/commit/83a7a4d643d33a8b74a42229346b7ed7139fcef9
[2] https://lore.kernel.org/all/[email protected]/

Reiji Watanabe (2):
KVM: arm64: PMU: Restore the host's PMUSERENR_EL0
KVM: arm64: PMU: Don't overwrite PMUSERENR with vcpu loaded

 arch/arm/include/asm/arm_pmuv3.h        |  5 +++++
 arch/arm64/include/asm/kvm_host.h       |  7 +++++++
 arch/arm64/kvm/hyp/include/hyp/switch.h | 15 ++++++++++++--
 arch/arm64/kvm/hyp/vhe/switch.c         | 14 +++++++++++++
 arch/arm64/kvm/pmu.c                    | 27 +++++++++++++++++++++++++
 drivers/perf/arm_pmuv3.c                | 21 ++++++++++++++++---
 6 files changed, 84 insertions(+), 5 deletions(-)

base-commit: 7877cb91f1081754a1487c144d85dc0d2e2e7fc4

Signature · 1 line
2.41.0.rc0.172.g3f132b7071-goog
#1 lore raw

PATCH v5 1/2 KVM: arm64: PMU: Restore the host's PMUSERENR_EL0

Restore the host's PMUSERENR_EL0 value instead of clearing it, before returning back to userspace, as the host's EL0 might have a direct access to PMU registers (some bits of PMUSERENR_EL0 for might not be zero for the host EL0).

Fixes 83a7a4d643d3 ("arm64: perf: Enable PMU counter userspace access for perf event")
---
 arch/arm64/kvm/hyp/include/hyp/switch.h | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +11 -2
index e78a08a72a3c..51229d847103 100644
@@ -82,7 +82,12 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 	 * EL1 instead of being trapped to EL2.
 	 */
 	if (kvm_arm_support_pmu_v3()) {
+		struct kvm_cpu_context *hctxt;
+
 		write_sysreg(0, pmselr_el0);
+
+		hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
+		ctxt_sys_reg(hctxt, PMUSERENR_EL0) = read_sysreg(pmuserenr_el0);
 		write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
 	}
 
@@ -106,8 +111,12 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
 	write_sysreg(vcpu->arch.mdcr_el2_host, mdcr_el2);
 
 	write_sysreg(0, hstr_el2);
-	if (kvm_arm_support_pmu_v3())
-		write_sysreg(0, pmuserenr_el0);
+	if (kvm_arm_support_pmu_v3()) {
+		struct kvm_cpu_context *hctxt;
+
+		hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
+		write_sysreg(ctxt_sys_reg(hctxt, PMUSERENR_EL0), pmuserenr_el0);
+	}
 
 	if (cpus_have_final_cap(ARM64_SME)) {
 		sysreg_clear_set_s(SYS_HFGRTR_EL2, 0,
Signature · 1 line
2.41.0.rc0.172.g3f132b7071-goog
#2 lore raw

PATCH v5 2/2 KVM: arm64: PMU: Don't overwrite PMUSERENR with vcpu loaded

In reply to Reiji Watanabe

Currently, with VHE, KVM sets ER, CR, SW and EN bits of PMUSERENR_EL0 to 1 on vcpu_load(), and saves and restores the register value for the host on vcpu_load() and vcpu_put(). If the value of those bits are cleared on a pCPU with a vCPU loaded (armv8pmu_start() would do that when PMU counters are programmed for the guest), PMU access from the guest EL0 might be trapped to the guest EL1 directly regardless of the current PMUSERENR_EL0 value of the vCPU.

Fix this by not letting armv8pmu_start() overwrite PMUSERENR_EL0 on the pCPU where PMUSERENR_EL0 for the guest is loaded, and instead updating the saved shadow register value for the host so that the value can be restored on vcpu_put() later. While vcpu_{put,load}() are manipulating PMUSERENR_EL0, disable IRQs to prevent a race condition between these processes and IPIs that attempt to update PMUSERENR_EL0 for the host EL0.

Fixes 83a7a4d643d3 ("arm64: perf: Enable PMU counter userspace access for perf event")
---
 arch/arm/include/asm/arm_pmuv3.h        |  5 +++++
 arch/arm64/include/asm/kvm_host.h       |  7 +++++++
 arch/arm64/kvm/hyp/include/hyp/switch.h |  2 ++
 arch/arm64/kvm/hyp/vhe/switch.c         | 14 +++++++++++++
 arch/arm64/kvm/pmu.c                    | 27 +++++++++++++++++++++++++
 drivers/perf/arm_pmuv3.c                | 21 ++++++++++++++++---
 6 files changed, 73 insertions(+), 3 deletions(-)
arch/arm/include/asm/arm_pmuv3.h +5 -0
index f4db3e75d75f..f3cd04ff022d 100644
@@ -222,6 +222,11 @@ static inline bool kvm_pmu_counter_deferred(struct perf_event_attr *attr)
 	return false;
 }
 
+static inline bool kvm_set_pmuserenr(u64 val)
+{
+	return false;
+}
+
 /* PMU Version in DFR Register */
 #define ARMV8_PMU_DFR_VER_NI        0
 #define ARMV8_PMU_DFR_VER_V3P4      0x5
arch/arm64/include/asm/kvm_host.h +7 -0
index 7e7e19ef6993..9787503ff43f 100644
@@ -699,6 +699,8 @@ struct kvm_vcpu_arch {
 #define SYSREGS_ON_CPU		__vcpu_single_flag(sflags, BIT(4))
 /* Software step state is Active-pending */
 #define DBG_SS_ACTIVE_PENDING	__vcpu_single_flag(sflags, BIT(5))
+/* PMUSERENR for the guest EL0 is on physical CPU */
+#define PMUSERENR_ON_CPU	__vcpu_single_flag(sflags, BIT(6))
 
 
 /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */
@@ -1065,9 +1067,14 @@ void kvm_arch_vcpu_put_debug_state_flags(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_KVM
 void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr);
 void kvm_clr_pmu_events(u32 clr);
+bool kvm_set_pmuserenr(u64 val);
 #else
 static inline void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr) {}
 static inline void kvm_clr_pmu_events(u32 clr) {}
+static inline bool kvm_set_pmuserenr(u64 val)
+{
+	return false;
+}
 #endif
 
 void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu);
arch/arm64/kvm/hyp/include/hyp/switch.h +2 -0
index 51229d847103..c530cf6b4bef 100644
@@ -89,6 +89,7 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 		hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
 		ctxt_sys_reg(hctxt, PMUSERENR_EL0) = read_sysreg(pmuserenr_el0);
 		write_sysreg(ARMV8_PMU_USERENR_MASK, pmuserenr_el0);
+		vcpu_set_flag(vcpu, PMUSERENR_ON_CPU);
 	}
 
 	vcpu->arch.mdcr_el2_host = read_sysreg(mdcr_el2);
@@ -116,6 +117,7 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
 
 		hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
 		write_sysreg(ctxt_sys_reg(hctxt, PMUSERENR_EL0), pmuserenr_el0);
+		vcpu_clear_flag(vcpu, PMUSERENR_ON_CPU);
 	}
 
 	if (cpus_have_final_cap(ARM64_SME)) {
arch/arm64/kvm/hyp/vhe/switch.c +14 -0
index 3d868e84c7a0..3f0b5756c956 100644
@@ -92,14 +92,28 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu)
 }
 NOKPROBE_SYMBOL(__deactivate_traps);
 
+/*
+ * Disable IRQs in {activate,deactivate}_traps_vhe_{load,put}() to
+ * prevent a race condition between context switching of PMUSERENR_EL0
+ * in __{activate,deactivate}_traps_common() and IPIs that attempts to
+ * update PMUSERENR_EL0. See also kvm_set_pmuserenr().
+ */
 void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
 {
+	unsigned long flags;
+
+	local_irq_save(flags);
 	__activate_traps_common(vcpu);
+	local_irq_restore(flags);
 }
 
 void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu)
 {
+	unsigned long flags;
+
+	local_irq_save(flags);
 	__deactivate_traps_common(vcpu);
+	local_irq_restore(flags);
 }
 
 static const exit_handler_fn hyp_exit_handlers[] = {
arch/arm64/kvm/pmu.c +27 -0
index 7887133d15f0..121f1a14c829 100644
@@ -209,3 +209,30 @@ void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu)
 	kvm_vcpu_pmu_enable_el0(events_host);
 	kvm_vcpu_pmu_disable_el0(events_guest);
 }
+
+/*
+ * With VHE, keep track of the PMUSERENR_EL0 value for the host EL0 on the pCPU
+ * where PMUSERENR_EL0 for the guest is loaded, since PMUSERENR_EL0 is switched
+ * to the value for the guest on vcpu_load().  The value for the host EL0
+ * will be restored on vcpu_put(), before returning to userspace.
+ * This isn't necessary for nVHE, as the register is context switched for
+ * every guest enter/exit.
+ *
+ * Return true if KVM takes care of the register. Otherwise return false.
+ */
+bool kvm_set_pmuserenr(u64 val)
+{
+	struct kvm_cpu_context *hctxt;
+	struct kvm_vcpu *vcpu;
+
+	if (!kvm_arm_support_pmu_v3() || !has_vhe())
+		return false;
+
+	vcpu = kvm_get_running_vcpu();
+	if (!vcpu || !vcpu_get_flag(vcpu, PMUSERENR_ON_CPU))
+		return false;
+
+	hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
+	ctxt_sys_reg(hctxt, PMUSERENR_EL0) = val;
+	return true;
+}
drivers/perf/arm_pmuv3.c +18 -3
index c98e4039386d..93b7edb5f1e7 100644
@@ -677,9 +677,25 @@ static inline u32 armv8pmu_getreset_flags(void)
 	return value;
 }
 
+static void update_pmuserenr(u64 val)
+{
+	lockdep_assert_irqs_disabled();
+
+	/*
+	 * The current PMUSERENR_EL0 value might be the value for the guest.
+	 * If that's the case, have KVM keep tracking of the register value
+	 * for the host EL0 so that KVM can restore it before returning to
+	 * the host EL0. Otherwise, update the register now.
+	 */
+	if (kvm_set_pmuserenr(val))
+		return;
+
+	write_pmuserenr(val);
+}
+
 static void armv8pmu_disable_user_access(void)
 {
-	write_pmuserenr(0);
+	update_pmuserenr(0);
 }
 
 static void armv8pmu_enable_user_access(struct arm_pmu *cpu_pmu)
@@ -695,8 +711,7 @@ static void armv8pmu_enable_user_access(struct arm_pmu *cpu_pmu)
 			armv8pmu_write_evcntr(i, 0);
 	}
 
-	write_pmuserenr(0);
-	write_pmuserenr(ARMV8_PMU_USERENR_ER | ARMV8_PMU_USERENR_CR);
+	update_pmuserenr(ARMV8_PMU_USERENR_ER | ARMV8_PMU_USERENR_CR);
 }
 
 static void armv8pmu_enable_event(struct perf_event *event)
Signature · 1 line
2.41.0.rc0.172.g3f132b7071-goog
maintainer Applied
#3 lore raw
In reply to Reiji Watanabe
Quoted 9 lines · Reiji Watanabe

This series will fix bugs in KVM's handling of PMUSERENR_EL0.

With PMU access support from EL0 [1], the perf subsystem would set CR and ER bits of PMUSERENR_EL0 as needed to allow EL0 to have a direct access to PMU counters. However, KVM appears to assume that the register value is always zero for the host EL0, and has the following two problems in handling the register.

[...]

Applied to fixes, thanks!

[1/2] KVM: arm64: PMU: Restore the host's PMUSERENR_EL0

      commit: 8681f71759010503892f9e3ddb05f65c0f21b690
[2/2] KVM: arm64: PMU: Don't overwrite PMUSERENR with vcpu loaded
      commit: 0c2f9acf6ae74118385f7a7d48f4b2d93637b628

Cheers,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.