perf/x86/intel: Don't update PEBS_ENABLE MSR in NMI context
Subsystems (2)
PATCH 1/1 perf/x86/intel: Don't update PEBS_ENABLE MSR in NMI context
Writing MSR_IA32_PEBS_ENABLE in handle_pmi_common() when a PEBS event is throttled is problematic for KVM, which may have an older value for the MSR (previously obtained via perf_guest_get_msrs()) stored in the VM-exit MSR-load list.
It isn't necessary to update the MSR in the PMI handler, because the throttled counter's PerfEvtSel ENABLE bit has been cleared, so its PEBS_ENABLE bit is irrelevant. However, the MSR must be updated before the stale bit becomes relevant again (i.e. when the PMC starts counting a new perf event).
When the PEBS_ENABLE MSR is rendered stale in the PMI handler, just record that fact in a new cpu_hw_events boolean, pebs_stale. Extend intel_pmu_pebs_enable_all() to write MSR_IA32_PEBS_ENABLE when pebs_stale is set and to clear pebs_stale. This ensures stale bits are cleared during the normal PMU enable path, before PERF_GLOBAL_CTRL is restored and any new event can start counting.
--- arch/x86/events/intel/core.c | 28 ++++++++++++++++++++++------ arch/x86/events/intel/ds.c | 4 +++- arch/x86/events/perf_event.h | 1 + 3 files changed, 26 insertions(+), 7 deletions(-)
arch/x86/events/intel/core.c +22 -6
@@ -3548,14 +3548,26 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)static_call (x86_pmu_drain_pebs)(regs, &data);/* -* PMI throttle may be triggered, which stops the PEBS event. -* Although cpuc->pebs_enabled is updated accordingly, the -* MSR_IA32_PEBS_ENABLE is not updated. Because the -* cpuc->enabled has been forced to 0 in PMI. -* Update the MSR if pebs_enabled is changed. +* PMI throttling may be triggered, which stops the PEBS +* event. Although cpuc->pebs_enabled was updated +* accordingly, MSR_IA32_PEBS_ENABLE has not been updated. +* +* The MSR must not be updated in NMI context, because KVM +* may be in a critical region on its way to VM-entry, with +* the now-stale MSR value stored in the VM-exit MSR-load +* list. On VM-exit, the CPU will restore the stale value. +* +* Deferring the MSR update is harmless because the +* throttled event's PerfEvtSel ENABLE bit has been +* cleared, so the stale PEBS_ENABLE bit is irrelevant for +* now. +* +* Track when MSR_IA32_PEBS_ENABLE is stale, so that +* pebs_enable_all() will write cpuc->pebs_enabled to the +* MSR, even when cpuc->pebs_enabled is 0. */ if (pebs_enabled != cpuc->pebs_enabled) -wrmsrq (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); + cpuc->pebs_stale = true;/* * Above PEBS handler (PEBS counters snapshotting) has updated fixed
@@ -4978,6 +4990,10 @@ static int intel_pmu_hw_config(struct perf_event *event)* These values have nothing to do with the emulated values the guest sees * when it uses {RD,WR}MSR, which should be handled by the KVM context, * specifically in the intel_pmu_{get,set}_msr(). +* +* Note that MSRs returned from this function must not be modified in NMI +* context. Doing so could result in a stale host value being restored at +* the next VM-exit. */ static struct perf_guest_switch_msr *intel_guest_get_msrs (int *nr,void *data) {
arch/x86/events/intel/ds.c +3 -1
@@ -1982,8 +1982,10 @@ void intel_pmu_pebs_enable_all(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -if (cpuc->pebs_enabled) +if (cpuc->pebs_enabled || cpuc->pebs_stale)wrmsrq (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); + + cpuc->pebs_stale = false; }void intel_pmu_pebs_disable_all (void )
arch/x86/events/perf_event.h +1 -0
@@ -298,6 +298,7 @@ struct cpu_hw_events {/* DS based PEBS or arch-PEBS buffer address */ void *pebs_vaddr;u64 pebs_enabled; +bool pebs_stale;int n_pebs;int n_large_pebs;int n_pebs_via_pt;
Signature · 1 line
2.53.0.959.g497ff81fa9-goog
Quoted 103 lines · Jim Mattson
Writing
MSR_IA32_PEBS_ENABLEinhandle_pmi_common()when a PEBS event is throttled is problematic for KVM, which may have an older value for the MSR (previously obtained viaperf_guest_get_msrs()) stored in the VM-exit MSR-load list.It isn't necessary to update the MSR in the PMI handler, because the throttled counter's PerfEvtSel ENABLE bit has been cleared, so its
PEBS_ENABLEbit is irrelevant. However, the MSR must be updated before the stale bit becomes relevant again (i.e. when the PMC starts counting a new perf event).When the
PEBS_ENABLEMSR is rendered stale in the PMI handler, just record that fact in a newcpu_hw_eventsboolean,pebs_stale. Extendintel_pmu_pebs_enable_all()to writeMSR_IA32_PEBS_ENABLEwhenpebs_staleis set and to clearpebs_stale. This ensures stale bits are cleared during the normal PMU enable path, beforePERF_GLOBAL_CTRLis restored and any new event can start counting.Signed-off-by Jim Mattson <[email protected]>--- arch/x86/events/intel/core.c | 28 ++++++++++++++++++++++------ arch/x86/events/intel/ds.c | 4 +++- arch/x86/events/perf_event.h | 1 + 3 files changed, 26 insertions(+), 7 deletions(-)arch/x86/events/intel/core.c +22
-6@@ -3548,14 +3548,26 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)static_call (x86_pmu_drain_pebs)(regs, &data);/* -* PMI throttle may be triggered, which stops the PEBS event. -* Although cpuc->pebs_enabled is updated accordingly, the -* MSR_IA32_PEBS_ENABLE is not updated. Because the -* cpuc->enabled has been forced to 0 in PMI. -* Update the MSR if pebs_enabled is changed. +* PMI throttling may be triggered, which stops the PEBS +* event. Although cpuc->pebs_enabled was updated +* accordingly, MSR_IA32_PEBS_ENABLE has not been updated. +* +* The MSR must not be updated in NMI context, because KVM +* may be in a critical region on its way to VM-entry, with +* the now-stale MSR value stored in the VM-exit MSR-load +* list. On VM-exit, the CPU will restore the stale value. +* +* Deferring the MSR update is harmless because the +* throttled event's PerfEvtSel ENABLE bit has been +* cleared, so the stale PEBS_ENABLE bit is irrelevant for +* now. +* +* Track when MSR_IA32_PEBS_ENABLE is stale, so that +* pebs_enable_all() will write cpuc->pebs_enabled to the +* MSR, even when cpuc->pebs_enabled is 0. */ if (pebs_enabled != cpuc->pebs_enabled) -wrmsrq (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); + cpuc->pebs_stale = true;/* * Above PEBS handler (PEBS counters snapshotting) has updated fixed @@ -4978,6 +4990,10 @@ static int intel_pmu_hw_config(struct perf_event *event)* These values have nothing to do with the emulated values the guest sees * when it uses {RD,WR}MSR, which should be handled by the KVM context, * specifically in the intel_pmu_{get,set}_msr(). +* +* Note that MSRs returned from this function must not be modified in NMI +* context. Doing so could result in a stale host value being restored at +* the next VM-exit. */ static struct perf_guest_switch_msr *intel_guest_get_msrs (int *nr,void *data) {arch/x86/events/intel/ds.c +3
-1@@ -1982,8 +1982,10 @@ void intel_pmu_pebs_enable_all(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events); -if (cpuc->pebs_enabled) +if (cpuc->pebs_enabled || cpuc->pebs_stale)wrmsrq (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); + + cpuc->pebs_stale = false; }void intel_pmu_pebs_disable_all (void )arch/x86/events/perf_event.h +1
-1@@ -298,6 +298,7 @@ struct cpu_hw_events {/* DS based PEBS or arch-PEBS buffer address */ void *pebs_vaddr;u64 pebs_enabled; +bool pebs_stale;int n_pebs;int n_large_pebs;int n_pebs_via_pt; --
Quoted 2 lines
2.53.0.959.g497ff81fa9-goog
Apologies. Resend to fix Peter Zijlstra's address.
Quoted 33 lines · Jim Mattson
Quoted 32 lines · Jim Mattson
Writing
MSR_IA32_PEBS_ENABLEinhandle_pmi_common()when a PEBS event is throttled is problematic for KVM, which may have an older value for the MSR (previously obtained viaperf_guest_get_msrs()) stored in the VM-exit MSR-load list.It isn't necessary to update the MSR in the PMI handler, because the throttled counter's PerfEvtSel ENABLE bit has been cleared, so its
PEBS_ENABLEbit is irrelevant. However, the MSR must be updated before the stale bit becomes relevant again (i.e. when the PMC starts counting a new perf event).When the
PEBS_ENABLEMSR is rendered stale in the PMI handler, just record that fact in a newcpu_hw_eventsboolean,pebs_stale. Extendintel_pmu_pebs_enable_all()to writeMSR_IA32_PEBS_ENABLEwhenpebs_staleis set and to clearpebs_stale. This ensures stale bits are cleared during the normal PMU enable path, beforePERF_GLOBAL_CTRLis restored and any new event can start counting.Signed-off-by Jim Mattson <[email protected]>--- arch/x86/events/intel/core.c | 28 ++++++++++++++++++++++------ arch/x86/events/intel/ds.c | 4 +++- arch/x86/events/perf_event.h | 1 + 3 files changed, 26 insertions(+), 7 deletions(-)arch/x86/events/intel/core.c +0
-0@@ -3548,14 +3548,26 @@ static int handle_pmi_common(struct pt_regs *regs, u64 status)static_call (x86_pmu_drain_pebs)(regs, &data);
Quoted 14 lines
Quoted 4 lines
arch/x86/events/intel/core.c +0
-3/* -* PMI throttle may be triggered, which stops the PEBS event. -* Although cpuc->pebs_enabled is updated accordingly, the -* MSR_IA32_PEBS_ENABLE is not updated. Because the arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 3558
Quoted 10 lines
arch/x86/events/intel/core.c +8
-2-* cpuc->enabled has been forced to 0 in PMI. -* Update the MSR if pebs_enabled is changed. +* PMI throttling may be triggered, which stops the PEBS +* event. Although cpuc->pebs_enabled was updated +* accordingly, MSR_IA32_PEBS_ENABLE has not been updated. +* +* The MSR must not be updated in NMI context, because KVM +* may be in a critical region on its way to VM-entry, with +* the now-stale MSR value stored in the VM-exit MSR-load +* list. On VM-exit, the CPU will restore the stale value.
Hmm, that looks like a race condition between KVM and PMI handler. Suppose the issue could only happen in the legacy perf-based vPMU enabled case, right? For the mediated vPMU, the host events would be rescheduled on each VM-exit and the MSR_IA32_PEBS_ENABLE MSR would be always written.
Quoted 5 lines
Quoted 5 lines
arch/x86/events/intel/core.c +5
-0+* +* Deferring the MSR update is harmless because the +* throttled event's PerfEvtSel ENABLE bit has been +* cleared, so the stale PEBS_ENABLE bit is irrelevant for +* now.
Quoted 8 lines
arch/x86/events/intel/core.c static int handle_pmi_common(struct pt_regs *regs, u64 status) · line 3570
Quoted 8 lines
arch/x86/events/intel/core.c +5
-1+* +* Track when MSR_IA32_PEBS_ENABLE is stale, so that +* pebs_enable_all() will write cpuc->pebs_enabled to the +* MSR, even when cpuc->pebs_enabled is 0. */ if (pebs_enabled != cpuc->pebs_enabled) -wrmsrq (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); + cpuc->pebs_stale = true;
Deferring the MSR_IA32_PEBS_ENABLE writing would cause a mismatch between the value of MSR_IA32_PEBS_ENABLE and cpuc->pebs_enabled until intel_pmu_pebs_enable_all() is called again. But suppose it would be fine since the event has been stopped and intel_pmu_pebs_enable_all() would be called in next unthrottle.
Quoted 22 lines
Quoted 22 lines
arch/x86/events/intel/core.c +4
-0/* * Above PEBS handler (PEBS counters snapshotting) has updated fixed @@ -4978,6 +4990,10 @@ static int intel_pmu_hw_config(struct perf_event *event)* These values have nothing to do with the emulated values the guest sees * when it uses {RD,WR}MSR, which should be handled by the KVM context, * specifically in the intel_pmu_{get,set}_msr(). +* +* Note that MSRs returned from this function must not be modified in NMI +* context. Doing so could result in a stale host value being restored at +* the next VM-exit. */ static struct perf_guest_switch_msr *intel_guest_get_msrs (int *nr,void *data) {arch/x86/events/intel/ds.c +0
-0@@ -1982,8 +1982,10 @@ void intel_pmu_pebs_enable_all(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);
Quoted 5 lines
arch/x86/events/intel/ds.c void intel_pmu_pebs_enable_all(void) · line 1988
Quoted 5 lines
arch/x86/events/intel/ds.c +3
-1-if (cpuc->pebs_enabled) +if (cpuc->pebs_enabled || cpuc->pebs_stale)wrmsrq (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); + + cpuc->pebs_stale = false;
Sashiko comments
"
Can this race with an NMI?
If a PMI fires exactly between wrmsrq() and cpuc->pebs_stale = false:
CPU0intel_pmu_pebs_enable_all()
wrmsrq(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled);
<--- NMI (PMI) fires here
handle_pmi_common()
... throttles event ...
cpuc->pebs_stale = true;
<--- NMI returns
cpuc->pebs_stale = false;
Would this clobber the pebs_stale flag set by the NMI, leaving the software state desynchronized from the hardware MSR and potentially causing the PMU to skip the MSR update on the next enable?
"
It looks reasonable.
https://sashiko.dev/#/patchset/20260320152928.1916447-1-jmattson%40google.com
Thanks.
Quoted 20 lines
Quoted 16 lines
arch/x86/events/intel/ds.c +0
-0}void intel_pmu_pebs_disable_all (void )arch/x86/events/perf_event.h +1
-1@@ -298,6 +298,7 @@ struct cpu_hw_events {/* DS based PEBS or arch-PEBS buffer address */ void *pebs_vaddr;u64 pebs_enabled; +bool pebs_stale;int n_pebs;int n_large_pebs;int n_pebs_via_pt; --Quoted 2 lines
2.53.0.959.g497ff81fa9-goog
Apologies. Resend to fix Peter Zijlstra's address.
Quoted 39 lines · Dapeng
Quoted 9 lines · Jim Mattson
Quoted 8 lines · Jim Mattson
arch/x86/events/intel/ds.c +0
-0@@ -1982,8 +1982,10 @@ void intel_pmu_pebs_enable_all(void) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);Quoted 5 lines
arch/x86/events/intel/ds.c void intel_pmu_pebs_enable_all(void) · line 1988
Quoted 5 lines
arch/x86/events/intel/ds.c +3
-1-if (cpuc->pebs_enabled) +if (cpuc->pebs_enabled || cpuc->pebs_stale)wrmsrq (MSR_IA32_PEBS_ENABLE , cpuc->pebs_enabled); + + cpuc->pebs_stale = false;Sashiko comments
"
Can this race with an NMI?
If a PMI fires exactly betweenwrmsrq()andcpuc->pebs_stale= false:
CPU0intel_pmu_pebs_enable_all()wrmsrq(MSR_IA32_PEBS_ENABLE, cpuc->pebs_enabled); <--- NMI (PMI) fires here handle_pmi_common() ... throttles event ... cpuc->pebs_stale = true; <--- NMI returns cpuc->pebs_stale = false;Would this clobber the
pebs_staleflag set by the NMI, leaving the software state desynchronized from the hardware MSR and potentially causing the PMU to skip the MSR update on the next enable?
Quoted 6 lines
"
It looks reasonable.
https://sashiko.dev/#/patchset/20260320152928.1916447-1-jmattson%40google.comThanks.
I believe this function is only called while the PMU is disabled (on the path to re-enabling it), so there should be no PMIs.
Sadly, though, this patch has not fixed our problem.