KVM: x86: vPMU: truncate counter value to allowed width
Subsystems (4)
PATCH 1/1 KVM: x86: vPMU: truncate counter value to allowed width
Performance counters are defined to have width less than 64 bits. The vPMU code maintains the counters in u64 variables but assumes the value to fit within the defined width. However, for Intel non-full-width counters (MSR_IA32_PERFCTRx) the value receieved from the guest is truncated to 32 bits and then sign-extended to full 64 bits. If a negative value is set, it's sign-extended to 64 bits, but then in kvm_pmu_incr_counter() it's incremented, truncated, and compared to the previous value for overflow detection.
That previous value is not truncated, so it always evaluates bigger than the truncated new one, and a PMI is injected. If the PMI handler writes a negative counter value itself, the vCPU never quits the PMI loop.
Turns out that Linux PMI handler actually does write the counter with the value just read with RDPMC, so when no full-width support is exposed via MSR_IA32_PERF_CAPABILITIES, and the guest initializes the counter to a negative value, it locks up.
We observed this in the field, for example, when the guest configures atop to use perfevents and runs two instances of it simultaneously.
To address the problem, maintain the invariant that the counter value always fits in the defined bit width, by truncating the received value in the respective set_msr methods. For better readability, factor this out into a helper function, pmc_set_counter, shared by vmx and svm parts.
--- arch/x86/kvm/pmu.h | 6 ++++++ arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 4 ++-- 3 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/kvm/pmu.h +6 -0
@@ -60,6 +60,12 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_set_counter (struct kvm_pmc *pmc,u64 val) +{ + pmc->counter += val -pmc_read_counter (pmc); + pmc->counter &=pmc_bitmask (pmc); +} +static inline void pmc_release_perf_event (struct kvm_pmc *pmc) {if (pmc->perf_event) {
arch/x86/kvm/svm/pmu.c +1 -1
@@ -151,7 +151,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_set_counter (pmc, data);pmc_update_sample_period (pmc);return 0 ; }
arch/x86/kvm/vmx/pmu_intel.c +2 -2
@@ -467,11 +467,11 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); +pmc_set_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_set_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) {
Signature · 12 lines
2.34.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879
Quoted 8 lines · Roman Kagan
Performance counters are defined to have width less than 64 bits. The vPMU code maintains the counters in u64 variables but assumes the value to fit within the defined width. However, for Intel non-full-width counters (
MSR_IA32_PERFCTRx) the value receieved from the guest is truncated to 32 bits and then sign-extended to full 64 bits. If a negative value is set, it's sign-extended to 64 bits, but then inkvm_pmu_incr_counter()it's incremented, truncated, and compared to the previous value for overflow detection.
Thanks for reporting this issue. An easier-to-understand fix could be:
arch/x86/kvm/pmu.c +1 -1
@@ -718,7 +718,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)static void kvm_pmu_incr_counter (struct kvm_pmc *pmc) { - pmc->prev_counter = pmc->counter; + pmc->prev_counter = pmc->counter &pmc_bitmask (pmc); pmc->counter = (pmc->counter +1 ) &pmc_bitmask (pmc);kvm_pmu_request_counter_reprogram (pmc); }
Considering that the pmu code uses pmc_bitmask(pmc) everywhere to wrap around, I would prefer to use this fix above first and then do a more thorough cleanup based on your below diff. What do you think ?
Quoted 8 lines
That previous value is not truncated, so it always evaluates bigger than the truncated new one, and a PMI is injected. If the PMI handler writes a negative counter value itself, the vCPU never quits the PMI loop.
Turns out that Linux PMI handler actually does write the counter with the value just read with RDPMC, so when no full-width support is exposed via
MSR_IA32_PERF_CAPABILITIES, and the guest initializes the counter to a negative value, it locks up.
Not really sure what the behavioral difference is between "it locks up" and "the vCPU never quits the PMI loop".
Quoted 3 lines
We observed this in the field, for example, when the guest configures atop to use perfevents and runs two instances of it simultaneously.
A more essential case I found is this:
kvm_msr: msr_write CTR1 = 0xffffffffffffffea
rdpmc on host: CTR1, value 0xffffffffffe3
kvm_exit: vcpu 0 reason EXCEPTION_NMI
kvm_msr: msr_read CTR1 = 0x83 // nmi_handler
There are two typical issues here:
- the emulated counter value changed from 0xffffffffffffffffea to 0xffffffffffffe3, triggering __kvm_perf_overflow(pmc, false);
- PMI-handler should not reset the counter to a value that is easily overflowed, in order to avoid overflow here before iret;
Please confirm whether your usage scenarios consist of these two types, or more.
Quoted 9 lines
To address the problem, maintain the invariant that the counter value always fits in the defined bit width, by truncating the received value in the respective
set_msrmethods. For better readability, factor this out into a helper function,pmc_set_counter, shared by vmx and svm parts.Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")Signed-off-by Roman Kagan <[email protected]>
I prefer to use pmc_bitmask(pmc) to wrap around pmc->prev_counter as the first step.
Quoted 54 lines
--- arch/x86/kvm/pmu.h | 6 ++++++ arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 4 ++-- 3 files changed, 9 insertions(+), 3 deletions(-)arch/x86/kvm/pmu.h +6
-0@@ -60,6 +60,12 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_set_counter (struct kvm_pmc *pmc,u64 val) +{ + pmc->counter += val -pmc_read_counter (pmc); + pmc->counter &=pmc_bitmask (pmc); +} +static inline void pmc_release_perf_event (struct kvm_pmc *pmc) {if (pmc->perf_event) {arch/x86/kvm/svm/pmu.c +1
-1@@ -151,7 +151,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_set_counter (pmc, data);pmc_update_sample_period (pmc);return 0 ; }arch/x86/kvm/vmx/pmu_intel.c +2
-2@@ -467,11 +467,11 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); +pmc_set_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_set_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) {
Quoted 30 lines · Like Xu
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 8 lines · Roman Kagan
Performance counters are defined to have width less than 64 bits. The vPMU code maintains the counters in u64 variables but assumes the value to fit within the defined width. However, for Intel non-full-width counters (
MSR_IA32_PERFCTRx) the value receieved from the guest is truncated to 32 bits and then sign-extended to full 64 bits. If a negative value is set, it's sign-extended to 64 bits, but then inkvm_pmu_incr_counter()it's incremented, truncated, and compared to the previous value for overflow detection.Thanks for reporting this issue. An easier-to-understand fix could be:
arch/x86/kvm/pmu.c +1
-1@@ -718,7 +718,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)static void kvm_pmu_incr_counter (struct kvm_pmc *pmc) { - pmc->prev_counter = pmc->counter; + pmc->prev_counter = pmc->counter &pmc_bitmask (pmc); pmc->counter = (pmc->counter +1 ) &pmc_bitmask (pmc);kvm_pmu_request_counter_reprogram (pmc); }
Quoted 3 lines
Considering that the pmu code uses
pmc_bitmask(pmc)everywhere to wrap around, I would prefer to use this fix above first and then do a more thorough cleanup based on your below diff. What do you think ?
I did exactly this at first. However, it felt more natural and easier to reason about and less error-prone going forward, to maintain the invariant that pmc->counter always fits in the assumed width.
Quoted 11 lines
Quoted 8 lines
That previous value is not truncated, so it always evaluates bigger than the truncated new one, and a PMI is injected. If the PMI handler writes a negative counter value itself, the vCPU never quits the PMI loop.
Turns out that Linux PMI handler actually does write the counter with the value just read with RDPMC, so when no full-width support is exposed via
MSR_IA32_PERF_CAPABILITIES, and the guest initializes the counter to a negative value, it locks up.Not really sure what the behavioral difference is between "it locks up" and "the vCPU never quits the PMI loop".
No difference, the second paragraph just illustrates the specific case with Linux PMI handler and the system not exposing full-width counter support so the problematic code path is triggered.
Quoted 5 lines
Quoted 2 lines
We observed this in the field, for example, when the guest configures atop to use perfevents and runs two instances of it simultaneously.
A more essential case I found is this:
Quoted 7 lines
kvm_msr:msr_writeCTR1 =0xffffffffffffffea
rdpmc on host: CTR1, value0xffffffffffe3
kvm_exit: vcpu 0 reasonEXCEPTION_NMI
kvm_msr:msr_readCTR1 =0x83//nmi_handlerThere are two typical issues here:
- the emulated counter value changed from0xffffffffffffffffeato0xffffffffffffe3,
Strange, why would the counter go backwards (disregarding the extra high bits)?
Quoted 3 lines
triggering
__kvm_perf_overflow(pmc, false);
- PMI-handler should not reset the counter to a value that is easily overflowed, in order to avoid overflow here before iret;
This is a legitimate guest behavior, isn't it? The problem I'm trying to fix is when the value written is sufficiently far from overflowing, but it still ends up looking as an overflow and triggers a PMI.
Quoted 1 line
Please confirm whether your usage scenarios consist of these two types, or more.
The *usage* scenario is the one I wrote above. I've identified an easier repro since then:
perf stat -e instructions -C 0 & sleep 1 && perf stat -e instructions,cycles -C 0 sleep 0.1 && kill -INT %
Please also see the kvm-unit-test I posted at
https://lore.kernel.org/kvm/[email protected]
Quoted 11 lines
Quoted 8 lines
To address the problem, maintain the invariant that the counter value always fits in the defined bit width, by truncating the received value in the respective
set_msrmethods. For better readability, factor this out into a helper function,pmc_set_counter, shared by vmx and svm parts.Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")Signed-off-by Roman Kagan <[email protected]>Tested-by Like Xu <[email protected]>I prefer to use
pmc_bitmask(pmc)to wrap aroundpmc->prev_counteras the first step.
My preference is to keep the counter within the limits all the time, so I'd leave it up to the maintainers to decide.
Thanks,
Roman.
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Sean Christopherson
maintainerQuoted 27 lines · Roman Kagan
Quoted 26 lines · Like Xu
Quoted 8 lines · Roman Kagan
Performance counters are defined to have width less than 64 bits. The vPMU code maintains the counters in u64 variables but assumes the value to fit within the defined width. However, for Intel non-full-width counters (
MSR_IA32_PERFCTRx) the value receieved from the guest is truncated to 32 bits and then sign-extended to full 64 bits. If a negative value is set, it's sign-extended to 64 bits, but then inkvm_pmu_incr_counter()it's incremented, truncated, and compared to the previous value for overflow detection.Thanks for reporting this issue. An easier-to-understand fix could be:
arch/x86/kvm/pmu.c +1
-1@@ -718,7 +718,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)static void kvm_pmu_incr_counter (struct kvm_pmc *pmc) { - pmc->prev_counter = pmc->counter; + pmc->prev_counter = pmc->counter &pmc_bitmask (pmc); pmc->counter = (pmc->counter +1 ) &pmc_bitmask (pmc);kvm_pmu_request_counter_reprogram (pmc); }
Quoted 7 lines
Quoted 3 lines
Considering that the pmu code uses
pmc_bitmask(pmc)everywhere to wrap around, I would prefer to use this fix above first and then do a more thorough cleanup based on your below diff. What do you think ?I did exactly this at first. However, it felt more natural and easier to reason about and less error-prone going forward, to maintain the invariant that
pmc->counteralways fits in the assumed width.
Agreed, KVM shouldn't store information that's not supposed to exist.
Sean Christopherson
maintainerarch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 65
Quoted 11 lines · Roman Kagan
arch/x86/kvm/pmu.h +3
-0@@ -60,6 +60,12 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_set_counter (struct kvm_pmc *pmc,u64 val) +{ + pmc->counter += val -pmc_read_counter (pmc);
Ugh, not your code, but I don't see how the current code can possibly be correct.
The above unpacks to
counter = pmc->counter; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter += val - (counter & pmc_bitmask(pmc));
which distills down to
counter = 0; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val - (counter & pmc_bitmask(pmc));
or more succinctly
if (pmc->perf_event && !pmc->is_paused) val -= perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val;
which is obviously wrong. E.g. if the guest writes '0' to an active counter, the adjustment will cause pmc->counter to be loaded with a large (in unsigned terms) value, and thus quickly overflow after a write of '0'.
I assume the intent it to purge any accumulated counts that occurred since the last read, but *before* the write, but then shouldn't this just be:
/* Purge any events that were acculumated before the write. */ if (pmc->perf_event && !pmc->is_paused) (void)perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val & pmc_bitmask(pmc);
Am I missing something?
I'd like to get this sorted out before applying this patch, because I really want to document what on earth this new helper is doing. Seeing a bizarre partial RMW operation in a helper with "set" as the verb is super weird.
Quoted 34 lines · Sean Christopherson
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 65
Quoted 11 lines · Roman Kagan
arch/x86/kvm/pmu.h +3
-0@@ -60,6 +60,12 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_set_counter (struct kvm_pmc *pmc,u64 val) +{ + pmc->counter += val -pmc_read_counter (pmc);Ugh, not your code, but I don't see how the current code can possibly be correct.
The above unpacks to
counter = pmc->counter; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter += val - (counter & pmc_bitmask(pmc));which distills down to
counter = 0; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val - (counter & pmc_bitmask(pmc));or more succinctly
Quoted 8 lines
if (pmc->perf_event && !pmc->is_paused) val -= perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val;which is obviously wrong. E.g. if the guest writes '0' to an active counter, the adjustment will cause
pmc->counterto be loaded with a large (in unsigned terms) value, and thus quickly overflow after a write of '0'.
This weird construct goes all the way back to commit f5132b01386b ("KVM: Expose a version 2 architectural PMU to a guests"). Paolo killed it in commit 2924b52117b2 ("KVM: x86/pmu: do not mask the value that is written to fixed PMUs"), perhaps by accident. Eric then resurrected it in commit 4400cf546b4b ("KVM: x86: Fix perfctr WRMSR for running counters").
It makes no sense to me. WRMSR should just set the new value of the counter, regardless of the old value or whether or not it is running.
Quoted 14 lines
I assume the intent it to purge any accumulated counts that occurred since the last read, but *before* the write, but then shouldn't this just be:
/* Purge any events that were acculumated before the write. */ if (pmc->perf_event && !pmc->is_paused) (void)perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val & pmc_bitmask(pmc);Am I missing something?
I'd like to get this sorted out before applying this patch, because I really want to document what on earth this new helper is doing. Seeing a bizarre partial RMW operation in a helper with "set" as the verb is super weird.
Sean Christopherson
maintainer+Mingwei
Quoted 51 lines · Jim Mattson
Quoted 34 lines · Sean Christopherson
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 65
Quoted 11 lines · Roman Kagan
arch/x86/kvm/pmu.h +3
-0@@ -60,6 +60,12 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_set_counter (struct kvm_pmc *pmc,u64 val) +{ + pmc->counter += val -pmc_read_counter (pmc);Ugh, not your code, but I don't see how the current code can possibly be correct.
The above unpacks to
counter = pmc->counter; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter += val - (counter & pmc_bitmask(pmc));which distills down to
counter = 0; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val - (counter & pmc_bitmask(pmc));or more succinctly
Quoted 8 lines
if (pmc->perf_event && !pmc->is_paused) val -= perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val;which is obviously wrong. E.g. if the guest writes '0' to an active counter, the adjustment will cause
pmc->counterto be loaded with a large (in unsigned terms) value, and thus quickly overflow after a write of '0'.This weird construct goes all the way back to commit
f5132b01386b("KVM: Expose a version 2 architectural PMU to a guests"). Paolo killed it in commit2924b52117b2("KVM: x86/pmu: do not mask the value that is written to fixed PMUs"), perhaps by accident. Eric then resurrected it in commit4400cf546b4b("KVM: x86: Fix perfctr WRMSR for running counters").
Quoted 2 lines
It makes no sense to me. WRMSR should just set the new value of the counter, regardless of the old value or whether or not it is running.
Heh, didn't stop you from giving Eric's patch a thumbs-up[*] :-)
Thanks to Eric's testcase [Wow, tests do help! We should try writing more of them!], I finally figured out what's going on. I wrongly assumed perf_event_read_value() is destructive, but it's not, it just reads the current value. So on a WRMSR, KVM offsets the value with the current perf event, and then *mostly* adjusts for it when reading the counter.
But that is obviously super fragile because it means pmc->counter must never be read directly unless the perf event is paused and the accumulated counter has been propagated to pmc->counter. Blech.
I fiddled with a variety of things, but AFAICT the easiest solution is also the most obviously correct: set perf's count to the guest's count. Lightly tested patch below.
On a related topic, Mingwei also appears to have found another bug: prev_counter needs to be set when the counter is written, i.e. my proposed pmc_write_counter() also needs to update prev_counter.
Though that also raises the question of whether or not zeroing prev_counter in reprogram_counter() is correct. Unless I'm missing something, reprogram_counter() should also set pmc->prev_counter to pmc->counter when the counter is successfully (re)enabled.
And Jim also pointed out that prev_counter needs to be set even when KVM fails to enable a perf event (software counting should still work).
[*] https://lore.kernel.org/all/CALMp9eRfeFFb6n22Uf4R2Pf8WW7BVLX_Vuf04WFwiMtrk14Y-Q@mail.gmail.com
--- arch/x86/kvm/pmu.h | 8 ++++++++ arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 4 ++-- include/linux/perf_event.h | 2 ++ kernel/events/core.c | 11 +++++++++++ 5 files changed, 24 insertions(+), 3 deletions(-)
arch/x86/kvm/pmu.h +8 -0
@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val; +} +static inline void pmc_release_perf_event (struct kvm_pmc *pmc) {if (pmc->perf_event) {
arch/x86/kvm/svm/pmu.c +1 -1
@@ -160,7 +160,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);return 0 ; }
arch/x86/kvm/vmx/pmu_intel.c +2 -2
@@ -406,11 +406,11 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) {
include/linux/perf_event.h +2 -0
@@ -1677,6 +1677,7 @@ extern void perf_event_disable_inatomic(struct perf_event *event);extern void perf_event_task_tick (void );extern int perf_event_account_interrupt (struct perf_event *event);extern int perf_event_period (struct perf_event *event,u64 value); +extern void perf_event_set_count (struct perf_event *event,u64 count);extern u64 perf_event_pause (struct perf_event *event,bool reset);#else /* !CONFIG_PERF_EVENTS: */ static inline void *
@@ -1760,6 +1761,7 @@ static inline int perf_event_period(struct perf_event *event, u64 value) {return -EINVAL ; } +static inline perf_event_set_count(struct perf_event *event, u64 count) { }static inline u64 perf_event_pause (struct perf_event *event,bool reset) {return 0 ;
kernel/events/core.c +11 -0
@@ -5646,6 +5646,17 @@ static void _perf_event_reset(struct perf_event *event)perf_event_update_userpage (event); } +void perf_event_set_count (struct perf_event *event,u64 count) +{ +struct perf_event_context *ctx; + + ctx =perf_event_ctx_lock (event); + (void )perf_event_read (event, false); +local64_set (&event->count, count); +perf_event_ctx_unlock (event, ctx); +} +EXPORT_SYMBOL_GPL (perf_event_set_count); +/* Assume it's not an event with inherit set. */ u64 perf_event_pause (struct perf_event *event,bool reset) {
base-commit: 5ae85a1bd17b959796f6cc4c1153ceada2cf8f24
Quoted 55 lines · Sean Christopherson
+Mingwei
Quoted 51 lines · Jim Mattson
Quoted 34 lines · Sean Christopherson
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 65
Quoted 11 lines · Roman Kagan
arch/x86/kvm/pmu.h +3
-0@@ -60,6 +60,12 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_set_counter (struct kvm_pmc *pmc,u64 val) +{ + pmc->counter += val -pmc_read_counter (pmc);Ugh, not your code, but I don't see how the current code can possibly be correct.
The above unpacks to
counter = pmc->counter; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter += val - (counter & pmc_bitmask(pmc));which distills down to
counter = 0; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val - (counter & pmc_bitmask(pmc));or more succinctly
Quoted 8 lines
if (pmc->perf_event && !pmc->is_paused) val -= perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val;which is obviously wrong. E.g. if the guest writes '0' to an active counter, the adjustment will cause
pmc->counterto be loaded with a large (in unsigned terms) value, and thus quickly overflow after a write of '0'.This weird construct goes all the way back to commit
f5132b01386b("KVM: Expose a version 2 architectural PMU to a guests"). Paolo killed it in commit2924b52117b2("KVM: x86/pmu: do not mask the value that is written to fixed PMUs"), perhaps by accident. Eric then resurrected it in commit4400cf546b4b("KVM: x86: Fix perfctr WRMSR for running counters").
Quoted 4 lines
Quoted 2 lines
It makes no sense to me. WRMSR should just set the new value of the counter, regardless of the old value or whether or not it is running.
Heh, didn't stop you from giving Eric's patch a thumbs-up[*] :-)
I was smarter then, and probably understood what was going on.
Quoted 6 lines
Thanks to Eric's testcase [Wow, tests do help! We should try writing more of them!], I finally figured out what's going on. I wrongly assumed
perf_event_read_value()is destructive, but it's not, it just reads the current value. So on a WRMSR, KVM offsets the value with the current perf event, and then *mostly* adjusts for it when reading the counter.
Quoted 7 lines
But that is obviously super fragile because it means
pmc->countermust never be read directly unless the perf event is paused and the accumulated counter has been propagated topmc->counter. Blech.I fiddled with a variety of things, but AFAICT the easiest solution is also the most obviously correct: set perf's count to the guest's count. Lightly tested patch below.
That seems correct to me. :)
Quoted 11 lines
On a related topic, Mingwei also appears to have found another bug:
prev_counterneeds to be set when the counter is written, i.e. my proposedpmc_write_counter()also needs to updateprev_counter.Though that also raises the question of whether or not zeroing
prev_counterinreprogram_counter()is correct. Unless I'm missing something,reprogram_counter()should also setpmc->prev_countertopmc->counterwhen the counter is successfully (re)enabled.And Jim also pointed out that
prev_counterneeds to be set even when KVM fails to enable a perf event (software counting should still work).
This whole prev_counter business is a mess. The software counting increments by at most one per VM-exit. The only time we need to concern ourselves about overflow is when there has been a software increment and the paused counter value is -1. That is the only situation in which software counting can trigger a PMI. Even if the VM-exit was for a WRMSR to the PMC, assuming that the value just written by the guest is the value we see when we pause the counter. Or am I missing something?
Quoted 105 lines
[*] https://lore.kernel.org/all/CALMp9eRfeFFb6n22Uf4R2Pf8WW7BVLX_Vuf04WFwiMtrk14Y-Q@mail.gmail.com
--- arch/x86/kvm/pmu.h | 8 ++++++++ arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 4 ++-- include/linux/perf_event.h | 2 ++ kernel/events/core.c | 11 +++++++++++ 5 files changed, 24 insertions(+), 3 deletions(-)arch/x86/kvm/pmu.h +8
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val; +} +static inline void pmc_release_perf_event (struct kvm_pmc *pmc) {if (pmc->perf_event) {arch/x86/kvm/svm/pmu.c +1
-1@@ -160,7 +160,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);return 0 ; }arch/x86/kvm/vmx/pmu_intel.c +2
-2@@ -406,11 +406,11 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) {include/linux/perf_event.h +2
-0@@ -1677,6 +1677,7 @@ extern void perf_event_disable_inatomic(struct perf_event *event);extern void perf_event_task_tick (void );extern int perf_event_account_interrupt (struct perf_event *event);extern int perf_event_period (struct perf_event *event,u64 value); +extern void perf_event_set_count (struct perf_event *event,u64 count);extern u64 perf_event_pause (struct perf_event *event,bool reset);#else /* !CONFIG_PERF_EVENTS: */ static inline void *@@ -1760,6 +1761,7 @@ static inline int perf_event_period(struct perf_event *event, u64 value) {return -EINVAL ; } +static inline perf_event_set_count(struct perf_event *event, u64 count) { }static inline u64 perf_event_pause (struct perf_event *event,bool reset) {return 0 ;kernel/events/core.c +11
-0@@ -5646,6 +5646,17 @@ static void _perf_event_reset(struct perf_event *event)perf_event_update_userpage (event); } +void perf_event_set_count (struct perf_event *event,u64 count) +{ +struct perf_event_context *ctx; + + ctx =perf_event_ctx_lock (event); + (void )perf_event_read (event, false); +local64_set (&event->count, count); +perf_event_ctx_unlock (event, ctx); +} +EXPORT_SYMBOL_GPL (perf_event_set_count); +/* Assume it's not an event with inherit set. */ u64 perf_event_pause (struct perf_event *event,bool reset) {base-commit:
5ae85a1bd17b959796f6cc4c1153ceada2cf8f24--
Quoted 100 lines · Sean Christopherson
+Mingwei
Quoted 51 lines · Jim Mattson
Quoted 34 lines · Sean Christopherson
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 65
Quoted 11 lines · Roman Kagan
arch/x86/kvm/pmu.h +3
-0@@ -60,6 +60,12 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_set_counter (struct kvm_pmc *pmc,u64 val) +{ + pmc->counter += val -pmc_read_counter (pmc);Ugh, not your code, but I don't see how the current code can possibly be correct.
The above unpacks to
counter = pmc->counter; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter += val - (counter & pmc_bitmask(pmc));which distills down to
counter = 0; if (pmc->perf_event && !pmc->is_paused) counter += perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val - (counter & pmc_bitmask(pmc));or more succinctly
Quoted 8 lines
if (pmc->perf_event && !pmc->is_paused) val -= perf_event_read_value(pmc->perf_event, &enabled, &running); pmc->counter = val;which is obviously wrong. E.g. if the guest writes '0' to an active counter, the adjustment will cause
pmc->counterto be loaded with a large (in unsigned terms) value, and thus quickly overflow after a write of '0'.This weird construct goes all the way back to commit
f5132b01386b("KVM: Expose a version 2 architectural PMU to a guests"). Paolo killed it in commit2924b52117b2("KVM: x86/pmu: do not mask the value that is written to fixed PMUs"), perhaps by accident. Eric then resurrected it in commit4400cf546b4b("KVM: x86: Fix perfctr WRMSR for running counters").Quoted 2 lines
It makes no sense to me. WRMSR should just set the new value of the counter, regardless of the old value or whether or not it is running.
Heh, didn't stop you from giving Eric's patch a thumbs-up[*] :-)
Thanks to Eric's testcase [Wow, tests do help! We should try writing more of them!], I finally figured out what's going on. I wrongly assumed
perf_event_read_value()is destructive, but it's not, it just reads the current value. So on a WRMSR, KVM offsets the value with the current perf event, and then *mostly* adjusts for it when reading the counter.But that is obviously super fragile because it means
pmc->countermust never be read directly unless the perf event is paused and the accumulated counter has been propagated topmc->counter. Blech.I fiddled with a variety of things, but AFAICT the easiest solution is also the most obviously correct: set perf's count to the guest's count. Lightly tested patch below.
On a related topic, Mingwei also appears to have found another bug:
prev_counterneeds to be set when the counter is written, i.e. my proposedpmc_write_counter()also needs to updateprev_counter.Though that also raises the question of whether or not zeroing
prev_counterinreprogram_counter()is correct. Unless I'm missing something,reprogram_counter()should also setpmc->prev_countertopmc->counterwhen the counter is successfully (re)enabled.And Jim also pointed out that
prev_counterneeds to be set even when KVM fails to enable a perf event (software counting should still work).[*] https://lore.kernel.org/all/CALMp9eRfeFFb6n22Uf4R2Pf8WW7BVLX_Vuf04WFwiMtrk14Y-Q@mail.gmail.com
--- arch/x86/kvm/pmu.h | 8 ++++++++ arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 4 ++-- include/linux/perf_event.h | 2 ++ kernel/events/core.c | 11 +++++++++++ 5 files changed, 24 insertions(+), 3 deletions(-)arch/x86/kvm/pmu.h +0
-0
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 9 lines
arch/x86/kvm/pmu.h +6
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); } +static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val;
Doesn't this still have the original problem of storing wider value than allowed?
Thanks,
Roman.
Quoted 79 lines
arch/x86/kvm/pmu.h +2
-0+} +static inline void pmc_release_perf_event (struct kvm_pmc *pmc) {if (pmc->perf_event) {arch/x86/kvm/svm/pmu.c +1
-1@@ -160,7 +160,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);return 0 ; }arch/x86/kvm/vmx/pmu_intel.c +2
-2@@ -406,11 +406,11 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); +pmc_write_counter (pmc, data);pmc_update_sample_period (pmc);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) {include/linux/perf_event.h +2
-0@@ -1677,6 +1677,7 @@ extern void perf_event_disable_inatomic(struct perf_event *event);extern void perf_event_task_tick (void );extern int perf_event_account_interrupt (struct perf_event *event);extern int perf_event_period (struct perf_event *event,u64 value); +extern void perf_event_set_count (struct perf_event *event,u64 count);extern u64 perf_event_pause (struct perf_event *event,bool reset);#else /* !CONFIG_PERF_EVENTS: */ static inline void *@@ -1760,6 +1761,7 @@ static inline int perf_event_period(struct perf_event *event, u64 value) {return -EINVAL ; } +static inline perf_event_set_count(struct perf_event *event, u64 count) { }static inline u64 perf_event_pause (struct perf_event *event,bool reset) {return 0 ;kernel/events/core.c +11
-0@@ -5646,6 +5646,17 @@ static void _perf_event_reset(struct perf_event *event)perf_event_update_userpage (event); } +void perf_event_set_count (struct perf_event *event,u64 count) +{ +struct perf_event_context *ctx; + + ctx =perf_event_ctx_lock (event); + (void )perf_event_read (event, false); +local64_set (&event->count, count); +perf_event_ctx_unlock (event, ctx); +} +EXPORT_SYMBOL_GPL (perf_event_set_count); +/* Assume it's not an event with inherit set. */ u64 perf_event_pause (struct perf_event *event,bool reset) {
Quoted 2 lines
base-commit:
5ae85a1bd17b959796f6cc4c1153ceada2cf8f24--
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Sean Christopherson
maintainerQuoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }
Quoted 9 lines
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
arch/x86/kvm/pmu.h +6
-0+static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }Quoted 9 lines
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
arch/x86/kvm/pmu.h +6
-0+static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continue
in the guests when perf is used. These messages disappear when 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.
Thanks,
Roman.
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Sean Christopherson
maintainerQuoted 24 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }Quoted 9 lines
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
arch/x86/kvm/pmu.h +6
-0+static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
Quoted 7 lines
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continuein the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.
Can you reverting de0f619564f4 ("KVM: x86/pmu: Defer counter emulated overflow via pmc->prev_counter")? I suspect the problem is the prev_counter mess.
Quoted 30 lines · Sean Christopherson
Quoted 28 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
(quoted diff) +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc) return counter & pmc_bitmask(pmc); }Quoted 9 lines
Review static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
(quoted diff) +6
-0+static inline void pmc_write_counter(struct kvm_pmc *pmc, u64 val) +{ + if (pmc->perf_event && !pmc->is_paused) + perf_event_set_count(pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continue
Quoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Can you reverting
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter")? I suspect the problem is theprev_countermess.
For sure it is prev_counter issue, I have done some instrumentation as follows:
arch/x86/kvm/pmu.c +4 -1
@@ -322,8 +322,11 @@ static void reprogram_counter(struct kvm_pmc *pmc)if (!pmc_event_is_allowed (pmc)) goto reprogram_complete; -if (pmc->counter < pmc->prev_counter) +if (pmc->counter < pmc->prev_counter) { +pr_info ("pmc->counter: %llx\tpmc->prev_counter: %llx\n" , + pmc->counter, pmc->prev_counter);__kvm_perf_overflow (pmc, false); + }if (eventsel &ARCH_PERFMON_EVENTSEL_PIN_CONTROL )printk_once ("kvm pmu: pin control bit is ignored\n" );
I find some interesting changes on prev_counter:
[ +7.295348] pmc->counter: 12 pmc->prev_counter: fffffffffb3d [ +0.622991] pmc->counter: 3 pmc->prev_counter: fffffffffb1a [ +6.943282] pmc->counter: 1 pmc->prev_counter: fffffffff746 [ +4.483523] pmc->counter: 0 pmc->prev_counter: ffffffffffff
[ +12.817772] pmc->counter: 0 pmc->prev_counter: ffffffffffff [ +21.721233] pmc->counter: 0 pmc->prev_counter: ffffffffffff
The first 3 logs will generate this:
[ +11.811925] Uhhuh. NMI received for unknown reason 20 on CPU 2. [ +0.000003] Dazed and confused, but trying to continue
While the last 3 logs won't. This is quite reasonable as looking into de0f619564f4 ("KVM: x86/pmu: Defer counter emulated overflow via pmc->prev_counter"), counter and prev_counter should only have 1 diff in value.
So, the reasonable suspect should be the stale prev_counter. There might be several potential reasons behind this. Jim's theory is the highly reasonable one as I did another experiment and found that KVM may leave pmu->global_status as '0' when injecting an NMI.
Quoted 70 lines · Mingwei Zhang
Quoted 30 lines · Sean Christopherson
Quoted 28 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
(quoted diff) +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc) return counter & pmc_bitmask(pmc); }Quoted 9 lines
Review static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
(quoted diff) +6
-0+static inline void pmc_write_counter(struct kvm_pmc *pmc, u64 val) +{ + if (pmc->perf_event && !pmc->is_paused) + perf_event_set_count(pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continueQuoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Can you reverting
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter")? I suspect the problem is theprev_countermess.For sure it is
prev_counterissue, I have done some instrumentation as follows:arch/x86/kvm/pmu.c +4
-1@@ -322,8 +322,11 @@ static void reprogram_counter(struct kvm_pmc *pmc)if (!pmc_event_is_allowed (pmc)) goto reprogram_complete; -if (pmc->counter < pmc->prev_counter) +if (pmc->counter < pmc->prev_counter) { +pr_info ("pmc->counter: %llx\tpmc->prev_counter: %llx\n" , + pmc->counter, pmc->prev_counter);__kvm_perf_overflow (pmc, false); + }if (eventsel &ARCH_PERFMON_EVENTSEL_PIN_CONTROL )printk_once ("kvm pmu: pin control bit is ignored\n" );I find some interesting changes on
prev_counter:[ +7.295348] pmc->counter: 12 pmc->prev_counter: fffffffffb3d [ +0.622991] pmc->counter: 3 pmc->prev_counter: fffffffffb1a [ +6.943282] pmc->counter: 1 pmc->prev_counter: fffffffff746 [ +4.483523] pmc->counter: 0 pmc->prev_counter: ffffffffffff[ +12.817772]
pmc->counter: 0pmc->prev_counter: ffffffffffff [ +21.721233]pmc->counter: 0pmc->prev_counter: ffffffffffffThe first 3 logs will generate this:
Quoted 7 lines
[ +11.811925] Uhhuh. NMI received for unknown reason 20 on CPU 2. [ +0.000003] Dazed and confused, but trying to continue
While the last 3 logs won't. This is quite reasonable as looking into
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter"), counter andprev_countershould only have 1 diff in value.
prev_counter isn't actually sync'ed at this point, is it? This comes back to that "setting a running counter" nonsense. We want to add 1 to the current counter, but we don't actually know what the current counter is.
My interpretation of the above is that, in the first three cases, PMU hardware has already detected an overflow. In the last three cases, software counting has detected an overflow.
If the last three occur while executing the guest's PMI handler (i.e. NMIs are blocked), then this could corroborate my conjecture about IA32_DEBUGCTL.Freeze_PerfMon_On_PMI.
Quoted 4 lines
So, the reasonable suspect should be the stale
prev_counter. There might be several potential reasons behind this. Jim's theory is the highly reasonable one as I did another experiment and found that KVM may leavepmu->global_statusas '0' when injecting an NMI.
Quoted 85 lines · Jim Mattson
Quoted 70 lines · Mingwei Zhang
Quoted 30 lines · Sean Christopherson
Quoted 28 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
(quoted diff) +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc) return counter & pmc_bitmask(pmc); }Quoted 9 lines
Review static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
(quoted diff) +6
-0+static inline void pmc_write_counter(struct kvm_pmc *pmc, u64 val) +{ + if (pmc->perf_event && !pmc->is_paused) + perf_event_set_count(pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continueQuoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Can you reverting
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter")? I suspect the problem is theprev_countermess.For sure it is
prev_counterissue, I have done some instrumentation as follows:arch/x86/kvm/pmu.c +4
-1@@ -322,8 +322,11 @@ static void reprogram_counter(struct kvm_pmc *pmc)if (!pmc_event_is_allowed (pmc)) goto reprogram_complete; -if (pmc->counter < pmc->prev_counter) +if (pmc->counter < pmc->prev_counter) { +pr_info ("pmc->counter: %llx\tpmc->prev_counter: %llx\n" , + pmc->counter, pmc->prev_counter);__kvm_perf_overflow (pmc, false); + }if (eventsel &ARCH_PERFMON_EVENTSEL_PIN_CONTROL )printk_once ("kvm pmu: pin control bit is ignored\n" );I find some interesting changes on
prev_counter:[ +7.295348] pmc->counter: 12 pmc->prev_counter: fffffffffb3d [ +0.622991] pmc->counter: 3 pmc->prev_counter: fffffffffb1a [ +6.943282] pmc->counter: 1 pmc->prev_counter: fffffffff746 [ +4.483523] pmc->counter: 0 pmc->prev_counter: ffffffffffff[ +12.817772]
pmc->counter: 0pmc->prev_counter: ffffffffffff [ +21.721233]pmc->counter: 0pmc->prev_counter: ffffffffffffThe first 3 logs will generate this:
Quoted 7 lines
[ +11.811925] Uhhuh. NMI received for unknown reason 20 on CPU 2. [ +0.000003] Dazed and confused, but trying to continue
While the last 3 logs won't. This is quite reasonable as looking into
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter"), counter andprev_countershould only have 1 diff in value.
prev_counterisn't actually sync'ed at this point, is it? This comes back to that "setting a running counter" nonsense. We want to add 1 to the current counter, but we don't actually know what the current counter is.
Quoted 8 lines
My interpretation of the above is that, in the first three cases, PMU hardware has already detected an overflow. In the last three cases, software counting has detected an overflow.
If the last three occur while executing the guest's PMI handler (i.e. NMIs are blocked), then this could corroborate my conjecture about IA32_DEBUGCTL.Freeze_PerfMon_On_PMI.
I see. I wonder if we can just do this:
arch/x86/kvm/pmu.c +1 -1
@@ -322,7 +322,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)if (!pmc_event_is_allowed (pmc)) goto reprogram_complete; -if (pmc->counter < pmc->prev_counter) +if (pmc->counter ==0 )__kvm_perf_overflow (pmc, false);if (eventsel &ARCH_PERFMON_EVENTSEL_PIN_CONTROL )
Since this is software emulation, we (KVM) should only handle overflow by plusing one?
Quoted 14 lines
arch/x86/kvm/pmu.c +1
-1@@ -322,7 +322,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)if (!pmc_event_is_allowed (pmc)) goto reprogram_complete; -if (pmc->counter < pmc->prev_counter) +if (pmc->counter ==0 )__kvm_perf_overflow (pmc, false);if (eventsel &ARCH_PERFMON_EVENTSEL_PIN_CONTROL )
Quoted 2 lines
Since this is software emulation, we (KVM) should only handle overflow by plusing one?
Sign. Please ignore this as it is not only hacky but also not working.
Sean Christopherson
maintainerQuoted 29 lines · Sean Christopherson
Quoted 28 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
(quoted diff) +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc) return counter & pmc_bitmask(pmc); }Quoted 9 lines
Review static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
(quoted diff) +6
-0+static inline void pmc_write_counter(struct kvm_pmc *pmc, u64 val) +{ + if (pmc->perf_event && !pmc->is_paused) + perf_event_set_count(pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continue
Quoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Can you reverting
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter")? I suspect the problem is theprev_countermess.
Ugh, yeah, de0f619564f4 created a bit of a mess. The underlying issue that it was solving is that perf_event_read_value() and friends might sleep (yay mutex), and so can't be called from KVM's fastpath (IRQs disabled).
However, detecting overflow requires reading perf_event_read_value() to gather the accumulated count from the hardware event in order to add it to the emulated count from software. E.g. if pmc->counter is X and the perf event counter is Y, KVM needs to factor in Y because X+Y+1 might overflow even if X+1 does not.
Trying to snapshot the previous counter value is a bit of a mess. It could probably made to work, but it's hard to reason about what the snapshot actually contains and when it should be cleared, especially when factoring in the wrapping logic.
Rather than snapshot the previous counter, I think it makes sense to:
1) Track the number of emulated counter events
2) Accumulate and reset the counts from perf_event and emulated_counter into
pmc->counter when pausing the PMC
3) Pause and reprogram the PMC on writes (instead of the current approach of
blindly updating the sample period)
4) Pause the counter when stopping the perf_event to ensure pmc->counter is
fresh (instead of manually updating pmc->counter)IMO, that yields more intuitive logic, and makes it easier to reason about correctness since the behavior is easily define: pmc->counter holds the counts that have been gathered and processed, perf_event and emulated_counter hold outstanding counts on top. E.g. on a WRMSR to the counter, both the emulated counter and the hardware counter are reset, because whatever counts existed previously are irrelevant.
Pausing the counter _might_ make WRMSR slower, but we need to get this all functionally correct before worrying too much about performance.
Diff below for what I'm thinking (needs to be split into multiple patches). It's *very* lightly tested.
I'm about to disappear for a week, I'll pick this back up when I get return. In the meantime, any testing and/or input would be much appreciated!
--- arch/x86/include/asm/kvm-x86-pmu-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 11 ++- arch/x86/kvm/pmu.c | 94 ++++++++++++++++++++++---- arch/x86/kvm/pmu.h | 53 +++------------ arch/x86/kvm/svm/pmu.c | 19 +----- arch/x86/kvm/vmx/pmu_intel.c | 26 +------ 6 files changed, 103 insertions(+), 102 deletions(-)
arch/x86/include/asm/kvm-x86-pmu-ops.h +1 -1
@@ -22,7 +22,7 @@ KVM_X86_PMU_OP(get_msr) KVM_X86_PMU_OP(set_msr) KVM_X86_PMU_OP(refresh) KVM_X86_PMU_OP(init) -KVM_X86_PMU_OP(reset) +KVM_X86_PMU_OP_OPTIONAL(reset) KVM_X86_PMU_OP_OPTIONAL(deliver_pmi) KVM_X86_PMU_OP_OPTIONAL(cleanup)
arch/x86/include/asm/kvm_host.h +10 -1
@@ -492,8 +492,17 @@ struct kvm_pmc {u8 idx;bool is_paused;bool intr; +/* +* Value of the PMC counter that has been gathered from the associated +* perf_event and from emulated_counter. This is *not* the current +* value as seen by the guest or userspace. +*/ u64 counter; -u64 prev_counter; +/* +* PMC events triggered by KVM emulation that haven't been fully +* procssed, e.g. haven't undergone overflow detection. +*/ +u64 emulated_counter;u64 eventsel;struct perf_event *perf_event;struct kvm_vcpu *vcpu;
arch/x86/kvm/pmu.c +79 -15
165 lines not on this page · raw message
arch/x86/kvm/pmu.h +10 -43
84 lines not on this page · raw message
arch/x86/kvm/svm/pmu.c +1 -18
@@ -160,8 +160,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);return 0 ; }/* MSR_EVNTSELn */
@@ -233,21 +232,6 @@ static void amd_pmu_init(struct kvm_vcpu *vcpu) } } -static void amd_pmu_reset (struct kvm_vcpu *vcpu) -{ -struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); -int i; - -for (i =0 ; i <KVM_AMD_PMC_MAX_GENERIC ; i++) { -struct kvm_pmc *pmc = &pmu->gp_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter = pmc->eventsel =0 ; - } - - pmu->global_ctrl = pmu->global_status =0 ; -} -struct kvm_pmu_ops amd_pmu_ops__initdata = { .hw_event_available = amd_hw_event_available, .pmc_idx_to_pmc = amd_pmc_idx_to_pmc,
@@ -259,7 +243,6 @@ struct kvm_pmu_ops amd_pmu_ops __initdata = { .set_msr = amd_pmu_set_msr, .refresh = amd_pmu_refresh, .init = amd_pmu_init, - .reset = amd_pmu_reset, .EVENTSEL_EVENT = AMD64_EVENTSEL_EVENT, .MAX_NR_GP_COUNTERS = KVM_AMD_PMC_MAX_GENERIC, .MIN_NR_GP_COUNTERS = AMD64_NUM_COUNTERS,
arch/x86/kvm/vmx/pmu_intel.c +2 -24
@@ -406,12 +406,10 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) { reserved_bits = pmu->reserved_bits;
@@ -603,26 +601,6 @@ static void intel_pmu_init(struct kvm_vcpu *vcpu)static void intel_pmu_reset (struct kvm_vcpu *vcpu) { -struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); -struct kvm_pmc *pmc =NULL ; -int i; - -for (i =0 ; i <KVM_INTEL_PMC_MAX_GENERIC ; i++) { - pmc = &pmu->gp_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter = pmc->eventsel =0 ; - } - -for (i =0 ; i <KVM_PMC_MAX_FIXED ; i++) { - pmc = &pmu->fixed_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter =0 ; - } - - pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status =0 ; -intel_pmu_release_guest_lbr_event (vcpu); }
base-commit: 88bb466c9dec4f70d682cf38c685324e7b1b3d60
Quoted 67 lines · Sean Christopherson
Quoted 29 lines · Sean Christopherson
Quoted 28 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }Quoted 9 lines
Review static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
(quoted diff) +6
-0+static inline void pmc_write_counter(struct kvm_pmc *pmc, u64 val) +{ + if (pmc->perf_event && !pmc->is_paused) + perf_event_set_count(pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continueQuoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Can you reverting
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter")? I suspect the problem is theprev_countermess.Ugh, yeah,
de0f619564f4created a bit of a mess. The underlying issue that it was solving is thatperf_event_read_value()and friends might sleep (yay mutex), and so can't be called from KVM's fastpath (IRQs disabled).However, detecting overflow requires reading
perf_event_read_value()to gather the accumulated count from the hardware event in order to add it to the emulated count from software. E.g. ifpmc->counteris X and the perf event counter is Y, KVM needs to factor in Y because X+Y+1 might overflow even if X+1 does not.Trying to snapshot the previous counter value is a bit of a mess. It could probably made to work, but it's hard to reason about what the snapshot actually contains and when it should be cleared, especially when factoring in the wrapping logic.
Rather than snapshot the previous counter, I think it makes sense to:
1) Track the number of emulated counter events
2) Accumulate and reset the counts fromperf_eventandemulated_counterintopmc->counter when pausing the PMC 3) Pause and reprogram the PMC on writes (instead of the current approach of blindly updating the sample period) 4) Pause the counter when stopping the perf_event to ensure pmc->counter is fresh (instead of manually updating pmc->counter)IMO, that yields more intuitive logic, and makes it easier to reason about correctness since the behavior is easily define:
pmc->counterholds the counts that have been gathered and processed,perf_eventandemulated_counterhold outstanding counts on top. E.g. on a WRMSR to the counter, both the emulated counter and the hardware counter are reset, because whatever counts existed previously are irrelevant.
Quoted 9 lines
Pausing the counter _might_ make WRMSR slower, but we need to get this all functionally correct before worrying too much about performance.
Diff below for what I'm thinking (needs to be split into multiple patches). It's *very* lightly tested.
I'm about to disappear for a week, I'll pick this back up when I get return. In the meantime, any testing and/or input would be much appreciated!
Sean, I have tested this change with QEMU on the tip of kvmx86/next. hmm, my observation is that this one still causes spurious NMIs. The volume of spurious NMIs are even more...
Command I am using inside VM:/usr/bin/perf record -N -B -T --sample-cpu --compression-level=1 --threads --clockid=CLOCK_MONOTONIC_RAW -e $events
Events:
cpu/period=0xcdfe60,event=0x3c,name='CPU_CLK_UNHALTED.THREAD'/Duk cpu/period=0xcdfe60,umask=0x3,name='CPU_CLK_UNHALTED.REF_TSC'/Duk cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk cpu/period=0xcdfe60,event=0xec,umask=0x2,name='CPU_CLK_UNHALTED.DISTRIBUTED'/uk cpu/period=0x98968f,event=0x3c,name='CPU_CLK_UNHALTED.THREAD_P'/uk cpu/period=0x98968f,event=0xa6,umask=0x21,cmask=0x5,name='EXE_ACTIVITY.BOUND_ON_LOADS'/uk cpu/period=0x4c4b4f,event=0x9c,umask=0x1,name='IDQ_UOPS_NOT_DELIVERED.CORE'/uk cpu/period=0x4c4b4f,event=0xad,umask=0x10,name='INT_MISC.UOP_DROPPING'/uk cpu/period=0x4c4b4f,event=0x47,umask=0x5,cmask=0x5,name='MEMORY_ACTIVITY.STALLS_L2_MISS'/uk cpu/period=0x7a143,event=0xd1,umask=0x40,name='MEM_LOAD_RETIRED.FB_HIT'/uk cpu/period=0xf424f,event=0xd1,umask=0x8,name='MEM_LOAD_RETIRED.L1_MISS'/uk cpu/period=0x4c4b4f,event=0x20,umask=0x8,cmask=0x4,name='OFFCORE_REQUESTS_OUTSTANDING.ALL_DATA_RD_cmask_4'/uk cpu/period=0x2faf090,event=0xa4,umask=0x2,name='TOPDOWN.BACKEND_BOUND_SLOTS'/uk cpu/period=0x2faf090,event=0xa4,umask=0x10,name='TOPDOWN.MEMORY_BOUND_SLOTS'/uk cpu/period=0x98968f,event=0xc2,umask=0x2,name='UOPS_RETIRED.SLOTS'/uk cpu/period=0x7a12f,event=0xd0,umask=0x82,name='MEM_INST_RETIRED.ALL_STORES'/uk cpu/period=0x7a12f,event=0xd0,umask=0x81,name='MEM_INST_RETIRED.ALL_LOADS'/uk
In addition, I noticed a periodic performance overhead at the host level (invisible to guest). It seems all vCPU threads are spending 60% of the time on the mutex inside perf_event_contex. So a significant amount of time was spend on kvm_pmu_handle_event() and in particular perf_event_ctx_lock{,_nested}() and perf_event_ctx_unlock() for a couple of seconds periodically.
Looking deeper, the perf_event_context might be
the perf_cpu_context->task_ctx. But still I cannot convince myself that all vCPU threads may be competing against this one, since it is still should be a per task_struct mutext...
Anyway, the performance issue may be off the current topic but it persist across many kernel versions when using vPMU with the above event set in sampling mode.
Quoted 411 lines
--- arch/x86/include/asm/kvm-x86-pmu-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 11 ++- arch/x86/kvm/pmu.c | 94 ++++++++++++++++++++++---- arch/x86/kvm/pmu.h | 53 +++------------ arch/x86/kvm/svm/pmu.c | 19 +----- arch/x86/kvm/vmx/pmu_intel.c | 26 +------ 6 files changed, 103 insertions(+), 102 deletions(-)arch/x86/include/asm/kvm-x86-pmu-ops.h +1
-1@@ -22,7 +22,7 @@ KVM_X86_PMU_OP(get_msr) KVM_X86_PMU_OP(set_msr) KVM_X86_PMU_OP(refresh) KVM_X86_PMU_OP(init) -KVM_X86_PMU_OP(reset) +KVM_X86_PMU_OP_OPTIONAL(reset) KVM_X86_PMU_OP_OPTIONAL(deliver_pmi) KVM_X86_PMU_OP_OPTIONAL(cleanup)arch/x86/include/asm/kvm_host.h +10
-1@@ -492,8 +492,17 @@ struct kvm_pmc {u8 idx;bool is_paused;bool intr; +/* +* Value of the PMC counter that has been gathered from the associated +* perf_event and from emulated_counter. This is *not* the current +* value as seen by the guest or userspace. +*/ u64 counter; -u64 prev_counter; +/* +* PMC events triggered by KVM emulation that haven't been fully +* procssed, e.g. haven't undergone overflow detection. +*/ +u64 emulated_counter;u64 eventsel;struct perf_event *perf_event;struct kvm_vcpu *vcpu;arch/x86/kvm/pmu.c +79
-15@@ -148,9 +148,9 @@ static void kvm_perf_overflow(struct perf_event *perf_event,struct kvm_pmc *pmc = perf_event->overflow_handler_context;/* -* Ignore overflow events for counters that are scheduled to be -* reprogrammed, e.g. if a PMI for the previous event races with KVM's -* handling of a related guest WRMSR. +* Ignore asynchronous overflow events for counters that are scheduled +* to be reprogrammed, e.g. if a PMI for the previous event races with +* KVM's handling of a related guest WRMSR. */ if (test_and_set_bit (pmc->idx,pmc_to_pmu (pmc)->reprogram_pmi))return ;@@ -182,6 +182,21 @@ static u64 pmc_get_pebs_precise_level(struct kvm_pmc *pmc)return 1 ; } +static u64 pmc_get_sample_period (struct kvm_pmc *pmc) +{ +u64 sample_period = (-pmc->counter) &pmc_bitmask (pmc); + +/* +* Verify pmc->counter is fresh, i.e. that the perf event is paused and +* emulated events have been gathered. +*/ +WARN_ON_ONCE (pmc->emulated_counter || (pmc->perf_event && !pmc->is_paused)); + +if (!sample_period) + sample_period =pmc_bitmask (pmc) +1 ; +return sample_period; +} +static int pmc_reprogram_counter (struct kvm_pmc *pmc,u32 type,u64 config,bool exclude_user,bool exclude_kernel,bool intr)@@ -200,7 +215,7 @@ static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config, };bool pebs =test_bit (pmc->idx, (unsigned long *)&pmu->pebs_enable); - attr.sample_period =get_sample_period (pmc, pmc->counter); + attr.sample_period =pmc_get_sample_period (pmc);if ((attr.config &HSW_IN_TX_CHECKPOINTED ) &&guest_cpuid_is_intel (pmc->vcpu)) {@@ -238,13 +253,19 @@ static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config,static void pmc_pause_counter (struct kvm_pmc *pmc) { -u64 counter = pmc->counter; +/* +* Accumulate emulated events, even if the PMC was already paused, e.g. +* if KVM emulated an event after a WRMSR, but before reprogramming, or +* if KVM couldn't create a perf event. +*/ +u64 counter = pmc->counter + pmc->emulated_counter; -if (!pmc->perf_event || pmc->is_paused) -return ; + pmc->emulated_counter =0 ;/* update counter, reset event value to avoid redundant accumulation */ - counter +=perf_event_pause (pmc->perf_event, true); +if (pmc->perf_event && !pmc->is_paused) + counter +=perf_event_pause (pmc->perf_event, true); + pmc->counter = counter &pmc_bitmask (pmc); pmc->is_paused = true; }@@ -256,8 +277,7 @@ static bool pmc_resume_counter(struct kvm_pmc *pmc)/* recalibrate sample period and check if it's accepted by perf core */ if (is_sampling_event (pmc->perf_event) && -perf_event_period (pmc->perf_event, -get_sample_period (pmc, pmc->counter))) +perf_event_period (pmc->perf_event,pmc_get_sample_period (pmc)))return false; if (test_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->pebs_enable) !=@@ -395,6 +415,32 @@ static bool check_pmu_event_filter(struct kvm_pmc *pmc)return is_fixed_event_allowed (filter, pmc->idx); } +void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +pmc_pause_counter (pmc); + pmc->counter = val &pmc_bitmask (pmc); +kvm_pmu_request_counter_reprogram (pmc); +} +EXPORT_SYMBOL_GPL (pmc_write_counter); + +static void pmc_release_perf_event (struct kvm_pmc *pmc) +{ +if (pmc->perf_event) { +perf_event_release_kernel (pmc->perf_event); + pmc->perf_event =NULL ; + pmc->current_config =0 ; +pmc_to_pmu (pmc)->event_count--; + } +} + +static void pmc_stop_counter (struct kvm_pmc *pmc) +{ +if (pmc->perf_event) { +pmc_pause_counter (pmc); +pmc_release_perf_event (pmc); + } +} +static bool pmc_event_is_allowed (struct kvm_pmc *pmc) { return pmc_is_globally_enabled(pmc) && pmc_speculative_in_use(pmc) &&@@ -404,6 +450,7 @@ static bool pmc_event_is_allowed(struct kvm_pmc *pmc)static void reprogram_counter (struct kvm_pmc *pmc) {struct kvm_pmu *pmu =pmc_to_pmu (pmc); +u64 prev_counter = pmc->counter;u64 eventsel = pmc->eventsel;u64 new_config = eventsel;u8 fixed_ctr_ctrl;@@ -413,7 +460,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)if (!pmc_event_is_allowed (pmc)) goto reprogram_complete; -if (pmc->counter < pmc->prev_counter) +if (pmc->counter < prev_counter)__kvm_perf_overflow (pmc, false);if (eventsel &ARCH_PERFMON_EVENTSEL_PIN_CONTROL )@@ -453,7 +500,6 @@ static void reprogram_counter(struct kvm_pmc *pmc) reprogram_complete:clear_bit (pmc->idx, (unsigned long *)&pmc_to_pmu (pmc)->reprogram_pmi); - pmc->prev_counter =0 ; }void kvm_pmu_handle_event (struct kvm_vcpu *vcpu)@@ -678,9 +724,28 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)void kvm_pmu_reset (struct kvm_vcpu *vcpu) {struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); +struct kvm_pmc *pmc; +int i;irq_work_sync (&pmu->irq_work); -static_call (kvm_x86_pmu_reset)(vcpu); + +bitmap_zero (pmu->reprogram_pmi,X86_PMC_IDX_MAX ); + +for_each_set_bit (i, pmu->all_valid_pmc_idx,X86_PMC_IDX_MAX ) { + pmc =static_call (kvm_x86_pmu_pmc_idx_to_pmc)(pmu, i); +if (!pmc) +continue ; + +pmc_stop_counter (pmc); + pmc->counter =0 ; + +if (pmc_is_gp (pmc)) + pmc->eventsel =0 ; + }; + + pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status =0 ; + +static_call_cond (kvm_x86_pmu_reset)(vcpu); }void kvm_pmu_init (struct kvm_vcpu *vcpu)@@ -727,8 +792,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)static void kvm_pmu_incr_counter (struct kvm_pmc *pmc) { - pmc->prev_counter = pmc->counter; - pmc->counter = (pmc->counter +1 ) &pmc_bitmask (pmc); + pmc->emulated_counter++;kvm_pmu_request_counter_reprogram (pmc); }arch/x86/kvm/pmu.h +10
-43@@ -55,6 +55,12 @@ static inline bool kvm_pmu_has_perf_global_ctrl(struct kvm_pmu *pmu)return pmu->version >1 ; } +static inline void kvm_pmu_request_counter_reprogram (struct kvm_pmc *pmc) +{ +set_bit (pmc->idx,pmc_to_pmu (pmc)->reprogram_pmi); +kvm_make_request (KVM_REQ_PMU , pmc->vcpu); +} +static inline u64 pmc_bitmask (struct kvm_pmc *pmc) {struct kvm_pmu *pmu =pmc_to_pmu (pmc);@@ -66,31 +72,17 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc) {u64 counter, enabled, running; - counter = pmc->counter; + counter = pmc->counter + pmc->emulated_counter; +if (pmc->perf_event && !pmc->is_paused) counter +=perf_event_read_value (pmc->perf_event, &enabled, &running); +/* FIXME: Scaling needed? */ return counter &pmc_bitmask (pmc); } -static inline void pmc_release_perf_event (struct kvm_pmc *pmc) -{ -if (pmc->perf_event) { -perf_event_release_kernel (pmc->perf_event); - pmc->perf_event =NULL ; - pmc->current_config =0 ; -pmc_to_pmu (pmc)->event_count--; - } -} - -static inline void pmc_stop_counter (struct kvm_pmc *pmc) -{ -if (pmc->perf_event) { - pmc->counter =pmc_read_counter (pmc); -pmc_release_perf_event (pmc); - } -} +void pmc_write_counter (struct kvm_pmc *pmc,u64 val);static inline bool pmc_is_gp (struct kvm_pmc *pmc) {@@ -140,25 +132,6 @@ static inline struct kvm_pmc *get_fixed_pmc(struct kvm_pmu *pmu, u32 msr)return NULL ; } -static inline u64 get_sample_period (struct kvm_pmc *pmc,u64 counter_value) -{ -u64 sample_period = (-counter_value) &pmc_bitmask (pmc); - -if (!sample_period) - sample_period =pmc_bitmask (pmc) +1 ; -return sample_period; -} - -static inline void pmc_update_sample_period (struct kvm_pmc *pmc) -{ -if (!pmc->perf_event || pmc->is_paused || - !is_sampling_event (pmc->perf_event)) -return ; - -perf_event_period (pmc->perf_event, -get_sample_period (pmc, pmc->counter)); -} -static inline bool pmc_speculative_in_use (struct kvm_pmc *pmc) {struct kvm_pmu *pmu =pmc_to_pmu (pmc);@@ -214,12 +187,6 @@ static inline void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops) KVM_PMC_MAX_FIXED); } -static inline void kvm_pmu_request_counter_reprogram (struct kvm_pmc *pmc) -{ -set_bit (pmc->idx,pmc_to_pmu (pmc)->reprogram_pmi); -kvm_make_request (KVM_REQ_PMU , pmc->vcpu); -} -static inline void reprogram_counters (struct kvm_pmu *pmu,u64 diff) {int bit;arch/x86/kvm/svm/pmu.c +1
-18@@ -160,8 +160,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);return 0 ; }/* MSR_EVNTSELn */ @@ -233,21 +232,6 @@ static void amd_pmu_init(struct kvm_vcpu *vcpu) } } -static void amd_pmu_reset (struct kvm_vcpu *vcpu) -{ -struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); -int i; - -for (i =0 ; i <KVM_AMD_PMC_MAX_GENERIC ; i++) { -struct kvm_pmc *pmc = &pmu->gp_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter = pmc->eventsel =0 ; - } - - pmu->global_ctrl = pmu->global_status =0 ; -} -struct kvm_pmu_ops amd_pmu_ops__initdata = { .hw_event_available = amd_hw_event_available, .pmc_idx_to_pmc = amd_pmc_idx_to_pmc,@@ -259,7 +243,6 @@ struct kvm_pmu_ops amd_pmu_ops __initdata = { .set_msr = amd_pmu_set_msr, .refresh = amd_pmu_refresh, .init = amd_pmu_init, - .reset = amd_pmu_reset, .EVENTSEL_EVENT = AMD64_EVENTSEL_EVENT, .MAX_NR_GP_COUNTERS = KVM_AMD_PMC_MAX_GENERIC, .MIN_NR_GP_COUNTERS = AMD64_NUM_COUNTERS,arch/x86/kvm/vmx/pmu_intel.c +2
-24@@ -406,12 +406,10 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) { reserved_bits = pmu->reserved_bits;@@ -603,26 +601,6 @@ static void intel_pmu_init(struct kvm_vcpu *vcpu)static void intel_pmu_reset (struct kvm_vcpu *vcpu) { -struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); -struct kvm_pmc *pmc =NULL ; -int i; - -for (i =0 ; i <KVM_INTEL_PMC_MAX_GENERIC ; i++) { - pmc = &pmu->gp_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter = pmc->eventsel =0 ; - } - -for (i =0 ; i <KVM_PMC_MAX_FIXED ; i++) { - pmc = &pmu->fixed_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter =0 ; - } - - pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status =0 ; -intel_pmu_release_guest_lbr_event (vcpu); }base-commit:
88bb466c9dec4f70d682cf38c685324e7b1b3d60
Quoted 71 lines · Sean Christopherson
Date: Fri, 30 Jun 2023 14:26:02 -0700
From: Sean Christopherson <[email protected]>
Subject: Re: [PATCH] KVM: x86: vPMU: truncate counter value to allowed widthQuoted 29 lines · Sean Christopherson
Quoted 28 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }Quoted 9 lines
Review static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
(quoted diff) +6
-0+static inline void pmc_write_counter(struct kvm_pmc *pmc, u64 val) +{ + if (pmc->perf_event && !pmc->is_paused) + perf_event_set_count(pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continueQuoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Can you reverting
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter")? I suspect the problem is theprev_countermess.Ugh, yeah,
de0f619564f4created a bit of a mess. The underlying issue that it was solving is thatperf_event_read_value()and friends might sleep (yay mutex), and so can't be called from KVM's fastpath (IRQs disabled).However, detecting overflow requires reading
perf_event_read_value()to gather the accumulated count from the hardware event in order to add it to the emulated count from software. E.g. ifpmc->counteris X and the perf event counter is Y, KVM needs to factor in Y because X+Y+1 might overflow even if X+1 does not.Trying to snapshot the previous counter value is a bit of a mess. It could probably made to work, but it's hard to reason about what the snapshot actually contains and when it should be cleared, especially when factoring in the wrapping logic.
Rather than snapshot the previous counter, I think it makes sense to:
1) Track the number of emulated counter events
2) Accumulate and reset the counts fromperf_eventandemulated_counterintopmc->counter when pausing the PMC 3) Pause and reprogram the PMC on writes (instead of the current approach of blindly updating the sample period) 4) Pause the counter when stopping the perf_event to ensure pmc->counter is fresh (instead of manually updating pmc->counter)IMO, that yields more intuitive logic, and makes it easier to reason about correctness since the behavior is easily define:
pmc->counterholds the counts that have been gathered and processed,perf_eventandemulated_counterhold outstanding counts on top. E.g. on a WRMSR to the counter, both the emulated counter and the hardware counter are reset, because whatever counts existed previously are irrelevant.
Quoted 8 lines
Pausing the counter _might_ make WRMSR slower, but we need to get this all functionally correct before worrying too much about performance.
Diff below for what I'm thinking (needs to be split into multiple patches). It's *very* lightly tested.
I'm about to disappear for a week, I'll pick this back up when I get return. In the meantime, any testing and/or input would be much appreciated!
We also observed this PMI injection deadloop issue, especially when the counters are in non-full-width mode and the MSB of counter value is 1, the deadloop issue is quite easily triggered. The PMI injection deadloop would cause PMI storm and exhaust all CPU resource eventually.
We verified either Roman or Sean's patch can fix this PMI deadloop issue on Intel Sapphire Rapids platform.
The Perf command in validation comes from Mingwei and it is
sudo ./perf record -N -B -T --sample-cpu \
--clockid=CLOCK_MONOTONIC_RAW \ -e cpu/period=0xcdfe60,event=0x3c,name='CPU_CLK_UNHALTED.THREAD'/Duk \ -e cpu/period=0xcdfe60,umask=0x3,name='CPU_CLK_UNHALTED.REF_TSC'/Duk \ -e cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY'/Duk \ -e cpu/period=0xcdfe60,event=0xec,umask=0x2,name='CPU_CLK_UNHALTED.DISTRIBUTED'/uk \ -e cpu/period=0x98968f,event=0x3c,name='CPU_CLK_UNHALTED.THREAD_P'/uk \ -e cpu/period=0x98968f,event=0xa6,umask=0x21,cmask=0x5,name='EXE_ACTIVITY.BOUND_ON_LOADS'/uk \ -e cpu/period=0x4c4b4f,event=0x9c,umask=0x1,name='IDQ_UOPS_NOT_DELIVERED.CORE'/uk \ -e cpu/period=0x4c4b4f,event=0xad,umask=0x10,name='INT_MISC.UOP_DROPPING'/uk \ -e cpu/period=0x4c4b4f,event=0x47,umask=0x3,cmask=0x3,name='MEMORY_ACTIVITY.STALLS_L1D_MISS'/uk \ -e cpu/period=0x4c4b4f,event=0x47,umask=0x5,cmask=0x5,name='MEMORY_ACTIVITY.STALLS_L2_MISS'/uk \ -e cpu/period=0x4c4b4f,event=0x47,umask=0x9,cmask=0x9,name='MEMORY_ACTIVITY.STALLS_L3_MISS'/uk \ -e cpu/period=0x7a143,event=0xd3,umask=0x1,name='MEM_LOAD_L3_MISS_RETIRED.LOCAL_DRAM'/uk \ -e cpu/period=0x4c4b4f,event=0xd3,umask=0x2,name='MEM_LOAD_L3_MISS_RETIRED.REMOTE_DRAM'/uk \ -e cpu/period=0x7a143,event=0xd3,umask=0x8,name='MEM_LOAD_L3_MISS_RETIRED.REMOTE_FWD'/uk \ -e cpu/period=0x4c4b4f,event=0xd3,umask=0x4,name='MEM_LOAD_L3_MISS_RETIRED.REMOTE_HITM'/uk \ -e cpu/period=0x7a143,event=0xd3,umask=0x10,name='MEM_LOAD_L3_MISS_RETIRED.REMOTE_PMM'/uk \ -e cpu/period=0x7a143,event=0xd1,umask=0x40,name='MEM_LOAD_RETIRED.FB_HIT'/uk \ -e cpu/period=0x4c4b4f,event=0xd1,umask=0x1,name='MEM_LOAD_RETIRED.L1_HIT'/uk \ -e cpu/period=0xf424f,event=0xd1,umask=0x8,name='MEM_LOAD_RETIRED.L1_MISS'/uk \ -e cpu/period=0xf424f,event=0xd1,umask=0x2,name='MEM_LOAD_RETIRED.L2_HIT'/uk \ -e cpu/period=0x7a189,event=0xd1,umask=0x4,name='MEM_LOAD_RETIRED.L3_HIT'/uk \ -e cpu/period=0x3d0f9,event=0xd1,umask=0x20,name='MEM_LOAD_RETIRED.L3_MISS'/uk \ -e cpu/period=0x4c4b4f,event=0xd1,umask=0x80,name='MEM_LOAD_RETIRED.LOCAL_PMM'/uk \ -e cpu/period=0x4c4b4f,event=0x20,umask=0x8,cmask=0x4,name='OFFCORE_REQUESTS_OUTSTANDING.ALL_DATA_RD_cmask_4'/uk \ -e cpu/period=0x4c4b4f,event=0x20,umask=0x8,cmask=0x1,name='OFFCORE_REQUESTS_OUTSTANDING.CYCLES_WITH_DATA_RD'/uk \ -e cpu/period=0x2faf090,event=0xa4,umask=0x2,name='TOPDOWN.BACKEND_BOUND_SLOTS'/uk \ -e cpu/period=0x2faf090,event=0xa4,umask=0x10,name='TOPDOWN.MEMORY_BOUND_SLOTS'/uk \ -e cpu/period=0x98968f,event=0xc2,umask=0x2,name='UOPS_RETIRED.SLOTS'/uk \ -e cpu/period=0x7a12f,event=0xd0,umask=0x82,name='MEM_INST_RETIRED.ALL_STORES'/uk \ -e cpu/period=0x7a12f,event=0xd0,umask=0x81,name='MEM_INST_RETIRED.ALL_LOADS'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x2,name='FP_ARITH_INST_RETIRED.SCALAR_SINGLE'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x8,name='FP_ARITH_INST_RETIRED.128B_PACKED_SINGLE'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x20,name='FP_ARITH_INST_RETIRED.256B_PACKED_SINGLE'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x1,name='FP_ARITH_INST_RETIRED.SCALAR_DOUBLE'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x4,name='FP_ARITH_INST_RETIRED.128B_PACKED_DOUBLE'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x10,name='FP_ARITH_INST_RETIRED.256B_PACKED_DOUBLE'/uk \ -e cpu/period=0x98968f,event=0xb1,umask=0x10,name='UOPS_EXECUTED.X87'/uk \ -e cpu/period=0x98968f,event=0xb1,umask=0x1,name='UOPS_EXECUTED.THREAD'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x40,name='FP_ARITH_INST_RETIRED.512B_PACKED_DOUBLE'/uk \ -e cpu/period=0x98968f,event=0xc7,umask=0x80,name='FP_ARITH_INST_RETIRED.512B_PACKED_SINGLE'/uk \ -e cpu/period=0x98968f,event=0xce,umask=0x1,name='AMX_OPS_RETIRED.INT8'/uk \ -e cpu/period=0x98968f,event=0xce,umask=0x2,name='AMX_OPS_RETIRED.BF16'/uk \ -e cpu/period=0x98968f,event=0xcf,umask=0x1,name='FP_ARITH_INST_RETIRED2.SCALAR_HALF'/uk \ -e cpu/period=0x98968f,event=0xcf,umask=0x4,name='FP_ARITH_INST_RETIRED2.128B_PACKED_HALF'/uk \ -e cpu/period=0x98968f,event=0xcf,umask=0x8,name='FP_ARITH_INST_RETIRED2.256B_PACKED_HALF'/uk \ -e cpu/period=0x98968f,event=0xcf,umask=0x10,name='FP_ARITH_INST_RETIRED2.512B_PACKED_HALF'/uk \ -e cpu/period=0xcdfe60,event=0xc0,name='INST_RETIRED.ANY_P'/uk \
-e cpu/period=0x2faf090,event=0xa4,umask=0x1,name='TOPDOWN.SLOTS_P'/uk
Quoted 412 lines
--- arch/x86/include/asm/kvm-x86-pmu-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 11 ++- arch/x86/kvm/pmu.c | 94 ++++++++++++++++++++++---- arch/x86/kvm/pmu.h | 53 +++------------ arch/x86/kvm/svm/pmu.c | 19 +----- arch/x86/kvm/vmx/pmu_intel.c | 26 +------ 6 files changed, 103 insertions(+), 102 deletions(-)arch/x86/include/asm/kvm-x86-pmu-ops.h +1
-1@@ -22,7 +22,7 @@ KVM_X86_PMU_OP(get_msr) KVM_X86_PMU_OP(set_msr) KVM_X86_PMU_OP(refresh) KVM_X86_PMU_OP(init) -KVM_X86_PMU_OP(reset) +KVM_X86_PMU_OP_OPTIONAL(reset) KVM_X86_PMU_OP_OPTIONAL(deliver_pmi) KVM_X86_PMU_OP_OPTIONAL(cleanup)arch/x86/include/asm/kvm_host.h +10
-1@@ -492,8 +492,17 @@ struct kvm_pmc {u8 idx;bool is_paused;bool intr; +/* +* Value of the PMC counter that has been gathered from the associated +* perf_event and from emulated_counter. This is *not* the current +* value as seen by the guest or userspace. +*/ u64 counter; -u64 prev_counter; +/* +* PMC events triggered by KVM emulation that haven't been fully +* procssed, e.g. haven't undergone overflow detection. +*/ +u64 emulated_counter;u64 eventsel;struct perf_event *perf_event;struct kvm_vcpu *vcpu;arch/x86/kvm/pmu.c +79
-15@@ -148,9 +148,9 @@ static void kvm_perf_overflow(struct perf_event *perf_event,struct kvm_pmc *pmc = perf_event->overflow_handler_context;/* -* Ignore overflow events for counters that are scheduled to be -* reprogrammed, e.g. if a PMI for the previous event races with KVM's -* handling of a related guest WRMSR. +* Ignore asynchronous overflow events for counters that are scheduled +* to be reprogrammed, e.g. if a PMI for the previous event races with +* KVM's handling of a related guest WRMSR. */ if (test_and_set_bit (pmc->idx,pmc_to_pmu (pmc)->reprogram_pmi))return ;@@ -182,6 +182,21 @@ static u64 pmc_get_pebs_precise_level(struct kvm_pmc *pmc)return 1 ; } +static u64 pmc_get_sample_period (struct kvm_pmc *pmc) +{ +u64 sample_period = (-pmc->counter) &pmc_bitmask (pmc); + +/* +* Verify pmc->counter is fresh, i.e. that the perf event is paused and +* emulated events have been gathered. +*/ +WARN_ON_ONCE (pmc->emulated_counter || (pmc->perf_event && !pmc->is_paused)); + +if (!sample_period) + sample_period =pmc_bitmask (pmc) +1 ; +return sample_period; +} +static int pmc_reprogram_counter (struct kvm_pmc *pmc,u32 type,u64 config,bool exclude_user,bool exclude_kernel,bool intr)@@ -200,7 +215,7 @@ static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config, };bool pebs =test_bit (pmc->idx, (unsigned long *)&pmu->pebs_enable); - attr.sample_period =get_sample_period (pmc, pmc->counter); + attr.sample_period =pmc_get_sample_period (pmc);if ((attr.config &HSW_IN_TX_CHECKPOINTED ) &&guest_cpuid_is_intel (pmc->vcpu)) {@@ -238,13 +253,19 @@ static int pmc_reprogram_counter(struct kvm_pmc *pmc, u32 type, u64 config,static void pmc_pause_counter (struct kvm_pmc *pmc) { -u64 counter = pmc->counter; +/* +* Accumulate emulated events, even if the PMC was already paused, e.g. +* if KVM emulated an event after a WRMSR, but before reprogramming, or +* if KVM couldn't create a perf event. +*/ +u64 counter = pmc->counter + pmc->emulated_counter; -if (!pmc->perf_event || pmc->is_paused) -return ; + pmc->emulated_counter =0 ;/* update counter, reset event value to avoid redundant accumulation */ - counter +=perf_event_pause (pmc->perf_event, true); +if (pmc->perf_event && !pmc->is_paused) + counter +=perf_event_pause (pmc->perf_event, true); + pmc->counter = counter &pmc_bitmask (pmc); pmc->is_paused = true; }@@ -256,8 +277,7 @@ static bool pmc_resume_counter(struct kvm_pmc *pmc)/* recalibrate sample period and check if it's accepted by perf core */ if (is_sampling_event (pmc->perf_event) && -perf_event_period (pmc->perf_event, -get_sample_period (pmc, pmc->counter))) +perf_event_period (pmc->perf_event,pmc_get_sample_period (pmc)))return false; if (test_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->pebs_enable) !=@@ -395,6 +415,32 @@ static bool check_pmu_event_filter(struct kvm_pmc *pmc)return is_fixed_event_allowed (filter, pmc->idx); } +void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +pmc_pause_counter (pmc); + pmc->counter = val &pmc_bitmask (pmc); +kvm_pmu_request_counter_reprogram (pmc); +} +EXPORT_SYMBOL_GPL (pmc_write_counter); + +static void pmc_release_perf_event (struct kvm_pmc *pmc) +{ +if (pmc->perf_event) { +perf_event_release_kernel (pmc->perf_event); + pmc->perf_event =NULL ; + pmc->current_config =0 ; +pmc_to_pmu (pmc)->event_count--; + } +} + +static void pmc_stop_counter (struct kvm_pmc *pmc) +{ +if (pmc->perf_event) { +pmc_pause_counter (pmc); +pmc_release_perf_event (pmc); + } +} +static bool pmc_event_is_allowed (struct kvm_pmc *pmc) { return pmc_is_globally_enabled(pmc) && pmc_speculative_in_use(pmc) &&@@ -404,6 +450,7 @@ static bool pmc_event_is_allowed(struct kvm_pmc *pmc)static void reprogram_counter (struct kvm_pmc *pmc) {struct kvm_pmu *pmu =pmc_to_pmu (pmc); +u64 prev_counter = pmc->counter;u64 eventsel = pmc->eventsel;u64 new_config = eventsel;u8 fixed_ctr_ctrl;@@ -413,7 +460,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)if (!pmc_event_is_allowed (pmc)) goto reprogram_complete; -if (pmc->counter < pmc->prev_counter) +if (pmc->counter < prev_counter)__kvm_perf_overflow (pmc, false);if (eventsel &ARCH_PERFMON_EVENTSEL_PIN_CONTROL )@@ -453,7 +500,6 @@ static void reprogram_counter(struct kvm_pmc *pmc) reprogram_complete:clear_bit (pmc->idx, (unsigned long *)&pmc_to_pmu (pmc)->reprogram_pmi); - pmc->prev_counter =0 ; }void kvm_pmu_handle_event (struct kvm_vcpu *vcpu)@@ -678,9 +724,28 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)void kvm_pmu_reset (struct kvm_vcpu *vcpu) {struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); +struct kvm_pmc *pmc; +int i;irq_work_sync (&pmu->irq_work); -static_call (kvm_x86_pmu_reset)(vcpu); + +bitmap_zero (pmu->reprogram_pmi,X86_PMC_IDX_MAX ); + +for_each_set_bit (i, pmu->all_valid_pmc_idx,X86_PMC_IDX_MAX ) { + pmc =static_call (kvm_x86_pmu_pmc_idx_to_pmc)(pmu, i); +if (!pmc) +continue ; + +pmc_stop_counter (pmc); + pmc->counter =0 ; + +if (pmc_is_gp (pmc)) + pmc->eventsel =0 ; + }; + + pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status =0 ; + +static_call_cond (kvm_x86_pmu_reset)(vcpu); }void kvm_pmu_init (struct kvm_vcpu *vcpu)@@ -727,8 +792,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)static void kvm_pmu_incr_counter (struct kvm_pmc *pmc) { - pmc->prev_counter = pmc->counter; - pmc->counter = (pmc->counter +1 ) &pmc_bitmask (pmc); + pmc->emulated_counter++;kvm_pmu_request_counter_reprogram (pmc); }arch/x86/kvm/pmu.h +10
-43@@ -55,6 +55,12 @@ static inline bool kvm_pmu_has_perf_global_ctrl(struct kvm_pmu *pmu)return pmu->version >1 ; } +static inline void kvm_pmu_request_counter_reprogram (struct kvm_pmc *pmc) +{ +set_bit (pmc->idx,pmc_to_pmu (pmc)->reprogram_pmi); +kvm_make_request (KVM_REQ_PMU , pmc->vcpu); +} +static inline u64 pmc_bitmask (struct kvm_pmc *pmc) {struct kvm_pmu *pmu =pmc_to_pmu (pmc);@@ -66,31 +72,17 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc) {u64 counter, enabled, running; - counter = pmc->counter; + counter = pmc->counter + pmc->emulated_counter; +if (pmc->perf_event && !pmc->is_paused) counter +=perf_event_read_value (pmc->perf_event, &enabled, &running); +/* FIXME: Scaling needed? */ return counter &pmc_bitmask (pmc); } -static inline void pmc_release_perf_event (struct kvm_pmc *pmc) -{ -if (pmc->perf_event) { -perf_event_release_kernel (pmc->perf_event); - pmc->perf_event =NULL ; - pmc->current_config =0 ; -pmc_to_pmu (pmc)->event_count--; - } -} - -static inline void pmc_stop_counter (struct kvm_pmc *pmc) -{ -if (pmc->perf_event) { - pmc->counter =pmc_read_counter (pmc); -pmc_release_perf_event (pmc); - } -} +void pmc_write_counter (struct kvm_pmc *pmc,u64 val);static inline bool pmc_is_gp (struct kvm_pmc *pmc) {@@ -140,25 +132,6 @@ static inline struct kvm_pmc *get_fixed_pmc(struct kvm_pmu *pmu, u32 msr)return NULL ; } -static inline u64 get_sample_period (struct kvm_pmc *pmc,u64 counter_value) -{ -u64 sample_period = (-counter_value) &pmc_bitmask (pmc); - -if (!sample_period) - sample_period =pmc_bitmask (pmc) +1 ; -return sample_period; -} - -static inline void pmc_update_sample_period (struct kvm_pmc *pmc) -{ -if (!pmc->perf_event || pmc->is_paused || - !is_sampling_event (pmc->perf_event)) -return ; - -perf_event_period (pmc->perf_event, -get_sample_period (pmc, pmc->counter)); -} -static inline bool pmc_speculative_in_use (struct kvm_pmc *pmc) {struct kvm_pmu *pmu =pmc_to_pmu (pmc);@@ -214,12 +187,6 @@ static inline void kvm_init_pmu_capability(const struct kvm_pmu_ops *pmu_ops) KVM_PMC_MAX_FIXED); } -static inline void kvm_pmu_request_counter_reprogram (struct kvm_pmc *pmc) -{ -set_bit (pmc->idx,pmc_to_pmu (pmc)->reprogram_pmi); -kvm_make_request (KVM_REQ_PMU , pmc->vcpu); -} -static inline void reprogram_counters (struct kvm_pmu *pmu,u64 diff) {int bit;arch/x86/kvm/svm/pmu.c +1
-18@@ -160,8 +160,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)/* MSR_PERFCTRn */ pmc =get_gp_pmc_amd (pmu, msr,PMU_TYPE_COUNTER );if (pmc) { - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);return 0 ; }/* MSR_EVNTSELn */ @@ -233,21 +232,6 @@ static void amd_pmu_init(struct kvm_vcpu *vcpu) } } -static void amd_pmu_reset (struct kvm_vcpu *vcpu) -{ -struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); -int i; - -for (i =0 ; i <KVM_AMD_PMC_MAX_GENERIC ; i++) { -struct kvm_pmc *pmc = &pmu->gp_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter = pmc->eventsel =0 ; - } - - pmu->global_ctrl = pmu->global_status =0 ; -} -struct kvm_pmu_ops amd_pmu_ops__initdata = { .hw_event_available = amd_hw_event_available, .pmc_idx_to_pmc = amd_pmc_idx_to_pmc,@@ -259,7 +243,6 @@ struct kvm_pmu_ops amd_pmu_ops __initdata = { .set_msr = amd_pmu_set_msr, .refresh = amd_pmu_refresh, .init = amd_pmu_init, - .reset = amd_pmu_reset, .EVENTSEL_EVENT = AMD64_EVENTSEL_EVENT, .MAX_NR_GP_COUNTERS = KVM_AMD_PMC_MAX_GENERIC, .MIN_NR_GP_COUNTERS = AMD64_NUM_COUNTERS,arch/x86/kvm/vmx/pmu_intel.c +2
-24@@ -406,12 +406,10 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)if (!msr_info->host_initiated && !(msr &MSR_PMC_FULL_WIDTH_BIT )) data = (s64 )(s32 )data; - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);break ; }else if ((pmc =get_fixed_pmc (pmu, msr))) { - pmc->counter += data -pmc_read_counter (pmc); -pmc_update_sample_period (pmc); +pmc_write_counter (pmc, data);break ; }else if ((pmc =get_gp_pmc (pmu, msr,MSR_P6_EVNTSEL0 ))) { reserved_bits = pmu->reserved_bits;@@ -603,26 +601,6 @@ static void intel_pmu_init(struct kvm_vcpu *vcpu)static void intel_pmu_reset (struct kvm_vcpu *vcpu) { -struct kvm_pmu *pmu =vcpu_to_pmu (vcpu); -struct kvm_pmc *pmc =NULL ; -int i; - -for (i =0 ; i <KVM_INTEL_PMC_MAX_GENERIC ; i++) { - pmc = &pmu->gp_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter = pmc->eventsel =0 ; - } - -for (i =0 ; i <KVM_PMC_MAX_FIXED ; i++) { - pmc = &pmu->fixed_counters[i]; - -pmc_stop_counter (pmc); - pmc->counter = pmc->prev_counter =0 ; - } - - pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status =0 ; -intel_pmu_release_guest_lbr_event (vcpu); }base-commit:
88bb466c9dec4f70d682cf38c685324e7b1b3d60
Signature · 2 lines
Thanks, Dapeng Mi
Quoted 3 lines · Sean Christopherson
Ugh, yeah,
de0f619564f4created a bit of a mess. The underlying issue that it was solving is thatperf_event_read_value()and friends might sleep (yay mutex), and so can't be called from KVM's fastpath (IRQs disabled).
Updating pmu counters for emulated instructions cause troubles.
Quoted 6 lines
However, detecting overflow requires reading
perf_event_read_value()to gather the accumulated count from the hardware event in order to add it to the emulated count from software. E.g. ifpmc->counteris X and the perf event counter is Y, KVM needs to factor in Y because X+Y+1 might overflow even if X+1 does not.
Quoted 7 lines
Trying to snapshot the previous counter value is a bit of a mess. It could probably made to work, but it's hard to reason about what the snapshot actually contains and when it should be cleared, especially when factoring in the wrapping logic.
Rather than snapshot the previous counter, I think it makes sense to:
1) Track the number of emulated counter events
If events are counted separately, the challenge here is to correctly time the emulation of counter overflows, which can occur on both sides of the counter values out of sync.
Quoted 4 lines
2) Accumulate and reset the counts from
perf_eventandemulated_counterintopmc->counter when pausing the PMC3) Pause and reprogram the PMC on writes (instead of the current approach of blindly updating the sample period)
Updating the sample period is the only interface for KVM to configure hw behaviour on hw-ctr. I note that perf_event_set_count() will be proposed, and I'm pessimistic about this change.
Quoted 9 lines
4) Pause the counter when stopping the
perf_eventto ensurepmc->counterisfresh (instead of manually updating pmc->counter)IMO, that yields more intuitive logic, and makes it easier to reason about correctness since the behavior is easily define:
pmc->counterholds the counts that have been gathered and processed,perf_eventandemulated_counterhold outstanding counts on top. E.g. on a WRMSR to the counter, both the emulated counter and the hardware counter are reset, because whatever counts existed previously are irrelevant.
If we take the hardware view, a counter, emulated or not, just increments and overflows at the threshold. The missing logic here is when the counter is truncated when writing high bit-width values, and how to deal with the value of pmc->prev_counter was before pmc->counter was truncated.
Quoted 3 lines
Pausing the
counter_might_make WRMSR slower, but we need to get this all functionally correct before worrying too much about performance.
Performance, security and correctness should all be considered at the beginning.
Quoted 3 lines
Diff below for what I'm thinking (needs to be split into multiple patches). It's *very* lightly tested.
It saddens me that no one has come up with an actual low-level counter-test for this issue.
Quoted 3 lines
I'm about to disappear for a week, I'll pick this back up when I get return. In the meantime, any testing and/or input would be much appreciated!
How about accepting Roman's original fix and then exercising the rewriting genius ?
Quoted 50 lines · Like Xu
Quoted 3 lines · Sean Christopherson
Ugh, yeah,
de0f619564f4created a bit of a mess. The underlying issue that it was solving is thatperf_event_read_value()and friends might sleep (yay mutex), and so can't be called from KVM's fastpath (IRQs disabled).Updating pmu counters for emulated instructions cause troubles.
Quoted 6 lines
However, detecting overflow requires reading
perf_event_read_value()to gather the accumulated count from the hardware event in order to add it to the emulated count from software. E.g. ifpmc->counteris X and the perf event counter is Y, KVM needs to factor in Y because X+Y+1 might overflow even if X+1 does not.Quoted 7 lines
Trying to snapshot the previous counter value is a bit of a mess. It could probably made to work, but it's hard to reason about what the snapshot actually contains and when it should be cleared, especially when factoring in the wrapping logic.
Rather than snapshot the previous counter, I think it makes sense to:
1) Track the number of emulated counter events
If events are counted separately, the challenge here is to correctly time the emulation of counter overflows, which can occur on both sides of the counter values out of sync.
Quoted 4 lines
2) Accumulate and reset the counts from
perf_eventandemulated_counterintopmc->counter when pausing the PMC3) Pause and reprogram the PMC on writes (instead of the current approach of blindly updating the sample period)
Updating the sample period is the only interface for KVM to configure hw behaviour on hw-ctr. I note that
perf_event_set_count()will be proposed, and I'm pessimistic about this change.Quoted 9 lines
4) Pause the counter when stopping the
perf_eventto ensurepmc->counterisfresh (instead of manually updating pmc->counter)IMO, that yields more intuitive logic, and makes it easier to reason about correctness since the behavior is easily define:
pmc->counterholds the counts that have been gathered and processed,perf_eventandemulated_counterhold outstanding counts on top. E.g. on a WRMSR to the counter, both the emulated counter and the hardware counter are reset, because whatever counts existed previously are irrelevant.If we take the hardware view, a counter, emulated or not, just increments and overflows at the threshold. The missing logic here is when the counter is truncated when writing high bit-width values, and how to deal with the value of
pmc->prev_counterwas beforepmc->counterwas truncated.Quoted 1 line
Quoted 5 lines
Quoted 2 lines
Pausing the
counter_might_make WRMSR slower, but we need to get this all functionally correct before worrying too much about performance.Performance, security and correctness should all be considered at the beginning.
+1 on the performance part.
I did several rounds of performance testing, pausing the counter is fast, but restarting the counter is *super* slow. The extra overhead might just make vPMU useless especially when the guest workload takes full CPU/memory resources in a VM (like SPEC2017 does).
Quoted 4 lines
Quoted 3 lines
Diff below for what I'm thinking (needs to be split into multiple patches). It's *very* lightly tested.
Quoted 8 lines
It saddens me that no one has come up with an actual low-level counter-test for this issue.
Quoted 3 lines
I'm about to disappear for a week, I'll pick this back up when I get return. In the meantime, any testing and/or input would be much appreciated!
How about accepting Roman's original fix and then exercising the rewriting genius ?
+1
I think the best option would be to just apply the fix in a short term and put the refactoring of the emulated counter in the next series.
Quoted 29 lines · Sean Christopherson
Quoted 28 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }Quoted 9 lines
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
arch/x86/kvm/pmu.h +6
-0+static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continue
Quoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Can you reverting
de0f619564f4("KVM: x86/pmu: Defer counter emulated overflow viapmc->prev_counter")? I suspect the problem is theprev_countermess.
We observe the problem on a branch that predates this commit
Thanks,
Roman.
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Quoted 25 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }Quoted 9 lines
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
arch/x86/kvm/pmu.h +6
-0+static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
Quoted 7 lines
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continuein the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.
Maybe this is because KVM doesn't virtualize
IA32_DEBUGCTL.Freeze_PerfMon_On_PMI?
Consider:
1. PMC0 overflows, GLOBAL_STATUS[0] is set, and an NMI is delivered.
2. Before the guest's PMI handler clears GLOBAL_CTRL, PMC1 overflows, GLOBAL_STATUS[1] is set, and an NMI is queued for delivery after the next IRET.
3. The guest's PMI handler clears GLOBAL_CTRL, reads 3 from GLOBAL_STATUS, writes 3 to GLOBAL_OVF_CTRL, re-enables GLOBAL_CTRL, and IRETs.
4. The queued NMI is delivered, but GLOBAL_STATUS is now 0. No one claims the NMI, so we get the spurious NMI message.
I don't know why this would require counting the retirement of emulated instructions. It seems that hardware PMC overflow in the early part of the guest's PMI handler would also be a problem.
Quoted 15 lines
Thanks,
Roman.Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Quoted 31 lines · Jim Mattson
Quoted 29 lines · Roman Kagan
Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Roman Kagan
Quoted 4 lines · Sean Christopherson
arch/x86/kvm/pmu.h +0
-0@@ -74,6 +74,14 @@ static inline u64 pmc_read_counter(struct kvm_pmc *pmc)return counter &pmc_bitmask (pmc); }Quoted 9 lines
arch/x86/kvm/pmu.h static inline u64 pmc_read_counter(struct kvm_pmc *pmc) · line 82
Quoted 6 lines
arch/x86/kvm/pmu.h +6
-0+static inline void pmc_write_counter (struct kvm_pmc *pmc,u64 val) +{ +if (pmc->perf_event && !pmc->is_paused) +perf_event_set_count (pmc->perf_event, val); + + pmc->counter = val;Doesn't this still have the original problem of storing wider value than allowed?
Quoted 2 lines
Yes, this was just to fix the counter offset weirdness. My plan is to apply your patch on top. Sorry for not making that clear.
Ah, got it, thanks!
Also I'm now chasing a problem that we occasionally see
[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43. [3939579.462836] Do you have a strange power saving mode enabled? [3939579.462836] Dazed and confused, but trying to continue
Quoted 6 lines
Quoted 3 lines
in the guests when perf is used. These messages disappear when
9cd803d496e7("KVM: x86: Update vPMCs when retiring instructions") is reverted. I haven't yet figured out where exactly the culprit is.Maybe this is because KVM doesn't virtualize
IA32_DEBUGCTL.Freeze_PerfMon_On_PMI?
Never mind. Linux doesn't set IA32_DEBUGCTL.Freeze_PerfMon_On_PMI.
Quoted 31 lines
Consider:
1. PMC0 overflows, GLOBAL_STATUS[0] is set, and an NMI is delivered.
2. Before the guest's PMI handler clearsGLOBAL_CTRL, PMC1 overflows, GLOBAL_STATUS[1] is set, and an NMI is queued for delivery after the next IRET.
3. The guest's PMI handler clearsGLOBAL_CTRL, reads 3 fromGLOBAL_STATUS, writes 3 toGLOBAL_OVF_CTRL, re-enablesGLOBAL_CTRL, and IRETs.
4. The queued NMI is delivered, butGLOBAL_STATUSis now 0. No one claims the NMI, so we get the spurious NMI message.I don't know why this would require counting the retirement of emulated instructions. It seems that hardware PMC overflow in the early part of the guest's PMI handler would also be a problem.
Quoted 15 lines
Thanks,
Roman.Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Quoted 12 lines · Roman Kagan
Performance counters are defined to have width less than 64 bits. The vPMU code maintains the counters in u64 variables but assumes the value to fit within the defined width. However, for Intel non-full-width counters (
MSR_IA32_PERFCTRx) the value receieved from the guest is truncated to 32 bits and then sign-extended to full 64 bits. If a negative value is set, it's sign-extended to 64 bits, but then inkvm_pmu_incr_counter()it's incremented, truncated, and compared to the previous value for overflow detection.
That previous value is not truncated, so it always evaluates bigger than the truncated new one, and a PMI is injected. If the PMI handler writes a negative counter value itself, the vCPU never quits the PMI loop.
Quoted 1 line
[...]
Applied to kvm-x86 pmu, with a slightly massaged changelog. Thanks! And sorry for the horrendous delay...
[1/1] KVM: x86/pmu: Truncate counter value to allowed width on write
https://github.com/kvm-x86/linux/commit/b29a2acd36dd