Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU"

RFC PATCH Applied Vitaly Kuznetsov 3 participants 4 messages Jun 19, 2020

Subsystems (3)

x86 KVM KVM x86

lore

Share & tools

lore

Vitaly Kuznetsov

author

2020-06-19 09:42 UTC #0 lore raw

RFC PATCH 1/1 Revert "KVM: VMX: Micro-optimize vmexit time when not exposing PMU"

Guest crashes are observed on a Cascade Lake system when 'perf top' is launched on the host, e.g.

 BUG: unable to handle kernel paging request at fffffe0000073038
 PGD 7ffa7067 P4D 7ffa7067 PUD 7ffa6067 PMD 7ffa5067 PTE ffffffffff120
 Oops: 0000 [#1] SMP PTI
 CPU: 1 PID: 1 Comm: systemd Not tainted 4.18.0+ #380
...
 Call Trace:
  serial8250_console_write+0xfe/0x1f0
  call_console_drivers.constprop.0+0x9d/0x120
  console_unlock+0x1ea/0x460

Call traces are different but the crash is imminent. The problem was blindly bisected to the commit 041bc42ce2d0 ("KVM: VMX: Micro-optimize vmexit time when not exposing PMU"). It was also confirmed that the issue goes away if PMU is exposed to the guest.

With some instrumentation of the guest we can see what is being switched (when we do atomic_switch_perf_msrs()):

vmx_vcpu_run: switching 2 msrs
vmx_vcpu_run: switching MSR38f guest: 70000000d host: 70000000f
vmx_vcpu_run: switching MSR3f1 guest: 0 host: 2

The current guess is that PEBS (MSR_IA32_PEBS_ENABLE, 0x3f1) is to blame. Regardless of whether PMU is exposed to the guest or not, PEBS needs to be disabled upon switch.

This reverts commit 041bc42ce2d0efac3b85bbb81dea8c74b81f4ef9.

Reported-by Maxime Coquelin <[email protected]>

Signed-off-by Vitaly Kuznetsov <[email protected]>

---
- Perf/KVM interractions are a mystery to me, thus RFC.

---
 arch/x86/kvm/vmx/vmx.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

arch/x86/kvm/vmx/vmx.c +1 -2

index 36c771728c8c..b1a23ad986ff 100644
@@ -6728,8 +6728,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
 
 	pt_guest_enter(vmx);
 
-	if (vcpu_to_pmu(vcpu)->version)
-		atomic_switch_perf_msrs(vmx);
+	atomic_switch_perf_msrs(vmx);
 	atomic_switch_umwait_control_msr(vmx);
 
 	if (enable_preemption_timer)

Signature · 1 line

2.25.4

Xu, Like

2020-06-19 10:03 UTC #1 lore raw

Quoted 2 lines · Vitaly Kuznetsov

Guest crashes are observed on a Cascade Lake system when 'perf top' is launched on the host, e.g.

Interesting, is it specific to Cascade Lake?

Would you mind sharing the output of
"cpuid -r -l 1 -1" and "cat /proc/cpuinfo| grep microcode | uniq" with us ?

Thanks,
Like Xu

Quoted 51 lines

  BUG: unable to handle kernel paging request at fffffe0000073038
  PGD 7ffa7067 P4D 7ffa7067 PUD 7ffa6067 PMD 7ffa5067 PTE ffffffffff120
  Oops: 0000 [#1] SMP PTI
  CPU: 1 PID: 1 Comm: systemd Not tainted 4.18.0+ #380
...
  Call Trace:
   serial8250_console_write+0xfe/0x1f0
   call_console_drivers.constprop.0+0x9d/0x120
   console_unlock+0x1ea/0x460
Call traces are different but the crash is imminent. The problem was blindly bisected to the commit 041bc42ce2d0 ("KVM: VMX: Micro-optimize vmexit time when not exposing PMU"). It was also confirmed that the issue goes away if PMU is exposed to the guest.
With some instrumentation of the guest we can see what is being switched (when we do atomic_switch_perf_msrs()):
vmx_vcpu_run: switching 2 msrs
vmx_vcpu_run: switching MSR38f guest: 70000000d host: 70000000f
vmx_vcpu_run: switching MSR3f1 guest: 0 host: 2
The current guess is that PEBS (MSR_IA32_PEBS_ENABLE, 0x3f1) is to blame. Regardless of whether PMU is exposed to the guest or not, PEBS needs to be disabled upon switch.
This reverts commit 041bc42ce2d0efac3b85bbb81dea8c74b81f4ef9.
Reported-by Maxime Coquelin <[email protected]>
Signed-off-by Vitaly Kuznetsov <[email protected]>
---
- Perf/KVM interractions are a mystery to me, thus RFC.
---
  arch/x86/kvm/vmx/vmx.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +1 -2
index 36c771728c8c..b1a23ad986ff 100644
@@ -6728,8 +6728,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
 
  	pt_guest_enter(vmx);
 
-	if (vcpu_to_pmu(vcpu)->version)
-		atomic_switch_perf_msrs(vmx);
+	atomic_switch_perf_msrs(vmx);
  	atomic_switch_umwait_control_msr(vmx);
 
  	if (enable_preemption_timer)

Vitaly Kuznetsov

author

2020-06-19 11:29 UTC #2 lore raw

Quoted 5 lines · Like

Quoted 2 lines · Vitaly Kuznetsov
Guest crashes are observed on a Cascade Lake system when 'perf top' is launched on the host, e.g.
Interesting, is it specific to Cascade Lake?

Actually no, just reproduced this on a Haswell system. If you run the guest with "-cpu host,-pmu" and do 'perf top' (on the host) the guest crashes immediately.

Quoted 2 lines

Would you mind sharing the output of
"cpuid -r -l 1 -1" and "cat /proc/cpuinfo| grep microcode | uniq" with us ?

Sure (this is probably unrelated because the issue also reproduces on Haswell but still):

# cpuid -r -l 1 -1

CPU:
0x00000001 0x00: eax=0x00050657 ebx=0x03200800 ecx=0x7ffefbff edx=0xbfebfbff

# cat /proc/cpuinfo| grep microcode | uniq
microcode	: 0x500002c

This is "Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz"

Signature · 1 line

Vitaly

Paolo Bonzini

maintainer Applied

2020-06-19 12:14 UTC #3 lore raw

In reply to Vitaly Kuznetsov

Quoted 45 lines · Vitaly Kuznetsov

Guest crashes are observed on a Cascade Lake system when 'perf top' is launched on the host, e.g.
 BUG: unable to handle kernel paging request at fffffe0000073038
 PGD 7ffa7067 P4D 7ffa7067 PUD 7ffa6067 PMD 7ffa5067 PTE ffffffffff120
 Oops: 0000 [#1] SMP PTI
 CPU: 1 PID: 1 Comm: systemd Not tainted 4.18.0+ #380
...
 Call Trace:
  serial8250_console_write+0xfe/0x1f0
  call_console_drivers.constprop.0+0x9d/0x120
  console_unlock+0x1ea/0x460
Call traces are different but the crash is imminent. The problem was blindly bisected to the commit 041bc42ce2d0 ("KVM: VMX: Micro-optimize vmexit time when not exposing PMU"). It was also confirmed that the issue goes away if PMU is exposed to the guest.
With some instrumentation of the guest we can see what is being switched (when we do atomic_switch_perf_msrs()):
vmx_vcpu_run: switching 2 msrs
vmx_vcpu_run: switching MSR38f guest: 70000000d host: 70000000f
vmx_vcpu_run: switching MSR3f1 guest: 0 host: 2
The current guess is that PEBS (MSR_IA32_PEBS_ENABLE, 0x3f1) is to blame. Regardless of whether PMU is exposed to the guest or not, PEBS needs to be disabled upon switch.
This reverts commit 041bc42ce2d0efac3b85bbb81dea8c74b81f4ef9.
Reported-by Maxime Coquelin <[email protected]>
Signed-off-by Vitaly Kuznetsov <[email protected]>
---
- Perf/KVM interractions are a mystery to me, thus RFC.
---
 arch/x86/kvm/vmx/vmx.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kvm/vmx/vmx.c +0 -0
index 36c771728c8c..b1a23ad986ff 100644

arch/x86/kvm/vmx/vmx.c static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) · line 6734

Quoted 9 lines

arch/x86/kvm/vmx/vmx.c +1 -2

@@ -6728,8 +6728,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu)
 
 	pt_guest_enter(vmx);
 
-	if (vcpu_to_pmu(vcpu)->version)
-		atomic_switch_perf_msrs(vmx);
+	atomic_switch_perf_msrs(vmx);
 	atomic_switch_umwait_control_msr(vmx);
 
 	if (enable_preemption_timer)

Queued, thanks.

Paolo