Skip to contentSkip to conversation

KVM: VMX: Fix x2apic check in vmx_msr_bitmap_mode()

PATCH In review Joerg Roedel 6 participants 7 messages Feb 21, 2019 – Feb 22, 2019 Stable branch

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: VMX: Fix x2apic check in vmx_msr_bitmap_mode()

From: Joerg Roedel <[email protected]>

The stable backport of upstream commit

904e14fb7cb96 KVM: VMX: make MSR bitmaps per-VCPU

has a bug in vmx_msr_bitmap_mode(). It enables the x2apic MSR-bitmap when the kernel emulates x2apic for the guest in software. The upstream version of the commit checkes whether the hardware has virtualization enabled for x2apic emulation.

Since KVM emulates x2apic for guests even when the host does not support x2apic in hardware, this causes the intercept of at least the X2APIC_TASKPRI MSR to be disabled on machines not supporting that MSR. The result is undefined behavior, on some machines (Intel Westmere based) it causes a crash of the guest kernel when it tries to access that MSR.

Change the check in vmx_msr_bitmap_mode() to match the upstream code. This fixes the guest crashes observed with stable kernels starting with v4.4.168 through v4.4.175.

---
 arch/x86/kvm/vmx.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx.c +3 -1
index aee2886a387c..14553f6c03a6 100644
@@ -4628,7 +4628,9 @@ static u8 vmx_msr_bitmap_mode(struct kvm_vcpu *vcpu)
 {
 	u8 mode = 0;
 
-	if (irqchip_in_kernel(vcpu->kvm) && apic_x2apic_mode(vcpu->arch.apic)) {
+	if (cpu_has_secondary_exec_ctrls() &&
+	    (vmcs_read32(SECONDARY_VM_EXEC_CONTROL) &
+	     SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE)) {
 		mode |= MSR_BITMAP_MODE_X2APIC;
 		if (enable_apicv)
 			mode |= MSR_BITMAP_MODE_X2APIC_APICV;
Signature · 1 line
2.16.3
#1 lore raw
Quoted 46 lines · Joerg Roedel

From: Joerg Roedel <[email protected]>

The stable backport of upstream commit

904e14fb7cb96 KVM: VMX: make MSR bitmaps per-VCPU

has a bug in vmx_msr_bitmap_mode(). It enables the x2apic MSR-bitmap when the kernel emulates x2apic for the guest in software. The upstream version of the commit checkes whether the hardware has virtualization enabled for x2apic emulation.

Since KVM emulates x2apic for guests even when the host does not support x2apic in hardware, this causes the intercept of at least the X2APIC_TASKPRI MSR to be disabled on machines not supporting that MSR. The result is undefined behavior, on some machines (Intel Westmere based) it causes a crash of the guest kernel when it tries to access that MSR.

Change the check in vmx_msr_bitmap_mode() to match the upstream code. This fixes the guest crashes observed with stable kernels starting with v4.4.168 through v4.4.175.

---
 arch/x86/kvm/vmx.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx.c +3 -1
index aee2886a387c..14553f6c03a6 100644
@@ -4628,7 +4628,9 @@ static u8 vmx_msr_bitmap_mode(struct kvm_vcpu *vcpu)
 {
 	u8 mode = 0;
 
-	if (irqchip_in_kernel(vcpu->kvm) && apic_x2apic_mode(vcpu->arch.apic)) {
+	if (cpu_has_secondary_exec_ctrls() &&
+	    (vmcs_read32(SECONDARY_VM_EXEC_CONTROL) &
+	     SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE)) {
 		mode |= MSR_BITMAP_MODE_X2APIC;
 		if (enable_apicv)
 			mode |= MSR_BITMAP_MODE_X2APIC_APICV;
Signature · 1 line
2.16.3

Ugh, good catch!

Any hint as to what type of testing that you did that caught this? I keep asking people to run some kvm tests, but so far no one is :(

thanks,

greg k-h

#2 lore raw
Quoted 4 lines · Greg Kroah-Hartman

Ugh, good catch!

Any hint as to what type of testing that you did that caught this? I keep asking people to run some kvm tests, but so far no one is :(

We caught this at SUSE while testing candidate kernel updates for one of our service packs using a 4.4-based kernel and debugging turned out that this is issue came in via stable-updates. We also build a vanilla-flavour of the kernel which is nearly identical to the upstream stable tree, but what usually ends up in testing is the full tree with other backports.

This particular issue was found by updating some openstack machines with the candidate kernel, which then triggered the problem in some guests. It is also a very special one, since I was only able to trigger the problem on Westmere-based machines with a specific guest-config.

Regards,

Joerg

#3 lore raw
Quoted 13 lines · Joerg Roedel
Quoted 4 lines · Greg Kroah-Hartman

Ugh, good catch!

Any hint as to what type of testing that you did that caught this? I keep asking people to run some kvm tests, but so far no one is :(

We caught this at SUSE while testing candidate kernel updates for one of our service packs using a 4.4-based kernel and debugging turned out that this is issue came in via stable-updates. We also build a vanilla-flavour of the kernel which is nearly identical to the upstream stable tree, but what usually ends up in testing is the full tree with other backports.

Quoted 4 lines

This particular issue was found by updating some openstack machines with the candidate kernel, which then triggered the problem in some guests. It is also a very special one, since I was only able to trigger the problem on Westmere-based machines with a specific guest-config.

Nice work. Any chance that "test" could be added to the kvm testing scripts that I think are being worked on somewhere? Ideally we would have caught this before it ever hit the stable tree. Due to the lack of good KVM testing, that's one of the areas I am always most worried about :(

thanks,

greg k-h

#4 lore raw
Quoted 14 lines · Greg Kroah-Hartman
Quoted 13 lines · Joerg Roedel
Quoted 4 lines · Greg Kroah-Hartman

Ugh, good catch!

Any hint as to what type of testing that you did that caught this?  I keep asking people to run some kvm tests, but so far no one is :(

We caught this at SUSE while testing candidate kernel updates for one of our service packs using a 4.4-based kernel and debugging turned out that this is issue came in via stable-updates. We also build a vanilla-flavour of the kernel which is nearly identical to the upstream stable tree, but what usually ends up in testing is the full tree with other backports.

Quoted 8 lines
Quoted 4 lines

This particular issue was found by updating some openstack machines with the candidate kernel, which then triggered the problem in some guests. It is also a very special one, since I was only able to trigger the problem on Westmere-based machines with a specific guest-config.

Nice work.  Any chance that "test" could be added to the kvm testing scripts that I think are being worked on somewhere?  Ideally we would have caught this before it ever hit the stable tree.

If I understood correctly, the bug is specific to my backport.

Quoted 3 lines

Due to the lack of
good KVM testing, that's one of the areas I am always most worried about :(

Since the behaviour in this area depends on the host CPU model this might not help much.

Ben.

Signature · 3 lines
Ben Hutchings, Software Developer                         Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom
#5 lore raw
In reply to Greg Kroah-Hartman
Quoted 19 lines · Greg Kroah-Hartman
Quoted 13 lines · Joerg Roedel
Quoted 4 lines · Greg Kroah-Hartman

Ugh, good catch!

Any hint as to what type of testing that you did that caught this? I keep asking people to run some kvm tests, but so far no one is :(

We caught this at SUSE while testing candidate kernel updates for one of our service packs using a 4.4-based kernel and debugging turned out that this is issue came in via stable-updates. We also build a vanilla-flavour of the kernel which is nearly identical to the upstream stable tree, but what usually ends up in testing is the full tree with other backports.

Quoted 4 lines

This particular issue was found by updating some openstack machines with the candidate kernel, which then triggered the problem in some guests. It is also a very special one, since I was only able to trigger the problem on Westmere-based machines with a specific guest-config.

Quoted 4 lines

Nice work. Any chance that "test" could be added to the kvm testing scripts that I think are being worked on somewhere? Ideally we would have caught this before it ever hit the stable tree. Due to the lack of good KVM testing, that's one of the areas I am always most worried about

This bug exists only in the 4.4.y backport; upstream, 4.9.y and 4.14.y all had the correct code from the get-go. And there is already a KVM unit test that *should* hit this, albeit somewhat indirectly. I'll verify the tests that touch the TPR actually run with x2APIC enabled.

Assuming the KVM unit test actually works, it's not a stretch for the bug to esacpe, e.g. if the tests weren't run on 4.4.y at all, or were only run on hardware with x2APIC.

#6 lore raw
Quoted 4 lines · Sean Christopherson

This bug exists only in the 4.4.y backport; upstream, 4.9.y and 4.14.y all had the correct code from the get-go. And there is already a KVM unit test that *should* hit this, albeit somewhat indirectly. I'll verify the tests that touch the TPR actually run with x2APIC enabled.

eventinj from kvm-unit-tests should trigger it. There are other tests that touch the TPR, but they use cr8 so they don't show the bug.

Quoted 3 lines

Assuming the KVM unit test actually works, it's not a stretch for the bug to esacpe, e.g. if the tests weren't run on 4.4.y at all, or were only run on hardware with x2APIC.

Yeah, you should be able to see this with kvm_intel.enable_apicv=0 on newer processors. But I've never run the tests for 4.4.y.

Paolo