Skip to contentSkip to conversation

KVM: nVMX: guest reset fixes

PATCH In review Vitaly Kuznetsov 5 participants 10 messages Jul 25, 2019 – Jul 29, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover KVM: nVMX: guest reset fixes

Few patches were recently marked for stable@ but commits are not backportable as-is and require a few tweaks. Here is 4.19 stable backport.

Jan Kiszka (1):
KVM: nVMX: Clear pending KVM_REQ_GET_VMCS12_PAGES when leaving nested

Paolo Bonzini (1):
KVM: nVMX: do not use dangling shadow VMCS after guest reset

 arch/x86/kvm/vmx.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/2 KVM: nVMX: do not use dangling shadow VMCS after guest reset

From: Paolo Bonzini <[email protected]>

[ Upstream commit 88dddc11a8d6b09201b4db9d255b3394d9bc9e57 ]

If a KVM guest is reset while running a nested guest, free_nested will disable the shadow VMCS execution control in the vmcs01. However, on the next KVM_RUN vmx_vcpu_run would nevertheless try to sync the VMCS12 to the shadow VMCS which has since been freed.

This causes a vmptrld of a NULL pointer on my machime, but Jan reports the host to hang altogether. Let's see how much this trivial patch fixes.

---
 arch/x86/kvm/vmx.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx.c +7 -1
index 73d6d585dd66..880bc36a0d5d 100644
@@ -8457,6 +8457,7 @@ static void vmx_disable_shadow_vmcs(struct vcpu_vmx *vmx)
 {
 	vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL, SECONDARY_EXEC_SHADOW_VMCS);
 	vmcs_write64(VMCS_LINK_POINTER, -1ull);
+	vmx->nested.sync_shadow_vmcs = false;
 }
 
 static inline void nested_release_vmcs12(struct vcpu_vmx *vmx)
@@ -8468,7 +8469,6 @@ static inline void nested_release_vmcs12(struct vcpu_vmx *vmx)
 		/* copy to memory all shadowed fields in case
 		   they were modified */
 		copy_shadow_to_vmcs12(vmx);
-		vmx->nested.sync_shadow_vmcs = false;
 		vmx_disable_shadow_vmcs(vmx);
 	}
 	vmx->nested.posted_intr_nv = -1;
@@ -8668,6 +8668,9 @@ static void copy_shadow_to_vmcs12(struct vcpu_vmx *vmx)
 	u64 field_value;
 	struct vmcs *shadow_vmcs = vmx->vmcs01.shadow_vmcs;
 
+	if (WARN_ON(!shadow_vmcs))
+		return;
+
 	preempt_disable();
 
 	vmcs_load(shadow_vmcs);
@@ -8706,6 +8709,9 @@ static void copy_vmcs12_to_shadow(struct vcpu_vmx *vmx)
 	u64 field_value = 0;
 	struct vmcs *shadow_vmcs = vmx->vmcs01.shadow_vmcs;
 
+	if (WARN_ON(!shadow_vmcs))
+		return;
+
 	vmcs_load(shadow_vmcs);
 
 	for (q = 0; q < ARRAY_SIZE(fields); q++) {
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH stable-4.19 1/2] KVM: nVMX: do not use dangling shadow VMCS after guest reset

Quoted 10 lines · Vitaly Kuznetsov <[email protected]> 于2019年7月25日周四 下午3:29写道:

From: Paolo Bonzini <[email protected]>

[ Upstream commit 88dddc11a8d6b09201b4db9d255b3394d9bc9e57 ]

If a KVM guest is reset while running a nested guest, free_nested will disable the shadow VMCS execution control in the vmcs01. However, on the next KVM_RUN vmx_vcpu_run would nevertheless try to sync the VMCS12 to the shadow VMCS which has since been freed.

Quoted 7 lines

This causes a vmptrld of a NULL pointer on my machime, but Jan reports the host to hang altogether. Let's see how much this trivial patch fixes.

Hi all,

Do we need to backport the fix also to stable 4.14? It applies cleanly and compiles fine.

Regards,
Jack Wang

maintainer
#3 lore raw

Re: [PATCH stable-4.19 1/2] KVM: nVMX: do not use dangling shadow VMCS after guest reset

Quoted 19 lines · Jack Wang
Quoted 10 lines · Vitaly Kuznetsov <[email protected]> 于2019年7月25日周四 下午3:29写道:

From: Paolo Bonzini <[email protected]>

[ Upstream commit 88dddc11a8d6b09201b4db9d255b3394d9bc9e57 ]

If a KVM guest is reset while running a nested guest, free_nested will disable the shadow VMCS execution control in the vmcs01. However, on the next KVM_RUN vmx_vcpu_run would nevertheless try to sync the VMCS12 to the shadow VMCS which has since been freed.

Quoted 7 lines

This causes a vmptrld of a NULL pointer on my machime, but Jan reports the host to hang altogether. Let's see how much this trivial patch fixes.

Quoted 4 lines

Hi all,

Do we need to backport the fix also to stable 4.14? It applies cleanly and compiles fine.

The reproducer required newer kernels that support KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE, so it would be hard to test it. However, the patch itself should be safe.

Paolo

#4 lore raw

Re: [PATCH stable-4.19 1/2] KVM: nVMX: do not use dangling shadow VMCS after guest reset

Quoted 23 lines · Paolo Bonzini <[email protected]> 于2019年7月29日周一 上午11:10写道:
Quoted 21 lines · Jack Wang
Quoted 10 lines · Vitaly Kuznetsov <[email protected]> 于2019年7月25日周四 下午3:29写道:

From: Paolo Bonzini <[email protected]>

[ Upstream commit 88dddc11a8d6b09201b4db9d255b3394d9bc9e57 ]

If a KVM guest is reset while running a nested guest, free_nested will disable the shadow VMCS execution control in the vmcs01. However, on the next KVM_RUN vmx_vcpu_run would nevertheless try to sync the VMCS12 to the shadow VMCS which has since been freed.

Quoted 7 lines

This causes a vmptrld of a NULL pointer on my machime, but Jan reports the host to hang altogether. Let's see how much this trivial patch fixes.

Hi all,

Quoted 8 lines
Quoted 2 lines

Do we need to backport the fix also to stable 4.14? It applies cleanly and compiles fine.

The reproducer required newer kernels that support KVM_GET_NESTED_STATE and KVM_SET_NESTED_STATE, so it would be hard to test it. However, the patch itself should be safe.

Paolo

Thanks Paolo for confirmation. I'm asking because we had one incident in our production with 4.14.129 kernel,
System is Skylake Gold cpu, first kvm errors, host hung afterwards

kernel: [1186161.091160] kvm: vmptrld (null)/6bfc00000000 failed
kernel: [1186161.091537] kvm: vmclear fail: (null)/6bfc00000000
kernel: [1186186.490300] watchdog: BUG: soft lockup - CPU#54 stuck for 23s! [qemu:16639]

Hi Sasha, hi Greg,

Would be great if you can pick this patch also to 4.14 kernel.

Best regards,
Jack Wang

maintainer Acked-by
#5 lore raw

Re: [PATCH stable-4.19 1/2] KVM: nVMX: do not use dangling shadow VMCS after guest reset

Quoted 4 lines · Jack Wang

Thanks Paolo for confirmation. I'm asking because we had one incident in our production with 4.14.129 kernel,
System is Skylake Gold cpu, first kvm errors, host hung afterwards

Quoted 8 lines

kernel: [1186161.091160] kvm: vmptrld (null)/6bfc00000000 failed
kernel: [1186161.091537] kvm: vmclear fail: (null)/6bfc00000000
kernel: [1186186.490300] watchdog: BUG: soft lockup - CPU#54 stuck for 23s! [qemu:16639]

Hi Sasha, hi Greg,

Would be great if you can pick this patch also to 4.14 kernel.

#6 lore raw

Re: [PATCH stable-4.19 1/2] KVM: nVMX: do not use dangling shadow VMCS after guest reset

Quoted 10 lines · Paolo Bonzini
Quoted 9 lines · Jack Wang

Thanks Paolo for confirmation. I'm asking because we had one incident in our production with 4.14.129 kernel,
System is Skylake Gold cpu, first kvm errors, host hung afterwards

kernel: [1186161.091160] kvm: vmptrld (null)/6bfc00000000 failed
kernel: [1186161.091537] kvm: vmclear fail: (null)/6bfc00000000
kernel: [1186186.490300] watchdog: BUG: soft lockup - CPU#54 stuck for 23s! [qemu:16639]

Quoted 5 lines
Quoted 3 lines

Hi Sasha, hi Greg,

Would be great if you can pick this patch also to 4.14 kernel.

Ok, now queued up, thanks.

greg k-h

#7 lore raw

PATCH 2/2 KVM: nVMX: Clear pending KVM_REQ_GET_VMCS12_PAGES when leaving nested

In reply to Vitaly Kuznetsov

From: Jan Kiszka <[email protected]>

[ Upstream commit cf64527bb33f6cec2ed50f89182fc4688d0056b6 ]

Letting this pend may cause nested_get_vmcs12_pages to run against an invalid state, corrupting the effective vmcs of L1.

This was triggerable in QEMU after a guest corruption in L2, followed by a L1 reset.

Fixes 7f7f1ba33cf2 ("KVM: x86: do not load vmcs12 pages while still in SMM")
---
 arch/x86/kvm/vmx.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kvm/vmx.c +2 -0
index 880bc36a0d5d..4cf16378dffe 100644
@@ -8490,6 +8490,8 @@ static void free_nested(struct vcpu_vmx *vmx)
 	if (!vmx->nested.vmxon && !vmx->nested.smm.vmxon)
 		return;
 
+	kvm_clear_request(KVM_REQ_GET_VMCS12_PAGES, &vmx->vcpu);
+
 	hrtimer_cancel(&vmx->nested.preemption_timer);
 	vmx->nested.vmxon = false;
 	vmx->nested.smm.vmxon = false;
Signature · 1 line
2.20.1
maintainer Acked-by on the v1 series
#8 lore raw
In reply to Vitaly Kuznetsov
Quoted 12 lines · Vitaly Kuznetsov

Few patches were recently marked for stable@ but commits are not backportable as-is and require a few tweaks. Here is 4.19 stable backport.

Jan Kiszka (1):
KVM: nVMX: Clear pending KVM_REQ_GET_VMCS12_PAGES when leaving nested

Paolo Bonzini (1):
KVM: nVMX: do not use dangling shadow VMCS after guest reset

 arch/x86/kvm/vmx.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

Thanks Vitaly for helping out.

Paolo

#9 lore raw
In reply to Vitaly Kuznetsov
Quoted 11 lines · Vitaly Kuznetsov

Few patches were recently marked for stable@ but commits are not backportable as-is and require a few tweaks. Here is 4.19 stable backport.

Jan Kiszka (1):
KVM: nVMX: Clear pending KVM_REQ_GET_VMCS12_PAGES when leaving nested

Paolo Bonzini (1):
KVM: nVMX: do not use dangling shadow VMCS after guest reset

 arch/x86/kvm/vmx.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

All now applied, thanks!

greg k-h