Skip to contentSkip to conversation

nVMX: Simple posted interrupts test

PATCH In review Jim Mattson 4 participants 10 messages Dec 11, 2023 – Oct 16, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover nVMX: Simple posted interrupts test

I reported recently that commit 26844fee6ade ("KVM: x86: never write to memory from kvm_vcpu_check_block()") broke delivery of a virtualized posted interrupt from an L1 vCPU to a halted L2 vCPU (see
https://lore.kernel.org/all/[email protected]/).
The test that exposed the regression is the final patch of this series. The others are prerequisites.

It would make sense to add "vmx_posted_interrupts_test" to the set of tests to be run under the unit test name, "vmx_apicv_test," but that is non-trivial. The vmx_posted_interrupts_test requires "smp = 2," but I find that adding that to the vmx_apicv_tests causes virt_x2apic_mode_test to fail with:

FAIL: x2apic - reading 0x310: x86/vmx_tests.c:2151: Assertion failed: (expected) == (actual)

	LHS: 0x0000000000000012 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010 - 18
	RHS: 0x0000000000000001 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001 - 1
Expected VMX_VMCALL, got VMX_EXTINT.
	STACK: 406ef8 40725a 41299f 402036 403f59 4001bd

I haven't investigated.

Jim Mattson (1):
nVMX: Enable x2APIC mode for virtual-interrupt delivery tests

Marc Orr (Google) (3):
nVMX: test nested "virtual-interrupt delivery"
nVMX: test nested EOI virtualization
nVMX: add self-IPI tests to vmx_basic_vid_test

Oliver Upton (1):
nVMX: add test for posted interrupts

 lib/x86/apic.h       |   5 +
 lib/x86/asm/bitops.h |   8 +
 x86/unittests.cfg    |  10 +-
 x86/vmx_tests.c      | 423 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 445 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.0.472.g3155946c3a-goog
#1 lore raw

PATCH 1/5 nVMX: Enable x2APIC mode for virtual-interrupt delivery tests

Since "virtualize x2APIC mode" is enabled for these tests, call enable_x2apic() so that the x2apic_ops function table will be installed.

---
 x86/vmx_tests.c | 1 +
 1 file changed, 1 insertion(+)
x86/vmx_tests.c +1 -0
index c1540d396da8..e5ed79b7da4a 100644
@@ -9305,6 +9305,7 @@ static void enable_vid(void)
 
 	assert(cpu_has_apicv());
 
+	enable_x2apic();
 	disable_intercept_for_x2apic_msrs();
 
 	virtual_apic_page = alloc_page();
Signature · 1 line
2.43.0.472.g3155946c3a-goog
#2 lore raw

PATCH 2/5 nVMX: test nested "virtual-interrupt delivery"

In reply to Jim Mattson

From: "Marc Orr (Google)" <[email protected]>

Add test coverage for recognizing and delivering virtual interrupts via VMX's "virtual-interrupt delivery" feature, in the following two scenarios:

1. There's a pending interrupt at VM-entry.
2. There's a pending interrupt during TPR virtualization.

---
 lib/x86/apic.h    |   5 ++
 x86/unittests.cfg |   2 +-
 x86/vmx_tests.c   | 165 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 171 insertions(+), 1 deletion(-)
lib/x86/apic.h +5 -0
index c389d40e169a..8df889b2d1e4 100644
@@ -81,6 +81,11 @@ static inline bool apic_lvt_entry_supported(int idx)
 	return GET_APIC_MAXLVT(apic_read(APIC_LVR)) >= idx;
 }
 
+static inline u8 task_priority_class(u8 vector)
+{
+	return vector >> 4;
+}
+
 enum x2apic_reg_semantics {
 	X2APIC_INVALID	= 0,
 	X2APIC_READABLE	= BIT(0),
x86/unittests.cfg +1 -1
index 3fe59449b650..dd086d9e2bf4 100644
@@ -361,7 +361,7 @@ timeout = 10
 
 [vmx_apicv_test]
 file = vmx.flat
-extra_params = -cpu max,+vmx -append "apic_reg_virt_test virt_x2apic_mode_test"
+extra_params = -cpu max,+vmx -append "apic_reg_virt_test virt_x2apic_mode_test vmx_basic_vid_test"
 arch = x86_64
 groups = vmx
 timeout = 10
x86/vmx_tests.c +165 -0

177 lines not on this page · raw message

Signature · 1 line
2.43.0.472.g3155946c3a-goog
#3 lore raw

PATCH 3/5 nVMX: test nested EOI virtualization

In reply to Jim Mattson

From: "Marc Orr (Google)" <[email protected]>

Add a test for nested VMs that invoke EOI virtualization. Specifically, check that a pending low-priority interrupt, masked by a higher-priority interrupt, is scheduled via "virtual-interrupt delivery," after the higher-priority interrupt executes EOI.

---
 x86/unittests.cfg |   2 +-
 x86/vmx_tests.c   | 161 ++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 136 insertions(+), 27 deletions(-)
x86/unittests.cfg +1 -1
index dd086d9e2bf4..f307168b0e01 100644
@@ -361,7 +361,7 @@ timeout = 10
 
 [vmx_apicv_test]
 file = vmx.flat
-extra_params = -cpu max,+vmx -append "apic_reg_virt_test virt_x2apic_mode_test vmx_basic_vid_test"
+extra_params = -cpu max,+vmx -append "apic_reg_virt_test virt_x2apic_mode_test vmx_basic_vid_test vmx_eoi_virt_test"
 arch = x86_64
 groups = vmx
 timeout = 10
x86/vmx_tests.c +135 -26

240 lines not on this page · raw message

Signature · 1 line
2.43.0.472.g3155946c3a-goog
#4 lore raw

PATCH 4/5 nVMX: add self-IPI tests to vmx_basic_vid_test

In reply to Jim Mattson

From: "Marc Orr (Google)" <[email protected]>

Extend the VMX "virtual-interrupt delivery test", vmx_basic_vid_test, to verify that virtual-interrupt delivery is triggered by a self-IPI in L2.

---
 x86/vmx_tests.c | 35 +++++++++++++++++++++++++----------
 1 file changed, 25 insertions(+), 10 deletions(-)
x86/vmx_tests.c +25 -10
index ce480431bf58..a26f77e92f72 100644
@@ -10720,6 +10720,7 @@ enum Vid_op {
 	VID_OP_SET_ISR,
 	VID_OP_NOP,
 	VID_OP_SET_CR8,
+	VID_OP_SELF_IPI,
 	VID_OP_TERMINATE,
 };
 
@@ -10779,6 +10780,9 @@ static void vmx_basic_vid_test_guest(void)
 		case VID_OP_SET_CR8:
 			write_cr8(nr);
 			break;
+		case VID_OP_SELF_IPI:
+			vmx_x2apic_write(APIC_SELF_IPI, nr);
+			break;
 		default:
 			break;
 		}
@@ -10817,7 +10821,7 @@ static void set_isrs_for_vmx_basic_vid_test(void)
  *   tpr_virt: If true, then test VID during TPR virtualization. Otherwise,
  *       test VID during VM-entry.
  */
-static void test_basic_vid(u8 nr, u8 tpr, bool tpr_virt, u32 isr_exec_cnt_want,
+static void test_basic_vid(u8 nr, u8 tpr, enum Vid_op op, u32 isr_exec_cnt_want,
 			   bool eoi_exit_induced)
 {
 	volatile struct vmx_basic_vid_test_guest_args *args =
@@ -10838,15 +10842,23 @@ static void test_basic_vid(u8 nr, u8 tpr, bool tpr_virt, u32 isr_exec_cnt_want,
 	 * However, PPR virtualization, which occurs before virtual interrupt
 	 * delivery, sets VPPR to VTPR, when SVI is 0.
 	 */
-	vmcs_write(GUEST_INT_STATUS, nr);
 	args->isr_exec_cnt = 0;
-	if (tpr_virt) {
-		args->op = VID_OP_SET_CR8;
+	args->op = op;
+	switch (op) {
+	case VID_OP_SELF_IPI:
+		vmcs_write(GUEST_INT_STATUS, 0);
+		args->nr = nr;
+		set_vtpr(0);
+		break;
+	case VID_OP_SET_CR8:
+		vmcs_write(GUEST_INT_STATUS, nr);
 		args->nr = task_priority_class(tpr);
 		set_vtpr(0xff);
-	} else {
-		args->op = VID_OP_NOP;
+		break;
+	default:
+		vmcs_write(GUEST_INT_STATUS, nr);
 		set_vtpr(tpr);
+		break;
 	}
 
 	enter_guest();
@@ -10903,15 +10915,18 @@ static void vmx_basic_vid_test(void)
 			if (nr == 0x20)
 				continue;
 
+			test_basic_vid(nr, /*tpr=*/0, VID_OP_SELF_IPI,
+				       /*isr_exec_cnt_want=*/1,
+				       /*eoi_exit_induced=*/false);
 			for (tpr = 0; tpr < 256; tpr++) {
 				u32 isr_exec_cnt_want =
 					task_priority_class(nr) >
 					task_priority_class(tpr) ? 1 : 0;
 
-				test_basic_vid(nr, tpr, /*tpr_virt=*/false,
+				test_basic_vid(nr, tpr, VID_OP_NOP,
 					       isr_exec_cnt_want,
 					       /*eoi_exit_induced=*/false);
-				test_basic_vid(nr, tpr, /*tpr_virt=*/true,
+				test_basic_vid(nr, tpr, VID_OP_SET_CR8,
 					       isr_exec_cnt_want,
 					       /*eoi_exit_induced=*/false);
 			}
@@ -10930,8 +10945,8 @@ static void test_eoi_virt(u8 nr, u8 lo_pri_nr, bool eoi_exit_induced)
 	u32 *virtual_apic_page = get_vapic_page();
 
 	set_virr_bit(virtual_apic_page, lo_pri_nr);
-	test_basic_vid(nr, /*tpr=*/0, /*tpr_virt=*/false,
-		       /*isr_exec_cnt_want=*/2, eoi_exit_induced);
+	test_basic_vid(nr, /*tpr=*/0, VID_OP_NOP, /*isr_exec_cnt_want=*/2,
+		       eoi_exit_induced);
 	TEST_ASSERT(!get_virr_bit(virtual_apic_page, lo_pri_nr));
 	TEST_ASSERT(!get_virr_bit(virtual_apic_page, nr));
 }
Signature · 1 line
2.43.0.472.g3155946c3a-goog
#5 lore raw

PATCH 5/5 nVMX: add test for posted interrupts

In reply to Jim Mattson

From: Oliver Upton <[email protected]>

Test virtual posted interrupts under the following conditions:

- vTPR[7:4] >= VECTOR[7:4]: Expect the L2 interrupt to be blocked. The bit corresponding to the posted interrupt should be set in L2's vIRR. Test with a running guest.

- vTPR[7:4] < VECTOR[7:4]: Expect the interrupt to be delivered and the ISR to execute once. Test with a running and halted guest.

---
 lib/x86/asm/bitops.h |   8 +++
 x86/unittests.cfg    |   8 +++
 x86/vmx_tests.c      | 133 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 149 insertions(+)
lib/x86/asm/bitops.h +8 -0
index 13a25ec9853d..54ec9c424cd6 100644
@@ -13,4 +13,12 @@
 
 #define HAVE_BUILTIN_FLS 1
 
+static inline void test_and_set_bit(long nr, unsigned long *addr)
+{
+	asm volatile("lock; bts %1,%0"
+		     : "+m" (*addr)
+		     : "Ir" (nr)
+		     : "memory");
+}
+
 #endif
x86/unittests.cfg +8 -0
index f307168b0e01..9598c61ef7ac 100644
@@ -366,6 +366,14 @@ arch = x86_64
 groups = vmx
 timeout = 10
 
+[vmx_posted_intr_test]
+file = vmx.flat
+smp = 2
+extra_params = -cpu max,+vmx -append "vmx_posted_interrupts_test"
+arch = x86_64
+groups = vmx
+timeout = 10
+
 [vmx_apic_passthrough_thread]
 file = vmx.flat
 smp = 2
x86/vmx_tests.c +133 -0

205 lines not on this page · raw message

Signature · 1 line
2.43.0.472.g3155946c3a-goog
#6 lore raw
In reply to Jim Mattson
Quoted 8 lines · Jim Mattson

I reported recently that commit 26844fee6ade ("KVM: x86: never write to memory from kvm_vcpu_check_block()") broke delivery of a virtualized posted interrupt from an L1 vCPU to a halted L2 vCPU (see
https://lore.kernel.org/all/[email protected]/).
The test that exposed the regression is the final patch of this series. The others are prerequisites.

[...]

Applied to kvm-x86 next. I tweaked the commits to exclude the new tests from
the base VMX test, i.e. added:

-vmx_basic_vid_test -vmx_eoi_virt_test -vmx_posted_interrupts_test

to not duplicate coverage, and so that only the dedicated vmx_posted_intr_test config fails.

I am hoping that landing vmx_posted_intr_test motivates me (or someone) to actually fix that issue.

[1/5] nVMX: Enable x2APIC mode for virtual-interrupt delivery tests

      https://github.com/kvm-x86/kvm-unit-tests/commit/eef1e3d21e5a
[2/5] nVMX: test nested "virtual-interrupt delivery"
      https://github.com/kvm-x86/kvm-unit-tests/commit/a917f7c7e1e2
[3/5] nVMX: test nested EOI virtualization
      https://github.com/kvm-x86/kvm-unit-tests/commit/d485a75d8c52
[4/5] nVMX: add self-IPI tests to vmx_basic_vid_test
      https://github.com/kvm-x86/kvm-unit-tests/commit/3238da6bc526
[5/5] nVMX: add test for posted interrupts
      https://github.com/kvm-x86/kvm-unit-tests/commit/765b349b6c4b

--
https://github.com/kvm-x86/kvm-unit-tests/tree/next

#7 lore raw
In reply to Jim Mattson
Quoted 13 lines · Jim Mattson

I reported recently that commit 26844fee6ade ("KVM: x86: never write to memory from kvm_vcpu_check_block()") broke delivery of a virtualized posted interrupt from an L1 vCPU to a halted L2 vCPU (see
https://lore.kernel.org/all/[email protected]/).
The test that exposed the regression is the final patch of this series. The others are prerequisites.

It would make sense to add "vmx_posted_interrupts_test" to the set of tests to be run under the unit test name, "vmx_apicv_test," but that is non-trivial. The vmx_posted_interrupts_test requires "smp = 2," but I find that adding that to the vmx_apicv_tests causes virt_x2apic_mode_test to fail with:

Quoted 7 lines

FAIL: x2apic - reading 0x310: x86/vmx_tests.c:2151: Assertion failed: (expected) == (actual)

	LHS: 0x0000000000000012 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010 - 18
	RHS: 0x0000000000000001 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001 - 1
Expected VMX_VMCALL, got VMX_EXTINT.
	STACK: 406ef8 40725a 41299f 402036 403f59 4001bd

I haven't investigated.

This vmx_apicv_test test still fails when 'ept=N' (SPR + v6.12-rc2):

--- Virtualize APIC accesses + Use TPR shadow test ---
FAIL: xapic - reading 0x080: read 0x0, expected 0x70.
FAIL: xapic - writing 0x12345678 to 0x080: exitless write; val is 0x0, want 0x70

--- APIC-register virtualization test ---
FAIL: xapic - reading 0x020: read 0x0, expected 0x12345678.
FAIL: xapic - writing 0x12345678 to 0x020: x86/vmx_tests.c:2164: Assertion failed: (expected) == (actual)

	LHS: 0x0000000000000038 - 
0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0011'1000 - 56
	RHS: 0x0000000000000012 - 
0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010 - 18
Expected VMX_APIC_WRITE, got VMX_VMCALL.
	STACK: 406f7f 40d178 40202f 403f54 4001bd
Quoted 19 lines

Jim Mattson (1):
nVMX: Enable x2APIC mode for virtual-interrupt delivery tests

Marc Orr (Google) (3):
nVMX: test nested "virtual-interrupt delivery"
nVMX: test nested EOI virtualization
nVMX: add self-IPI tests to vmx_basic_vid_test

Oliver Upton (1):
nVMX: add test for posted interrupts

  lib/x86/apic.h       |   5 +
  lib/x86/asm/bitops.h |   8 +
  x86/unittests.cfg    |  10 +-
  x86/vmx_tests.c      | 423 +++++++++++++++++++++++++++++++++++++++++++
  4 files changed, 445 insertions(+), 1 deletion(-)
#8 lore raw
Quoted 33 lines · Like Xu
Quoted 13 lines · Jim Mattson

I reported recently that commit 26844fee6ade ("KVM: x86: never write to memory from kvm_vcpu_check_block()") broke delivery of a virtualized posted interrupt from an L1 vCPU to a halted L2 vCPU (see
https://lore.kernel.org/all/[email protected]/).
The test that exposed the regression is the final patch of this series. The others are prerequisites.

It would make sense to add "vmx_posted_interrupts_test" to the set of tests to be run under the unit test name, "vmx_apicv_test," but that is non-trivial. The vmx_posted_interrupts_test requires "smp = 2," but I find that adding that to the vmx_apicv_tests causes virt_x2apic_mode_test to fail with:

Quoted 7 lines

FAIL: x2apic - reading 0x310: x86/vmx_tests.c:2151: Assertion failed: (expected) == (actual)

	LHS: 0x0000000000000012 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010 - 18
	RHS: 0x0000000000000001 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001 - 1
Expected VMX_VMCALL, got VMX_EXTINT.
	STACK: 406ef8 40725a 41299f 402036 403f59 4001bd

I haven't investigated.

This vmx_apicv_test test still fails when 'ept=N' (SPR + v6.12-rc2):

--- Virtualize APIC accesses + Use TPR shadow test ---
FAIL: xapic - reading 0x080: read 0x0, expected 0x70.
FAIL: xapic - writing 0x12345678 to 0x080: exitless write; val is 0x0, want 0x70

--- APIC-register virtualization test ---
FAIL: xapic - reading 0x020: read 0x0, expected 0x12345678.
FAIL: xapic - writing 0x12345678 to 0x020: x86/vmx_tests.c:2164: Assertion
failed: (expected) == (actual)

Quoted 6 lines
	LHS: 0x0000000000000038 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0011'1000
- 56
	RHS: 0x0000000000000012 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010
- 18
Expected VMX_APIC_WRITE, got VMX_VMCALL.
	STACK: 406f7f 40d178 40202f 403f54 4001bd

These failures occur because KVM flushes TLB with the wrong VPID, causing TLB for the 'APIC-access page' to be retained across nested transitions. This TLB entry exists because L1 writes to that page before entering the guest, see test_xapic_rd():

        /* Setup virtual APIC page */
        if (!expectation->virtualize_apic_accesses) {
                apic_access_address[apic_reg_index(reg)] = val;
                virtual_apic_page[apic_reg_index(reg)] = 0;
        } else if (exit_reason_want == VMX_VMCALL) {
                apic_access_address[apic_reg_index(reg)] = 0;
                virtual_apic_page[apic_reg_index(reg)] = val;
        }

Specifically, in the failing scenario, EPT is disabled, and VPID is enabled in L0 but disabled in L1. As a result, vmcs01 and vmcs02 share the same VPID (vmx->vpid, see prepare_vmcs02_early_rare()), and vmx->nested.vpid02 is never used. But during nested transitions, KVM incorrectly flushes TLB using vmx->nested.vpid02. The sequence is as follows:

	nested_vmx_transition_tlb_flush ->
	  kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu) ->
	    kvm_vcpu_flush_tlb_guest ->	
	      vmx_flush_tlb_guest ->
		vmx_get_current_vpid ->

With the diff below applied, these failures disappear.

arch/x86/kvm/vmx/nested.h +10 -7
index cce4e2aa30fb..246d9c6e20d0 100644
@@ -61,13 +61,6 @@ static inline int vmx_has_valid_vmcs12(struct kvm_vcpu *vcpu)
 		nested_vmx_is_evmptr12_set(vmx);
 }
 
-static inline u16 nested_get_vpid02(struct kvm_vcpu *vcpu)
-{
-	struct vcpu_vmx *vmx = to_vmx(vcpu);
-
-	return vmx->nested.vpid02 ? vmx->nested.vpid02 : vmx->vpid;
-}
-
 static inline unsigned long nested_ept_get_eptp(struct kvm_vcpu *vcpu)
 {
 	/* return the page table to be shadowed - in our case, EPT12 */
@@ -187,6 +180,16 @@ static inline bool nested_cpu_has_vpid(struct vmcs12 *vmcs12)
 	return nested_cpu_has2(vmcs12, SECONDARY_EXEC_ENABLE_VPID);
 }
 
+static inline u16 nested_get_vpid02(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+
+	if (nested_cpu_has_vpid(get_vmcs12(vcpu)) && vmx->nested.vpid02)
+		return vmx->nested.vpid02;
+
+	return vmx->vpid;
+}
+
 static inline bool nested_cpu_has_apic_reg_virt(struct vmcs12 *vmcs12)
 {
 	return nested_cpu_has2(vmcs12, SECONDARY_EXEC_APIC_REGISTER_VIRT);
#9 lore raw
Quoted 89 lines · Chao Gao
Quoted 33 lines · Like Xu
Quoted 13 lines · Jim Mattson

I reported recently that commit 26844fee6ade ("KVM: x86: never write to memory from kvm_vcpu_check_block()") broke delivery of a virtualized posted interrupt from an L1 vCPU to a halted L2 vCPU (see
https://lore.kernel.org/all/[email protected]/).
The test that exposed the regression is the final patch of this series. The others are prerequisites.

It would make sense to add "vmx_posted_interrupts_test" to the set of tests to be run under the unit test name, "vmx_apicv_test," but that is non-trivial. The vmx_posted_interrupts_test requires "smp = 2," but I find that adding that to the vmx_apicv_tests causes virt_x2apic_mode_test to fail with:

Quoted 7 lines

FAIL: x2apic - reading 0x310: x86/vmx_tests.c:2151: Assertion failed: (expected) == (actual)

	LHS: 0x0000000000000012 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010 - 18
	RHS: 0x0000000000000001 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001 - 1
Expected VMX_VMCALL, got VMX_EXTINT.
	STACK: 406ef8 40725a 41299f 402036 403f59 4001bd

I haven't investigated.

This vmx_apicv_test test still fails when 'ept=N' (SPR + v6.12-rc2):

--- Virtualize APIC accesses + Use TPR shadow test ---
FAIL: xapic - reading 0x080: read 0x0, expected 0x70.
FAIL: xapic - writing 0x12345678 to 0x080: exitless write; val is 0x0, want 0x70

--- APIC-register virtualization test ---
FAIL: xapic - reading 0x020: read 0x0, expected 0x12345678.
FAIL: xapic - writing 0x12345678 to 0x020: x86/vmx_tests.c:2164: Assertion
failed: (expected) == (actual)

Quoted 6 lines
	LHS: 0x0000000000000038 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0011'1000
- 56
	RHS: 0x0000000000000012 - 0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0000'0001'0010
- 18
Expected VMX_APIC_WRITE, got VMX_VMCALL.
	STACK: 406f7f 40d178 40202f 403f54 4001bd

These failures occur because KVM flushes TLB with the wrong VPID, causing TLB for the 'APIC-access page' to be retained across nested transitions. This TLB entry exists because L1 writes to that page before entering the guest, see test_xapic_rd():

        /* Setup virtual APIC page */
        if (!expectation->virtualize_apic_accesses) {
                apic_access_address[apic_reg_index(reg)] = val;
                virtual_apic_page[apic_reg_index(reg)] = 0;
        } else if (exit_reason_want == VMX_VMCALL) {
                apic_access_address[apic_reg_index(reg)] = 0;
                virtual_apic_page[apic_reg_index(reg)] = val;
        }

Specifically, in the failing scenario, EPT is disabled, and VPID is enabled in L0 but disabled in L1. As a result, vmcs01 and vmcs02 share the same VPID (vmx->vpid, see prepare_vmcs02_early_rare()), and vmx->nested.vpid02 is never used. But during nested transitions, KVM incorrectly flushes TLB using vmx->nested.vpid02. The sequence is as follows:

	nested_vmx_transition_tlb_flush ->
	  kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu) ->
	    kvm_vcpu_flush_tlb_guest ->	
	      vmx_flush_tlb_guest ->
		vmx_get_current_vpid ->

With the diff below applied, these failures disappear.

arch/x86/kvm/vmx/nested.h +0 -7
index cce4e2aa30fb..246d9c6e20d0 100644
@@ -61,13 +61,6 @@ static inline int vmx_has_valid_vmcs12(struct kvm_vcpu *vcpu)
 		nested_vmx_is_evmptr12_set(vmx);
 }
 
-static inline u16 nested_get_vpid02(struct kvm_vcpu *vcpu)
-{
-	struct vcpu_vmx *vmx = to_vmx(vcpu);
-
-	return vmx->nested.vpid02 ? vmx->nested.vpid02 : vmx->vpid;
-}
-
 static inline unsigned long nested_ept_get_eptp(struct kvm_vcpu *vcpu)
 {
 	/* return the page table to be shadowed - in our case, EPT12 */

arch/x86/kvm/vmx/nested.h static inline bool nested_cpu_has_vpid(struct vmcs12 *vmcs12) · line 187

Quoted 8 lines
arch/x86/kvm/vmx/nested.h +5 -0
@@ -187,6 +180,16 @@ static inline bool nested_cpu_has_vpid(struct vmcs12 *vmcs12)
 	return nested_cpu_has2(vmcs12, SECONDARY_EXEC_ENABLE_VPID);
 }
 
+static inline u16 nested_get_vpid02(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_vmx *vmx = to_vmx(vcpu);
+
+	if (nested_cpu_has_vpid(get_vmcs12(vcpu)) && vmx->nested.vpid02)

This isn't quite right. When KVM emulates INVVPID for L1, there is no way to know which vmcs12 will be used and thus no way to know if KVM should invalidate vpid02 or vpid01. So I'm fairly certain the correct fix is:

arch/x86/kvm/vmx/vmx.c +1 -1
index 1a4438358c5e..896f0fea0306 100644
@@ -3216,7 +3216,7 @@ void vmx_flush_tlb_all(struct kvm_vcpu *vcpu)
 
 static inline int vmx_get_current_vpid(struct kvm_vcpu *vcpu)
 {
-       if (is_guest_mode(vcpu))
+       if (is_guest_mode(vcpu) && nested_cpu_has_vpid(get_vmcs12(vcpu)))
                return nested_get_vpid02(vcpu);
        return to_vmx(vcpu)->vpid;
 }

Note, it's tempting, but subtly wrong (though not a violation of the architecture), to use vpid02 when VPID is disabled in vmcs12, i.e. this would also resolve the issue, but would result in over-flushing.

arch/x86/kvm/vmx/nested.c +1 -1
index a8e7bc04d9bf..ce89e2e27681 100644
@@ -2316,7 +2316,7 @@ static void prepare_vmcs02_early_rare(struct vcpu_vmx *vmx,
        vmcs_write64(VMCS_LINK_POINTER, INVALID_GPA);
 
        if (enable_vpid) {
-               if (nested_cpu_has_vpid(vmcs12) && vmx->nested.vpid02)
+               if (vmx->nested.vpid02)
                        vmcs_write16(VIRTUAL_PROCESSOR_ID, vmx->nested.vpid02);
                else
                        vmcs_write16(VIRTUAL_PROCESSOR_ID, vmx->vpid);

And it's wrong because the comment in nested_vmx_transition_tlb_flush() is wrong:

* If vmcs12 doesn't use VPID, L1 expects linear and combined mappings
* for *all* contexts to be flushed on VM-Enter/VM-Exit, i.e. it's a
* full TLB flush from the guest's perspective

Per the SDM, only VPID=0 is flushed:

If the “enable VPID” VM-execution control is 0, VM entries and VM exits invalidate linear mappings and combined mappings associated with VPID 0000H (for all PCIDs). Combined mappings for VPID 0000H are invalidated for all EPTRTAs.

so using vpid01, which mimics using L1's host VPID (of '0'), is correct. As above, the fallout is simply that KVM will over-flush, e.g. if L1 runs L2 X with VPID=1, then L2 Y with VPID disabled, and then runs X again, it's legal to retain TLB entries for VPID=1 even though there was a VMX transition with VPID disabled.

I'll post a proper patch with lots of comments.

Thanks Chao!