Skip to contentSkip to conversation

Regression test for L1 LDTR persistence bug

PATCH In review Jim Mattson 3 participants 9 messages Oct 15, 2021 – Nov 30, 2021 KVM

lore
Share & tools
lore
#0 lore raw

v2 cover Regression test for L1 LDTR persistence bug

In Linux commit afc8de0118be ("KVM: nVMX: Set LDTR to its architecturally defined value on nested VM-Exit"), Sean suggested that this bug was likely benign, but it turns out that--for us, at least--it can result in live migration failures. On restore, we call KVM_SET_SREGS before KVM_SET_NESTED_STATE, so when L2 is active at the time of save/restore, the target vmcs01 is temporarily populated with L2 values. Hence, the LDTR visible to L1 after the next emulated VM-exit is L2's, rather than its own.

This issue is significant enough that it warrants a regression test. Unfortunately, at the moment, the best we can do is check for the LDTR persistence bug. I'd like to be able to trigger a save/restore from within the L2 guest, but AFAICT, there's no way to do that under qemu. Does anyone want to implement a qemu ISA test device that triggers a save/restore when its configured I/O port is written to?

Jim Mattson (3):
x86: Fix operand size for lldt
x86: Make set_gdt_entry usable in 64-bit mode
x86: Add a regression test for L1 LDTR persistence bug

v1 -> v2:
Reworded report messages at Sean's suggestion.

 lib/x86/desc.c      | 41 +++++++++++++++++++++++++++++++----------
 lib/x86/desc.h      |  3 ++-
 lib/x86/processor.h |  2 +-
 x86/cstart64.S      |  1 +
 x86/vmx_tests.c     | 39 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 74 insertions(+), 12 deletions(-)
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
#1 lore raw

PATCH v2 1/3 x86: Fix operand size for lldt

The lldt instruction takes an r/m16 operand.

Fixes 7d36db351752 ("Initial commit from qemu-kvm.git kvm/test/")
---
 lib/x86/processor.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/x86/processor.h +1 -1
index eaf24d491499..fe5add548261 100644
@@ -443,7 +443,7 @@ static inline void sidt(struct descriptor_table_ptr *ptr)
     asm volatile ("sidt %0" : "=m"(*ptr));
 }
 
-static inline void lldt(unsigned val)
+static inline void lldt(u16 val)
 {
     asm volatile ("lldt %0" : : "rm"(val));
 }
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
#2 lore raw

PATCH v2 2/3 x86: Make set_gdt_entry usable in 64-bit mode

In reply to Jim Mattson

The omission of a 64-bit implementation of set_gdt_entry was probably just an oversight, since the declaration is in scope.

---
 lib/x86/desc.c | 41 +++++++++++++++++++++++++++++++----------
 lib/x86/desc.h |  3 ++-
 x86/cstart64.S |  1 +
 3 files changed, 34 insertions(+), 11 deletions(-)
lib/x86/desc.c +31 -10
index e7378c1d372a..87e2850f46c6 100644
@@ -279,23 +279,39 @@ bool exception_rflags_rf(void)
 
 static char intr_alt_stack[4096];
 
-#ifndef __x86_64__
-void set_gdt_entry(int sel, u32 base,  u32 limit, u8 access, u8 gran)
+static void __set_gdt_entry(int sel, uintptr_t base, u32 limit, u8 access,
+			    u8 gran, gdt_entry_t *gdt)
 {
 	int num = sel >> 3;
 
 	/* Setup the descriptor base address */
-	gdt32[num].base_low = (base & 0xFFFF);
-	gdt32[num].base_middle = (base >> 16) & 0xFF;
-	gdt32[num].base_high = (base >> 24) & 0xFF;
+	gdt[num].base_low = (base & 0xFFFF);
+	gdt[num].base_middle = (base >> 16) & 0xFF;
+	gdt[num].base_high = (base >> 24) & 0xFF;
 
 	/* Setup the descriptor limits */
-	gdt32[num].limit_low = (limit & 0xFFFF);
-	gdt32[num].granularity = ((limit >> 16) & 0x0F);
+	gdt[num].limit_low = (limit & 0xFFFF);
+	gdt[num].granularity = ((limit >> 16) & 0x0F);
+
+	/* Penultimately, set up the granularity and access flags */
+	gdt[num].granularity |= (gran & 0xF0);
+	gdt[num].access = access;
+
+#ifdef __x86_64__
+	/* 64-bit system descriptors take up two slots */
+	if (!(access & 0x10000)) {
+		u32 *p = (u32 *)&gdt[num + 1];
 
-	/* Finally, set up the granularity and access flags */
-	gdt32[num].granularity |= (gran & 0xF0);
-	gdt32[num].access = access;
+		p[0] = base >> 32;
+		p[1] = 0;
+	}
+#endif
+}
+
+#ifndef __x86_64__
+void set_gdt_entry(int sel, uintptr_t base, u32 limit, u8 access, u8 gran)
+{
+	__set_gdt_entry(sel, base, limit, access, gran, gdt32);
 }
 
 void set_gdt_task_gate(u16 sel, u16 tss_sel)
@@ -366,6 +382,11 @@ void print_current_tss_info(void)
 		       tr, tr ? "interrupt" : "main", tss.prev, tss_intr.prev);
 }
 #else
+void set_gdt_entry(int sel, uintptr_t base, u32 limit, u8 access, u8 gran)
+{
+	__set_gdt_entry(sel, base, limit, access, gran, gdt64);
+}
+
 void set_intr_alt_stack(int e, void *addr)
 {
 	set_idt_entry(e, addr, 0);
lib/x86/desc.h +2 -1
index a6ffb38c79a1..994f1d791130 100644
@@ -208,6 +208,7 @@ void set_idt_task_gate(int vec, u16 sel);
 void set_intr_task_gate(int vec, void *fn);
 void setup_tss32(void);
 #else
+extern gdt_entry_t gdt64[];
 extern tss64_t tss;
 #endif
 
@@ -218,7 +219,7 @@ bool exception_rflags_rf(void);
 void set_idt_entry(int vec, void *addr, int dpl);
 void set_idt_sel(int vec, u16 sel);
 void set_idt_dpl(int vec, u16 dpl);
-void set_gdt_entry(int sel, u32 base,  u32 limit, u8 access, u8 gran);
+void set_gdt_entry(int sel, uintptr_t base,  u32 limit, u8 access, u8 gran);
 void set_intr_alt_stack(int e, void *fn);
 void print_current_tss_info(void);
 handler handle_exception(u8 v, handler fn);
x86/cstart64.S +1 -0
index 5c6ad38543cc..62ace3512bb0 100644
@@ -62,6 +62,7 @@ gdt64_desc:
 	.word gdt64_end - gdt64 - 1
 	.quad gdt64
 
+.globl gdt64
 gdt64:
 	.quad 0
 	.quad 0x00af9b000000ffff // 64-bit code segment
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
#3 lore raw

PATCH v2 3/3 x86: Add a regression test for L1 LDTR persistence bug

In reply to Jim Mattson

Add a regression test for Linux commit afc8de0118be ("KVM: nVMX: Set LDTR to its architecturally defined value on nested VM-Exit"). L1's LDTR should be 0 after an emulated VM-exit from L2.

---
 x86/vmx_tests.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
x86/vmx_tests.c +39 -0
index 3b97cfa6ed10..6093a90fd4ac 100644
@@ -8363,6 +8363,44 @@ static void vmentry_movss_shadow_test(void)
 	vmcs_write(GUEST_RFLAGS, X86_EFLAGS_FIXED);
 }
 
+static void vmx_ldtr_test_guest(void)
+{
+	u16 ldtr = sldt();
+
+	report(ldtr == NP_SEL, "Expected %x for L2 LDTR selector (got %x)",
+	       NP_SEL, ldtr);
+}
+
+/*
+ * Ensure that the L1 LDTR is set to 0 on VM-exit.
+ */
+static void vmx_ldtr_test(void)
+{
+	const u8 ldt_ar = 0x82; /* Present LDT */
+	u16 sel = FIRST_SPARE_SEL;
+
+	/* Set up a non-zero L1 LDTR prior to VM-entry. */
+	set_gdt_entry(sel, 0, 0, ldt_ar, 0);
+	lldt(sel);
+
+	test_set_guest(vmx_ldtr_test_guest);
+	/*
+	 * Set up a different LDTR for L2. The actual GDT contents are
+	 * irrelevant, since we stuff the hidden descriptor state
+	 * straight into the VMCS rather than reading it from the GDT.
+	 */
+	vmcs_write(GUEST_SEL_LDTR, NP_SEL);
+	vmcs_write(GUEST_AR_LDTR, ldt_ar);
+	enter_guest();
+
+	/*
+	 * VM-exit should clear LDTR (and make it unusable, but we
+	 * won't verify that here).
+	 */
+	sel = sldt();
+	report(!sel, "Expected 0 for L1 LDTR selector (got %x)", sel);
+}
+
 static void vmx_single_vmcall_guest(void)
 {
 	vmcall();
@@ -10724,6 +10762,7 @@ struct vmx_test vmx_tests[] = {
 	/* VMCS Shadowing tests */
 	TEST(vmx_vmcs_shadow_test),
 	/* Regression tests */
+	TEST(vmx_ldtr_test),
 	TEST(vmx_cr_load_test),
 	TEST(vmx_cr4_osxsave_test),
 	TEST(vmx_nm_test),
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
#4 lore raw
In reply to Jim Mattson
Quoted 33 lines · Jim Mattson

In Linux commit afc8de0118be ("KVM: nVMX: Set LDTR to its architecturally defined value on nested VM-Exit"), Sean suggested that this bug was likely benign, but it turns out that--for us, at least--it can result in live migration failures. On restore, we call KVM_SET_SREGS before KVM_SET_NESTED_STATE, so when L2 is active at the time of save/restore, the target vmcs01 is temporarily populated with L2 values. Hence, the LDTR visible to L1 after the next emulated VM-exit is L2's, rather than its own.

This issue is significant enough that it warrants a regression test. Unfortunately, at the moment, the best we can do is check for the LDTR persistence bug. I'd like to be able to trigger a save/restore from within the L2 guest, but AFAICT, there's no way to do that under qemu. Does anyone want to implement a qemu ISA test device that triggers a save/restore when its configured I/O port is written to?

Jim Mattson (3):
x86: Fix operand size for lldt
x86: Make set_gdt_entry usable in 64-bit mode
x86: Add a regression test for L1 LDTR persistence bug

v1 -> v2:
Reworded report messages at Sean's suggestion.

 lib/x86/desc.c      | 41 +++++++++++++++++++++++++++++++----------
 lib/x86/desc.h      |  3 ++-
 lib/x86/processor.h |  2 +-
 x86/cstart64.S      |  1 +
 x86/vmx_tests.c     | 39 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 74 insertions(+), 12 deletions(-)
Quoted 3 lines

--
2.33.0.1079.g6e70778dc9-goog

Ping.

maintainer
#5 lore raw
In reply to Jim Mattson
Quoted 7 lines · Jim Mattson

This issue is significant enough that it warrants a regression test. Unfortunately, at the moment, the best we can do is check for the LDTR persistence bug. I'd like to be able to trigger a save/restore from within the L2 guest, but AFAICT, there's no way to do that under qemu. Does anyone want to implement a qemu ISA test device that triggers a save/restore when its configured I/O port is written to?

The selftests infrastructure already has save/restore tests at instruction granularity (state_test.c) so you should have more luck that way; these tests are worthwhile anyway.

Paolo

maintainer
#8 lore raw
In reply to Jim Mattson
Quoted 25 lines · Jim Mattson

In Linux commit afc8de0118be ("KVM: nVMX: Set LDTR to its architecturally defined value on nested VM-Exit"), Sean suggested that this bug was likely benign, but it turns out that--for us, at least--it can result in live migration failures. On restore, we call KVM_SET_SREGS before KVM_SET_NESTED_STATE, so when L2 is active at the time of save/restore, the target vmcs01 is temporarily populated with L2 values. Hence, the LDTR visible to L1 after the next emulated VM-exit is L2's, rather than its own.

This issue is significant enough that it warrants a regression test. Unfortunately, at the moment, the best we can do is check for the LDTR persistence bug. I'd like to be able to trigger a save/restore from within the L2 guest, but AFAICT, there's no way to do that under qemu. Does anyone want to implement a qemu ISA test device that triggers a save/restore when its configured I/O port is written to?

Jim Mattson (3):
x86: Fix operand size for lldt
x86: Make set_gdt_entry usable in 64-bit mode
x86: Add a regression test for L1 LDTR persistence bug

v1 -> v2:
Reworded report messages at Sean's suggestion.

Quoted 7 lines
  lib/x86/desc.c      | 41 +++++++++++++++++++++++++++++++----------
  lib/x86/desc.h      |  3 ++-
  lib/x86/processor.h |  2 +-
  x86/cstart64.S      |  1 +
  x86/vmx_tests.c     | 39 +++++++++++++++++++++++++++++++++++++++
  5 files changed, 74 insertions(+), 12 deletions(-)

Queued patches 1 and 3; the function of 2 is also done by my own GDT/IDT cleanup series, which is also present in Aaron's #PF tests.

Paolo

#6 lore raw
In reply to Paolo Bonzini
Quoted 10 lines · Paolo Bonzini
Quoted 7 lines · Jim Mattson

This issue is significant enough that it warrants a regression test. Unfortunately, at the moment, the best we can do is check for the LDTR persistence bug. I'd like to be able to trigger a save/restore from within the L2 guest, but AFAICT, there's no way to do that under qemu. Does anyone want to implement a qemu ISA test device that triggers a save/restore when its configured I/O port is written to?

Quoted 3 lines

The selftests infrastructure already has save/restore tests at instruction granularity (state_test.c) so you should have more luck that way; these tests are worthwhile anyway.

There is also (I just discovered) support for guest-triggerable migration in the kvm-unit-tests [1]. You could use that to trigger a save/restore.

[1] Example: https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/blob/master/arm/gic.c#L798

Quoted 3 lines

Paolo

maintainer
#7 lore raw
Quoted 6 lines · David Matlack
Quoted 3 lines

The selftests infrastructure already has save/restore tests at instruction granularity (state_test.c) so you should have more luck that way; these tests are worthwhile anyway.

There is also (I just discovered) support for guest-triggerable migration in the kvm-unit-tests [1]. You could use that to trigger a save/restore.

There is, but I don't think it's good. Save/restore bugs almost invariably depend on very specific processor state, and it's hard to ensure that in kvm-unit-tests (except probabilistically, but we can do better).

Paolo

Quoted 2 lines

[1] Example:https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/blob/master/arm/gic.c#L798