Skip to contentSkip to conversation

nVMX: Improve DEBUGCTL test coverage

PATCH Applied Sean Christopherson 3 participants 8 messages May 27 – Sep 25 KVM

lore
Share & tools
lore

Sean Christopherson

author reviewer
#0 lore raw

v3 cover nVMX: Improve DEBUGCTL test coverage

Current nested debugctls test focuses on DR7 value and fakes the result of IA32_DEBUGCTLMSR in comment. Although the test can pass because DR7 is the only criterion to determine the result, there are some error messages appearing in dmesg due to accessing to BTF | LBR bits in IA32_DEBUGCTLMSR with report_ignored_msrs KVM parameter enabled.

Clean up the old test so that it focuses on DR7, then add a separate test for DEBUGCTLMSR bits that are actually supported and can be validated during VMX transitions.

v3:
- Massage changelog in patch 1. [Xiaoyao]
- Combine all DEBUGCTL tests into one testcase. [Sean]
- Validate all supported combinations. [Sean]
- Independently validate LOAD/SAVE controls. [Sean]
- Drop unnecessary configuration entry. [Sean]

v2:
- Modify the commit message in patch 1 (Xiaoyao)
- Add dedicated tests (for BUS_LOCK_DETECT and LBR) covering the DEBUGCTL reads and writes during VMX transitions (Sean)
- Add one additional test for RTM_DEBUG bit that was not mentioned in v1.

v1: https://lore.kernel.org/kvm/[email protected]

Chenyi Qiang (2):
nVMX: Remove the IA32_DEBUGCTLMSR access in debugctls test x86/vmx: (Re)Add a nVMX test to validate load/save of DEBUGCTL

 lib/x86/msr.h   |   2 +
 x86/vmx_tests.c | 196 ++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 166 insertions(+), 32 deletions(-)

base-commit: 05b7715ffd4ae4ef8f0a5dea869f1543d89f936d

Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog
author reviewer
#1 lore raw

PATCH v3 1/2 nVMX: Remove the IA32_DEBUGCTLMSR access in debugctls test

From: Chenyi Qiang <[email protected]>

"debug controls" test was added by commit dc5c01f17b1a ("VMX: Test behavior on set and cleared save/load debug controls") to verify that "VM-Entry load debug controls" and "VM-Exit save debug controls" are correctly emulated by KVM for nested VMX. But due to the limitation that KVM didn't support MSR_IA32_DEBUGCTL for guest at that time, the test commented out all the value comparison of MSR_IA32_DEBUGCTL and leave it to future when KVM supports the MSR.

The test doesn't check the functionality of save/restore guest MSR_IA32_DEBUGCTL on vm-exit/entry, but it keeps the write of MSR_IA32_DEBUGCTL. It leads to

kvm_intel: kvm [18663]: vcpu0, guest rIP: 0x407de7 Unhandled WRMSR(0x1d9) = 0x1
kvm_intel: kvm [18663]: vcpu0, guest rIP: 0x0 Unhandled WRMSR(0x1d9) = 0x2
kvm_intel: kvm [18663]: vcpu0, guest rIP: 0x40936f Unhandled WRMSR(0x1d9) = 0x3
kvm_intel: kvm [18663]: vcpu0, guest rIP: 0x40cf09 Unhandled WRMSR(0x1d9) = 0x1
kvm_intel: kvm [18663]: vcpu0, guest rIP: 0x40940d Unhandled WRMSR(0x1d9) = 0x3

to the kernel log. Though it doesn't break the test, the log confuses people to think something wrong happened in the test case, and the test isn't actually validating anything useful.

Now that KVM supports some MSR_IA32_DEBUGCTL based features (dependent on the underlying hardware), drop the old, half-baked code for DEBUGCTL in anticipation of adding a dedicated functional test for DEBUTCTL. Keep the existing DR7 coverage, but rename the testcase to explicitly scope it to just DR7.

[sean: massage changelog, name testcase dbgctls_dr7]
---
 x86/vmx_tests.c | 46 ++++++++++++++--------------------------------
 1 file changed, 14 insertions(+), 32 deletions(-)
x86/vmx_tests.c +14 -32
index 7aa3194c..e1f53fbe 100644
@@ -1858,21 +1858,18 @@ static int nmi_hlt_exit_handler(union exit_reason exit_reason)
 }
 
 
-static int dbgctls_init(struct vmcs *vmcs)
+static int dbgctls_dr7_init(struct vmcs *vmcs)
 {
 	u64 dr7 = 0x402;
 	u64 zero = 0;
 
-	msr_bmp_init();
 	asm volatile(
 		"mov %0,%%dr0\n\t"
 		"mov %0,%%dr1\n\t"
 		"mov %0,%%dr2\n\t"
 		"mov %1,%%dr7\n\t"
 		: : "r" (zero), "r" (dr7));
-	wrmsr(MSR_IA32_DEBUGCTLMSR, 0x1);
 	vmcs_write(GUEST_DR7, 0x404);
-	vmcs_write(GUEST_DEBUGCTL, 0x2);
 
 	vmcs_write(ENT_CONTROLS, vmcs_read(ENT_CONTROLS) | ENT_LOAD_DBGCTLS);
 	vmcs_write(EXI_CONTROLS, vmcs_read(EXI_CONTROLS) | EXI_SAVE_DBGCTLS);
@@ -1880,23 +1877,19 @@ static int dbgctls_init(struct vmcs *vmcs)
 	return VMX_TEST_START;
 }
 
-static void dbgctls_main(void)
+static void dbgctls_dr7_main(void)
 {
-	u64 dr7, debugctl;
+	u64 dr7;
 
 	asm volatile("mov %%dr7,%0" : "=r" (dr7));
-	debugctl = rdmsr(MSR_IA32_DEBUGCTLMSR);
-	/* Commented out: KVM does not support DEBUGCTL so far */
-	(void)debugctl;
-	report(dr7 == 0x404, "Load debug controls" /* && debugctl == 0x2 */);
+	report(dr7 == 0x404, "DR7: Load debug controls");
 
 	dr7 = 0x408;
 	asm volatile("mov %0,%%dr7" : : "r" (dr7));
-	wrmsr(MSR_IA32_DEBUGCTLMSR, 0x3);
 
 	vmx_set_test_stage(0);
 	vmcall();
-	report(vmx_get_test_stage() == 1, "Save debug controls");
+	report(vmx_get_test_stage() == 1, "DR7: Save debug controls");
 
 	if (ctrl_enter_rev.set & ENT_LOAD_DBGCTLS ||
 	    ctrl_exit_rev.set & EXI_SAVE_DBGCTLS) {
@@ -1907,46 +1900,37 @@ static void dbgctls_main(void)
 	vmcall();
 
 	asm volatile("mov %%dr7,%0" : "=r" (dr7));
-	debugctl = rdmsr(MSR_IA32_DEBUGCTLMSR);
-	/* Commented out: KVM does not support DEBUGCTL so far */
-	(void)debugctl;
 	report(dr7 == 0x402,
-	       "Guest=host debug controls" /* && debugctl == 0x1 */);
+	       "DR7: Guest=host debug controls");
 
 	dr7 = 0x408;
 	asm volatile("mov %0,%%dr7" : : "r" (dr7));
-	wrmsr(MSR_IA32_DEBUGCTLMSR, 0x3);
 
 	vmx_set_test_stage(3);
 	vmcall();
-	report(vmx_get_test_stage() == 4, "Don't save debug controls");
+	report(vmx_get_test_stage() == 4, "DR7: Don't save debug controls");
 }
 
-static int dbgctls_exit_handler(union exit_reason exit_reason)
+static int dbgctls_dr7_exit_handler(union exit_reason exit_reason)
 {
 	u32 insn_len = vmcs_read(EXI_INST_LEN);
 	u64 guest_rip = vmcs_read(GUEST_RIP);
-	u64 dr7, debugctl;
+	u64 dr7;
 
 	asm volatile("mov %%dr7,%0" : "=r" (dr7));
-	debugctl = rdmsr(MSR_IA32_DEBUGCTLMSR);
 
 	switch (exit_reason.basic) {
 	case VMX_VMCALL:
 		switch (vmx_get_test_stage()) {
 		case 0:
-			if (dr7 == 0x400 && debugctl == 0 &&
-			    vmcs_read(GUEST_DR7) == 0x408 /* &&
-			    Commented out: KVM does not support DEBUGCTL so far
-			    vmcs_read(GUEST_DEBUGCTL) == 0x3 */)
+			if (dr7 == 0x400 &&
+			    vmcs_read(GUEST_DR7) == 0x408)
 				vmx_inc_test_stage();
 			break;
 		case 2:
 			dr7 = 0x402;
 			asm volatile("mov %0,%%dr7" : : "r" (dr7));
-			wrmsr(MSR_IA32_DEBUGCTLMSR, 0x1);
 			vmcs_write(GUEST_DR7, 0x404);
-			vmcs_write(GUEST_DEBUGCTL, 0x2);
 
 			vmcs_write(ENT_CONTROLS,
 				vmcs_read(ENT_CONTROLS) & ~ENT_LOAD_DBGCTLS);
@@ -1954,10 +1938,8 @@ static int dbgctls_exit_handler(union exit_reason exit_reason)
 				vmcs_read(EXI_CONTROLS) & ~EXI_SAVE_DBGCTLS);
 			break;
 		case 3:
-			if (dr7 == 0x400 && debugctl == 0 &&
-			    vmcs_read(GUEST_DR7) == 0x404 /* &&
-			    Commented out: KVM does not support DEBUGCTL so far
-			    vmcs_read(GUEST_DEBUGCTL) == 0x2 */)
+			if (dr7 == 0x400 &&
+			    vmcs_read(GUEST_DR7) == 0x404)
 				vmx_inc_test_stage();
 			break;
 		}
@@ -11813,7 +11795,7 @@ struct vmx_test vmx_tests[] = {
 	{ "PML", pml_init, pml_main, pml_exit_handler },
 	{ "interrupt", interrupt_init, interrupt_main, interrupt_exit_handler },
 	{ "nmi_hlt", nmi_hlt_init, nmi_hlt_main, nmi_hlt_exit_handler },
-	{ "debug controls", dbgctls_init, dbgctls_main, dbgctls_exit_handler },
+	{ "dbgctls_dr7", dbgctls_dr7_init, dbgctls_dr7_main, dbgctls_dr7_exit_handler },
 	{ "MSR switch", msr_switch_init, msr_switch_main,
 		msr_switch_exit_handler, msr_switch_entry_failure },
 	{ "vmmcall", vmmcall_init, vmmcall_main, vmmcall_exit_handler },
Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog
author reviewer
#2 lore raw

PATCH v3 2/2 x86/vmx: (Re)Add a nVMX test to validate load/save of DEBUGCTL

In reply to Sean Christopherson

From: Chenyi Qiang <[email protected]>

Add a nVMX test to verify that DEBUGCTL is/isn't loaded/saved on entry/exit as per vmcs12's controls. Validate all combinations of legal values (only three bits are supported, i.e. there are only 8 unique combinations), for both the host and the guest, along with '0'.

Validate that:

- DEBUGCTL is loaded from vmcs.GUEST_DEBUGCTL iff "Load debug controls" is set, otherwise the guest should see the host's value.
- DEBUGCTL is saved to vmcs.GUEST_DEBUGCTL iff "Save debug controls" is set, otherwise the VMCS should retain its previous value.
- DEBUGCTL is *always* zeroed on VM-Exit, irrespective of guest controls.

---
 lib/x86/msr.h   |   2 +
 x86/vmx_tests.c | 150 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 152 insertions(+)
lib/x86/msr.h +2 -0
index 7397809c..8b019f7e 100644
@@ -103,12 +103,14 @@
 /* DEBUGCTLMSR bits (others vary by model): */
 #define DEBUGCTLMSR_LBR			(1UL <<  0) /* last branch recording */
 #define DEBUGCTLMSR_BTF			(1UL <<  1) /* single-step on branches */
+#define DEBUGCTLMSR_BUS_LOCK_DETECT	(1UL <<  2)
 #define DEBUGCTLMSR_TR			(1UL <<  6)
 #define DEBUGCTLMSR_BTS			(1UL <<  7)
 #define DEBUGCTLMSR_BTINT		(1UL <<  8)
 #define DEBUGCTLMSR_BTS_OFF_OS		(1UL <<  9)
 #define DEBUGCTLMSR_BTS_OFF_USR		(1UL << 10)
 #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI	(1UL << 11)
+#define DEBUGCTLMSR_RTM_DEBUG		(1UL << 15)
 
 #define MSR_LBR_NHM_FROM	0x00000680
 #define MSR_LBR_NHM_TO		0x000006c0
x86/vmx_tests.c +150 -0

161 lines not on this page · raw message

Signature · 1 line
2.54.0.823.g6e5bcc1fc9-goog
#3 lore raw

Re: [kvm-unit-tests PATCH v3 2/2] x86/vmx: (Re)Add a nVMX test to validate load/save of DEBUGCTL

x86/vmx_tests.c static void vmx_cet_test(void) · line 11922

Quoted 11 lines · Sean Christopherson
x86/vmx_tests.c +11 -0
+	for (i = 1; i <= supported_bits; i++) {
+		if ((i & supported_bits) != i)
+			continue;
+
+		for (j = 1; j <= supported_bits; j++) {
+			if ((j & supported_bits) != j)
+				continue;
+
+			__vmx_debugctl_test(i, j);
+		}
+	}

What a clever idea!

A minor flaw: Both i and j should start from 0, since 0 is a valid value.

(I will go through the whole patch tomorrow.)

author reviewer
#4 lore raw

Re: [kvm-unit-tests PATCH v3 2/2] x86/vmx: (Re)Add a nVMX test to validate load/save of DEBUGCTL

Quoted 15 lines · Xiaoyao Li

x86/vmx_tests.c static void vmx_cet_test(void) · line 11922

Quoted 11 lines · Sean Christopherson
x86/vmx_tests.c +11 -0
+	for (i = 1; i <= supported_bits; i++) {
+		if ((i & supported_bits) != i)
+			continue;
+
+		for (j = 1; j <= supported_bits; j++) {
+			if ((j & supported_bits) != j)
+				continue;
+
+			__vmx_debugctl_test(i, j);
+		}
+	}

What a clever idea!

Quoted 1 line

A minor flaw: Both i and j should start from 0, since 0 is a valid value.

Heh, I knew I should have added a comment. I deliberately started at '1', because '0' is used as a "base" value, to be able to detect changes. I.e. '0' is getting valiated on every test anyways.

#5 lore raw

Re: [kvm-unit-tests PATCH v3 2/2] x86/vmx: (Re)Add a nVMX test to validate load/save of DEBUGCTL

In reply to Sean Christopherson
Quoted 101 lines · Sean Christopherson

From: Chenyi Qiang <[email protected]>

Add a nVMX test to verify that DEBUGCTL is/isn't loaded/saved on entry/exit as per vmcs12's controls. Validate all combinations of legal values (only three bits are supported, i.e. there are only 8 unique combinations), for both the host and the guest, along with '0'.

Validate that:

- DEBUGCTL is loaded from vmcs.GUEST_DEBUGCTL iff "Load debug controls" is set, otherwise the guest should see the host's value.
- DEBUGCTL is saved to vmcs.GUEST_DEBUGCTL iff "Save debug controls" is set, otherwise the VMCS should retain its previous value.
- DEBUGCTL is *always* zeroed on VM-Exit, irrespective of guest controls.

---
 lib/x86/msr.h   |   2 +
 x86/vmx_tests.c | 150 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 152 insertions(+)
lib/x86/msr.h +2 -0
index 7397809c..8b019f7e 100644
@@ -103,12 +103,14 @@
 /* DEBUGCTLMSR bits (others vary by model): */
 #define DEBUGCTLMSR_LBR			(1UL <<  0) /* last branch recording */
 #define DEBUGCTLMSR_BTF			(1UL <<  1) /* single-step on branches */
+#define DEBUGCTLMSR_BUS_LOCK_DETECT	(1UL <<  2)
 #define DEBUGCTLMSR_TR			(1UL <<  6)
 #define DEBUGCTLMSR_BTS			(1UL <<  7)
 #define DEBUGCTLMSR_BTINT		(1UL <<  8)
 #define DEBUGCTLMSR_BTS_OFF_OS		(1UL <<  9)
 #define DEBUGCTLMSR_BTS_OFF_USR		(1UL << 10)
 #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI	(1UL << 11)
+#define DEBUGCTLMSR_RTM_DEBUG		(1UL << 15)
 
 #define MSR_LBR_NHM_FROM	0x00000680
 #define MSR_LBR_NHM_TO		0x000006c0
x86/vmx_tests.c +51 -0
index e1f53fbe..f914924d 100644
@@ -11774,6 +11774,155 @@ static void vmx_cet_test(void)
 	test_set_guest_finished();
 }
 
+static u64 vmx_debugctl_test_val;
+
+static void vmx_debugctl_test_guest(void)
+{
+	for (;;) {
+		vmx_debugctl_test_val = rdmsr(MSR_IA32_DEBUGCTLMSR);
+		vmcall();
+
+		wrmsr(MSR_IA32_DEBUGCTLMSR, vmx_debugctl_test_val);
+		vmcall();
+	}
+}
+
+static void __run_vmx_debugctl_guest(void)
+{
+	u64 val;
+
+	enter_guest();
+	skip_exit_vmcall();
+
+	/* Verify DEBUGCTL is unconditionally zeroed on VM-Exit. */
+	val = rdmsr(MSR_IA32_DEBUGCTLMSR);
+	if (val)
+		report_fail("DEBUGCTL = 0x%lx (not zeroed on VM-Exit)", val);
+}
+
+static u64 run_vmx_debugctl_guest(u64 msr_val, u64 vmcs_val, u64 write_val)
+{
+	u64 read_val;
+
+	wrmsr(MSR_IA32_DEBUGCTLMSR, msr_val);
+	vmcs_write(GUEST_DEBUGCTL, vmcs_val);
+
+	__run_vmx_debugctl_guest();
+	read_val = vmx_debugctl_test_val;
+
+	vmx_debugctl_test_val = write_val;
+	__run_vmx_debugctl_guest();
+
+	return read_val;
+}
+
+static void __vmx_debugctl_test(u64 host_val, u64 guest_val)
+{
+	u64 val, rand;
+
+	/* Validate VM-Entrywhen save/load debug controls are set */
+	vmcs_set_bits(ENT_CONTROLS, ENT_LOAD_DBGCTLS);
+	vmcs_set_bits(EXI_CONTROLS, EXI_SAVE_DBGCTLS);
+
+	/* Verify guest DEBUGCTL is loaded/saved when the controls are set. */

x86/vmx_tests.c static void vmx_cet_test(void) · line 11837

Quoted 10 lines
x86/vmx_tests.c +10 -0
+	val = run_vmx_debugctl_guest(host_val, guest_val, 0);
+	report(val == guest_val, "Load DEBUGCTL = 0x%lx, guest RDMSR = 0x%lx", guest_val, val);
+	val = vmcs_read(GUEST_DEBUGCTL);
+	report(!val, "Save DEBUGCTL = 0x0, guest WRMSR = 0x%lx", val);
+
+	/* Rerun the test with the guest values flipped. */
+	val = run_vmx_debugctl_guest(host_val, 0, guest_val);
+	report(!val, "Load DEBUGCTL = 0x0, guest RDMSR = 0x%lx", val);
+	val = vmcs_read(GUEST_DEBUGCTL);
+	report(val == guest_val, "Save DEBUGCTL = 0x0%lx, guest WRMSR = 0x%lx", val, guest_val);
                                                    ^

See an extra "0" here during my test :)

And as mentioned in another thread, it requires the "migratable=no" thing in "vmx" testcase to cover LBR bit on my Skylake test.

Reviewed-by on v3 2/2
#6 lore raw

Re: [kvm-unit-tests PATCH v3 2/2] x86/vmx: (Re)Add a nVMX test to validate load/save of DEBUGCTL

In reply to Sean Christopherson
Quoted 15 lines · Sean Christopherson

From: Chenyi Qiang <[email protected]>

Add a nVMX test to verify that DEBUGCTL is/isn't loaded/saved on entry/exit as per vmcs12's controls. Validate all combinations of legal values (only three bits are supported, i.e. there are only 8 unique combinations), for both the host and the guest, along with '0'.

Validate that:

- DEBUGCTL is loaded from vmcs.GUEST_DEBUGCTL iff "Load debug controls" is set, otherwise the guest should see the host's value.
- DEBUGCTL is saved to vmcs.GUEST_DEBUGCTL iff "Save debug controls" is set, otherwise the VMCS should retain its previous value.
- DEBUGCTL is *always* zeroed on VM-Exit, irrespective of guest controls.

Quoted 3 lines

x86/vmx_tests.c static void vmx_cet_test(void) · line 11823

Quoted 5 lines
x86/vmx_tests.c +5 -0
+static void __vmx_debugctl_test(u64 host_val, u64 guest_val)
+{
+	u64 val, rand;
+
+	/* Validate VM-Entrywhen save/load debug controls are set */

missing a space between "VM-Entry" and "when".

But I don't understand why we need such a comment at all. Does it explain the purpose of following vmcs write is to validate that the two vmcs bits are allowed/supported on VM-Entry?

Quoted 2 lines
(quoted diff) +2 -0
+	vmcs_set_bits(ENT_CONTROLS, ENT_LOAD_DBGCTLS);
+	vmcs_set_bits(EXI_CONTROLS, EXI_SAVE_DBGCTLS);
author reviewer Applied
#7 lore raw
In reply to Sean Christopherson
Quoted 11 lines · Sean Christopherson

Current nested debugctls test focuses on DR7 value and fakes the result of IA32_DEBUGCTLMSR in comment. Although the test can pass because DR7 is the only criterion to determine the result, there are some error messages appearing in dmesg due to accessing to BTF | LBR bits in IA32_DEBUGCTLMSR with report_ignored_msrs KVM parameter enabled.

Clean up the old test so that it focuses on DR7, then add a separate test for DEBUGCTLMSR bits that are actually supported and can be validated during VMX transitions.

[...]

Applied to kvm-x86 next, with a comment to explain starting at '1', and a fix for the redudnant '0'.

[1/2] nVMX: Remove the IA32_DEBUGCTLMSR access in debugctls test

      https://github.com/kvm-x86/kvm-unit-tests/commit/071c710553e0
[2/2] x86/vmx: (Re)Add a nVMX test to validate load/save of DEBUGCTL
      https://github.com/kvm-x86/kvm-unit-tests/commit/067685479059

--
https://github.com/kvm-x86/kvm-unit-tests/tree/next