Skip to contentSkip to conversation

Add functional test for AMD VMSAVE/VMLOAD instructions

PATCH Superseded Martin Doucha 2 participants 18 messages Apr 30, 2024 – May 7, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Add functional test for AMD VMSAVE/VMLOAD instructions

Thanks to a minor bug in LTP KVM library, LTP test kvm_svm02 found a kernel bug in emulation of VMSAVE and VMLOAD instructions in nested VMs. Add a thorough functional test for both instructions which can detect and pinpoint emulation bugs.

Also implement basic printf-like formatting for tst_res() and tst_brk() so that the test can print incorrect register values to simplify result analysis. Only standard integer and character conversions are supported. Floating point number conversions, field alignment and padding are not implemented.

Martin Doucha (9):
KVM: Disable EBP register use in 32bit code
KVM: Implement strchr() and basic sprintf()
KVM: Implement printf-like formatting for tst_res() and tst_brk()
kvm_svm02: Fix saved stack segment index value kvm_find_free_descriptor(): Skip descriptor 0
KVM: Add system control MSR constants
KVM: Add VMSAVE/VMLOAD functions to x86 SVM library
KVM: Add functional test for VMSAVE/VMLOAD instructions
KVM: Move kvm_pagefault01 to the end of KVM runfile

 runtest/kvm                                |   4 +-
 testcases/kernel/kvm/.gitignore            |   1 +
 testcases/kernel/kvm/Makefile              |   2 +-
 testcases/kernel/kvm/include/kvm_guest.h   |  23 +-
 testcases/kernel/kvm/include/kvm_x86.h     |  10 +
 testcases/kernel/kvm/include/kvm_x86_svm.h |   6 +
 testcases/kernel/kvm/kvm_svm02.c           |  14 +-
 testcases/kernel/kvm/kvm_svm04.c           | 307 ++++++++++++++++++
 testcases/kernel/kvm/lib_guest.c           | 348 ++++++++++++++++++++-
 testcases/kernel/kvm/lib_x86.c             |  20 +-
 10 files changed, 713 insertions(+), 22 deletions(-)
 create mode 100644 testcases/kernel/kvm/kvm_svm04.c
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

PATCH 1/9 KVM: Disable EBP register use in 32bit code

The EBP register points to the stack segment by default but GCC uses it to access data segment without the proper prefix. This works fine on most systems because the stack and data segments are usually identical. However, KVM environment intentionally enforces strict limits on the stack segment and access to the data segment using unprefixed EBP would trigger stack segment fault exception in 32bit LTP builds (stack segment limits are ignored in 64bit mode).

---
 testcases/kernel/kvm/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/kvm/Makefile +1 -1
index ce4a5ede2..c85790e11 100644
@@ -22,7 +22,7 @@ ifeq ($(HOST_CPU),x86_64)
 endif
 
 ifeq ($(HOST_CPU),x86)
-	GUEST_CFLAGS += -m32
+	GUEST_CFLAGS += -m32 -ffixed-ebp
 	ASFLAGS += --32
 endif
 
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 1/9
#2 lore raw

Re: [LTP] [PATCH 1/9] KVM: Disable EBP register use in 32bit code

Hi Martin,

Quoted 18 lines

The EBP register points to the stack segment by default but GCC uses it to access data segment without the proper prefix. This works fine on most systems because the stack and data segments are usually identical. However, KVM environment intentionally enforces strict limits on the stack segment and access to the data segment using unprefixed EBP would trigger stack segment fault exception in 32bit LTP builds (stack segment limits are ignored in 64bit mode).

---
 testcases/kernel/kvm/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/kvm/Makefile +0 -0
index ce4a5ede2..c85790e11 100644

testcases/kernel/kvm/Makefile ifeq ($(HOST_CPU),x86_64) · line 25

Quoted 5 lines
testcases/kernel/kvm/Makefile +1 -1
@@ -22,7 +22,7 @@ ifeq ($(HOST_CPU),x86_64)
 endif
 
 ifeq ($(HOST_CPU),x86)
-	GUEST_CFLAGS += -m32
+	GUEST_CFLAGS += -m32 -ffixed-ebp

FYI this will fail on 32 bit build on clang:

clang: error: unknown argument: '-ffixed-ebp'

I don't want to block this patchset which brings important test, but it'd be great to fix it.

Is there clang equivalent? Or is it even needed for clang?

Either way, we need to detect clang. I don't think simple

ifeq ($(CXX),clang)

would be enough, because cc can be alias to clang.

Maybe wrap it with version detection:

ifeq ($(shell $(CC) -v 2>&1 | grep "clang version"), 1)

	GUEST_CFLAGS += -ffixed-ebp

endif

Kind regards,
Petr

Quoted 2 lines
testcases/kernel/kvm/Makefile +0 -0
 	ASFLAGS += --32
 endif
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw

Re: [LTP] [PATCH 1/9] KVM: Disable EBP register use in 32bit code

Quoted 11 lines · Petr Vorel

testcases/kernel/kvm/Makefile ifeq ($(HOST_CPU),x86_64) · line 25

Quoted 3 lines
  ifeq ($(HOST_CPU),x86)
testcases/kernel/kvm/Makefile +1 -1
-	GUEST_CFLAGS += -m32
+	GUEST_CFLAGS += -m32 -ffixed-ebp

FYI this will fail on 32 bit build on clang:

clang: error: unknown argument: '-ffixed-ebp'

I don't want to block this patchset which brings important test, but it'd be great to fix it.

Quoted 7 lines

Is there clang equivalent? Or is it even needed for clang?

Either way, we need to detect clang. I don't think simple

ifeq ($(CXX),clang)

would be enough, because cc can be alias to clang.

Hmm, I need to fix this. I guess that configure should just check for -ffixed-ebp support. Fortunately, clang doesn't generate code that would trigger stack segment fault so the workaround is only needed for GCC.

Could you review and merge the trivial patches (4, 5, 6, 7) so that I don't need to resubmit everything?

Signature · 11 lines
Martin Doucha   [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw

Re: [LTP] [PATCH 1/9] KVM: Disable EBP register use in 32bit code

Quoted 14 lines · Petr Vorel
Quoted 12 lines

testcases/kernel/kvm/Makefile ifeq ($(HOST_CPU),x86_64) · line 25

Quoted 3 lines
  ifeq ($(HOST_CPU),x86)
testcases/kernel/kvm/Makefile +1 -1
-	GUEST_CFLAGS += -m32
+	GUEST_CFLAGS += -m32 -ffixed-ebp

FYI this will fail on 32 bit build on clang:

clang: error: unknown argument: '-ffixed-ebp'

I don't want to block this patchset which brings important test, but it'd be great to fix it.

Is there clang equivalent? Or is it even needed for clang?

Quoted 7 lines
Quoted 5 lines

Either way, we need to detect clang. I don't think simple

ifeq ($(CXX),clang)

would be enough, because cc can be alias to clang.

Hmm, I need to fix this. I guess that configure should just check for -ffixed-ebp support.

Yes, that would be ideal solution which I definitely don't want to force you.

Quoted 2 lines

Fortunately, clang doesn't generate code that would trigger stack segment fault so the workaround is only needed for GCC.

Great!

Quoted 2 lines

Could you review and merge the trivial patches (4, 5, 6, 7) so that I don't need to resubmit everything?

Sure!

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

Re: [LTP] [PATCH 1/9] KVM: Disable EBP register use in 32bit code

In reply to Martin Doucha

Hi Martin,

...

Quoted 2 lines

Could you review and merge the trivial patches (4, 5, 6, 7) so that I don't need to resubmit everything?

I could even rebase 9th (missing kvm_svm04) to merge it before. Please let me know if change order was actually needed only for kvm_svm04 (it does not look like from the commit message).

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw

Re: [LTP] [PATCH 1/9] KVM: Disable EBP register use in 32bit code

Quoted 9 lines · Petr Vorel

Hi Martin,

...

Quoted 2 lines

Could you review and merge the trivial patches (4, 5, 6, 7) so that I don't need to resubmit everything?

I could even rebase 9th (missing kvm_svm04) to merge it before. Please let me know if change order was actually needed only for kvm_svm04 (it does not look like from the commit message).

Feel free to rebase it. It's needed for kvm_svm01 and kvm_svm02 as well.

Signature · 11 lines
Martin Doucha   [email protected]
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw

PATCH 2/9 KVM: Implement strchr() and basic sprintf()

In reply to Martin Doucha

Add basic implementation of sprintf() that supports string, pointer and integer arguments but without advanced formatting options like field alignment and padding.

---
 testcases/kernel/kvm/include/kvm_guest.h |  11 +
 testcases/kernel/kvm/lib_guest.c         | 312 +++++++++++++++++++++++
 2 files changed, 323 insertions(+)
testcases/kernel/kvm/include/kvm_guest.h +11 -0
index 96f246155..f19bacc39 100644
@@ -39,9 +39,15 @@ typedef unsigned int uint32_t;
 typedef long long int64_t;
 typedef unsigned long long uint64_t;
 typedef unsigned long uintptr_t;
+typedef __builtin_va_list va_list;
 
 #define NULL ((void *)0)
 
+#define va_start __builtin_va_start
+#define va_arg __builtin_va_arg
+#define va_end __builtin_va_end
+#define va_copy __builtin_va_copy
+
 void *memset(void *dest, int val, size_t size);
 void *memzero(void *dest, size_t size);
 void *memcpy(void *dest, const void *src, size_t size);
@@ -49,6 +55,11 @@ void *memcpy(void *dest, const void *src, size_t size);
 char *strcpy(char *dest, const char *src);
 char *strcat(char *dest, const char *src);
 size_t strlen(const char *str);
+char *strchr(const char *s, int c);
+char *strrchr(const char *s, int c);
+
+int vsprintf(char *dest, const char *fmt, va_list ap);
+int sprintf(char *dest, const char *fmt, ...);
 
 /* Exit the VM by looping on a HLT instruction forever */
 void kvm_exit(void) __attribute__((noreturn));
testcases/kernel/kvm/lib_guest.c +312 -0

324 lines not on this page · raw message

Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#8 lore raw

PATCH 3/9 KVM: Implement printf-like formatting for tst_res() and tst_brk()

In reply to Martin Doucha
---
 testcases/kernel/kvm/include/kvm_guest.h | 12 +++++---
 testcases/kernel/kvm/lib_guest.c         | 36 +++++++++++++++++++++---
 2 files changed, 40 insertions(+), 8 deletions(-)
testcases/kernel/kvm/include/kvm_guest.h +8 -4
index f19bacc39..080d0ac2b 100644
@@ -68,12 +68,16 @@ void kvm_exit(void) __attribute__((noreturn));
 void kvm_yield(void);
 
 void tst_res_(const char *file, const int lineno, int result,
-	const char *message);
-#define tst_res(result, msg) tst_res_(__FILE__, __LINE__, (result), (msg))
+	const char *fmt, ...)
+	__attribute__ ((format (printf, 4, 5)));
+#define tst_res(result, fmt, ...) \
+	tst_res_(__FILE__, __LINE__, (result), (fmt), ##__VA_ARGS__)
 
 void tst_brk_(const char *file, const int lineno, int result,
-	const char *message) __attribute__((noreturn));
-#define tst_brk(result, msg) tst_brk_(__FILE__, __LINE__, (result), (msg))
+	const char *fmt, ...) __attribute__((noreturn))
+	__attribute__ ((format (printf, 4, 5)));
+#define tst_brk(result, fmt, ...) \
+	tst_brk_(__FILE__, __LINE__, (result), (fmt), ##__VA_ARGS__)
 
 /*
  * Send asynchronous notification to host without stopping VM execution and
testcases/kernel/kvm/lib_guest.c +32 -4
index 73a76ccb1..2e3e9cb6e 100644
@@ -443,6 +443,7 @@ static void tst_fatal_error(const char *file, const int lineno,
 	test_result->result = TBROK;
 	test_result->lineno = lineno;
 	test_result->file_addr = (uintptr_t)file;
+	/* Avoid sprintf() here in case of bugs */
 	strcpy(test_result->message, message);
 	strcat(test_result->message, " at address 0x");
 	ptr2hex(test_result->message + strlen(test_result->message), ip);
@@ -451,19 +452,46 @@ static void tst_fatal_error(const char *file, const int lineno,
 }
 
 void tst_res_(const char *file, const int lineno, int result,
-	const char *message)
+	const char *fmt, ...)
 {
+	va_list args;
+	int ret;
+
+	va_start(args, fmt);
 	test_result->result = result;
 	test_result->lineno = lineno;
 	test_result->file_addr = (uintptr_t)file;
-	strcpy(test_result->message, message);
+	ret = vsprintf(test_result->message, fmt, args);
+	va_end(args);
+
+	if (ret < 0) {
+		tst_brk_(file, lineno, TBROK, "Invalid tst_res() format: %s",
+			fmt);
+	}
+
 	kvm_yield();
 }
 
 void tst_brk_(const char *file, const int lineno, int result,
-	const char *message)
+	const char *fmt, ...)
 {
-	tst_res_(file, lineno, result, message);
+	va_list args;
+	int ret;
+
+	va_start(args, fmt);
+	test_result->result = result;
+	test_result->lineno = lineno;
+	test_result->file_addr = (uintptr_t)file;
+	ret = vsprintf(test_result->message, fmt, args);
+	va_end(args);
+
+	if (ret < 0) {
+		test_result->result = TBROK;
+		strcpy(test_result->message, "Invalid tst_brk() format: ");
+		strcat(test_result->message, fmt);
+	}
+
+	kvm_yield();
 	kvm_exit();
 }
 
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#9 lore raw

PATCH 4/9 kvm_svm02: Fix saved stack segment index value

In reply to Martin Doucha

The VMCB init helper function expects GDT item index while the VMCB segment selector values are byte offsets. Shift the byte offset of saved stack segment selector to get the correct GDT index.

---
 testcases/kernel/kvm/kvm_svm02.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/kvm/kvm_svm02.c +1 -1
index 5d2e2ce37..f72fb3812 100644
@@ -96,7 +96,7 @@ void main(void)
 	vmsave_buf = kvm_alloc_vmcb();
 
 	/* Save allocated stack for later VM reinit */
-	ss = vcpu->vmcb->ss.selector;
+	ss = vcpu->vmcb->ss.selector >> 3;
 	rsp = vcpu->vmcb->rsp;
 
 	/* Load partial state from vmsave_buf and save it to vcpu->vmcb */
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#10 lore raw

PATCH 5/9 kvm_find_free_descriptor(): Skip descriptor 0

In reply to Martin Doucha

The GDT/LDT descriptor 0 should always be empty. Start search for free descriptor table entry at index 1.

---
 testcases/kernel/kvm/lib_x86.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/kvm/lib_x86.c +1 -1
index 3e6656f11..1c0e629c3 100644
@@ -174,7 +174,7 @@ int kvm_find_free_descriptor(const struct segment_descriptor *table,
 	const struct segment_descriptor *ptr;
 	size_t i;
 
-	for (i = 0, ptr = table; i < size; i++, ptr++) {
+	for (i = 1, ptr = table + 1; i < size; i++, ptr++) {
 		if (!(ptr->flags_lo & SEGFLAG_PRESENT))
 			return i;
 
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#11 lore raw

PATCH 6/9 KVM: Add system control MSR constants

In reply to Martin Doucha
---
 testcases/kernel/kvm/include/kvm_x86.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
testcases/kernel/kvm/include/kvm_x86.h +10 -0
index bc36c0e0f..08d3f6759 100644
@@ -68,7 +68,17 @@
 
 
 /* Model-specific CPU register constants */
+#define MSR_SYSENTER_CS 0x174
+#define MSR_SYSENTER_ESP 0x175
+#define MSR_SYSENTER_EIP 0x176
 #define MSR_EFER 0xc0000080
+#define MSR_STAR 0xc0000081
+#define MSR_LSTAR 0xc0000082
+#define MSR_CSTAR 0xc0000083
+#define MSR_SFMASK 0xc0000084
+#define MSR_FS_BASE 0xc0000100
+#define MSR_GS_BASE 0xc0000101
+#define MSR_KERNEL_GS_BASE 0xc0000102
 #define MSR_VM_CR 0xc0010114
 #define MSR_VM_HSAVE_PA 0xc0010117
 
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#12 lore raw

PATCH 7/9 KVM: Add VMSAVE/VMLOAD functions to x86 SVM library

In reply to Martin Doucha
---
 testcases/kernel/kvm/include/kvm_x86_svm.h |  6 ++++++
 testcases/kernel/kvm/kvm_svm02.c           | 12 ++----------
 testcases/kernel/kvm/lib_x86.c             | 18 ++++++++++++++++++
 3 files changed, 26 insertions(+), 10 deletions(-)
testcases/kernel/kvm/include/kvm_x86_svm.h +6 -0
index b4b1b80e2..73563ed2d 100644
@@ -163,4 +163,10 @@ struct kvm_svm_vcpu *kvm_create_svm_vcpu(int (*guest_main)(void),
 
 void kvm_svm_vmrun(struct kvm_svm_vcpu *cpu);
 
+/* Load FS, GS, TR and LDTR state from vmsave_buf */
+void kvm_svm_vmload(struct kvm_vmcb *buf);
+
+/* Save current FS, GS, TR and LDTR state to vmsave_buf */
+void kvm_svm_vmsave(struct kvm_vmcb *buf);
+
 #endif /* KVM_X86_SVM_H_ */
testcases/kernel/kvm/kvm_svm02.c +2 -10
index f72fb3812..6914fdcba 100644
@@ -33,22 +33,14 @@ static void *vmsave_buf;
 /* Load FS, GS, TR and LDTR state from vmsave_buf */
 static int guest_vmload(void)
 {
-	asm (
-		"vmload %0\n"
-		:
-		: "a" (vmsave_buf)
-	);
+	kvm_svm_vmload(vmsave_buf);
 	return 0;
 }
 
 /* Save current FS, GS, TR and LDTR state to vmsave_buf */
 static int guest_vmsave(void)
 {
-	asm (
-		"vmsave %0\n"
-		:
-		: "a" (vmsave_buf)
-	);
+	kvm_svm_vmsave(vmsave_buf);
 	return 0;
 }
 
testcases/kernel/kvm/lib_x86.c +18 -0
index 1c0e629c3..8db3abd3f 100644
@@ -393,3 +393,21 @@ struct kvm_svm_vcpu *kvm_create_svm_vcpu(int (*guest_main)(void),
 	ret->vmcb = vmcb;
 	return ret;
 }
+
+void kvm_svm_vmload(struct kvm_vmcb *buf)
+{
+	asm (
+		"vmload %0\n"
+		:
+		: "a" (buf)
+	);
+}
+
+void kvm_svm_vmsave(struct kvm_vmcb *buf)
+{
+	asm (
+		"vmsave %0\n"
+		:
+		: "a" (buf)
+	);
+}
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#13 lore raw

Re: [LTP] [PATCH 7/9] KVM: Add VMSAVE/VMLOAD functions to x86 SVM library

Hi Martin,

merged patches 4, 5, 6, 7 as we discussed in the first patch.

Thanks!

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#14 lore raw

PATCH 8/9 KVM: Add functional test for VMSAVE/VMLOAD instructions

In reply to Martin Doucha
---
 runtest/kvm                      |   1 +
 testcases/kernel/kvm/.gitignore  |   1 +
 testcases/kernel/kvm/kvm_svm04.c | 307 +++++++++++++++++++++++++++++++
 3 files changed, 309 insertions(+)
 create mode 100644 testcases/kernel/kvm/kvm_svm04.c
runtest/kvm +1 -0
index 4094a21a8..0e1b2e555 100644
@@ -2,3 +2,4 @@ kvm_pagefault01 kvm_pagefault01
 kvm_svm01 kvm_svm01
 kvm_svm02 kvm_svm02
 kvm_svm03 kvm_svm03
+kvm_svm04 kvm_svm04
testcases/kernel/kvm/.gitignore +1 -0
index 9638a6fc7..661472cae 100644
@@ -2,3 +2,4 @@
 /kvm_svm01
 /kvm_svm02
 /kvm_svm03
+/kvm_svm04
testcases/kernel/kvm/kvm_svm04.c new file +307 -0

307 lines not on this page · raw message

Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#15 lore raw

PATCH 9/9 KVM: Move kvm_pagefault01 to the end of KVM runfile

In reply to Martin Doucha

The kvm_pagefault01 test reloads the KVM module with special parameters which may change kernel code paths and hide open bugs from other KVM tests. Move the test to the end of KVM schedule to prevent interference.

---
 runtest/kvm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
runtest/kvm +2 -1
index 0e1b2e555..74a517add 100644
@@ -1,5 +1,6 @@
-kvm_pagefault01 kvm_pagefault01
 kvm_svm01 kvm_svm01
 kvm_svm02 kvm_svm02
 kvm_svm03 kvm_svm03
 kvm_svm04 kvm_svm04
+# Tests below may interfere with bug reproducibility
+kvm_pagefault01 kvm_pagefault01
Signature · 5 lines
2.44.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
Reviewed-by on v1 9/9
#16 lore raw

Re: [LTP] [PATCH 9/9] KVM: Move kvm_pagefault01 to the end of KVM runfile

Hi Martin,

Quoted 3 lines

The kvm_pagefault01 test reloads the KVM module with special parameters which may change kernel code paths and hide open bugs from other KVM tests. Move the test to the end of KVM schedule to prevent interference.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#17 lore raw

Re: [LTP] [PATCH 9/9] KVM: Move kvm_pagefault01 to the end of KVM runfile

In reply to Martin Doucha

Hi Martin,

Quoted 3 lines

The kvm_pagefault01 test reloads the KVM module with special parameters which may change kernel code paths and hide open bugs from other KVM tests. Move the test to the end of KVM schedule to prevent interference.

Merged rebased, as we discussed at the first patch. Thanks!

On a brief look the other patches looks very well (apart from that clang issue), but I need some time for actual review.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp