Skip to contentSkip to conversation

Introduce XSAVE feature self-test

RFC PATCH Superseded Pengfei Xu 2 participants 8 messages Dec 3, 2021 – Dec 13, 2021

Subsystems (2)

v5 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Pengfei Xu

author
#0 lore raw

v5 cover Introduce XSAVE feature self-test

The XSAVE feature set supports the saving and restoring of xstate components, which is used for process context switching. The state components include x87 state for FPU execution environment, SSE state, AVX state and so on. In order to ensure that XSAVE works correctly, add XSAVE basic test for XSAVE architecture functionality.

This patch set tests and verifies the basic functions of XSAVE in user space; it tests "FPU, AVX2, AVX512 opmask and PKRU" xstates(will call them "above xstates" instead) with following cases:
1. In nested signal processing, the signal handling will use each signal's own xstates, and the xstates of the signal handling under test should not be changed after another nested signal handling is completed; and the above mentioned xstates in the process should not change after the nested signal handling is complete.
2. Above xstates in child process should same as parent xstates; and after process switch, the above xstates contents in child process should not be changed.

This series introduces only the most basic XSAVE tests. In the future, the intention is to continue expanding the scope of these selftests to include more xstates and kernel XSAVE-related functionality tests.

========
- Change from v4 to v5:
- Moved code files into tools/testing/selftests/x86.
- Delete xsave instruction test, becaue it's not related to kernel.
- Improved case description.
- Added AVX512 opmask change and related XSAVE content verification.
- Added PKRU part xstate test into instruction and signal handling test.
- Added XSAVE process swich test for FPU, AVX2, AVX512 opmask and PKRU part.

- Change from v3 to v4:
- Improve the comment in patch 1.

- Change from v2 to v3:
- Improve the description of patch 2 git log.

- Change from v1 to v2:
- Improve the cover-letter. Thanks Dave Hansen's suggestion.

Pengfei Xu (2):
selftests/x86: add xsave test during and after signal handling selftests/x86: add xsave test after process switch

 tools/testing/selftests/x86/Makefile          |   4 +-
 tools/testing/selftests/x86/xsave_common.h    | 426 ++++++++++++++++++
 tools/testing/selftests/x86/xsave_fork_test.c | 127 ++++++
 .../selftests/x86/xsave_signal_handle.c       | 192 ++++++++
 4 files changed, 747 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/x86/xsave_common.h
 create mode 100644 tools/testing/selftests/x86/xsave_fork_test.c
 create mode 100644 tools/testing/selftests/x86/xsave_signal_handle.c
Signature · 1 line
2.27.0
author
#1 lore raw

RFC PATCH v5 1/2 selftests/x86: add xsave test during and after signal handling

In order to ensure that FPU, AVX2, AVX512 opmask and PKRU XSAVE content in the same process is not affected by signal handling, this case tests that:
1. In nested signal processing, the signal handling will use each signal's
own xstates, and the xstates of the signal handling under test should not be changed after another nested signal handling is completed;
2. The xstates content of the process should not change after the nested
signal handling. Process will change FPU, AVX2, AVX512, PKRU xstates before signal handling test to ensure that the entire processing of these xstates works as expected.

---
 tools/testing/selftests/x86/Makefile          |   4 +-
 tools/testing/selftests/x86/xsave_common.h    | 426 ++++++++++++++++++
 .../selftests/x86/xsave_signal_handle.c       | 192 ++++++++
 3 files changed, 620 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/selftests/x86/xsave_common.h
 create mode 100644 tools/testing/selftests/x86/xsave_signal_handle.c
tools/testing/selftests/x86/Makefile +2 -2
index 8a1f62ab3c8e..b51f02bc38c2 100644
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \
 			test_FCMOV test_FCOMI test_FISTTP \
 			vdso_restorer
 TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \
-			corrupt_xstate_header amx
+			corrupt_xstate_header amx xsave_signal_handle
 # Some selftests require 32bit support enabled also on 64bit systems
 TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
 
@@ -75,7 +75,7 @@ $(BINARIES_32): $(OUTPUT)/%_32: %.c helpers.h
 	$(CC) -m32 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl -lm
 
 $(BINARIES_64): $(OUTPUT)/%_64: %.c helpers.h
-	$(CC) -m64 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl
+	$(CC) -m64 -mxsave -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl
 
 # x86_64 users should be encouraged to install 32-bit libraries
 ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01)
tools/testing/selftests/x86/xsave_common.h new file +426 -0

426 lines not on this page · raw message

tools/testing/selftests/x86/xsave_signal_handle.c new file +192 -0

192 lines not on this page · raw message

Signature · 1 line
2.27.0
#2 lore raw

Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling

tools/testing/selftests/x86/xsave_common.h line 426

Quoted 7 lines · Pengfei Xu
tools/testing/selftests/x86/xsave_common.h new file +7 -0
+void populate_xstate_regs(void)
+{
+	set_fpu_reg();
+	set_avx2_ymm();
+	set_avx512_opmask();
+	set_pkru_xstate();
+}

Pengfei, as I mentioned several times, XMM and YMM registers are not preserved across function calls. This only works by chance. The compiler is free to clobber them at basically any time between when they are populated and the XSAVE happens.

author
#3 lore raw

Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling

Hi Dave,

Quoted 13 lines · Dave Hansen

tools/testing/selftests/x86/xsave_common.h line 426

Quoted 7 lines · Pengfei Xu
tools/testing/selftests/x86/xsave_common.h new file +7 -0
+void populate_xstate_regs(void)
+{
+	set_fpu_reg();
+	set_avx2_ymm();
+	set_avx512_opmask();
+	set_pkru_xstate();
+}

Pengfei, as I mentioned several times, XMM and YMM registers are not preserved across function calls. This only works by chance. The compiler is free to clobber them at basically any time between when they are populated and the XSAVE happens.

Thanks for suggestions!
Sorry, I read the link https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf
I have removed the XMM registers check as XMM registers are not preserved across function calls in above doc mentioned:
"
Register | Usage | Preserved across function calls

%xmm0–%xmm1  | used to pass and return floating point arguments | No
%xmm2–%xmm7  | used to pass floating point arguments | No
%xmm8–%xmm15 | temporary registers | No
%mmx0–%mmx7  | temporary registers | No

"

It seems that the reason why the contents of the XMM register changed by chance is:
As printf for example, after "printf" Glibc function:
do_lookup_x is triggered by printf:
"
0x00007ffff7fd5538 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2 ->
0x00007ffff7fd55d7 <+1367>: e8 94 7b 01 00 callq 0x7ffff7fed170 <strcmp>
...

   0x00007ffff7fee1af <+4159>:  90      nop
   0x00007ffff7fee1b0 <+4160>:  66 0f ef c0     pxor   %xmm0,%xmm0
   0x00007ffff7fee1b4 <+4164>:  66 0f 6f 17     movdqa (%rdi),%xmm2
=> 0x00007ffff7fee1b8 <+4168>:  66 0f 6f 0e     movdqa (%rsi),%xmm1
   0x00007ffff7fee1bc <+4172>:  66 0f 74 c1     pcmpeqb %xmm1,%xmm0

" And after printf, xmm1 and xmm2 value was changed. And seems some other glibc function will use XMM registers also.

YMM will not be preserved across function calls, but I'm not sure whether some Glibc will use YMM.
I will remove YMM xstates check until there is some other better way for YMM test or it's confirmed YMM will not be used by Glibc.

I think PKRU and AVX512 opmask will not be affected in the test. PKRU xstate will not be affected by registers, and Glibc will not use AVX512 instructions.

How about the following changes:
Will remove set_avx2_ymm() and will only check XSAVE_MASK_FP, XSAVE_MASK_OPMASK and XSAVE_MASK_PKRU xstates after signal handling and process switch, xstates will be controlled by XSAVE_TEST_MASK:
"
#define XSAVE_TEST_MASK (XSAVE_MASK_FP | XSAVE_MASK_YMM | XSAVE_MASK_OPMASK \
| XSAVE_MASK_PKRU)
"
will change to
"
#define XSAVE_TEST_MASK (XSAVE_MASK_FP | XSAVE_MASK_OPMASK | XSAVE_MASK_PKRU) "

FP will be used only by user space, and seems Glibc will not use FP in general. This is the xstates position for FP, XMM, Header, YMM, AVX512 opmask and PKRU:
It could be saved by xsave instruction, and mask could control which part will be saved(Header will be saved as mandatory):

FP               (0-159 bytes)
XMM              (160-415 bytes)
Reserved         (416-511 bytes  SDM vol1 13.4.1)
Header_used      (512-527 bytes)
Headed_reserved  (528-575 bytes must 00, otherwise xrstor will #GP)
YMM           (Offset:CPUID.(EAX=0D,ECX=2).EBX  Size:CPUID(EAX=0D,ECX=2).EAX)
AVX512 opmask (Offset:CPUID.(EAX=0D,ECX=5).EBX  Size:CPUID(EAX=0D,ECX=5).EAX)
PKRU          (Offset:CPUID.(EAX=0D,ECX=9).EBX  Size:CPUID(EAX=0D,ECX=9).EAX)

Thanks!
BR.

#4 lore raw

Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling

Quoted 3 lines · Pengfei Xu

How about the following changes:
Will remove set_avx2_ymm() and will only check XSAVE_MASK_FP, XSAVE_MASK_OPMASK and XSAVE_MASK_PKRU xstates after signal handling and process switch,

First and foremost, the whole point of these tests is to ensure that the kernel is properly maintaining register state. Removing registers from the test moves *away* from the primary goal of this test.

Second, you just listed three states there. Have you considered looking at whether those have the same problem as the XMM/YMM registers? Please do.

Third (and I've also suggested this before), we should explicitly tell the compiler not to use the FPU registers. This is what the kernel does, and it's what allows us to, for instance, make function calls in the kernel without clobbering userspace content in XSAVE-managed registers.

If we did that, then we would only have to worry about calls to things *outside* of the test program, like libc.

author
#5 lore raw

Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling

Hi Dave,

Quoted 9 lines · Dave Hansen
Quoted 3 lines · Pengfei Xu

How about the following changes:
Will remove set_avx2_ymm() and will only check XSAVE_MASK_FP, XSAVE_MASK_OPMASK and XSAVE_MASK_PKRU xstates after signal handling and process switch,

First and foremost, the whole point of these tests is to ensure that the kernel is properly maintaining register state. Removing registers from the test moves *away* from the primary goal of this test.

Thanks for suggestion!
Actually, I already removed any useless libc function before and after xsave action, only left the test action between xsave action:
"

	XSAVE(xsave_buf2, XSAVE_TEST_MASK);
	do raise signal or fork test
	XSAVE(xsave_buf3, XSAVE_TEST_MASK);

"
I found that after fork() function in virtual machine, XMM0 or XMM1 register
will be used and changed.
But in YMM xstate, I haven't see signal handline and fork action will use
and change YMM regiseters in the test. Seems we could keep YMM xstate test.
Seems it needs some other better way for XMM xstate.

Quoted 3 lines

Second, you just listed three states there. Have you considered looking at whether those have the same problem as the XMM/YMM registers? Please do.

I have tested FP, AVX512 opmask and pkru xstates on different platforms and
virtual machine, gdb these 3 xstates with fork and signal handling even printf, above 3 functions will not use and change above 3 xstates. I used previous xsave instruction tests to get the results.

Quoted 7 lines

Third (and I've also suggested this before), we should explicitly tell the compiler not to use the FPU registers. This is what the kernel does, and it's what allows us to, for instance, make function calls in the kernel without clobbering userspace content in XSAVE-managed registers.

If we did that, then we would only have to worry about calls to things *outside* of the test program, like libc.

Thanks! Yes, I will add "float a = 0.12, b = 0.34; a = a + b;" to tell
libc process, float points has been used.
Seems if there is no addition, subtraction, multiplication or division,
there is no change in FP xstate compared with no float definition. If there is above operation, mxcsr(xstate offset 0x18-0x1b bytes)will change from 801f to a01f. Rounding control bit change from 00 to 01.

Thanks!
BR.

author
#6 lore raw

Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling

Hi Dave,

Quoted 39 lines · Pengfei Xu

Hi Dave,

Quoted 9 lines · Dave Hansen
Quoted 3 lines · Pengfei Xu

How about the following changes:
Will remove set_avx2_ymm() and will only check XSAVE_MASK_FP, XSAVE_MASK_OPMASK and XSAVE_MASK_PKRU xstates after signal handling and process switch,

First and foremost, the whole point of these tests is to ensure that the kernel is properly maintaining register state. Removing registers from the test moves *away* from the primary goal of this test.

Thanks for suggestion!
Actually, I already removed any useless libc function before and after xsave action, only left the test action between xsave action:
"

	XSAVE(xsave_buf2, XSAVE_TEST_MASK);
	do raise signal or fork test
	XSAVE(xsave_buf3, XSAVE_TEST_MASK);

"
I found that after fork() function in virtual machine, XMM0 or XMM1 register
will be used and changed.
But in YMM xstate, I haven't see signal handline and fork action will use
and change YMM regiseters in the test. Seems we could keep YMM xstate test.
Seems it needs some other better way for XMM xstate.

Quoted 3 lines

Second, you just listed three states there. Have you considered looking at whether those have the same problem as the XMM/YMM registers? Please do.

I have tested FP, AVX512 opmask and pkru xstates on different platforms and
virtual machine, gdb these 3 xstates with fork and signal handling even printf, above 3 functions will not use and change above 3 xstates. I used previous xsave instruction tests to get the results.

Quoted 5 lines

Third (and I've also suggested this before), we should explicitly tell the compiler not to use the FPU registers. This is what the kernel does, and it's what allows us to, for instance, make function calls in the kernel without clobbering userspace content in XSAVE-managed registers.

Quoted 9 lines
Quoted 2 lines

If we did that, then we would only have to worry about calls to things *outside* of the test program, like libc.

Thanks! Yes, I will add "float a = 0.12, b = 0.34; a = a + b;" to tell
libc process, float points has been used.
Seems if there is no addition, subtraction, multiplication or division,
there is no change in FP xstate compared with no float definition. If there is above operation, mxcsr(xstate offset 0x18-0x1b bytes)will change from 801f to a01f. Rounding control bit change from 00 to 01.

Sorry, I misunderstood your meaning in the last email, I should directly add syscall function between the 2 xsave comparisons, it will not use XMM, YMM and so on xstates, and I could add them back.
For example, for "fork", I will use syscall(SYS_fork) instead of fork().

Quoted 2 lines

Thanks!
BR.

author
#7 lore raw

RFC PATCH v5 2/2 selftests/x86: add xsave test after process switch

In reply to Pengfei Xu

It will change FPU, AVX2, AVX512, PKRU xstates before process switch test to ensure that the entire processing of these xstates works as expected. In order to ensure the above XSAVE content is not affected after process switch, this case tests that:
1. The xstates content of the child process should same as xstates of parent process.
2. The xstates content of the child process should not change after process switch back, even grandchild process cleaned up it's own XMM, YMM and ZMM xstates.

---
 tools/testing/selftests/x86/Makefile          |   2 +-
 tools/testing/selftests/x86/xsave_fork_test.c | 127 ++++++++++++++++++
 2 files changed, 128 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/x86/xsave_fork_test.c
tools/testing/selftests/x86/Makefile +1 -1
index b51f02bc38c2..5b627ae5e385 100644
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \
 			test_FCMOV test_FCOMI test_FISTTP \
 			vdso_restorer
 TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \
-			corrupt_xstate_header amx xsave_signal_handle
+			corrupt_xstate_header amx xsave_signal_handle xsave_fork_test
 # Some selftests require 32bit support enabled also on 64bit systems
 TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
 
tools/testing/selftests/x86/xsave_fork_test.c new file +127 -0

127 lines not on this page · raw message

Signature · 1 line
2.27.0