Introduce XSAVE feature self-test
v5 cover Introduce XSAVE feature self-test
The XSAVE feature set supports the saving and restoring of xstate components, which is used for process context switching. The state components include x87 state for FPU execution environment, SSE state, AVX state and so on. In order to ensure that XSAVE works correctly, add XSAVE basic test for XSAVE architecture functionality.
This patch set tests and verifies the basic functions of XSAVE in user space; it tests "FPU, AVX2, AVX512 opmask and PKRU" xstates(will call them "above xstates" instead) with following cases:
1. In nested signal processing, the signal handling will use each signal's own xstates, and the xstates of the signal handling under test should not be changed after another nested signal handling is completed; and the above mentioned xstates in the process should not change after the nested signal handling is complete.
2. Above xstates in child process should same as parent xstates; and after process switch, the above xstates contents in child process should not be changed.
This series introduces only the most basic XSAVE tests. In the future, the intention is to continue expanding the scope of these selftests to include more xstates and kernel XSAVE-related functionality tests.
========
- Change from v4 to v5:
- Moved code files into tools/testing/selftests/x86.
- Delete xsave instruction test, becaue it's not related to kernel.
- Improved case description.
- Added AVX512 opmask change and related XSAVE content verification.
- Added PKRU part xstate test into instruction and signal handling test.
- Added XSAVE process swich test for FPU, AVX2, AVX512 opmask and PKRU part.
- Change from v3 to v4:
- Improve the comment in patch 1.
- Change from v2 to v3:
- Improve the description of patch 2 git log.
- Change from v1 to v2:
- Improve the cover-letter. Thanks Dave Hansen's suggestion.
Pengfei Xu (2):
selftests/x86: add xsave test during and after signal handling selftests/x86: add xsave test after process switch
tools/testing/selftests/x86/Makefile | 4 +- tools/testing/selftests/x86/xsave_common.h | 426 ++++++++++++++++++ tools/testing/selftests/x86/xsave_fork_test.c | 127 ++++++ .../selftests/x86/xsave_signal_handle.c | 192 ++++++++ 4 files changed, 747 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/x86/xsave_common.h create mode 100644 tools/testing/selftests/x86/xsave_fork_test.c create mode 100644 tools/testing/selftests/x86/xsave_signal_handle.c
Signature · 1 line
2.27.0
RFC PATCH v5 1/2 selftests/x86: add xsave test during and after signal handling
In order to ensure that FPU, AVX2, AVX512 opmask and PKRU XSAVE content in the same process is not affected by signal handling, this case tests that:
1. In nested signal processing, the signal handling will use each signal's
own xstates, and the xstates of the signal handling under test should not be changed after another nested signal handling is completed;
2. The xstates content of the process should not change after the nested
signal handling. Process will change FPU, AVX2, AVX512, PKRU xstates before signal handling test to ensure that the entire processing of these xstates works as expected.
--- tools/testing/selftests/x86/Makefile | 4 +- tools/testing/selftests/x86/xsave_common.h | 426 ++++++++++++++++++ .../selftests/x86/xsave_signal_handle.c | 192 ++++++++ 3 files changed, 620 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/x86/xsave_common.h create mode 100644 tools/testing/selftests/x86/xsave_signal_handle.c
tools/testing/selftests/x86/Makefile +2 -2
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx xsave_signal_handle # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
@@ -75,7 +75,7 @@ $(BINARIES_32): $(OUTPUT)/%_32: %.c helpers.h $(CC) -m32 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl -lm $(BINARIES_64): $(OUTPUT)/%_64: %.c helpers.h - $(CC) -m64 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl + $(CC) -m64 -mxsave -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $^ -lrt -ldl # x86_64 users should be encouraged to install 32-bit libraries ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01)
tools/testing/selftests/x86/xsave_common.h new file +426 -0
426 lines not on this page · raw message
tools/testing/selftests/x86/xsave_signal_handle.c new file +192 -0
192 lines not on this page · raw message
Signature · 1 line
2.27.0
RFC PATCH v5 2/2 selftests/x86: add xsave test after process switch
In reply to Pengfei XuIt will change FPU, AVX2, AVX512, PKRU xstates before process switch test to ensure that the entire processing of these xstates works as expected. In order to ensure the above XSAVE content is not affected after process switch, this case tests that:
1. The xstates content of the child process should same as xstates of parent process.
2. The xstates content of the child process should not change after process switch back, even grandchild process cleaned up it's own XMM, YMM and ZMM xstates.
--- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/xsave_fork_test.c | 127 ++++++++++++++++++ 2 files changed, 128 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/xsave_fork_test.c
tools/testing/selftests/x86/Makefile +1 -1
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx xsave_signal_handle + corrupt_xstate_header amx xsave_signal_handle xsave_fork_test # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
tools/testing/selftests/x86/xsave_fork_test.c new file +127 -0
127 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling
In reply to Pengfei Xutools/testing/selftests/x86/xsave_common.h line 426
Quoted 7 lines · Pengfei Xu
tools/testing/selftests/x86/xsave_common.h new file +7
-0+void populate_xstate_regs (void ) +{ +set_fpu_reg (); +set_avx2_ymm (); +set_avx512_opmask (); +set_pkru_xstate (); +}
Pengfei, as I mentioned several times, XMM and YMM registers are not preserved across function calls. This only works by chance. The compiler is free to clobber them at basically any time between when they are populated and the XSAVE happens.
Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling
Hi Dave,
Quoted 13 lines · Dave Hansen
tools/testing/selftests/x86/xsave_common.h line 426
Quoted 7 lines · Pengfei Xu
tools/testing/selftests/x86/xsave_common.h new file +7
-0+void populate_xstate_regs (void ) +{ +set_fpu_reg (); +set_avx2_ymm (); +set_avx512_opmask (); +set_pkru_xstate (); +}Pengfei, as I mentioned several times, XMM and YMM registers are not preserved across function calls. This only works by chance. The compiler is free to clobber them at basically any time between when they are populated and the XSAVE happens.
Thanks for suggestions!
Sorry, I read the link https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf
I have removed the XMM registers check as XMM registers are not preserved across function calls in above doc mentioned:
"
Register | Usage | Preserved across function calls
%xmm0–%xmm1 | used to pass and return floating point arguments | No %xmm2–%xmm7 | used to pass floating point arguments | No %xmm8–%xmm15 | temporary registers | No %mmx0–%mmx7 | temporary registers | No
"
It seems that the reason why the contents of the XMM register changed by chance is:
As printf for example, after "printf" Glibc function:do_lookup_x is triggered by printf:
"0x00007ffff7fd5538 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2 ->
0x00007ffff7fd55d7 <+1367>: e8 94 7b 01 00 callq 0x7ffff7fed170 <strcmp>
...
0x00007ffff7fee1af <+4159>: 90 nop 0x00007ffff7fee1b0 <+4160>: 66 0f ef c0 pxor %xmm0,%xmm0 0x00007ffff7fee1b4 <+4164>: 66 0f 6f 17 movdqa (%rdi),%xmm2 => 0x00007ffff7fee1b8 <+4168>: 66 0f 6f 0e movdqa (%rsi),%xmm1 0x00007ffff7fee1bc <+4172>: 66 0f 74 c1 pcmpeqb %xmm1,%xmm0
" And after printf, xmm1 and xmm2 value was changed. And seems some other glibc function will use XMM registers also.
YMM will not be preserved across function calls, but I'm not sure whether some Glibc will use YMM.
I will remove YMM xstates check until there is some other better way for YMM test or it's confirmed YMM will not be used by Glibc.
I think PKRU and AVX512 opmask will not be affected in the test. PKRU xstate will not be affected by registers, and Glibc will not use AVX512 instructions.
How about the following changes:
Will remove set_avx2_ymm() and will only check XSAVE_MASK_FP, XSAVE_MASK_OPMASK and XSAVE_MASK_PKRU xstates after signal handling and process switch, xstates will be controlled by XSAVE_TEST_MASK:
"
#define XSAVE_TEST_MASK (XSAVE_MASK_FP | XSAVE_MASK_YMM | XSAVE_MASK_OPMASK \
| XSAVE_MASK_PKRU)
"
will change to
"
#define XSAVE_TEST_MASK (XSAVE_MASK_FP | XSAVE_MASK_OPMASK | XSAVE_MASK_PKRU) "
FP will be used only by user space, and seems Glibc will not use FP in general. This is the xstates position for FP, XMM, Header, YMM, AVX512 opmask and PKRU:
It could be saved by xsave instruction, and mask could control which part will be saved(Header will be saved as mandatory):
FP (0-159 bytes) XMM (160-415 bytes) Reserved (416-511 bytes SDM vol1 13.4.1) Header_used (512-527 bytes) Headed_reserved (528-575 bytes must 00, otherwise xrstor will #GP)
YMM (Offset:CPUID.(EAX=0D,ECX=2).EBX Size:CPUID(EAX=0D,ECX=2).EAX) AVX512 opmask (Offset:CPUID.(EAX=0D,ECX=5).EBX Size:CPUID(EAX=0D,ECX=5).EAX) PKRU (Offset:CPUID.(EAX=0D,ECX=9).EBX Size:CPUID(EAX=0D,ECX=9).EAX)
Thanks!
BR.
Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling
Quoted 3 lines · Pengfei Xu
How about the following changes:
Will removeset_avx2_ymm()and will only checkXSAVE_MASK_FP,XSAVE_MASK_OPMASKandXSAVE_MASK_PKRUxstates after signal handling and process switch,
First and foremost, the whole point of these tests is to ensure that the kernel is properly maintaining register state. Removing registers from the test moves *away* from the primary goal of this test.
Second, you just listed three states there. Have you considered looking at whether those have the same problem as the XMM/YMM registers? Please do.
Third (and I've also suggested this before), we should explicitly tell the compiler not to use the FPU registers. This is what the kernel does, and it's what allows us to, for instance, make function calls in the kernel without clobbering userspace content in XSAVE-managed registers.
If we did that, then we would only have to worry about calls to things *outside* of the test program, like libc.
Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling
Hi Dave,
Quoted 9 lines · Dave Hansen
Quoted 3 lines · Pengfei Xu
How about the following changes:
Will removeset_avx2_ymm()and will only checkXSAVE_MASK_FP,XSAVE_MASK_OPMASKandXSAVE_MASK_PKRUxstates after signal handling and process switch,First and foremost, the whole point of these tests is to ensure that the kernel is properly maintaining register state. Removing registers from the test moves *away* from the primary goal of this test.
Thanks for suggestion!
Actually, I already removed any useless libc function before and after xsave action, only left the test action between xsave action:
"
XSAVE(xsave_buf2, XSAVE_TEST_MASK); do raise signal or fork test XSAVE(xsave_buf3, XSAVE_TEST_MASK);
"
I found that after fork() function in virtual machine, XMM0 or XMM1 register
will be used and changed.
But in YMM xstate, I haven't see signal handline and fork action will use
and change YMM regiseters in the test. Seems we could keep YMM xstate test.
Seems it needs some other better way for XMM xstate.
Quoted 3 lines
Second, you just listed three states there. Have you considered looking at whether those have the same problem as the XMM/YMM registers? Please do.
I have tested FP, AVX512 opmask and pkru xstates on different platforms and
virtual machine, gdb these 3 xstates with fork and signal handling even printf, above 3 functions will not use and change above 3 xstates. I used previous xsave instruction tests to get the results.
Quoted 7 lines
Third (and I've also suggested this before), we should explicitly tell the compiler not to use the FPU registers. This is what the kernel does, and it's what allows us to, for instance, make function calls in the kernel without clobbering userspace content in XSAVE-managed registers.
If we did that, then we would only have to worry about calls to things *outside* of the test program, like libc.
Thanks! Yes, I will add "float a = 0.12, b = 0.34; a = a + b;" to tell
libc process, float points has been used.
Seems if there is no addition, subtraction, multiplication or division,
there is no change in FP xstate compared with no float definition. If there is above operation, mxcsr(xstate offset 0x18-0x1b bytes)will change from 801f to a01f. Rounding control bit change from 00 to 01.
Thanks!
BR.
Re: [RFC PATCH v5 1/2] selftests/x86: add xsave test during and after signal handling
Hi Dave,
Quoted 39 lines · Pengfei Xu
Hi Dave,
Quoted 9 lines · Dave Hansen
Quoted 3 lines · Pengfei Xu
How about the following changes:
Will removeset_avx2_ymm()and will only checkXSAVE_MASK_FP,XSAVE_MASK_OPMASKandXSAVE_MASK_PKRUxstates after signal handling and process switch,First and foremost, the whole point of these tests is to ensure that the kernel is properly maintaining register state. Removing registers from the test moves *away* from the primary goal of this test.
Thanks for suggestion!
Actually, I already removed any useless libc function before and after xsave action, only left the test action between xsave action:
"XSAVE(xsave_buf2, XSAVE_TEST_MASK); do raise signal or fork test XSAVE(xsave_buf3, XSAVE_TEST_MASK);"
I found that afterfork()function in virtual machine, XMM0 or XMM1 register
will be used and changed.
But in YMM xstate, I haven't see signal handline and fork action will use
and change YMM regiseters in the test. Seems we could keep YMM xstate test.
Seems it needs some other better way for XMM xstate.Quoted 3 lines
Second, you just listed three states there. Have you considered looking at whether those have the same problem as the XMM/YMM registers? Please do.
I have tested FP, AVX512 opmask and pkru xstates on different platforms and
virtual machine, gdb these 3 xstates with fork and signal handling even printf, above 3 functions will not use and change above 3 xstates. I used previous xsave instruction tests to get the results.Quoted 5 lines
Third (and I've also suggested this before), we should explicitly tell the compiler not to use the FPU registers. This is what the kernel does, and it's what allows us to, for instance, make function calls in the kernel without clobbering userspace content in XSAVE-managed registers.
Quoted 9 lines
Quoted 2 lines
If we did that, then we would only have to worry about calls to things *outside* of the test program, like libc.
Thanks! Yes, I will add "float a = 0.12, b = 0.34; a = a + b;" to tell
libc process, float points has been used.
Seems if there is no addition, subtraction, multiplication or division,
there is no change in FP xstate compared with no float definition. If there is above operation, mxcsr(xstate offset 0x18-0x1b bytes)will change from 801f to a01f. Rounding control bit change from 00 to 01.
Sorry, I misunderstood your meaning in the last email, I should directly add syscall function between the 2 xsave comparisons, it will not use XMM, YMM and so on xstates, and I could add them back.
For example, for "fork", I will use syscall(SYS_fork) instead of fork().
Quoted 2 lines
Thanks!
BR.