Skip to contentSkip to conversation

x86/xsaves: Optimize xstate context switch by xsaves/xrstors

PATCH In review Fenghua Yu 4 participants 21 messages May 26, 2014

Subsystems (2)
lore
Share & tools
lore

Fenghua Yu

author
#0 lore raw

cover x86/xsaves: Optimize xstate context switch by xsaves/xrstors

From: Fenghua Yu <[email protected]>

With ever growing extended state registers (xstate) on x86 processors, kernel needs to cope with issue of growing memory space occupied by xstate. The xsave area is holding more and more xstate registers, growing from legacy FP and SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.

The recently introduced compacted format of xsave area saves xstates only for enabled states. This patch set saves the xsave area space per process in compacted format by xsaves/xrstors instructions.

xsaves/xrstors also supports modified optimization which tracks modified xstates and only saves modified xstates to xsave area to improve saving performance.

For more information about xsaves/xrstors instructions, please refer to latest Intel X86 SDM.

Fenghua Yu (15):
x86/xsaves: Detect xsaves/xrstors feature
x86/xsaves: Add kernel parameter to disable xsaves/xrstors x86/alternative: Add alternative_input_2
x86/xsaves: Change compacted format xsave area header x86/xsaves: Define macros for xsave instructions x86/xsaves: Define macro for handle xsave/xrstor fault x86/xsaves: Use xsaves/xrstors for saving and restoring xsave area x86/xsaves: Use xsaves/xrstors for context switch x86/xsaves: Use xsave/xrstor for saving and restoring user space
context
x86/xsaves: Clear reserved bits in xsave header x86/xsaves: Add xsaves and xrstors support for booting time x86/xsaves: Save xstate to task's xsave area in __save_fpu during
booting time
x86/xsaves: Call booting time xsaves and xrstors in setup_init_fpu_buf x86/xsaves: Enable xsaves/xrstors
Define kernel API to get address of each state in xsave area

 Documentation/kernel-parameters.txt   |   9 ++
 arch/x86/include/asm/alternative.h    |  13 ++
 arch/x86/include/asm/cpufeature.h     |   2 +
 arch/x86/include/asm/fpu-internal.h   |   9 +-
 arch/x86/include/asm/processor.h      |   4 +-
 arch/x86/include/asm/xsave.h          | 220 +++++++++++++++++++++++++---------
 arch/x86/include/uapi/asm/msr-index.h |   2 +
 arch/x86/kernel/cpu/common.c          |   8 ++
 arch/x86/kernel/cpu/scattered.c       |   1 +
 arch/x86/kernel/i387.c                |   2 +-
 arch/x86/kernel/process.c             |   1 +
 arch/x86/kernel/xsave.c               | 112 +++++++++++++++--
 12 files changed, 313 insertions(+), 70 deletions(-)
Signature · 1 line
1.8.1.2
author
#1 lore raw

PATCH 1/15 x86/xsaves: Detect xsaves/xrstors feature

From: Fenghua Yu <[email protected]>

Detect the xsaves feature in processor extended state enumberation sub-leaf (eax=0x0d, ecx=1):
Bit 03: Supports xsaves/xrstors and IA32_XSS if set

The IA32_XSS MSR (index DA0H) contains a state-component bitmap that specifies the state components that software has enabled xsaves and xrstors to manage. If the bit corresponding to a state component is clear in XCR0 | IA32_XSS, xsaves and xrstors will not operate on that state component, regardless of the value of the instruction mask.

---
 arch/x86/include/asm/cpufeature.h     | 2 ++
 arch/x86/include/uapi/asm/msr-index.h | 2 ++
 arch/x86/kernel/cpu/scattered.c       | 1 +
 3 files changed, 5 insertions(+)
arch/x86/include/asm/cpufeature.h +2 -0
index e265ff9..efede2e 100644
@@ -186,6 +186,7 @@
 #define X86_FEATURE_DTHERM	(7*32+ 7) /* Digital Thermal Sensor */
 #define X86_FEATURE_HW_PSTATE	(7*32+ 8) /* AMD HW-PState */
 #define X86_FEATURE_PROC_FEEDBACK (7*32+ 9) /* AMD ProcFeedbackInterface */
+#define X86_FEATURE_XSAVES	(7*32+13) /* xsaves/xrstors/xss */
 
 /* Virtualization flags: Linux defined, word 8 */
 #define X86_FEATURE_TPR_SHADOW  (8*32+ 0) /* Intel TPR Shadow */
@@ -328,6 +329,7 @@ extern const char * const x86_power_flags[32];
 #define cpu_has_x2apic		boot_cpu_has(X86_FEATURE_X2APIC)
 #define cpu_has_xsave		boot_cpu_has(X86_FEATURE_XSAVE)
 #define cpu_has_xsaveopt	boot_cpu_has(X86_FEATURE_XSAVEOPT)
+#define cpu_has_xsaves		boot_cpu_has(X86_FEATURE_XSAVES)
 #define cpu_has_osxsave		boot_cpu_has(X86_FEATURE_OSXSAVE)
 #define cpu_has_hypervisor	boot_cpu_has(X86_FEATURE_HYPERVISOR)
 #define cpu_has_pclmulqdq	boot_cpu_has(X86_FEATURE_PCLMULQDQ)
arch/x86/include/uapi/asm/msr-index.h +2 -0
index fcf2b3a..ce04632 100644
@@ -311,6 +311,8 @@
 #define MSR_IA32_UCODE_WRITE		0x00000079
 #define MSR_IA32_UCODE_REV		0x0000008b
 
+#define MSR_IA32_XSS			0x00000da0
+
 #define MSR_IA32_PERF_STATUS		0x00000198
 #define MSR_IA32_PERF_CTL		0x00000199
 #define MSR_AMD_PSTATE_DEF_BASE		0xc0010064
arch/x86/kernel/cpu/scattered.c +1 -0
index b6f794a..8b29ea2 100644
@@ -39,6 +39,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c)
 		{ X86_FEATURE_APERFMPERF,	CR_ECX, 0, 0x00000006, 0 },
 		{ X86_FEATURE_EPB,		CR_ECX, 3, 0x00000006, 0 },
 		{ X86_FEATURE_XSAVEOPT,		CR_EAX,	0, 0x0000000d, 1 },
+		{ X86_FEATURE_XSAVES,		CR_EAX,	3, 0x0000000d, 1 },
 		{ X86_FEATURE_HW_PSTATE,	CR_EDX, 7, 0x80000007, 0 },
 		{ X86_FEATURE_CPB,		CR_EDX, 9, 0x80000007, 0 },
 		{ X86_FEATURE_PROC_FEEDBACK,	CR_EDX,11, 0x80000007, 0 },
Signature · 1 line
1.8.1.2
author
#2 lore raw

PATCH 2/15 x86/xsaves: Add kernel parameter to disable xsaves/xrstors

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Add kernel parameter noxsaves to disable xsaves/xrstors feature.

Add kernel parameter noxsaveopt in doc.

---
 Documentation/kernel-parameters.txt | 9 +++++++++
 arch/x86/kernel/cpu/common.c        | 8 ++++++++
 2 files changed, 17 insertions(+)
Documentation/kernel-parameters.txt +9 -0
index 4384217..8541752 100644
@@ -2124,6 +2124,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			and restore using xsave. The kernel will fallback to
 			enabling legacy floating-point and sse state.
 
+	noxsaveopt	[X86] Disables x86 extended register state context
+			switch using xsaveopt. The kernel wil fallback to use
+			not optimized xsave to do context switch.
+
+	noxsaves	[X86] Disables x86 extended register state context
+			switch and save/restore compacted formated xsave
+			area using xsaves. Then the kernel uses standard
+			xsave area for xstate.
+
 	eagerfpu=	[X86]
 			on	enable eager fpu restore
 			off	disable eager fpu restore
arch/x86/kernel/cpu/common.c +8 -0
index a135239..409f47c 100644
@@ -146,6 +146,7 @@ static int __init x86_xsave_setup(char *s)
 {
 	setup_clear_cpu_cap(X86_FEATURE_XSAVE);
 	setup_clear_cpu_cap(X86_FEATURE_XSAVEOPT);
+	setup_clear_cpu_cap(X86_FEATURE_XSAVES);
 	setup_clear_cpu_cap(X86_FEATURE_AVX);
 	setup_clear_cpu_cap(X86_FEATURE_AVX2);
 	return 1;
@@ -159,6 +160,13 @@ static int __init x86_xsaveopt_setup(char *s)
 }
 __setup("noxsaveopt", x86_xsaveopt_setup);
 
+static int __init x86_xsaves_setup(char *s)
+{
+	setup_clear_cpu_cap(X86_FEATURE_XSAVES);
+	return 1;
+}
+__setup("noxsaves", x86_xsaves_setup);
+
 #ifdef CONFIG_X86_32
 static int cachesize_override = -1;
 static int disable_x86_serial_nr = 1;
Signature · 1 line
1.8.1.2
author
#3 lore raw

PATCH 3/15 x86/alternative: Add alternative_input_2

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Add alternative_input_2 to take two features and respective instructions.

If CPU has feature2, newinstr2 is used.

Otherwise, if CPU has feature1, newinstr1 is used.

Otherwise, oldinstr is used.

---
 arch/x86/include/asm/alternative.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)
arch/x86/include/asm/alternative.h +13 -0
index 0a3f9c9..eb8d80b 100644
@@ -161,6 +161,19 @@ static inline int alternatives_text_reserved(void *start, void *end)
 	asm volatile (ALTERNATIVE(oldinstr, newinstr, feature)		\
 		: : "i" (0), ## input)
 
+/*
+ * Like alternative_input, but there are two features and respective
+ * instructions.
+ * If CPU has feature2, newinstr2 is used.
+ * Otherwise, if CPU has feature1, newinstr1 is used.
+ * Otherwise, oldinstr is used.
+ */
+#define alternative_input_2(oldinstr, newinstr1, feature1, newinstr2,	     \
+			   feature2, input...)				     \
+	asm volatile(ALTERNATIVE_2(oldinstr, newinstr1, feature1,	     \
+		newinstr2, feature2)					     \
+		: : "i" (0), ## input)
+
 /* Like alternative_input, but with a single output argument */
 #define alternative_io(oldinstr, newinstr, feature, output, input...)	\
 	asm volatile (ALTERNATIVE(oldinstr, newinstr, feature)		\
Signature · 1 line
1.8.1.2
author
#4 lore raw

PATCH 4/15 x86/xsaves: Change compacted format xsave area header

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

With compacted format xsave area supported, the xsave area header is changed to support both compacted format and standard format xsave area.

The XSAVE header of an XSAVE area comprises the 64 bytes starting at offset 512 from the area\u2019s base address:
- Bytes 7:0 of the XSAVE header is a state-component bitmap called
XSTATE_BV. It identifies the state components in the XSAVE area.
- Bytes 15:8 of the XSAVE header is a state-component bitmap called
XCOMP_BV. It is used as follows:
- XCOMP_BV[63] indicates the format of the extended region of
the XSAVE area. If it is clear, the standard format is used. If it is set, the compacted format is used; XCOMP_BV[62:0] provide format specifics.
- XCOMP_BV[63] determines which form of the XRSTOR instruction
is used. If the bit is set, the compacted form is used; otherwise, the standard form is used. See Section 13.7.
- All bits in XCOMP_BV should be 0 if the processor does not support
the compaction extensions to the XSAVE feature set.
- Bytes 63:16 of the XSAVE header are reserved.

---
 arch/x86/include/asm/processor.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/processor.h +2 -2
index a4ea023..2c8d3b8 100644
@@ -386,8 +386,8 @@ struct bndcsr_struct {
 
 struct xsave_hdr_struct {
 	u64 xstate_bv;
-	u64 reserved1[2];
-	u64 reserved2[5];
+	u64 xcomp_bv;
+	u64 reserved[6];
 } __attribute__((packed));
 
 struct xsave_struct {
Signature · 1 line
1.8.1.2
author
#5 lore raw

PATCH 5/15 x86/xsaves: Define macros for xsave instructions

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Define macros for xsave, xsaveopt, xsaves, xrstor, xrstors inline instructions. The instructions will be used for saving and rstoring xstate.

---
 arch/x86/include/asm/xsave.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/xsave.h +6 -0
index d949ef2..71bdde4 100644
@@ -52,6 +52,12 @@ extern void xsave_init(void);
 extern void update_regset_xstate_info(unsigned int size, u64 xstate_mask);
 extern int init_fpu(struct task_struct *child);
 
+#define XSAVE		".byte " REX_PREFIX "0x0f,0xae,0x27"
+#define XSAVEOPT	".byte " REX_PREFIX "0x0f,0xae,0x37"
+#define XSAVES		".byte " REX_PREFIX "0x0f,0xc7,0x2f"
+#define XRSTOR		".byte " REX_PREFIX "0x0f,0xae,0x2f"
+#define XRSTORS		".byte " REX_PREFIX "0x0f,0xc7,0x1f"
+
 static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 {
 	int err;
Signature · 1 line
1.8.1.2
author
#6 lore raw

PATCH 6/15 x86/xsaves: Define macro for handle xsave/xrstor fault

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Define macro to handle fault generated by xsave, xsaveopt, xsaves, xrstor, xrstors instructions.

---
 arch/x86/include/asm/xsave.h | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/include/asm/xsave.h +7 -0
index 71bdde4..76c2459 100644
@@ -58,6 +58,13 @@ extern int init_fpu(struct task_struct *child);
 #define XRSTOR		".byte " REX_PREFIX "0x0f,0xae,0x2f"
 #define XRSTORS		".byte " REX_PREFIX "0x0f,0xc7,0x1f"
 
+#define xstate_fault	".section .fixup,\"ax\"\n"	\
+			"3:  movl $-1,%[err]\n"		\
+			"    jmp  2b\n"			\
+			".previous\n"			\
+			_ASM_EXTABLE(1b, 3b)		\
+			: [err] "=r" (err)
+
 static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 {
 	int err;
Signature · 1 line
1.8.1.2
author
#7 lore raw

PATCH 7/15 x86/xsaves: Use xsaves/xrstors for saving and restoring xsave area

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

If xsaves is eanbled, use xsaves/xrstors for saving and restoring xstate. xsaves supports compacted format, init optimization, modified optimization, and supervisor states.

---
 arch/x86/include/asm/xsave.h | 82 +++++++++++++++++++++++++++++++++-----------
 1 file changed, 62 insertions(+), 20 deletions(-)
arch/x86/include/asm/xsave.h +62 -20
index 76c2459..e538a0e 100644
@@ -65,6 +65,68 @@ extern int init_fpu(struct task_struct *child);
 			_ASM_EXTABLE(1b, 3b)		\
 			: [err] "=r" (err)
 
+/*
+ * Save processor xstate to xsave area.
+ */
+static inline void xsave_state(struct xsave_struct *fx, u64 mask)
+{
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+	int err = 0;
+
+	/*
+	 * If xsaves is enabled, xsaves replaces xsaveopt because
+	 * it supports compact format and supervisor states in addition to
+	 * modified optimization in xsaveopt.
+	 *
+	 * Otherwise, if xsaveopt is enabled, xsaveopt replaces xsave
+	 * because xsaveopt supports modified optimization which is not
+	 * supported by xsave.
+	 *
+	 * If none of xsaves and xsaveopt is enabled, use xsave.
+	 */
+	alternative_input_2(
+		"1:"XSAVE,
+		"1:"XSAVEOPT,
+		X86_FEATURE_XSAVEOPT,
+		"1:"XSAVES,
+		X86_FEATURE_XSAVES,
+		[fx] "D" (fx), "a" (lmask), "d" (hmask) :
+		"memory");
+	asm volatile("2:\n\t"
+		     xstate_fault
+		     : "0" (0)
+		     : "memory");
+}
+
+/*
+ * Restore processor xstate from xsave area.
+ */
+static inline int xrstor_state(struct xsave_struct *fx, u64 mask)
+{
+	int err = 0;
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+
+	/*
+	 * Use xrstors to restore context if it is enabled. xrstors supports
+	 * compacted format of xsave area which is not supported by xrstor.
+	 */
+	alternative_input(
+		"1: " XRSTOR,
+		"1: " XRSTORS,
+		X86_FEATURE_XSAVES,
+		"D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+		: "memory");
+
+	asm volatile("2:\n"
+		     xstate_fault
+		     : "0" (0)
+		     : "memory");
+
+	return err;
+}
+
 static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 {
 	int err;
@@ -130,26 +192,6 @@ static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 	return err;
 }
 
-static inline void xrstor_state(struct xsave_struct *fx, u64 mask)
-{
-	u32 lmask = mask;
-	u32 hmask = mask >> 32;
-
-	asm volatile(".byte " REX_PREFIX "0x0f,0xae,0x2f\n\t"
-		     : : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
-		     :   "memory");
-}
-
-static inline void xsave_state(struct xsave_struct *fx, u64 mask)
-{
-	u32 lmask = mask;
-	u32 hmask = mask >> 32;
-
-	asm volatile(".byte " REX_PREFIX "0x0f,0xae,0x27\n\t"
-		     : : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
-		     :   "memory");
-}
-
 static inline void fpu_xsave(struct fpu *fpu)
 {
 	/* This, however, we can work around by forcing the compiler to select
Signature · 1 line
1.8.1.2
author
#8 lore raw

PATCH 8/15 x86/xsaves: Use xsaves/xrstors for context switch

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

If xsaves is eanbled, use xsaves/xrstors for context switch to utilize compacted format xsave area and modified optimization.

---
 arch/x86/include/asm/xsave.h | 37 ++++++++++++-------------------------
 1 file changed, 12 insertions(+), 25 deletions(-)
arch/x86/include/asm/xsave.h +12 -25
index e538a0e..cfee67e 100644
@@ -127,22 +127,20 @@ static inline int xrstor_state(struct xsave_struct *fx, u64 mask)
 	return err;
 }
 
-static inline int fpu_xrstor_checking(struct xsave_struct *fx)
+/*
+ * Save xstate context for old process during context switch.
+ */
+static inline void fpu_xsave(struct fpu *fpu)
 {
-	int err;
-
-	asm volatile("1: .byte " REX_PREFIX "0x0f,0xae,0x2f\n\t"
-		     "2:\n"
-		     ".section .fixup,\"ax\"\n"
-		     "3:  movl $-1,%[err]\n"
-		     "    jmp  2b\n"
-		     ".previous\n"
-		     _ASM_EXTABLE(1b, 3b)
-		     : [err] "=r" (err)
-		     : "D" (fx), "m" (*fx), "a" (-1), "d" (-1), "0" (0)
-		     : "memory");
+	xsave_state(&fpu->state->xsave, -1);
+}
 
-	return err;
+/*
+ * Restore xstate context for new process during context switch.
+ */
+static inline int fpu_xrstor_checking(struct xsave_struct *fx)
+{
+	return xrstor_state(fx, -1);
 }
 
 static inline int xsave_user(struct xsave_struct __user *buf)
@@ -192,15 +190,4 @@ static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 	return err;
 }
 
-static inline void fpu_xsave(struct fpu *fpu)
-{
-	/* This, however, we can work around by forcing the compiler to select
-	   an addressing mode that doesn't require extended registers. */
-	alternative_input(
-		".byte " REX_PREFIX "0x0f,0xae,0x27",
-		".byte " REX_PREFIX "0x0f,0xae,0x37",
-		X86_FEATURE_XSAVEOPT,
-		[fx] "D" (&fpu->state->xsave), "a" (-1), "d" (-1) :
-		"memory");
-}
 #endif
Signature · 1 line
1.8.1.2
author
#9 lore raw

PATCH 9/15 x86/xsaves: Use xsave/xrstor for saving and restoring user space context

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

We use legacy xsave/xrstor for user space context. No xsaveopt or xsaves is used here.

We don't use modified optimization which is implemented in xsaveopt and xsaves because xrstor/xrstors might track a different application.

We don't use compacted format xsave area in xsaves for backward compatibility because old applications which don't understand compacted format of xsave area. This may cosume more user context memory than compacted format. But in this way legacy applications don't need any change to work with kernel.

---
 arch/x86/include/asm/xsave.h | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)
arch/x86/include/asm/xsave.h +18 -15
index cfee67e..a8f31d0 100644
@@ -143,6 +143,16 @@ static inline int fpu_xrstor_checking(struct xsave_struct *fx)
 	return xrstor_state(fx, -1);
 }
 
+/*
+ * Save xstate to user space xsave area.
+ *
+ * We don't use modified optimization because xrstor/xrstors might track
+ * a different application.
+ *
+ * We don't use compacted format xsave area for
+ * backward compatibility for old applications which don't understand
+ * compacted format of xsave area.
+ */
 static inline int xsave_user(struct xsave_struct __user *buf)
 {
 	int err;
@@ -156,35 +166,28 @@ static inline int xsave_user(struct xsave_struct __user *buf)
 		return -EFAULT;
 
 	__asm__ __volatile__(ASM_STAC "\n"
-			     "1: .byte " REX_PREFIX "0x0f,0xae,0x27\n"
+			     "1:"XSAVE"\n"
 			     "2: " ASM_CLAC "\n"
-			     ".section .fixup,\"ax\"\n"
-			     "3:  movl $-1,%[err]\n"
-			     "    jmp  2b\n"
-			     ".previous\n"
-			     _ASM_EXTABLE(1b,3b)
-			     : [err] "=r" (err)
+			     xstate_fault
 			     : "D" (buf), "a" (-1), "d" (-1), "0" (0)
 			     : "memory");
 	return err;
 }
 
+/*
+ * Restore xstate from user space xsave area.
+ */
 static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 {
-	int err;
+	int err = 0;
 	struct xsave_struct *xstate = ((__force struct xsave_struct *)buf);
 	u32 lmask = mask;
 	u32 hmask = mask >> 32;
 
 	__asm__ __volatile__(ASM_STAC "\n"
-			     "1: .byte " REX_PREFIX "0x0f,0xae,0x2f\n"
+			     "1:"XRSTOR"\n"
 			     "2: " ASM_CLAC "\n"
-			     ".section .fixup,\"ax\"\n"
-			     "3:  movl $-1,%[err]\n"
-			     "    jmp  2b\n"
-			     ".previous\n"
-			     _ASM_EXTABLE(1b,3b)
-			     : [err] "=r" (err)
+			     xstate_fault
 			     : "D" (xstate), "a" (lmask), "d" (hmask), "0" (0)
 			     : "memory");	/* memory required? */
 	return err;
Signature · 1 line
1.8.1.2
author
#10 lore raw

PATCH 10/15 x86/xsaves: Clear reserved bits in xsave header

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

The reserved bits (128~511) in the xsave header must be zero. Clear the bits here.

---
 arch/x86/kernel/i387.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/i387.c +1 -1
index d5dd808..a9a4229 100644
@@ -375,7 +375,7 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset,
 	/*
 	 * These bits must be zero.
 	 */
-	xsave_hdr->reserved1[0] = xsave_hdr->reserved1[1] = 0;
+	memset(xsave_hdr->reserved, 0, 48);
 
 	return ret;
 }
Signature · 1 line
1.8.1.2
author
#11 lore raw

PATCH 11/15 x86/xsaves: Add xsaves and xrstors support for booting time

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Since cpu caps are not enabled yet during early booting time, alternative can not be used in functions to access xsave area. Therefore, we define two new functions xrstor_state_booting and xsave_state_booting to access xsave area during early booting time.

Because xsave_state_booting() can be called by run-time function __save_fpu(), it's not defined as __init and will stay in memory during running time. But it's a small function and the memory cost is small.

---
 arch/x86/include/asm/xsave.h | 60 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
arch/x86/include/asm/xsave.h +60 -0
index a8f31d0..0de695d 100644
@@ -66,6 +66,66 @@ extern int init_fpu(struct task_struct *child);
 			: [err] "=r" (err)
 
 /*
+ * This function is called only during boot time when x86 caps are not set
+ * up and alternative can not be used yet.
+ */
+static void xsave_state_booting(struct xsave_struct *fx, u64 mask)
+{
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+	int err = 0;
+
+	WARN_ON(system_state != SYSTEM_BOOTING);
+
+	if (boot_cpu_has(X86_FEATURE_XSAVES))
+		asm volatile("1:"XSAVES"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+	else
+		asm volatile("1:"XSAVE"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+
+	asm volatile(xstate_fault
+		     : "0" (0)
+		     : "memory");
+
+	return err;
+}
+
+/*
+ * This function is called only during boot time when x86 caps are not set
+ * up and alternative can not be used yet.
+ */
+static inline int xrstor_state_booting(struct xsave_struct *fx, u64 mask)
+{
+	u32 lmask = mask;
+	u32 hmask = mask >> 32;
+	int err = 0;
+
+	WARN_ON(system_state != SYSTEM_BOOTING);
+
+	if (boot_cpu_has(X86_FEATURE_XSAVES))
+		asm volatile("1:"XRSTORS"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+	else
+		asm volatile("1:"XRSTOR"\n\t"
+			"2:\n\t"
+			: : "D" (fx), "m" (*fx), "a" (lmask), "d" (hmask)
+			:   "memory");
+
+	asm volatile(xstate_fault
+		     : "0" (0)
+		     : "memory");
+
+	return err;
+}
+
+/*
  * Save processor xstate to xsave area.
  */
 static inline void xsave_state(struct xsave_struct *fx, u64 mask)
Signature · 1 line
1.8.1.2
author
#12 lore raw

PATCH 12/15 x86/xsaves: Save xstate to task's xsave area in __save_fpu during booting time

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

__save_fpu() can be called during early booting time when cpu caps are not enabled and alternative can not be used yet. Therefore, it calls xsave_state_booting() to save xstate to task's xsave area.

---
 arch/x86/include/asm/fpu-internal.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
arch/x86/include/asm/fpu-internal.h +6 -3
index cea1c76..6099c0e 100644
@@ -508,9 +508,12 @@ static inline void user_fpu_begin(void)
 
 static inline void __save_fpu(struct task_struct *tsk)
 {
-	if (use_xsave())
-		xsave_state(&tsk->thread.fpu.state->xsave, -1);
-	else
+	if (use_xsave()) {
+		if (unlikely(system_state == SYSTEM_BOOTING))
+			xsave_state_booting(&tsk->thread.fpu.state->xsave, -1);
+		else
+			xsave_state(&tsk->thread.fpu.state->xsave, -1);
+	} else
 		fpu_fxsave(&tsk->thread.fpu);
 }
 
Signature · 1 line
1.8.1.2
author
#13 lore raw

PATCH 13/15 x86/xsaves: Call booting time xsaves and xrstors in setup_init_fpu_buf

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

setup_init_fpu_buf() calls booting time xsaves and xrstors to save/restore xstate in xsave area.

---
 arch/x86/kernel/xsave.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/x86/kernel/xsave.c +8 -2
index a4b451c..8fa7c7d 100644
@@ -496,15 +496,21 @@ static void __init setup_init_fpu_buf(void)
 
 	setup_xstate_features();
 
+	if (cpu_has_xsaves) {
+		init_xstate_buf->xsave_hdr.xcomp_bv =
+						(u64)1 << 63 | pcntxt_mask;
+		init_xstate_buf->xsave_hdr.xstate_bv = pcntxt_mask;
+	}
+
 	/*
 	 * Init all the features state with header_bv being 0x0
 	 */
-	xrstor_state(init_xstate_buf, -1);
+	xrstor_state_booting(init_xstate_buf, -1);
 	/*
 	 * Dump the init state again. This is to identify the init state
 	 * of any feature which is not represented by all zero's.
 	 */
-	xsave_state(init_xstate_buf, -1);
+	xsave_state_booting(init_xstate_buf, -1);
 }
 
 static enum { AUTO, ENABLE, DISABLE } eagerfpu = AUTO;
Signature · 1 line
1.8.1.2
author
#14 lore raw

PATCH 14/15 x86/xsaves: Enable xsaves/xrstors

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

If xsaves/xrstors is enabled, compacted format of xsave area will be used and less memory may be used for context switching per process. Modified optimization implemented in xsaves/xrstors improves performance of saving xstate as well.

---
 arch/x86/kernel/xsave.c | 38 ++++++++++++++++++++++++++++++++------
 1 file changed, 32 insertions(+), 6 deletions(-)
arch/x86/kernel/xsave.c +32 -6
index 8fa7c7d..2e9a77d 100644
@@ -8,6 +8,7 @@
 
 #include <linux/bootmem.h>
 #include <linux/compat.h>
+#include <linux/cpu.h>
 #include <asm/i387.h>
 #include <asm/fpu-internal.h>
 #include <asm/sigframe.h>
@@ -24,7 +25,9 @@ u64 pcntxt_mask;
 struct xsave_struct *init_xstate_buf;
 
 static struct _fpx_sw_bytes fx_sw_reserved, fx_sw_reserved_ia32;
-static unsigned int *xstate_offsets, *xstate_sizes, xstate_features;
+static unsigned int *xstate_offsets, *xstate_sizes;
+static unsigned int *xstate_comp_offsets, *xstate_comp_sizes;
+static unsigned int xstate_features;
 
 /*
  * If a processor implementation discern that a processor state component is
@@ -283,7 +286,7 @@ sanitize_restored_xstate(struct task_struct *tsk,
 
 	if (use_xsave()) {
 		/* These bits must be zero. */
-		xsave_hdr->reserved1[0] = xsave_hdr->reserved1[1] = 0;
+		memset(xsave_hdr->reserved, 0, 48);
 
 		/*
 		 * Init the state that is not present in the memory
@@ -526,6 +529,30 @@ static int __init eager_fpu_setup(char *s)
 }
 __setup("eagerfpu=", eager_fpu_setup);
 
+
+/*
+ * Calculate total size of enabled xstates in XCR0/pcntxt_mask.
+ */
+static void __init init_xstate_size(void)
+{
+	unsigned int eax, ebx, ecx, edx;
+	int i;
+
+	if (!cpu_has_xsaves) {
+		cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);
+		xstate_size = ebx;
+		return;
+	}
+
+	xstate_size = FXSAVE_SIZE + XSAVE_HDR_SIZE;
+	for (i = 2; i < 64; i++) {
+		if (test_bit(i, (unsigned long *)&pcntxt_mask)) {
+			cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx);
+			xstate_size += eax;
+		}
+	}
+}
+
 /*
  * Enable and initialize the xsave feature.
  */
@@ -557,8 +584,7 @@ static void __init xstate_enable_boot_cpu(void)
 	/*
 	 * Recompute the context size for enabled features
 	 */
-	cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx);
-	xstate_size = ebx;
+	init_xstate_size();
 
 	update_regset_xstate_info(xstate_size, pcntxt_mask);
 	prepare_fx_sw_frame();
@@ -578,8 +604,8 @@ static void __init xstate_enable_boot_cpu(void)
 		}
 	}
 
-	pr_info("enabled xstate_bv 0x%llx, cntxt size 0x%x\n",
-		pcntxt_mask, xstate_size);
+	pr_info("enabled xstate_bv 0x%llx, cntxt size 0x%x using %s\n",
+		pcntxt_mask, xstate_size, cpu_has_xsaves ? "xsaves" : "xsave");
 }
 
 /*
Signature · 1 line
1.8.1.2
author
#15 lore raw

PATCH 15/15 Define kernel API to get address of each state in xsave area

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

In standard format, each state is saved in the xsave area in fixed offset. In compacted format, offset of each saved state only can be calculated during run time because some xstates may not be saved.

get_xsave_addr() returns address of a given state saved in a xsave area.

It can be called in kernel to get address of each xstate in xsave area in either standard format or compacted format.

---
 arch/x86/include/asm/xsave.h |  3 +++
 arch/x86/kernel/process.c    |  1 +
 arch/x86/kernel/xsave.c      | 64 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+)
arch/x86/include/asm/xsave.h +3 -0
index 0de695d..575e156 100644
@@ -253,4 +253,7 @@ static inline int xrestore_user(struct xsave_struct __user *buf, u64 mask)
 	return err;
 }
 
+void *get_xsave_addr(struct xsave_struct *xsave, int xstate);
+void setup_xstate_comp(void);
+
 #endif
arch/x86/kernel/process.c +1 -0
index 4505e2a..f804dc9 100644
@@ -93,6 +93,7 @@ void arch_task_cache_init(void)
         	kmem_cache_create("task_xstate", xstate_size,
 				  __alignof__(union thread_xstate),
 				  SLAB_PANIC | SLAB_NOTRACK, NULL);
+	setup_xstate_comp();
 }
 
 /*
arch/x86/kernel/xsave.c +64 -0

73 lines not on this page · raw message

Signature · 1 line
1.8.1.2
#16 lore raw
In reply to Fenghua Yu
Quoted 10 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

With ever growing extended state registers (xstate) on x86 processors, kernel needs to cope with issue of growing memory space occupied by xstate. The xsave area is holding more and more xstate registers, growing from legacy FP and SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.

The recently introduced compacted format of xsave area saves xstates only for enabled states. This patch set saves the xsave area space per process in compacted format by xsaves/xrstors instructions.

Are we going to want to encourage userspace to do something like sticking vzeroupper right before each syscall to make any xsaves/xrestores faster?

--Andy

#17 lore raw
Quoted 13 lines

From: Andy Lutomirski [mailto:[email protected]]

Quoted 3 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

With ever growing extended state registers (xstate) on x86 processors,

kernel

Quoted 1 line

needs to cope with issue of growing memory space occupied by xstate.

The xsave

Quoted 1 line

area is holding more and more xstate registers, growing from legacy

FP and

Quoted 3 lines

SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.

The recently introduced compacted format of xsave area saves xstates

Quoted 8 lines

only

Quoted 1 line

for enabled states. This patch set saves the xsave area space per

process

Quoted 1 line

in compacted format by xsaves/xrstors instructions.

Are we going to want to encourage userspace to do something like sticking vzeroupper right before each syscall to make any xsaves/xrestores faster?

This patch set allow compacted format in kernel and standard format in user space. This works fine for both kernel and user application.

As said in the patch 9/15, there would be two issues for user applications to use xsaves/xrstors: one is modified optimization doesn't work for user applications and the other one is legacy applications don't know compacted format of xsave area.

So I don't see we need xsaves/xrstors in user space. But I could be wrong.

Thanks.

-Fenghua

#18 lore raw
Quoted 4 lines · Fenghua

So I don't see we need xsaves/xrstors in user space. But I could be wrong.

XSAVES/XRSTORS are privileged. The user-space equivalents are XSAVEC/XRSTOR.

-hpa

#19 lore raw
In reply to Yu, Fenghua
Quoted 16 lines · Fenghua
Quoted 16 lines

From: Andy Lutomirski [mailto:[email protected]]

Quoted 3 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

With ever growing extended state registers (xstate) on x86 processors,

kernel

Quoted 1 line

needs to cope with issue of growing memory space occupied by xstate.

The xsave

Quoted 1 line

area is holding more and more xstate registers, growing from legacy

FP and

Quoted 3 lines

SSE to AVX, AVX2, AVX-512, MPX, and Intel PT.

The recently introduced compacted format of xsave area saves xstates

only

Quoted 1 line

for enabled states. This patch set saves the xsave area space per

process

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

in compacted format by xsaves/xrstors instructions.

Are we going to want to encourage userspace to do something like sticking vzeroupper right before each syscall to make any xsaves/xrestores faster?

This patch set allow compacted format in kernel and standard format in user space. This works fine for both kernel and user application.

My question is purely about optimization: if userspace does a blocking system call, will it be significantly faster if userspace zeros out as much of the extended state as possible before doing the system call?

I think I tried this once with xsaveopt and decided that it didn't make much of a difference.

--Andy

#20 lore raw
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines
Quoted 4 lines

Are we going to want to encourage userspace to do something like sticking vzeroupper right before each syscall to make any xsaves/xrestores faster?

This patch set allow compacted format in kernel and standard format in user space. This works fine for both kernel and user application.

Quoted 7 lines

My question is purely about optimization: if userspace does a blocking system call, will it be significantly faster if userspace zeros out as much of the extended state as possible before doing the system call?

I think I tried this once with xsaveopt and decided that it didn't make much of a difference.

XSAVEOPT and XSAVES would be the same in this case. Keep in mind, too, that relatively few system calls result in context switches, which is the only time this ends up mattering.

-hpa