Enable LKGS instruction
Subsystems (3)
cover Enable LKGS instruction
LKGS instruction is introduced with Intel FRED (flexible return and event delivery) specification https://cdrdv2.intel.com/v1/dl/getContent/678938.
LKGS is independent of FRED, so we enable it as a standalone CPU feature.
LKGS behaves like the MOV to GS instruction except that it loads the base address into the IA32_KERNEL_GS_BASE MSR instead of the GS segment’s descriptor cache, which is exactly what Linux kernel does to load user level GS base. Thus, with LKGS, there is no need to SWAPGS away from the kernel GS base.
H. Peter Anvin (Intel) (6):
x86/cpufeature: add cpu feature bit for LKGS
x86/opcode: add LKGS instruction to x86-opcode-map x86/gsseg: make asm_load_gs_index() take an u16 x86/gsseg: move local_irq_save/restore() into asm_load_gs_index() x86/gsseg: move load_gs_index() to its own header file x86/gsseg: use the LKGS instruction if available for load_gs_index()
arch/x86/entry/entry_64.S | 28 +++++++++--- arch/x86/ia32/ia32_signal.c | 1 + arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/gsseg.h | 58 ++++++++++++++++++++++++ arch/x86/include/asm/mmu_context.h | 1 + arch/x86/include/asm/special_insns.h | 21 --------- arch/x86/kernel/paravirt.c | 1 + arch/x86/kernel/tls.c | 1 + arch/x86/lib/x86-opcode-map.txt | 1 + tools/arch/x86/include/asm/cpufeatures.h | 1 + tools/arch/x86/lib/x86-opcode-map.txt | 1 + 11 files changed, 88 insertions(+), 27 deletions(-) create mode 100644 arch/x86/include/asm/gsseg.h
Signature · 1 line
2.34.1
PATCH 1/6 x86/cpufeature: add cpu feature bit for LKGS
From: "H. Peter Anvin (Intel)" <[email protected]>
Add the CPU feature bit for LKGS (Load "Kernel" GS).
LKGS instruction is introduced with Intel FRED (flexible return and event delivery) specificaton
https://cdrdv2.intel.com/v1/dl/getContent/678938.
LKGS behaves like the MOV to GS instruction except that it loads the base address into the IA32_KERNEL_GS_BASE MSR instead of the GS segment’s descriptor cache, which is exactly what Linux kernel does to load a user level GS base. Thus, with LKGS, there is no need to SWAPGS away from the kernel GS base.
--- arch/x86/include/asm/cpufeatures.h | 1 + tools/arch/x86/include/asm/cpufeatures.h | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -308,6 +308,7 @@/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5)/* AVX512 BFLOAT16 instructions */ +#define X86_FEATURE_LKGS (12*32+ 18)/* Load "kernel" (userspace) gs */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0)/* CLZERO instruction */
tools/arch/x86/include/asm/cpufeatures.h +1 -0
@@ -308,6 +308,7 @@/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5)/* AVX512 BFLOAT16 instructions */ +#define X86_FEATURE_LKGS (12*32+ 18)/* Load "kernel" (userspace) gs */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0)/* CLZERO instruction */
Signature · 1 line
2.34.1
PATCH 2/6 x86/opcode: add LKGS instruction to x86-opcode-map
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add the instruction opcode used by LKGS.
Opcode number is per public FRED draft spec v3.0
https://cdrdv2.intel.com/v1/dl/getContent/678938.
--- arch/x86/lib/x86-opcode-map.txt | 1 + tools/arch/x86/lib/x86-opcode-map.txt | 1 + 2 files changed, 2 insertions(+)
arch/x86/lib/x86-opcode-map.txt +1 -0
@@ -1047,6 +1047,7 @@ GrpTable: Grp6 3: LTR Ew 4: VERR Ew 5: VERW Ew +6: LKGS Ew (F2) EndTable GrpTable: Grp7
tools/arch/x86/lib/x86-opcode-map.txt +1 -0
@@ -1047,6 +1047,7 @@ GrpTable: Grp6 3: LTR Ew 4: VERR Ew 5: VERW Ew +6: LKGS Ew (F2) EndTable GrpTable: Grp7
Signature · 1 line
2.34.1
PATCH 3/6 x86/gsseg: make asm_load_gs_index() take an u16
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Let gcc know that only the low 16 bits of load_gs_index() argument actually matter. It might allow it to create slightly better code. However, do not propagate this into the prototypes of functions that end up being paravirtualized, to avoid unnecessary changes.
--- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/special_insns.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +1 -1
@@ -779,7 +779,7 @@ _ASM_NOKPROBE(common_interrupt_return) /* * Reload gs selector with exception handling - * edi: new selector + * di: new selector * * Is in entry.text as it shouldn't be instrumented. */
arch/x86/include/asm/special_insns.h +1 -1
@@ -120,7 +120,7 @@ static inline void native_wbinvd(void) asmvolatile ("wbinvd" : : :"memory" ); } -extern asmlinkage void asm_load_gs_index(unsigned int selector); +extern asmlinkage void asm_load_gs_index(u16 selector);static inline void native_load_gs_index (unsigned int selector) {
Signature · 1 line
2.34.1
PATCH 4/6 x86/gsseg: move local_irq_save/restore() into asm_load_gs_index()
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
The need to disable/enable interrupts around asm_load_gs_index() is a consequence of the implementation. Prepare for using the LKGS instruction, which is locally atomic and therefore doesn't need interrupts disabled.
--- arch/x86/entry/entry_64.S | 26 +++++++++++++++++++++----- arch/x86/include/asm/special_insns.h | 4 ---- 2 files changed, 21 insertions(+), 9 deletions(-)
arch/x86/entry/entry_64.S +21 -5
@@ -778,19 +778,35 @@ SYM_CODE_END(common_interrupt_return) _ASM_NOKPROBE(common_interrupt_return) /* - * Reload gs selector with exception handling + * Reload gs selector with exception handling. This is used only on + * native, so using swapgs, pushf, popf, cli, sti, ... directly is fine. + * * di: new selector + * rax: scratch register * * Is in entry.text as it shouldn't be instrumented. + * + * Note: popf is slow, so use pushf to read IF and then execute cli/sti + * if necessary. */ SYM_FUNC_START(asm_load_gs_index) FRAME_BEGIN + pushf + pop %rax + andl $X86_EFLAGS_IF, %eax /* Interrupts enabled? */ + jz 1f + cli +1: swapgs .Lgs_change: ANNOTATE_NOENDBR // error_entry movl %edi, %gs 2: ALTERNATIVE "", "mfence", X86_BUG_SWAPGS_FENCE swapgs + testl %eax, %eax + jz 3f + sti +3: FRAME_END RET
@@ -799,12 +815,12 @@ SYM_FUNC_START(asm_load_gs_index) swapgs /* switch back to user gs */ .macro ZAP_GS /* This can't be a string because the preprocessor needs to see it. */ - movl $__USER_DS, %eax - movl %eax, %gs + movl $__USER_DS, %edi + movl %edi, %gs .endm ALTERNATIVE "", "ZAP_GS", X86_BUG_NULL_SEG - xorl %eax, %eax - movl %eax, %gs + xorl %edi, %edi + movl %edi, %gs jmp 2b _ASM_EXTABLE(.Lgs_change, .Lbad_gs)
arch/x86/include/asm/special_insns.h +0 -4
@@ -124,11 +124,7 @@ extern asmlinkage void asm_load_gs_index(u16 selector);static inline void native_load_gs_index (unsigned int selector) { -unsigned long flags; - -local_irq_save (flags);asm_load_gs_index (selector); -local_irq_restore (flags); }static inline unsigned long __read_cr4 (void )
Signature · 1 line
2.34.1
Re: [PATCH 4/6] x86/gsseg: move local_irq_save/restore() into asm_load_gs_index()
arch/x86/entry/entry_64.S SYM_CODE_END(common_interrupt_return) · line 799
Quoted 8 lines · Xin Li
SYM_FUNC_START(asm_load_gs_index)arch/x86/entry/entry_64.S +6
-0FRAME_BEGIN + pushf + pop %rax + andl $X86_EFLAGS_IF, %eax /* Interrupts enabled? */ + jz 1f + cli +1:
Why the pop,andl,jz ? AFAICT our arch_local_irq_save() doesn't even bother with that, why does this function.
H. Peter Anvin
reviewerRe: [PATCH 4/6] x86/gsseg: move local_irq_save/restore() into asm_load_gs_index()
Quoted 12 lines · Peter Zijlstra
arch/x86/entry/entry_64.S SYM_CODE_END(common_interrupt_return) · line 799
Quoted 8 lines · Xin Li
SYM_FUNC_START(asm_load_gs_index)arch/x86/entry/entry_64.S +6
-0FRAME_BEGIN + pushf + pop %rax + andl $X86_EFLAGS_IF, %eax /* Interrupts enabled? */ + jz 1f + cli +1:Why the pop,andl,jz ? AFAICT our
arch_local_irq_save()doesn't even bother with that, why does this function.
They pop and and are needed for the sti anyway, and so might as well be leveraged here, too.
PATCH 5/6 x86/gsseg: move load_gs_index() to its own header file
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
<asm/cpufeature.h> depends on <asm/special_insns.h>, so in order to be able to use alternatives in native_load_gs_index(), factor it out into a separate header file.
--- arch/x86/ia32/ia32_signal.c | 1 + arch/x86/include/asm/gsseg.h | 32 ++++++++++++++++++++++++++++ arch/x86/include/asm/mmu_context.h | 1 + arch/x86/include/asm/special_insns.h | 17 --------------- arch/x86/kernel/paravirt.c | 1 + arch/x86/kernel/tls.c | 1 + 6 files changed, 36 insertions(+), 17 deletions(-) create mode 100644 arch/x86/include/asm/gsseg.h
arch/x86/ia32/ia32_signal.c +1 -0
@@ -34,6 +34,7 @@#include <asm/sigframe.h> #include <asm/sighandling.h> #include <asm/smap.h> +#include <asm/gsseg.h> static inline void reload_segments (struct sigcontext_32 *sc) {
arch/x86/include/asm/gsseg.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _ASM_X86_GSSEG_H +#define _ASM_X86_GSSEG_H + +#include <linux/types.h> +#include <asm/processor.h> + +#ifdef CONFIG_X86_64 + +extern asmlinkage void asm_load_gs_index(u16 selector); + +static inline void native_load_gs_index (unsigned int selector) +{ +asm_load_gs_index (selector); +} + +#endif /* CONFIG_X86_64 */ + +#ifndef CONFIG_PARAVIRT_XXL + +static inline void load_gs_index (unsigned int selector) +{ +#ifdef CONFIG_X86_64 +native_load_gs_index (selector); +#else +loadsegment (gs, selector); +#endif +} + +#endif /* CONFIG_PARAVIRT_XXL */ + +#endif /* _ASM_X86_GSSEG_H */
arch/x86/include/asm/mmu_context.h +1 -0
@@ -12,6 +12,7 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> #include <asm/debugreg.h> +#include <asm/gsseg.h> extern atomic64_t last_mm_ctx_id;
arch/x86/include/asm/special_insns.h +0 -17
@@ -120,13 +120,6 @@ static inline void native_wbinvd(void) asmvolatile ("wbinvd" : : :"memory" ); } -extern asmlinkage void asm_load_gs_index(u16 selector); - -static inline void native_load_gs_index (unsigned int selector) -{ -asm_load_gs_index (selector); -} -static inline unsigned long __read_cr4 (void ) {return native_read_cr4 ();
@@ -180,16 +173,6 @@ static inline void wbinvd(void) native_wbinvd(); } - -static inline void load_gs_index (unsigned int selector) -{ -#ifdef CONFIG_X86_64 -native_load_gs_index (selector); -#else -loadsegment (gs, selector); -#endif -} -#endif /* CONFIG_PARAVIRT_XXL */ static inline void clflush (volatile void *__p)
arch/x86/kernel/paravirt.c +1 -0
@@ -32,6 +32,7 @@#include <asm/special_insns.h> #include <asm/tlb.h> #include <asm/io_bitmap.h> +#include <asm/gsseg.h> /* * nop stub, which must not clobber anything *including the stack* to
arch/x86/kernel/tls.c +1 -0
@@ -12,6 +12,7 @@#include <asm/ldt.h> #include <asm/processor.h> #include <asm/proto.h> +#include <asm/gsseg.h> #include "tls.h"
Signature · 1 line
2.34.1
Re: [PATCH 5/6] x86/gsseg: move load_gs_index() to its own header file
Quoted 9 lines · Xin Li
From: "H. Peter Anvin (Intel)" <[email protected]>
<
asm/cpufeature.h> depends on <asm/special_insns.h>, so in order to be able to use alternatives innative_load_gs_index(), factor it out into a separate header file.Signed-off-by H. Peter Anvin (Intel) <[email protected]>Signed-off-by Xin Li <[email protected]>
This could be moved into <asm/segment.h> instead of creating a new header file.
--
Brian Gerst
H. Peter Anvin
reviewerRe: [PATCH 5/6] x86/gsseg: move load_gs_index() to its own header file
Quoted 8 lines · Brian Gerst
Quoted 7 lines · Xin Li
From: "H. Peter Anvin (Intel)" <[email protected]>
<
asm/cpufeature.h> depends on <asm/special_insns.h>, so in order to be able to use alternatives innative_load_gs_index(), factor it out into a separate header file.
Quoted 7 lines
Quoted 2 lines
Signed-off-by H. Peter Anvin (Intel) <[email protected]>Signed-off-by Xin Li <[email protected]>This could be moved into <
asm/segment.h> instead of creating a new header file.--
Brian Gerst
At least at the time I wrote the code, it could not, without creating yet another circular header file dependency.
RE: [PATCH 5/6] x86/gsseg: move load_gs_index() to its own header file
Quoted 9 lines · Brian Gerst
Quoted 8 lines · Xin Li
Quoted 7 lines
From: "H. Peter Anvin (Intel)" <[email protected]>
<
asm/cpufeature.h> depends on <asm/special_insns.h>, so in order to be able to use alternatives innative_load_gs_index(), factor it out into a separate header file.
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by H. Peter Anvin (Intel) <[email protected]>Signed-off-by Xin Li <[email protected]>This could be moved into <
asm/segment.h> instead of creating a new headerfile.
Good suggestion. However I still prefer to keep GS segment in its own header file,
1) it's a special segment for x86_64 kernel, and it's more readable to keep it separated.
2) segment.h is included in too many files, but gsseg.h only 4 files. We avoid header pollution.
Maybe we should even factor GS segment out of segment.h for x86_64.
Xin
Quoted 6 lines
Quoted 3 lines
--
Brian GerstAt least at the time I wrote the code, it could not, without creating yet another circular header file dependency.
PATCH 6/6 x86/gsseg: use the LKGS instruction if available for load_gs_index()
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
The LKGS instruction atomically loads a segment descriptor into the %gs descriptor registers, *except* that %gs.base is unchanged, and the base is instead loaded into MSR_IA32_KERNEL_GS_BASE, which is exactly what we want this function to do.
--- arch/x86/include/asm/gsseg.h | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-)
arch/x86/include/asm/gsseg.h +27 -1
@@ -3,15 +3,41 @@#define _ASM_X86_GSSEG_H#include <linux/types.h> + +#include <asm/asm.h> +#include <asm/cpufeature.h> +#include <asm/alternative.h> #include <asm/processor.h> +#include <asm/nops.h> #ifdef CONFIG_X86_64 extern asmlinkage void asm_load_gs_index(u16 selector); +#define LKGS_DI _ASM_BYTES(0xf2,0x0f,0x00,0xf7) +static inline void native_load_gs_index (unsigned int selector) { -asm_load_gs_index (selector); +u16 sel = selector; + +/* +* Note: the fixup is used for the LKGS instruction, but +* it needs to be attached to the primary instruction sequence +* as it isn't something that gets patched. +* +* %rax is provided to the assembly routine as a scratch +* register. +*/ +alternative_io ("1: call asm_load_gs_index\n" +".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }#endif /* CONFIG_X86_64 */
Signature · 1 line
2.34.1
Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 13 lines · Xin Li
static inline void
native_load_gs_index(unsigned int selector){arch/x86/include/asm/gsseg.h +11
-0+u16 sel = selector; + +/* +* Note: the fixup is used for the LKGS instruction, but +* it needs to be attached to the primary instruction sequence +* as it isn't something that gets patched. +* +* %rax is provided to the assembly routine as a scratch +* register. +*/ + alternative_io("1: call asm_load_gs_index\n"
arch/x86/include/asm/gsseg.h line 41
Quoted 10 lines
arch/x86/include/asm/gsseg.h +9
-0+".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }
I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
e5eefda5aa51 ("x86: Remove .fixup section")
H. Peter Anvin
reviewerRe: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 29 lines · Peter Zijlstra
Quoted 13 lines · Xin Li
static inline void
native_load_gs_index(unsigned int selector){arch/x86/include/asm/gsseg.h +11
-0+u16 sel = selector; + +/* +* Note: the fixup is used for the LKGS instruction, but +* it needs to be attached to the primary instruction sequence +* as it isn't something that gets patched. +* +* %rax is provided to the assembly routine as a scratch +* register. +*/ + alternative_io("1: call asm_load_gs_index\n"arch/x86/include/asm/gsseg.h line 41
Quoted 10 lines
arch/x86/include/asm/gsseg.h +9
-0+".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
Quoted 1 line
e5eefda5aa51("x86: Remove .fixup section")
Xin, what did you use as the forward-porting baseline?
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 15 lines
Quoted 15 lines
arch/x86/include/asm/gsseg.h line 41
Quoted 11 lines
arch/x86/include/asm/gsseg.h +10
-0+alternative_io ("1: call asm_load_gs_index\n" +".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
Quoted 3 lines
Quoted 1 line
e5eefda5aa51("x86: Remove .fixup section")Xin, what did you use as the forward-porting baseline?
6.0 release, and my kernel dump shows me a fixup section is there, and a fixup section is created anyway if we do "pushsection "\.fixup\"".
H. Peter Anvin
reviewerRE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 15 lines · On October 7, 2022 11:07:34 AM PDT, "Li, Xin3" <[email protected]> wrote:
Quoted 15 lines
Quoted 15 lines
arch/x86/include/asm/gsseg.h line 41
Quoted 11 lines
arch/x86/include/asm/gsseg.h +10
-0+alternative_io ("1: call asm_load_gs_index\n" +".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
e5eefda5aa51("x86: Remove .fixup section")Xin, what did you use as the forward-porting baseline?
6.0 release, and my kernel dump shows me a fixup section is there, and a fixup section is created anyway if we do "pushsection "\.fixup\"".
Yeah. .fixup is really Just Another Text Section ™, so it is probably not surprising if it has crept back in.
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
In reply to Peter ZijlstraQuoted 6 lines
Quoted 6 lines
arch/x86/include/asm/gsseg.h +6
-0+ alternative_io("1: call asm_load_gs_index\n" + ".pushsection \".fixup\",\"ax\"\n" + "2: xorl %k[sel], %k[sel]\n" + " jmp 1b\n" + ".popsection\n" + _ASM_EXTABLE(1b, 2b),
Quoted 8 lines
arch/x86/include/asm/gsseg.h line 41
Quoted 5 lines
arch/x86/include/asm/gsseg.h +4
-0+ _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
Weird, did you ever check a kernel dump?
Quoted 2 lines
e5eefda5aa51("x86: Remove .fixup section")
Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 15 lines · On Fri, Oct 07, 2022 at 06:01:06PM +0000, Li, Xin3 wrote:
Quoted 6 lines
Quoted 6 lines
arch/x86/include/asm/gsseg.h +6
-0+ alternative_io("1: call asm_load_gs_index\n" + ".pushsection \".fixup\",\"ax\"\n" + "2: xorl %k[sel], %k[sel]\n" + " jmp 1b\n" + ".popsection\n" + _ASM_EXTABLE(1b, 2b),Quoted 8 lines
arch/x86/include/asm/gsseg.h line 41
Quoted 5 lines
arch/x86/include/asm/gsseg.h +4
-0+ _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
Quoted 1 line
Weird, did you ever check a kernel dump?
$ readelf -WS defconfig-build/vmlinux | grep fixup [ 5] .pci_fixup PROGBITS ffffffff826a5350 18a5350 003570 00 A 0 0 16 [ 6] .rela.pci_fixup RELA 0000000000000000 360c388 005028 18 I 60 5 8
In fact, when I add one I get:
ld: warning: orphan section .fixup' from arch/x86/kernel/traps.o' being placed in section `.fixup'
H. Peter Anvin
reviewerRe: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 18 lines · Peter Zijlstra
Quoted 15 lines · On Fri, Oct 07, 2022 at 06:01:06PM +0000, Li, Xin3 wrote:
Quoted 6 lines
Quoted 6 lines
arch/x86/include/asm/gsseg.h +6
-0+ alternative_io("1: call asm_load_gs_index\n" + ".pushsection \".fixup\",\"ax\"\n" + "2: xorl %k[sel], %k[sel]\n" + " jmp 1b\n" + ".popsection\n" + _ASM_EXTABLE(1b, 2b),Quoted 8 lines
arch/x86/include/asm/gsseg.h line 41
Quoted 5 lines
arch/x86/include/asm/gsseg.h +4
-0+ _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
Quoted 1 line
Weird, did you ever check a kernel dump?
Quoted 7 lines
$ readelf -WS defconfig-build/vmlinux | grep fixup [ 5] .pci_fixup PROGBITS ffffffff826a5350 18a5350 003570 00 A 0 0 16 [ 6] .rela.pci_fixup RELA 0000000000000000 360c388 005028 18 I 60 5 8In fact, when I add one I get:
ld: warning: orphan section
.fixup' fromarch/x86/kernel/traps.o' being placed in section `.fixup'
Perhaps the two of you need to compare confugurations?
Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 23 lines · H. Peter Anvin
Quoted 22 lines · Peter Zijlstra
Quoted 15 lines · On Fri, Oct 07, 2022 at 06:01:06PM +0000, Li, Xin3 wrote:
Quoted 6 lines
Quoted 6 lines
arch/x86/include/asm/gsseg.h +6
-0+ alternative_io("1: call asm_load_gs_index\n" + ".pushsection \".fixup\",\"ax\"\n" + "2: xorl %k[sel], %k[sel]\n" + " jmp 1b\n" + ".popsection\n" + _ASM_EXTABLE(1b, 2b),Quoted 8 lines
arch/x86/include/asm/gsseg.h line 41
Quoted 5 lines
arch/x86/include/asm/gsseg.h +4
-0+ _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }I'm very sure none of this was tested... the .fixup section hasn't existed for almost a year now.
Quoted 1 line
Weird, did you ever check a kernel dump?
$ readelf -WS defconfig-build/vmlinux | grep fixup [ 5] .pci_fixup PROGBITS ffffffff826a5350 18a5350 003570 00 A 0 0 16 [ 6] .rela.pci_fixup RELA 0000000000000000 360c388 005028 18 I 60 5 8
Quoted 5 lines
Quoted 3 lines
In fact, when I add one I get:
ld: warning: orphan section
.fixup' fromarch/x86/kernel/traps.o' being placed in section `.fixup'Perhaps the two of you need to compare confugurations?
Whatever for? I know the robots report this warning because there was one from the KVM cross-merge when the .fixup removal went in. It got reported and fixed and that was the last of it.
Anyway; try:
$ git grep "\.fixup" arch/x86/
there isn't a single usage.
Andrew Cooper suggested upgrading the orphan section warning to a hard link error, orphan sections are bad regardless.
H. Peter Anvin
reviewerRe: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines
Perhaps the two of you need to compare confugurations?
Whatever for? I know the robots report this warning because there was one from the KVM cross-merge when the .fixup removal went in. It got reported and fixed and that was the last of it.
To help track down what appears to be a problem?
Quoted 9 lines
Anyway; try:
$ git grep "\.fixup" arch/x86/there isn't a single usage.
Andrew Cooper suggested upgrading the orphan section warning to a hard link error, orphan sections are bad regardless.
Agreed 1000%. This is a no-brainer. From IRC:
<andyhhp> -LDFLAGS_vmlinux += --orphan-handling=warn <andyhhp> +LDFLAGS_vmlinux += --orphan-handling=error
-hpa
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 9 lines
Quoted 3 lines
Andrew Cooper suggested upgrading the orphan section warning to a hard link error, orphan sections are bad regardless.
Agreed 1000%. This is a no-brainer. From IRC:
<andyhhp> -LDFLAGS_vmlinux += --orphan-handling=warn <andyhhp> +LDFLAGS_vmlinux += --orphan-handling=error
Who is going to make the change?
H. Peter Anvin
reviewerRE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 4 lines · On October 7, 2022 10:32:31 PM PDT, "Li, Xin3" <[email protected]> wrote:
Quoted 4 lines
Quoted 3 lines
Andrew Cooper suggested upgrading the orphan section warning to a hard link error, orphan sections are bad regardless.
Quoted 8 lines
Quoted 5 lines
Agreed 1000%. This is a no-brainer. From IRC:
<andyhhp> -LDFLAGS_vmlinux += --orphan-handling=warn <andyhhp> +LDFLAGS_vmlinux += --orphan-handling=error
Who is going to make the change?
Why don't you?
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 8 lines · On October 7, 2022 10:32:31 PM PDT, "Li, Xin3" <[email protected]> wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Andrew Cooper suggested upgrading the orphan section warning to a hard link error, orphan sections are bad regardless.
Agreed 1000%. This is a no-brainer. From IRC:
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
<andyhhp> -LDFLAGS_vmlinux += --orphan-handling=warn <andyhhp> +LDFLAGS_vmlinux += --orphan-handling=error
Who is going to make the change?
Why don't you?
Will do.
Xin
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
In reply to H. Peter AnvinQuoted 9 lines
Quoted 3 lines
Andrew Cooper suggested upgrading the orphan section warning to a hard link error, orphan sections are bad regardless.
Agreed 1000%. This is a no-brainer. From IRC:
<andyhhp> -LDFLAGS_vmlinux += --orphan-handling=warn <andyhhp> +LDFLAGS_vmlinux += --orphan-handling=error
There is an arch independent config CONFIG_LD_ORPHAN_WARN, which forces linker to warn on implicit named sections, or there is even no warning.
CONFIG_LD_ORPHAN_WARN depends on ARCH_WANT_LD_ORPHAN_WARN, and some archs (arm/arm64/mips/x86/...) have it defined, and then ld generates warnings on orphan sections.
Should we promote warning to error only on x86?
Quoted 2 lines
-hpa
Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
In reply to Xin LiQuoted 1 line · Xin Li
+#define
LKGS_DI_ASM_BYTES(0xf2,0x0f,0x00,0xf7)
This should come with a comment that states the binutils version that
understands: "lkgs %di" (or however that ends being).
Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
In reply to Xin LiQuoted 13 lines · Xin Li
static inline void
native_load_gs_index(unsigned int selector){arch/x86/include/asm/gsseg.h +11
-0+u16 sel = selector; + +/* +* Note: the fixup is used for the LKGS instruction, but +* it needs to be attached to the primary instruction sequence +* as it isn't something that gets patched. +* +* %rax is provided to the assembly routine as a scratch +* register. +*/ + alternative_io("1: call asm_load_gs_index\n"
arch/x86/include/asm/gsseg.h line 41
Quoted 10 lines
arch/x86/include/asm/gsseg.h +9
-0+".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }
Something like so perhaps?
asm_inline volatile("1:\n"
ALTERNATIVE("call asm_load_gs_index\n",
LKGS_DI,
X86_FEATURE_LKGS)
_ASM_EXTABLE_TYPE_REG(1b, 1b, EX_TYPE_ZERO_REG, %k[sel])
: ASM_CALL_CONSTRAINT
: [sel] "D" (sel)
: "memory", _ASM_AX);(completely untested, not even seen a compiler upclose)
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 13 lines
Something like so perhaps?
asm_inline volatile("1:\n" ALTERNATIVE("call asm_load_gs_index\n", LKGS_DI, X86_FEATURE_LKGS) _ASM_EXTABLE_TYPE_REG(1b, 1b, EX_TYPE_ZERO_REG, %k[sel]) : ASM_CALL_CONSTRAINT : [sel] "D" (sel) : "memory", _ASM_AX);
Quoted 1 line
(completely untested, not even seen a compiler upclose)
It compiles (after adding "0x3e" to make it 5 bytes to match the size of "call asm_load_gs_index") and passes my fixup test, thanks!
Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
In reply to Xin LiQuoted 62 lines · Xin Li
From: "H. Peter Anvin (Intel)" <[email protected]>
The LKGS instruction atomically loads a segment descriptor into the %gs descriptor registers, *except* that %gs.base is unchanged, and the base is instead loaded into
MSR_IA32_KERNEL_GS_BASE, which is exactly what we want this function to do.Signed-off-by H. Peter Anvin (Intel) <[email protected]>Signed-off-by Xin Li <[email protected]>--- arch/x86/include/asm/gsseg.h | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-)arch/x86/include/asm/gsseg.h +27
-2@@ -3,15 +3,41 @@#define _ASM_X86_GSSEG_H#include <linux/types.h> + +#include <asm/asm.h> +#include <asm/cpufeature.h> +#include <asm/alternative.h> #include <asm/processor.h> +#include <asm/nops.h> #ifdef CONFIG_X86_64 extern asmlinkage void asm_load_gs_index(u16 selector); +#define LKGS_DI _ASM_BYTES(0xf2,0x0f,0x00,0xf7) +static inline void native_load_gs_index (unsigned int selector) { -asm_load_gs_index (selector); +u16 sel = selector; + +/* +* Note: the fixup is used for the LKGS instruction, but +* it needs to be attached to the primary instruction sequence +* as it isn't something that gets patched. +* +* %rax is provided to the assembly routine as a scratch +* register. +*/ +alternative_io ("1: call asm_load_gs_index\n" +".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }#endif /* CONFIG_X86_64 */ --
Quoted 1 line
2.34.1
There are not that many call sites, so using something like this (incorporating Peter Z's suggestion for the exception handler) would be better from a code readability perspective vs. a tiny increase in code size.
if (static_cpu_has(X86_FEATURE_LKGS))
asm volatile("1: " LKGS_DI
_ASM_EXTABLE_TYPE_REG(1b, 1b,
EX_TYPE_ZERO_REG, %k[sel])
: [sel] "+D" (sel) : : "memory");
else
asm_load_gs_index(sel);--
Brian Gerst
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 12 lines
Quoted 12 lines
arch/x86/include/asm/gsseg.h +10
-0+alternative_io ("1: call asm_load_gs_index\n" +".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }
Quoted 7 lines
Quoted 3 lines
#endif /*
CONFIG_X86_64*/
--
2.34.1There are not that many call sites, so using something like this (incorporating Peter Z's suggestion for the exception handler) would be better from a code readability perspective vs. a tiny increase in code size.
The existing approach patches the binary code thus we don't need to check it at runtime.
Quoted 8 lines
if (static_cpu_has(X86_FEATURE_LKGS))
asm volatile("1: " LKGS_DI
_ASM_EXTABLE_TYPE_REG(1b, 1b, EX_TYPE_ZERO_REG,
%k[sel])
: [sel] "+D" (sel) : : "memory");
else
asm_load_gs_index(sel);Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 17 lines · On Sat, Oct 8, 2022 at 1:40 AM Li, Xin3 <[email protected]> wrote:
Quoted 17 lines
Quoted 16 lines
arch/x86/include/asm/gsseg.h +10
-1+alternative_io ("1: call asm_load_gs_index\n" +".pushsection \".fixup\",\"ax\"\n" +"2: xorl %k[sel], %k[sel]\n" +" jmp 1b\n" +".popsection\n" + _ASM_EXTABLE(1b ,2b ), + _ASM_BYTES(0x3e) LKGS_DI, +X86_FEATURE_LKGS , + ASM_OUTPUT2([sel] "+D" (sel), ASM_CALL_CONSTRAINT), + ASM_NO_INPUT_CLOBBER(_ASM_AX)); }#endif /* CONFIG_X86_64 */ --2.34.1
Quoted 5 lines
Quoted 3 lines
There are not that many call sites, so using something like this (incorporating Peter Z's suggestion for the exception handler) would be better from a code readability perspective vs. a tiny increase in code size.
The existing approach patches the binary code thus we don't need to check it at runtime.
static_cpu_has() uses alternatives to patch the branch, so there is no runtime check after early boot.
--
Brian Gerst
RE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
There are not that many call sites, so using something like this (incorporating Peter Z's suggestion for the exception handler) would be better from a code readability perspective vs. a tiny increase in code size.
The existing approach patches the binary code thus we don't need to check it
at runtime.
static_cpu_has()uses alternatives to patch the branch, so there is no runtime check after early boot.
Sorry, didn't know it, thanks for point it out.
If we prefer static_cpu_has, are you asking to replace all alternative macros with it?
Xin
H. Peter Anvin
reviewerRE: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
Quoted 11 lines · On October 9, 2022 9:32:34 PM PDT, "Li, Xin3" <[email protected]> wrote:
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
There are not that many call sites, so using something like this (incorporating Peter Z's suggestion for the exception handler) would be better from a code readability perspective vs. a tiny increase in code size.
The existing approach patches the binary code thus we don't need to check it
at runtime.
static_cpu_has()uses alternatives to patch the branch, so there is no runtime check after early boot.
Quoted 7 lines
Sorry, didn't know it, thanks for point it out.
If we prefer
static_cpu_has, are you asking to replace all alternative macros with it?Xin
Honestly, it seems to me to be more than a bit excessive. The code might be nontrivial, but *with proper commenting* it should be perfectly understandable...
Re: [PATCH 6/6] x86/gsseg: use the LKGS instruction if available for load_gs_index()
In reply to Li, Xin3Quoted 7 lines · On Mon, Oct 10, 2022 at 04:32:34AM +0000, Li, Xin3 wrote:
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
There are not that many call sites, so using something like this (incorporating Peter Z's suggestion for the exception handler) would be better from a code readability perspective vs. a tiny increase in code size.
The existing approach patches the binary code thus we don't need to check it
at runtime.
Quoted 7 lines
Quoted 3 lines
static_cpu_has()uses alternatives to patch the branch, so there is no runtime check after early boot.Sorry, didn't know it, thanks for point it out.
If we prefer
static_cpu_has, are you asking to replace all alternative macros with it?
No; the only reason to do it here would be to unconfuse that exception thing. But even there I'm not convinced.
Yes, Brian's code is much easier to read, but code-gen is quite terrible (also, my binutils can't seem to decode this -- GNU objdump (GNU Binutils for Debian) 2.38.90.20220713)