percpu: Add a DEFINE_PER_CPU_2PAGE_ALIGNED
Subsystems (4)
PATCH 1/8 percpu: Add a DEFINE_PER_CPU_2PAGE_ALIGNED
From: Andi Kleen <[email protected]>
Needed in a followon patch which needs to naturally align a 8K stack. I just put it into the page aligned section. This may cause an extra 4K hole if we're unlucky.
--- include/linux/percpu-defs.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/percpu-defs.h +4 -0
@@ -163,6 +163,10 @@ DEFINE_PER_CPU_SECTION(type, name, "..page_aligned") \ __aligned(PAGE_SIZE) +#define DEFINE_PER_CPU_2PAGE_ALIGNED (type, name) \ + DEFINE_PER_CPU_SECTION(type, name, "..page_aligned") \ + __aligned(2*PAGE_SIZE) +/* * Declaration/definition used for per-CPU variables that must be read mostly. */
Signature · 1 line
1.9.3
PATCH 2/8 x86: Naturally align the debug IST stack
From: Andi Kleen <[email protected]>
The followon patch for RD*BASE requires the IST stacks to be naturally aligned because it uses and to access the bottom of the stack. This was true for everyone except for the debug ist stack. Make the debug ist stack naturally aligned too.
--- arch/x86/include/asm/page_64_types.h | 4 ++-- arch/x86/kernel/cpu/common.c | 8 +++++++- 2 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/include/asm/page_64_types.h +2 -2
@@ -17,8 +17,8 @@#define STACKFAULT_STACK 1#define DOUBLEFAULT_STACK 2#define NMI_STACK 3 -#define DEBUG_STACK 4 -#define MCE_STACK 5 +#define MCE_STACK 4 +#define DEBUG_STACK 5/* Must be always last */ #define N_EXCEPTION_STACKS 5/* hw limit: 7 */ #define PUD_PAGE_SIZE (_AC(1, UL) << PUD_SHIFT)
arch/x86/kernel/cpu/common.c +7 -1
@@ -1164,8 +1164,12 @@ static const unsigned int exception_stack_sizes[N_EXCEPTION_STACKS] = { [DEBUG_STACK -1 ] =DEBUG_STKSZ }; +/* The IST stacks must be naturally aligned */ + static DEFINE_PER_CPU_PAGE_ALIGNED(char, exception_stacks - [(N_EXCEPTION_STACKS - 1) * EXCEPTION_STKSZ + DEBUG_STKSZ]); + [(N_EXCEPTION_STACKS - 1) * EXCEPTION_STKSZ]); +static DEFINE_PER_CPU_2PAGE_ALIGNED(char, debug_stack + [DEBUG_STKSZ]);/* May not be marked __init: used by software suspend */ void syscall_init (void )
@@ -1349,6 +1353,8 @@ void cpu_init(void)char *estacks =per_cpu (exception_stacks, cpu);for (v =0 ; v <N_EXCEPTION_STACKS ; v++) { +if (v ==DEBUG_STACK -1 ) + estacks =per_cpu (debug_stack, cpu); estacks += exception_stack_sizes[v]; oist->ist[v] = t->x86_tss.ist[v] = (unsigned long )estacks;
Signature · 1 line
1.9.3
PATCH 3/8 x86: Add intrinsics/macros for new rd/wr fs/gs base instructions
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Add C intrinsics and assembler macros for the new rd/wr fs/gs base instructions and for swapgs.
Very straight forward. Used in followon patch.
For assembler only a few standard registers used by entry_64.S are defined.
--- arch/x86/include/asm/fsgs.h | 54 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 arch/x86/include/asm/fsgs.h
arch/x86/include/asm/fsgs.h new file +54 -0
@@ -0,0 +1,54 @@ +#ifndef _ASM_FSGS_H +#define _ASM_FSGS_H 1 + +#ifndef __ASSEMBLY__ + +static inline void swapgs (void ) +{ + asmvolatile ("swapgs" :::"memory" ); +} + +/* Must be protected by X86_FEATURE_FSGSBASE check. */ + +static inline unsigned long rdgsbase (void ) +{ +unsigned long gs; + asmvolatile (".byte 0xf3,0x48,0x0f,0xae,0xc8 # rdgsbaseq %%rax" + :"=a" (gs) + ::"memory" ); +return gs; +} + +static inline unsigned long rdfsbase (void ) +{ +unsigned long fs; + asmvolatile (".byte 0xf3,0x48,0x0f,0xae,0xc0 # rdfsbaseq %%rax" + :"=a" (fs) + ::"memory" ); +return fs; +} + +static inline void wrgsbase (unsigned long gs) +{ + asmvolatile (".byte 0xf3,0x48,0x0f,0xae,0xd8 # wrgsbaseq %%rax" + ::"a" (gs) + :"memory" ); +} + +static inline void wrfsbase (unsigned long fs) +{ + asmvolatile (".byte 0xf3,0x48,0x0f,0xae,0xd0 # wrfsbaseq %%rax" + ::"a" (fs) + :"memory" ); +} + +#else + +/* Handle old assemblers. */ +#define RDGSBASE_R15 .byte 0xf3,0x49,0x0f,0xae,0xcf +#define WRGSBASE_RDI .byte 0xf3,0x48,0x0f,0xae,0xdf +#define WRGSBASE_R15 .byte 0xf3,0x49,0x0f,0xae,0xdf + +#endif /* __ASSEMBLY__ */ + +#endif
Signature · 1 line
1.9.3
PATCH 4/8 x86: Add support for rd/wr fs/gs base
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Introduction:
IvyBridge added four new instructions to directly write the fs and gs 64bit base registers. Previously this had to be done with a system call to write to MSRs. The main use case is fast user space threading and switching the fs/gs registers quickly there. Another use case is having (relatively) cheap access to a new address register per thread.
The instructions are opt-in and have to be explicitely enabled by the OS.
For more details on how to use the instructions see Documentation/x86/fsgs.txt added in a followon patch.
Paranoid exception path changes:
===============================
The paranoid entry/exit code is used for any NMI like exception.
Previously Linux couldn't support the new instructions because the paranoid entry code relied on the gs base never being negative outside the kernel to decide when to use swaps. It would check the gs MSR value and assume it was already running in kernel if negative.
To get rid of this assumption we have to revamp the paranoid exception path to not rely on this. We can use the new instructions to get (relatively) quick access to the GS value, and use it directly.
This is also significantly faster than a MSR read, so will speed NMIs (critical for profiling)
The kernel gs for the paranoid path is now stored at the bottom of the IST stack (so that it can be derived from RSP). For this we need to know the size of the IST stack (4K or 8K), which is now passed in as a stack parameter to save_paranoid.
The original patch compared the gs with the kernel gs and assumed that if it was identical, swapgs was not needed (and no user space processing was needed). This was nice and simple and didn't need a lot of changes.
But this had the side effect that if a user process set its GS to the same as the kernel it may lose rescheduling checks (so a racing reschedule IPI would have been only acted upon the next non paranoid interrupt)
This version now switches to full save/restore of the GS. This requires quite some changes in the paranoid path. Unfortunately I didn't come up with a simpler scheme.
Previously we had a flag in EBX that indicated whether SWAPGS needs to be called later or not. In the new scheme this turns into a tristate, with a new "restore from R15" mode that is used when the FSGSBASE instructions are available. In this case the GS base is saved and restored.
The exit paths are all adjusted to handle this correctly.
There is one complication: to allow debuggers (especially from the int3 or debug vectors) access to the user GS we need to save it in the task struct. Normally the next context switch would overwrite it with the wrong value from kernel_gs, so we set new flag also in task_struct that prevents it. The flag is cleared on context switch.
[Even with the additional flag the new FS/GS context switch is vastly faster than the old MSR based version for bases >4GB]
To prevent recursive interrupts clobbering this state in the task_struct this is only done for interrupts coming directly from ring 3.
After a reschedule comes back we check if the flag is still set. If it wasn't set the GS is back in the (swapped) kernel gs so we revert to the SWAPGS mode, instead of restoring GS.
So the three possible states for the paranoid exit path are:
- Do nothing (pre FSGSBASE), if we interrupted the kernel as determined by the existing negative GS
- Do swapgs, if we interrupted user space with positive GS (pre FSGSBASE), or we saved gs, but it was overwritten later by a context switch (with FSGSBASE)
- Restore from R15 (with FSGSBASE), if the gs base was saved in R15, and not overwritten by a context switch.
Context switch changes:
======================
Then after these changes we need to also use the new instructions to save/restore fs and gs, so that the new values set by the users won't disappear. This is also significantly faster for the case when the 64bit base has to be switched (that is when GS is larger than 4GB), as we can replace the slow MSR write with a faster wr[fg]sbase execution.
The instructions do not context switch
the segment index, so the old invariant that fs or gs index have to be 0 for a different 64bit value to stick is still true. Previously it was enforced by arch_prctl, now the user program has to make sure it keeps the segment indexes zero. If it doesn't the changes may not stick.
[Yes, this implies that programs can find out when they got context switched. However they could already do that before by checking the timing.]
This is in term enables fast switching when there are enough threads that their TLS segment does not fit below 4GB, or alternatively programs that use fs as an additional base register will not get a sigificant context switch penalty.
It is all done in a single patch because there was no simple way to do it in pieces without having crash holes inbetween.
v2: Change to save/restore GS instead of using swapgs based on the value. Large scale changes.
v3: Fix wrong flag initialization in fallback path. Thanks 0day!
--- arch/x86/include/asm/processor.h | 4 + arch/x86/kernel/asm-offsets_64.c | 4 + arch/x86/kernel/cpu/common.c | 6 ++ arch/x86/kernel/entry_64.S | 163 +++++++++++++++++++++++++++++++-------- arch/x86/kernel/process_64.c | 31 +++++++- 5 files changed, 170 insertions(+), 38 deletions(-)
arch/x86/include/asm/processor.h +4 -0
@@ -522,6 +522,10 @@ struct thread_struct {* a short time */ unsigned char fpu_counter; +/* +* Set to one when gs is already saved. +*/ +unsigned char gs_saved; };/*
arch/x86/kernel/asm-offsets_64.c +4 -0
@@ -85,6 +85,10 @@ int main(void)DEFINE (__NR_ia32_syscall_max,sizeof (syscalls_ia32) -1 );DEFINE (IA32_NR_syscalls,sizeof (syscalls_ia32)); +BLANK (); + +OFFSET (task_thread_gs, task_struct, thread.gs); +OFFSET (task_thread_gs_saved, task_struct, thread.gs_saved);return 0 ; }
arch/x86/kernel/cpu/common.c +6 -0
@@ -953,6 +953,9 @@ static void identify_cpu(struct cpuinfo_x86 *c)#ifdef CONFIG_NUMA numa_add_cpu (smp_processor_id ());#endif + +if (cpu_has (c,X86_FEATURE_FSGSBASE )) +set_in_cr4 (X86_CR4_FSGSBASE ); }#ifdef CONFIG_X86_64
@@ -1351,10 +1354,13 @@ void cpu_init(void)*/ if (!oist->ist[0 ]) {char *estacks =per_cpu (exception_stacks, cpu); +void *gs =per_cpu (irq_stack_union.gs_base, cpu);for (v =0 ; v <N_EXCEPTION_STACKS ; v++) {if (v ==DEBUG_STACK -1 ) estacks =per_cpu (debug_stack, cpu); +/* Store GS at bottom of stack for bootstrap access */ + *(void **)estacks = gs; estacks += exception_stack_sizes[v]; oist->ist[v] = t->x86_tss.ist[v] = (unsigned long )estacks;
arch/x86/kernel/entry_64.S +129 -34
268 lines not on this page · raw message
arch/x86/kernel/process_64.c +27 -4
@@ -49,6 +49,7 @@#include <asm/syscalls.h> #include <asm/debugreg.h> #include <asm/switch_to.h> +#include <asm/fsgs.h> asmlinkage extern void ret_from_fork(void);
@@ -308,6 +309,16 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)*/ savesegment (fs, fsindex);savesegment (gs, gsindex); +if (static_cpu_has (X86_FEATURE_FSGSBASE )) { + prev->fs =rdfsbase (); +/* Interrupts are disabled here. */ +if (!prev->gs_saved) { +swapgs (); + prev->gs =rdgsbase (); +swapgs (); + } + prev->gs_saved =0 ; + }load_TLS (next, cpu);
@@ -338,8 +349,12 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) prev->fs =0 ; }/* when next process has a 64bit base use it */ -if (next->fs) -wrmsrl (MSR_FS_BASE , next->fs); +if (next->fs) { +if (static_cpu_has (X86_FEATURE_FSGSBASE )) +wrfsbase (next->fs); +else +wrmsrl (MSR_FS_BASE , next->fs); + } prev->fsindex = fsindex;if (unlikely (gsindex | next->gsindex | prev->gs)) {
@@ -347,8 +362,16 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)if (gsindex) prev->gs =0 ; } -if (next->gs) -wrmsrl (MSR_KERNEL_GS_BASE , next->gs); +if (next->gs) { +if (static_cpu_has (X86_FEATURE_FSGSBASE )) { +/* Interrupts are disabled here. */ +swapgs (); +wrgsbase (next->gs); +swapgs (); + }else { +wrmsrl (MSR_KERNEL_GS_BASE , next->gs); + } + } prev->gsindex = gsindex;switch_fpu_finish (next_p, fpu);
Signature · 1 line
1.9.3
Re: [PATCH 4/8] x86: Add support for rd/wr fs/gs base
Quoted 71 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Introduction:
IvyBridge added four new instructions to directly write the fs and gs 64bit base registers. Previously this had to be done with a system call to write to MSRs. The main use case is fast user space threading and switching the fs/gs registers quickly there. Another use case is having (relatively) cheap access to a new address register per thread.
The instructions are opt-in and have to be explicitely enabled by the OS.
For more details on how to use the instructions see
Documentation/x86/fsgs.txtadded in a followon patch.Paranoid exception path changes:
===============================The paranoid entry/exit code is used for any NMI like exception.
Previously Linux couldn't support the new instructions because the paranoid entry code relied on the gs base never being negative outside the kernel to decide when to use swaps. It would check the gs MSR value and assume it was already running in kernel if negative.
To get rid of this assumption we have to revamp the paranoid exception path to not rely on this. We can use the new instructions to get (relatively) quick access to the GS value, and use it directly.
This is also significantly faster than a MSR read, so will speed NMIs (critical for profiling)
The kernel gs for the paranoid path is now stored at the bottom of the IST stack (so that it can be derived from RSP). For this we need to know the size of the IST stack (4K or 8K), which is now passed in as a stack parameter to
save_paranoid.The original patch compared the gs with the kernel gs and assumed that if it was identical, swapgs was not needed (and no user space processing was needed). This was nice and simple and didn't need a lot of changes.
But this had the side effect that if a user process set its GS to the same as the kernel it may lose rescheduling checks (so a racing reschedule IPI would have been only acted upon the next non paranoid interrupt)
This version now switches to full save/restore of the GS. This requires quite some changes in the paranoid path. Unfortunately I didn't come up with a simpler scheme.
Previously we had a flag in EBX that indicated whether SWAPGS needs to be called later or not. In the new scheme this turns into a tristate, with a new "restore from R15" mode that is used when the FSGSBASE instructions are available. In this case the GS base is saved and restored.
The exit paths are all adjusted to handle this correctly.There is one complication: to allow debuggers (especially from the int3 or debug vectors) access to the user GS we need to save it in the task struct. Normally the next context switch would overwrite it with the wrong value from
kernel_gs, so we set new flag also intask_structthat prevents it. The flag is cleared on context switch.
Quoted 6 lines
[Even with the additional flag the new FS/GS context switch is vastly faster than the old MSR based version for bases >4GB]
To prevent recursive interrupts clobbering this state in the
task_structthis is only done for interrupts coming directly from ring 3.
Since this just came up in a different context today, I'd like to propose a different solution to this piece of the problem.
Can we change the paranoid entry to check if the entry came from ring 3 and to just switch stacks immediately to the standard kernel stack and run the non-paranoid entry code? This eliminates paranoid_userspace entirely, and there are no special gsbase machinations any more for the entry-from-userspace path.
In fact, I think that this will result in the MSR KERNEL_GS_BASE value *always* matching the userspace gs value from any C code in the kernel, since we'll always swapgs exactly once on entry from userspace.
If we make this change, then:
Quoted 9 lines
After a reschedule comes back we check if the flag is still set. If it wasn't set the GS is back in the (swapped) kernel gs so we revert to the SWAPGS mode, instead of restoring GS.
So the three possible states for the paranoid exit path are:
- Do nothing (pre FSGSBASE), if we interrupted the kernel as determined by the existing negative GS
This works the same way as before.
Quoted 3 lines
- Do swapgs, if we interrupted user space with positive GS (pre FSGSBASE), or we saved gs, but it was overwritten later by a context switch (with FSGSBASE)
This can never happen on paranoid return.
Quoted 3 lines
- Restore from R15 (with FSGSBASE), if the gs base was saved in R15, and not overwritten by a context switch.
I think that if we implement my suggestion, then we can't have a context switch here either.
Quoted 17 lines
Context switch changes:
======================Then after these changes we need to also use the new instructions to save/restore fs and gs, so that the new values set by the users won't disappear. This is also significantly faster for the case when the 64bit base has to be switched (that is when GS is larger than 4GB), as we can replace the slow MSR write with a faster wr[fg]sbase execution.
The instructions do not context switch
the segment index, so the old invariant that fs or gs index have to be 0 for a different 64bit value to stick is still true. Previously it was enforced byarch_prctl, now the user program has to make sure it keeps the segment indexes zero. If it doesn't the changes may not stick.
Quoted 3 lines
[Yes, this implies that programs can find out when they got context switched. However they could already do that before by checking the timing.]
I don't like this. Also, starting in 3.19 (if my CR4 patches get merged), then this isn't true in a tight enough seccomp sandbox.
What's wrong with switching the segment indexes separately? In the common case, this will be two reads of segment registers per context switch (to check if the previous segments registers were nonzero), which should be pretty cheap.
Also, it's probably very unsafe to let one task leak an LDT index in fs or gs into another task. This is asking for cross-task segfaults in the hands of malicious users.
Also, at some point hpa suggested that we should start saving and restoring fs, gs, fsbase, and gsbase in sigcontext when we enable this feature. Userspace code might want that.
arch/x86/kernel/cpu/common.c static void identify_cpu(struct cpuinfo_x86 *c) · line 958
Quoted 11 lines
arch/x86/kernel/cpu/common.c +3
-0@@ -953,6 +953,9 @@ static void identify_cpu(struct cpuinfo_x86 *c)#ifdef CONFIG_NUMA numa_add_cpu (smp_processor_id ());#endif + +if (cpu_has (c,X86_FEATURE_FSGSBASE )) +set_in_cr4 (X86_CR4_FSGSBASE );
This will conflict with my cr4 patches (the merge will fail to compile).
We can cross that bridge when we get to it.
Quoted 86 lines
arch/x86/kernel/cpu/common.c +59
-18+/* Values of the ebx flag: */ +#define DO_RESTORE_R15 2/* Restore GS at end */ +#define DO_SWAPGS 1/* Use SWAPGS at end */ +#define DO_NOTHING 0/* Back to ring 0 with same gs */ + +/* +* Stack layout: +* +16 pt_regs +* +8 stack mask for ist or 0 +* +0 return address. +*/ +#define OFF 16 +#define STKMSK 8 +ENTRY (save_paranoid) - XCPT_FRAME 1 RDI+8 + XCPT_FRAME 1 RDI+OFF cld - movq %rdi, RDI+8(%rsp) - movq %rsi, RSI+8(%rsp) - movq_cfi rdx, RDX+8 - movq_cfi rcx, RCX+8 - movq_cfi rax, RAX+8 - movq %r8, R8+8(%rsp) - movq %r9, R9+8(%rsp) - movq %r10, R10+8(%rsp) - movq %r11, R11+8(%rsp) - movq_cfi rbx, RBX+8 - movq %rbp, RBP+8(%rsp) - movq %r12, R12+8(%rsp) - movq %r13, R13+8(%rsp) - movq %r14, R14+8(%rsp) - movq %r15, R15+8(%rsp) - movl $1,%ebx + movq %rdi, RDI+OFF(%rsp) + movq %rsi, RSI+OFF(%rsp) + movq_cfi rdx, RDX+OFF + movq_cfi rcx, RCX+OFF + movq_cfi rax, RAX+OFF + movq %r8, R8+OFF(%rsp) + movq %r9, R9+OFF(%rsp) + movq %r10, R10+OFF(%rsp) + movq %r11, R11+OFF(%rsp) + movq_cfi rbx, RBX+OFF + movq %rbp, RBP+OFF(%rsp) + movq %r12, R12+OFF(%rsp) + movq %r13, R13+OFF(%rsp) + movq %r14, R14+OFF(%rsp) + movq %r15, R15+OFF(%rsp) + movq $-1,ORIG_RAX+OFF(%rsp) /* no syscall to restart */ +33: + ASM_NOP5 /* May be replaced with jump to paranoid_save_gs */ +34: + movl $DO_NOTHING,%ebx movl $MSR_GS_BASE,%ecx rdmsr testl %edx,%edx js 1f /* negative -> in kernel */ SWAPGS - xorl %ebx,%ebx + movl $DO_SWAPGS,%ebx 1: ret + + /* Patch in jump to paranoid_save_gs for X86_FEATURE_FSGSBASE */ + .section .altinstr_replacement,"ax" +35: .byte 0xe9 /* 32bit near jump */ + .long paranoid_save_gs-34b + .previous + .section .altinstructions,"a" + altinstruction_entry 33b,35b,X86_FEATURE_FSGSBASE,5,5 + .previous + +/* Faster version not using RDMSR, and also not assuming +* anything about the previous GS value. +* This allows the user to arbitarily change GS using +* WRGSBASE. + */ +paranoid_save_gs: + RDGSBASE_R15 # read previous gs + movq STKMSK(%rsp),%rax # get ist stack mask + andq %rsp,%rax # get bottom of stack + movq (%rax),%rdi # get expected GS + WRGSBASE_RDI # set gs for kernel + mov $DO_RESTORE_R15,%ebx # flag for exit code + testl $3,CS+OFF(%rsp) # did we come from ring 0?
arch/x86/kernel/cpu/common.c static void identify_cpu(struct cpuinfo_x86 *c)
Quoted 10 lines
arch/x86/kernel/cpu/common.c +10
-0+je paranoid_save_gs_kernel +/* +* Saving GS in the task struct allows a debugger to manipulate +* it. We only do this when coming from ring 3 to avoid recursion +* clobbering the state. + */ + movq PER_CPU_VAR(current_task),%rcx # get current + movb $1,task_thread_gs_saved(%rcx) # tell context switch gs is +# is already saved + movq %r15,task_thread_gs(%rcx) # save gs in task struct
As mentioned above, it seems like this would be simplified if this code could never happen.
Quoted 17 lines
arch/x86/kernel/cpu/common.c +5
-1+paranoid_save_gs_kernel: +ret + CFI_ENDPROCEND (save_paranoid)@@ -1040,7 +1096,8 @@ apicinterrupt IRQ_WORK_VECTOR \*/ #define INIT_TSS_IST (x) PER_CPU_VAR(init_tss) + (TSS_ist + ((x) - 1) * 8) -.macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 +.macro idtentry sym do_sym has_error_code:req paranoid=0 shift_ist=-1 \ + stack_mask=-EXCEPTION_STKSZ ENTRY(\sym)/* Sanity check */ .if \shift_ist != -1 && \paranoid == 0
arch/x86/kernel/cpu/common.c ENTRY(\sym) · line 1127
Quoted 7 lines
arch/x86/kernel/cpu/common.c +3
-0@@ -1064,7 +1121,10 @@ ENTRY(\sym) CFI_ADJUST_CFA_OFFSET ORIG_RAX-R15 .if \paranoid + pushq_cfi $\stack_mask /* ist stack size */ call save_paranoid + /* r15: previous gs */ + popq_cfi %rax /* Drop ist stack size */
IMO this is a bit ugly, but I don't see a better solution off the top of my head.
Quoted 35 lines
arch/x86/kernel/cpu/common.c +6
-4.else call error_entry .endif@@ -1099,7 +1159,7 @@ ENTRY(\sym) .endif .if \paranoid - jmp paranoid_exit /* %ebx: no swapgs flag */ + jmp paranoid_exit /* %ebx: no swapgs flag, r15: old gs */ .elsejmp error_exit/* %ebx: no swapgs flag */ .endif@@ -1287,8 +1347,10 @@ apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \ hyperv_callback_vector hyperv_vector_handler#endif /* CONFIG_HYPERV */ -idtentry debug do_debug has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK -idtentry int3 do_int3 has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK +idtentry debug do_debug has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK \ + stack_mask=-DEBUG_STKSZ +idtentry int3 do_int3 has_error_code=0 paranoid=1 shift_ist=DEBUG_STACK \ + stack_mask=-DEBUG_STKSZ idtentry stack_segment do_stack_segment has_error_code=1 paranoid=1 #ifdef CONFIG_XEN idtentry xen_debug do_debug has_error_code=0@@ -1317,49 +1379,69 @@ idtentry machine_check has_error_code=0 paranoid=1 do_sym=*machine_check_vector(* hard flags at once, atomically) */ -/* ebx: no swapgs flag */ +/* ebx: no swapgs flag, r15: gs if ebx == DO_RESTORE_R15 */ ENTRY (paranoid_exit) DEFAULT_FRAMEDISABLE_INTERRUPTS (CLBR_NONE)TRACE_IRQS_OFF_DEBUG
arch/x86/kernel/cpu/common.c idtentry machine_check has_error_code=0 paranoid=1 do_sym=*machine_check_vector( · line 1394
Quoted 10 lines
arch/x86/kernel/cpu/common.c +4
-2- testl %ebx,%ebx /* swapgs needed? */ - jnz paranoid_restore + cmpl $DO_NOTHING,%ebx /* swapgs needed? */ + je paranoid_restore testl $3,CS(%rsp) jnz paranoid_userspace paranoid_swapgs: TRACE_IRQS_IRETQ 0 + cmpl $DO_RESTORE_R15,%ebx /* restore gs? */ + je paranoid_restore_gs
If we switched off the IST for an entry from userspace, we wouldn't need the conditionals at all for the rdfsgsbase case, right?
Quoted 48 lines
arch/x86/kernel/cpu/common.c +16
-8SWAPGS_UNSAFE_STACK RESTORE_ALL 8 jmp irq_return +paranoid_restore_gs: + WRGSBASE_R15 + RESTORE_ALL 8 + jmp irq_return paranoid_restore: TRACE_IRQS_IRETQ_DEBUG 0 RESTORE_ALL 8 jmp irq_return paranoid_userspace: GET_THREAD_INFO(%rcx) - movl TI_flags(%rcx),%ebx - andl $_TIF_WORK_MASK,%ebx - jz paranoid_swapgs + movl TI_flags(%rcx),%ebp + andl $_TIF_WORK_MASK,%ebp + jz paranoid_clear_gs_flag movq %rsp,%rdi /* &pt_regs */ call sync_regs movq %rax,%rsp /* switch stack for scheduling */ - testl $_TIF_NEED_RESCHED,%ebx + testl $_TIF_NEED_RESCHED,%ebp jnz paranoid_schedule - movl %ebx,%edx /* arg3: thread flags */ + movl %ebp,%edx /* arg3: thread flags */ TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) xorl %esi,%esi /* arg2: oldset */ movq %rsp,%rdi /* arg1: &pt_regs */ call do_notify_resume - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - jmp paranoid_userspace + jmp paranoid_after_schedule +paranoid_clear_gs_flag: + movq PER_CPU_VAR(current_task),%rcx + movb $0,task_thread_gs_saved(%rcx) + jmp paranoid_swapgs paranoid_schedule: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_ANY) SCHEDULE_USER +paranoid_after_schedule: DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF + cmpl $DO_RESTORE_R15,%ebx
arch/x86/kernel/cpu/common.c idtentry machine_check has_error_code=0 paranoid=1 do_sym=*machine_check_vector( · line 1444
Quoted 10 lines
arch/x86/kernel/cpu/common.c +10
-0+ jne paranoid_userspace + movq PER_CPU_VAR(current_task),%rcx + cmpb $0,task_thread_gs_saved(%rcx) /* did we schedule? */ + jne paranoid_userspace /* did not schedule: keep our gs */ + /* + * Otherwise the context switch has put the correct gs into + * kernel_gs, so we can just end with swapgs and it + * will DTRT. + */ + mov $DO_SWAPGS,%ebx /* force swapgs after schedule */
OMG more special cases :( I'm liking my stack-switching proposal even more. Grr, entry code is a mess.
arch/x86/kernel/cpu/common.c idtentry machine_check has_error_code=0 paranoid=1 do_sym=*machine_check_vector(
Quoted 7 lines
arch/x86/kernel/cpu/common.c +7
-0+if (static_cpu_has (X86_FEATURE_FSGSBASE )) { + prev->fs =rdfsbase (); +/* Interrupts are disabled here. */ +if (!prev->gs_saved) { +swapgs (); + prev->gs =rdgsbase (); +swapgs ();
Is this adequately protected against kprobes?
--Andy
Re: [PATCH 4/8] x86: Add support for rd/wr fs/gs base
Quoted 14 lines · Andy Lutomirski
Quoted 3 lines · Andi Kleen
To prevent recursive interrupts clobbering this state in the
task_structthis is only done for interrupts coming directly from ring 3.Since this just came up in a different context today, I'd like to propose a different solution to this piece of the problem.
Can we change the paranoid entry to check if the entry came from ring 3 and to just switch stacks immediately to the standard kernel stack and run the non-paranoid entry code? This eliminates
paranoid_userspaceentirely, and there are no special gsbase machinations any more for the entry-from-userspace path.
Quoted 3 lines
In fact, I think that this will result in the MSR
KERNEL_GS_BASEvalue *always* matching the userspace gs value from any C code in the kernel, since we'll always swapgs exactly once on entry from userspace.
I'll send patches in a couple hours. I have it mostly working.
--Andy
PATCH 5/8 x86: Make old K8 swapgs workaround conditional
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Every gs selector/index reload always paid an extra MFENCE between the two SWAPGS. This was to work around an old bug in early K8 steppings. All other CPUs don't need the extra mfence. Patch the extra MFENCE only in for K8.
--- arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/amd.c | 3 +++ arch/x86/kernel/entry_64.S | 10 +++++++++- 3 files changed, 13 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeature.h +1 -0
@@ -106,6 +106,7 @@#define X86_FEATURE_APERFMPERF ( 3*32+28)/* APERFMPERF */ #define X86_FEATURE_EAGER_FPU ( 3*32+29)/* "eagerfpu" Non lazy FPU restore */ #define X86_FEATURE_NONSTOP_TSC_S3 ( 3*32+30)/* TSC doesn't stop in S3 state */ +#define X86_FEATURE_SWAPGS_MFENCE (3*32+31)/* SWAPGS may need MFENCE */ /* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */ #define X86_FEATURE_XMM3 ( 4*32+ 0)/* "pni" SSE-3 */
arch/x86/kernel/cpu/amd.c +3 -0
@@ -551,6 +551,9 @@ static void init_amd_k8(struct cpuinfo_x86 *c)if ((level >=0x0f48 && level <0x0f50 ) || level >=0x0f58 )set_cpu_cap (c,X86_FEATURE_REP_GOOD ); +/* Early steppings needed a mfence on swapgs. */ +set_cpu_cap (c,X86_FEATURE_SWAPGS_MFENCE ); +/* * Some BIOSes incorrectly force this feature, but only K8 revision D * (model = 0x14) and later actually support it.
arch/x86/kernel/entry_64.S +9 -1
@@ -1203,13 +1203,21 @@ ENTRY(native_load_gs_index) SWAPGS gs_change: movl %edi,%gs -2: mfence /* workaround */ +2: ASM_NOP3 /* may be replaced with mfence */ SWAPGS popfq_cfi ret CFI_ENDPROC END(native_load_gs_index) + /* Early K8 systems needed an mfence after swapgs to workaround a bug */ + .section .altinstr_replacement,"ax" +3: mfence + .previous + .section .altinstructions,"a" + altinstruction_entry 2b,3b,X86_FEATURE_SWAPGS_MFENCE,3,3 + .previous + _ASM_EXTABLE(gs_change,bad_gs) .section .fixup,"ax" /* running with kernelgs */
Signature · 1 line
1.9.3
Re: [PATCH 5/8] x86: Make old K8 swapgs workaround conditional
Quoted 19 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Every gs selector/index reload always paid an extra MFENCE between the two SWAPGS. This was to work around an old bug in early K8 steppings. All other CPUs don't need the extra mfence. Patch the extra MFENCE only in for K8.
Signed-off-by Andi Kleen <[email protected]>--- arch/x86/include/asm/cpufeature.h | 1 + arch/x86/kernel/cpu/amd.c | 3 +++ arch/x86/kernel/entry_64.S | 10 +++++++++- 3 files changed, 13 insertions(+), 1 deletion(-)arch/x86/include/asm/cpufeature.h +0
-0
arch/x86/include/asm/cpufeature.h line 110
Quoted 5 lines
arch/x86/include/asm/cpufeature.h +1
-0@@ -106,6 +106,7 @@#define X86_FEATURE_APERFMPERF ( 3*32+28)/* APERFMPERF */ #define X86_FEATURE_EAGER_FPU ( 3*32+29)/* "eagerfpu" Non lazy FPU restore */ #define X86_FEATURE_NONSTOP_TSC_S3 ( 3*32+30)/* TSC doesn't stop in S3 state */ +#define X86_FEATURE_SWAPGS_MFENCE (3*32+31)/* SWAPGS may need MFENCE */
Can this use the new X86_BUG thing?
--Andy
Quoted 44 lines
arch/x86/include/asm/cpufeature.h +0
-0/* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */ #define X86_FEATURE_XMM3 ( 4*32+ 0)/* "pni" SSE-3 */ arch/x86/kernel/cpu/amd.c +3
-0@@ -551,6 +551,9 @@ static void init_amd_k8(struct cpuinfo_x86 *c)if ((level >=0x0f48 && level <0x0f50 ) || level >=0x0f58 )set_cpu_cap (c,X86_FEATURE_REP_GOOD ); +/* Early steppings needed a mfence on swapgs. */ +set_cpu_cap (c,X86_FEATURE_SWAPGS_MFENCE ); +/* * Some BIOSes incorrectly force this feature, but only K8 revision D * (model = 0x14) and later actually support it. arch/x86/kernel/entry_64.S +9
-1@@ -1203,13 +1203,21 @@ ENTRY(native_load_gs_index) SWAPGS gs_change: movl %edi,%gs -2: mfence /* workaround */ +2: ASM_NOP3 /* may be replaced with mfence */ SWAPGS popfq_cfi ret CFI_ENDPROC END(native_load_gs_index) + /* Early K8 systems needed an mfence after swapgs to workaround a bug */ + .section .altinstr_replacement,"ax" +3: mfence + .previous + .section .altinstructions,"a" + altinstruction_entry 2b,3b,X86_FEATURE_SWAPGS_MFENCE,3,3 + .previous + _ASM_EXTABLE(gs_change,bad_gs) .section .fixup,"ax" /* running with kernelgs */
PATCH 6/8 x86: Enumerate kernel FSGS capability in AT_HWCAP2
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
The kernel needs to explicitely enable RD/WRFSBASE to handle context switch correctly. So the application needs to know if it can safely use these instruction. Just looking at the CPUID bit is not enough because it may be running in a kernel that does not enable the instructions.
One way for the application would be to just try and catch the SIGILL. But that is difficult to do in libraries which may not want to overwrite the signal handlers of the main application.
So we need to provide a way for the application to discover the kernel capability.
I used AT_HWCAP2 in the ELF aux vector which is already used by PPC for similar things. We define a new Linux defined bitmap returned in AT_HWCAP. Currently it has only one bit set, for kernel is FSGSBASE capable.
The application can then access it manually or using the getauxval() function in newer glibc.
--- arch/x86/include/asm/elf.h | 7 +++++++ arch/x86/include/uapi/asm/hwcap.h | 7 +++++++ arch/x86/kernel/cpu/common.c | 7 ++++++- 3 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/uapi/asm/hwcap.h
arch/x86/include/asm/elf.h +7 -0
@@ -257,6 +257,13 @@ extern int force_personality32;#define ELF_HWCAP (boot_cpu_data.x86_capability[0]) +extern unsigned kernel_enabled_features; + +/* HWCAP2 supplies kernel enabled CPU feature, so that the application +can know that it can safely use them. The bits are defined in +uapi/asm/hwcap.h. */ +#define ELF_HWCAP2 kernel_enabled_features; +/* This yields a string that ld.so will use to load implementation specific libraries for optimization. This is more specific in intent than poking at uname or /proc/cpuinfo.
arch/x86/include/uapi/asm/hwcap.h new file +7 -0
@@ -0,0 +1,7 @@ +#ifndef _ASM_HWCAP_H +#define _ASM_HWCAP_H 1 + +#define HWCAP2_FSGSBASE (1 << 0)/* Kernel enabled RD/WR FS/GS BASE */ +/* upto bit 31 free */ + +#endif
arch/x86/kernel/cpu/common.c +6 -1
@@ -33,6 +33,7 @@#include <asm/i387.h> #include <asm/fpu-internal.h> #include <asm/mtrr.h> +#include <asm/hwcap.h> #include <linux/numa.h> #include <asm/asm.h> #include <asm/cpu.h>
@@ -48,6 +49,8 @@#include "cpu.h" +unsigned kernel_enabled_features__read_mostly ; +/* all of these masks are initialized in setup_cpu_local_masks() */ cpumask_var_t cpu_initialized_mask;cpumask_var_t cpu_callout_mask;
@@ -954,8 +957,10 @@ static void identify_cpu(struct cpuinfo_x86 *c)numa_add_cpu (smp_processor_id ());#endif -if (cpu_has (c,X86_FEATURE_FSGSBASE )) +if (cpu_has (c,X86_FEATURE_FSGSBASE )) { + kernel_enabled_features |=HWCAP2_FSGSBASE ;set_in_cr4 (X86_CR4_FSGSBASE ); + } }#ifdef CONFIG_X86_64
Signature · 1 line
1.9.3
PATCH 7/8 x86: Add documentation for rd/wr fs/gs base
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
v2: Minor updates to documentation requested in review.
--- Documentation/x86/fsgs.txt | 76 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 Documentation/x86/fsgs.txt
Documentation/x86/fsgs.txt new file +76 -0
@@ -0,0 +1,76 @@ + +Using FS and GS prefixes on x86_64-linux + +The x86 architecture supports segment prefixes per instruction to add an +offset to an address. On 64bit x86, these are mostly nops, except for FS +and GS. + +This offers an efficient way to reference a global pointer. + +The compiler has to generate special code to use these base registers, +or they can be accessed with inline assembler. + + mov %gs:offset,%reg + mov %fs:offset,%reg + +FS is used to address the thread local segment (TLS), declared using +__thread. The compiler then automatically generates the correct prefixes and +relocations to access these values. + +FS is normally managed by the runtime code or the threading library. + +GS is freely available, but may need special (compiler or inline assembler) +code to use. + +Traditionally 64bit FS and GS could be set by the arch_prctl system call + + arch_prctl(ARCH_SET_GS, value) + arch_prctl(ARCH_SET_FS, value) + +[There was also an older method using modify_ldt(), inherited from 32bit, +but this is not discussed here.] + +However using a syscall is problematic for user space threading libraries +that want to context switch in user space. The whole point of them +is avoiding the overhead of a syscall. It's also cleaner for compilers +wanting to use the extra register to use instructions to write +it, or read it directly to compute addresses and offsets. + +Newer Intel CPUs (Ivy Bridge and later) added new instructions to directly +access these registers quickly from user context + + RDFSBASE %reg read the FS base (or _readfsbase_u64) + RDGSBASE %reg read the GS base (or _readgsbase_u64) + + WRFSBASE %reg write the FS base (or _writefsbase_u64) + WRGSBASE %reg write the GS base (or _writegsbase_u64) + +The instructions are supported by the CPU when the "fsgsbase" string is shown in +/proc/cpuinfo (or directly retrieved through the CPUID instruction). +The instructions are only available to 64bit binaries. + +However the kernel needs to explicitly enable these instructions, as it +may otherwise not correctly context switch the state. Newer Linux +kernels enable this. When the kernel did not enable the instruction +they will fault with an #UD exception. + +An FSGSBASE enabled kernel can be detected by checking the AT_HWCAP2 +bitmask in the aux vector. When the HWCAP2_FSGSBASE bit is set the +kernel supports RDFSGSBASE. + + #include <sys/auxv.h> + #include <elf.h> + + /* Will be eventually in asm/hwcap.h */ + #define HWCAP2_FSGSBASE (1 << 0) + + unsigned val = getauxval(AT_HWCAP2); + if (val & HWCAP2_FSGSBASE) { + asm("wrgsbase %0" :: "r" (ptr)); + } + +Another requirement is that the FS or GS selector has to be zero +(is normally true unless changed explicitly) + + +Andi Kleen
Signature · 1 line
1.9.3
PATCH 8/8 x86: Use rd/wr fs/gs base in arch_prctl
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Convert arch_prctl to use the new instructions to change fs/gs if available, instead of using MSRs.
This is merely a small performance optimization, no new functionality.
With the new instructions the syscall is really obsolete, as everything can be set directly in ring 3. But the syscall is widely used by existing software, so we still support it.
The syscall still enforces that the addresses are not in kernel space, even though that is not needed more. This is mainly so that the programs written for new CPUs do not suddenly fail on old CPUs.
With the new instructions available it prefers to use them in the context switch, instead of using the old "use GDT segment rewrite" trick.
--- arch/x86/kernel/process_64.c | 45 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-)
arch/x86/kernel/process_64.c +37 -8
@@ -483,15 +483,23 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr)int ret =0 ;int doit = task == current;int cpu; +int fast_seg =boot_cpu_has (X86_FEATURE_FSGSBASE );switch (code) {case ARCH_SET_GS : +/* +* With fast_seg we don't need that check anymore, +* but keep it so that programs do not suddenly +* start failing when run on older CPUs. +* If you really want to set a address in kernel space +* use WRGSBASE directly. +*/ if (addr >=TASK_SIZE_OF (task))return -EPERM ; cpu =get_cpu ();/* handle small bases via the GDT because that's faster to switch. */ -if (addr <=0xffffffff ) { +if (addr <=0xffffffff && !fast_seg) {set_32bit_tls (task,GS_TLS , addr);if (doit) {load_TLS (&task->thread, cpu);
@@ -503,8 +511,17 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr) task->thread.gsindex =0 ; task->thread.gs = addr;if (doit) { -load_gs_index (0 ); - ret =wrmsrl_safe (MSR_KERNEL_GS_BASE , addr); +if (fast_seg) { +local_irq_disable (); +swapgs (); +loadsegment (gs,0 ); +wrgsbase (addr); +swapgs (); +local_irq_enable (); + }else { +load_gs_index (0 ); + ret =wrmsrl_safe (MSR_KERNEL_GS_BASE , addr); + } } }put_cpu ();
@@ -517,7 +534,7 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr) cpu =get_cpu ();/* handle small bases via the GDT because that's faster to switch. */ -if (addr <=0xffffffff ) { +if (addr <=0xffffffff && !fast_seg) {set_32bit_tls (task,FS_TLS , addr);if (doit) {load_TLS (&task->thread, cpu);
@@ -532,7 +549,10 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr)/* set the selector to 0 to not confuse __switch_to */ loadsegment (fs,0 ); - ret =wrmsrl_safe (MSR_FS_BASE , addr); +if (fast_seg) +wrfsbase (addr); +else + ret =wrmsrl_safe (MSR_FS_BASE , addr); } }put_cpu ();
@@ -541,6 +561,8 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr)unsigned long base;if (task->thread.fsindex ==FS_TLS_SEL ) base =read_32bit_tls (task,FS_TLS ); +else if (doit && fast_seg) + base =rdfsbase ();else if (doit)rdmsrl (MSR_FS_BASE , base);else
@@ -555,9 +577,16 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr) base =read_32bit_tls (task,GS_TLS ); else if (doit) {savesegment (gs, gsindex); -if (gsindex) -rdmsrl (MSR_KERNEL_GS_BASE , base); -else +if (gsindex) { +if (fast_seg) { +local_irq_disable (); +swapgs (); + base =rdgsbase (); +swapgs (); +local_irq_enable (); + }else +rdmsrl (MSR_KERNEL_GS_BASE , base); + }else base = task->thread.gs; } else base = task->thread.gs;
Signature · 1 line
1.9.3
Re: [PATCH 8/8] x86: Use rd/wr fs/gs base in arch_prctl
Quoted 58 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Convert
arch_prctlto use the new instructions to change fs/gs if available, instead of using MSRs.This is merely a small performance optimization, no new functionality.
With the new instructions the syscall is really obsolete, as everything can be set directly in ring 3. But the syscall is widely used by existing software, so we still support it.
The syscall still enforces that the addresses are not in kernel space, even though that is not needed more. This is mainly so that the programs written for new CPUs do not suddenly fail on old CPUs.
With the new instructions available it prefers to use them in the context switch, instead of using the old "use GDT segment rewrite" trick.
Signed-off-by Andi Kleen <[email protected]>--- arch/x86/kernel/process_64.c | 45 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 8 deletions(-)arch/x86/kernel/process_64.c +9
-1@@ -483,15 +483,23 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr)int ret =0 ;int doit = task == current;int cpu; +int fast_seg =boot_cpu_has (X86_FEATURE_FSGSBASE );switch (code) {case ARCH_SET_GS : +/* +* With fast_seg we don't need that check anymore, +* but keep it so that programs do not suddenly +* start failing when run on older CPUs. +* If you really want to set a address in kernel space +* use WRGSBASE directly. +*/ if (addr >=TASK_SIZE_OF (task))return -EPERM ; cpu =get_cpu ();/* handle small bases via the GDT because that's faster to switch. */ -if (addr <=0xffffffff ) { +if (addr <=0xffffffff && !fast_seg) {set_32bit_tls (task,GS_TLS , addr);if (doit) {load_TLS (&task->thread, cpu);@@ -503,8 +511,17 @@ long do_arch_prctl(struct task_struct *task, int code, unsigned long addr) task->thread.gsindex =0 ; task->thread.gs = addr;
arch/x86/kernel/process_64.c long do_arch_prctl(struct task_struct *task, int code, unsigned long addr) · line 520
Quoted 10 lines
arch/x86/kernel/process_64.c +7
-2if (doit) { -load_gs_index (0 ); - ret =wrmsrl_safe (MSR_KERNEL_GS_BASE , addr); +if (fast_seg) { +local_irq_disable (); +swapgs (); +loadsegment (gs,0 ); +wrgsbase (addr); +swapgs (); +local_irq_enable ();
Does this (and the other copies of this) need kprobe protection?
--Andy