x86/asm/entry/64: Enable interrupts *after* we fetch PER_CPU_VAR(old_rsp)
PATCH 1/2 x86/asm/entry/64: Enable interrupts *after* we fetch PER_CPU_VAR(old_rsp)
Without this change, it is still not possible to get rid of PER_CPU_VAR(old_rsp) usage in switch_to: if preemption happens while we did not fetch PER_CPU_VAR(old_rsp) and stored it in pt_regs->sp, PER_CPU_VAR(old_rsp) gets corrupted by other task's user sp.
---
Run-tested, including with PARAVIRT on.
arch/x86/kernel/entry_64.S | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/kernel/entry_64.S +5 -5
@@ -241,17 +241,17 @@ GLOBAL(system_call_after_swapgs) movq %rsp,PER_CPU_VAR(old_rsp) /* kernel_stack is set so that 5 slots (iret frame) are preallocated */ movq PER_CPU_VAR(kernel_stack),%rsp - /* - * No need to follow this irqs off/on section - it's straight - * and short: - */ - ENABLE_INTERRUPTS(CLBR_NONE) ALLOC_PT_GPREGS_ON_STACK 8 /* +8: space for orig_ax */ movq %rcx,RIP(%rsp) movq PER_CPU_VAR(old_rsp),%rcx movq %r11,EFLAGS(%rsp) movq %rcx,RSP(%rsp) movq_cfi rax,ORIG_RAX + /* + * No need to follow this irqs off/on section - it's straight + * and short: + */ + ENABLE_INTERRUPTS(CLBR_RAX) SAVE_C_REGS_EXCEPT_RAX_RCX_R11 movq $-ENOSYS,RAX(%rsp) CFI_REL_OFFSET rip,RIP
Signature · 1 line
1.8.1.4
PATCH 2/2 x86/asm/entry/64: Remove unused thread_struct::usersp
All manipulations of PER_CPU(old_rsp) in C code are removed:
it is not used on SYSRET return, so storing anything there is pointless.
This also allows us to get rid of thread_struct::usersp, which was needed only to set PER_CPU(old_rsp) for correct return from fork/clone.
Tweak a few comments as well: we no longer have "partial stack frame", ever.
---
Run-tested, including with PARAVIRT on.
arch/x86/include/asm/processor.h | 6 ------ arch/x86/kernel/entry_64.S | 14 ++++++-------- arch/x86/kernel/process_64.c | 5 ----- 3 files changed, 6 insertions(+), 19 deletions(-)
arch/x86/include/asm/processor.h +0 -6
@@ -496,7 +496,6 @@ struct thread_struct {#ifdef CONFIG_X86_32 unsigned long sysenter_cs;#else -unsigned long usersp;/* Copy from PDA */ unsigned short es;unsigned short ds;unsigned short fsindex;
@@ -908,11 +907,6 @@ extern unsigned long thread_saved_pc(struct task_struct *tsk);#define task_pt_regs (tsk) ((struct pt_regs *)(tsk)->thread.sp0 - 1)extern unsigned long KSTK_ESP (struct task_struct *task); -/* -* User space RSP while inside the SYSCALL fast path -*/ -DECLARE_PER_CPU(unsigned long, old_rsp); -#endif /* CONFIG_X86_64 */ extern void start_thread(struct pt_regs *regs, unsigned long new_ip,
arch/x86/kernel/entry_64.S +6 -8
@@ -15,10 +15,8 @@ * after an interrupt and after each system call. * * A note on terminology: - * - top of stack: Architecture defined interrupt frame from SS to RIP + * - iret frame: Architecture defined interrupt frame from SS to RIP * at the top of the kernel process stack. - * - partial stack frame: partially saved registers up to R11. - * - full stack frame: Like partial stack frame, but all register saved. * * Some macro usage: * - CFI macros are used to generate dwarf2 unwind information for better
@@ -219,7 +217,7 @@ ENDPROC(native_usergs_sysret64) * Interrupts are off on entry. * Only called from user space. * - * When user can change the frames always force IRET. That is because + * When user can change pt_regs->foo always force IRET. That is because * it deals with uncanonical addresses better. SYSRET has trouble * with them due to bugs in both AMD and Intel CPUs. */
@@ -303,7 +301,7 @@ int_ret_from_sys_call_fixup: FIXUP_TOP_OF_STACK %r11 jmp int_ret_from_sys_call - /* Do syscall tracing */ + /* Do syscall entry tracing */ tracesys: movq %rsp, %rdi movq $AUDIT_ARCH_X86_64, %rsi
@@ -339,11 +337,11 @@ tracesys_phase2: movq %r10,%rcx /* fixup for C */ call *sys_call_table(,%rax,8) movq %rax,RAX(%rsp) - /* Use IRET because user could have changed frame */ + /* Use IRET because user could have changed pt_regs->foo */ /* * Syscall return path ending with IRET. - * Has correct top of stack, but partial stack frame. + * Has correct iret frame. */ GLOBAL(int_ret_from_sys_call) DISABLE_INTERRUPTS(CLBR_NONE)
@@ -374,7 +372,7 @@ int_careful: TRACE_IRQS_OFF jmp int_with_check - /* handle signals and tracing -- both require a full stack frame */ + /* handle signals and tracing -- both require a full pt_regs */ int_very_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE)
arch/x86/kernel/process_64.c +0 -5
@@ -161,7 +161,6 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, p->thread.sp0 = (unsigned long )task_stack_page (p) +THREAD_SIZE ; childregs =task_pt_regs (p); p->thread.sp = (unsigned long ) childregs; - p->thread.usersp = me->thread.usersp;set_tsk_thread_flag (p,TIF_FORK ); p->thread.io_bitmap_ptr =NULL ;
@@ -235,10 +234,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,loadsegment (es, _ds);loadsegment (ds, _ds);load_gs_index (0 ); - current->thread.usersp = new_sp; regs->ip = new_ip; regs->sp = new_sp; -this_cpu_write (old_rsp, new_sp); regs->cs = _cs; regs->ss = _ss; regs->flags =X86_EFLAGS_IF ;
@@ -398,8 +395,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)/* * Switch the PDA and FPU contexts. */ - prev->usersp =this_cpu_read (old_rsp); -this_cpu_write (old_rsp, next->usersp);this_cpu_write (current_task, next_p);/*
Signature · 1 line
1.8.1.4
Re: [PATCH 2/2] x86/asm/entry/64: Remove unused thread_struct::usersp
Quoted 10 lines · Denys Vlasenko
All manipulations of
PER_CPU(old_rsp)in C code are removed:
it is not used on SYSRET return, so storing anything there is pointless.This also allows us to get rid of
thread_struct::usersp, which was needed only to setPER_CPU(old_rsp)for correct return from fork/clone.Tweak a few comments as well: we no longer have "partial stack frame", ever.
So this doesn't really explain it very well, plus it still unnecessarily mixes up two changes in a single patch: please don't do that!
I've fixed it all up and split up the patch into two, easier to bisect parts.
Thanks,
Ingo
tip-bot for Ingo Molnar
merge bot[tip:x86/asm] x86/asm/entry/64: Simplify 'old_rsp' usage
In reply to Denys Vlasenkox86/asm/entry/64: Simplify 'old_rsp' usage
Commit-ID: 9854dd74c3f6af8d9d527de86c6074b7ed0495f1 Gitweb: http://git.kernel.org/tip/9854dd74c3f6af8d9d527de86c6074b7ed0495f1 Author: Ingo Molnar <[email protected]> AuthorDate: Tue, 17 Mar 2015 14:42:59 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 17 Mar 2015 16:01:41 +0100
x86/asm/entry/64: Simplify 'old_rsp' usage
Remove all manipulations of PER_CPU(old_rsp) in C code:
- it is not used on SYSRET return anymore, and system entries are atomic, so updating it from the fork and context switch paths is pointless.
- Tweak a few related comments as well: we no longer have a "partial stack frame" on entry, ever.
Based on (split out of) patch from Denys Vlasenko.
Originally-from: Denys Vlasenko <[email protected]>
--- arch/x86/include/asm/processor.h | 5 ----- arch/x86/kernel/process_64.c | 2 -- 2 files changed, 7 deletions(-)
arch/x86/include/asm/processor.h +0 -5
@@ -905,11 +905,6 @@ extern unsigned long thread_saved_pc(struct task_struct *tsk);#define task_pt_regs (tsk) ((struct pt_regs *)(tsk)->thread.sp0 - 1)extern unsigned long KSTK_ESP (struct task_struct *task); -/* -* User space RSP while inside the SYSCALL fast path -*/ -DECLARE_PER_CPU(unsigned long, old_rsp); -#endif /* CONFIG_X86_64 */ extern void start_thread(struct pt_regs *regs, unsigned long new_ip,
arch/x86/kernel/process_64.c +0 -2
@@ -238,7 +238,6 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip, current->thread.usersp = new_sp; regs->ip = new_ip; regs->sp = new_sp; -this_cpu_write (old_rsp, new_sp); regs->cs = _cs; regs->ss = _ss; regs->flags =X86_EFLAGS_IF ;
@@ -399,7 +398,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* Switch the PDA and FPU contexts. */ prev->usersp =this_cpu_read (old_rsp); -this_cpu_write (old_rsp, next->usersp);this_cpu_write (current_task, next_p);/*
Quoted 19 lines · Denys Vlasenko
Without this change, it is still not possible to get rid of
PER_CPU_VAR(old_rsp)usage inswitch_to: if preemption happens while we did not fetchPER_CPU_VAR(old_rsp)and stored it inpt_regs->sp,PER_CPU_VAR(old_rsp)gets corrupted by other task's user sp.Signed-off-by Denys Vlasenko <[email protected]>CC Linus Torvalds <[email protected]>CC Steven Rostedt <[email protected]>CC Ingo Molnar <[email protected]>CC Borislav Petkov <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Andy Lutomirski <[email protected]>CC Oleg Nesterov <[email protected]>CC Frederic Weisbecker <[email protected]>CC Alexei Starovoitov <[email protected]>CC Will Drewry <[email protected]>CC Kees Cook <[email protected]>
Quoted 3 lines
---
Run-tested, including with PARAVIRT on.
Well. The testing with PARAVIRT did work, however, I don't know why.
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 254
Quoted 6 lines
arch/x86/kernel/entry_64.S +5
-0movq_cfi rax,ORIG_RAX + /* + * No need to follow this irqs off/on section - it's straight + * and short: + */ + ENABLE_INTERRUPTS(CLBR_RAX)
Here I wrongly assumed that now I can clobber rax, since it is saved in pt_regs->orig_ax now. Wrong. Code below still wants to use rax register directly.
Looks like I was "lucky" and paravirt call happen to not change rax. I'll send a v2 patch without this ill-advised attempt of optimization. Sorry.
Quoted 5 lines · Denys Vlasenko
Without this change, it is still not possible to get rid of
PER_CPU_VAR(old_rsp)usage inswitch_to: if preemption happens while we did not fetchPER_CPU_VAR(old_rsp)and stored it inpt_regs->sp,PER_CPU_VAR(old_rsp)gets corrupted by other task's user sp.
[...]
Quoted 1 line
-
ENABLE_INTERRUPTS(CLBR_NONE)
[...]
Quoted 1 line
+
ENABLE_INTERRUPTS(CLBR_RAX)
In future patches, please don't sneak these things in. It makes it much harder to review. If you need to move code and change it like this, I think it should be clearly and explicitly mentioned in the changelog.
--Andy