x86/entry: Clean up entry code
Subsystems (2)
cover x86/entry: Clean up entry code
From: Lai Jiangshan <[email protected]>
This patchset moves the stack-switch code to the place where error_entry() return, distangles error_entry() from XENpv and makes entry_INT80_compat use idtentry macro.
This patchset is highly related to XENpv, because it does the extra cleanup to convert SWAPGS to swapgs after major cleanup is done.
After error_entry() is distangled from XENpv, a branch in sync_regs() is removed to.
Patch 1-10 are picked from the patchset
https://lore.kernel.org/lkml/[email protected]/
which coverts ASM code to C code. These patches are prepared for that purpose. But this patchset has it own value: it simplifies the stack switch, avoids leaving the old stack inside a function, and separates XENpv code with native code without adding new code which can lead to more cleanups.
Patch 11 is new in this patchset.
Lai Jiangshan (11):
x86/entry: Use swapgs and native_iret directly in
swapgs_restore_regs_and_return_to_usermode
x86/traps: Move pt_regs only in fixup_bad_iret() x86/entry: Switch the stack after error_entry() returns x86/entry: move PUSH_AND_CLEAR_REGS out of error_entry x86/entry: Move cld to the start of idtentry x86/entry: Don't call error_entry for XENPV x86/entry: Convert SWAPGS to swapgs in error_entry() x86/entry: Use idtentry macro for entry_INT80_compat x86/entry: Convert SWAPGS to swapgs in entry_SYSENTER_compat()
x86: Remove the definition of SWAPGS
x86/entry: Remove the branch in sync_regs()
arch/x86/entry/entry_64.S | 65 +++++++++++++------ arch/x86/entry/entry_64_compat.S | 104 +------------------------------ arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/irqflags.h | 2 - arch/x86/include/asm/proto.h | 4 -- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 25 ++++---- 7 files changed, 107 insertions(+), 142 deletions(-)
Signature · 1 line
2.19.1.6.gb485710b
PATCH 1/11 x86/entry: Use swapgs and native_iret directly in swapgs_restore_regs_and_return_to_usermode
From: Lai Jiangshan <[email protected]>
swapgs_restore_regs_and_return_to_usermode() is used in native code (non-xenpv) only now, so it doesn't need the PV-aware SWAPGS and INTERRUPT_RETURN.
--- arch/x86/entry/entry_64.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +2 -2
@@ -608,8 +608,8 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) /* Restore RDI. */ popq %rdi - SWAPGS - INTERRUPT_RETURN + swapgs + jmp native_iret SYM_INNER_LABEL(restore_regs_and_return_to_kernel, SYM_L_GLOBAL)
Signature · 1 line
2.19.1.6.gb485710b
Re: [PATCH 01/11] x86/entry: Use swapgs and native_iret directly in swapgs_restore_regs_and_return_to_usermode
Quoted 7 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
swapgs_restore_regs_and_return_to_usermode()is used in native code (non-xenpv) only now, so it doesn't need the PV-aware SWAPGS andINTERRUPT_RETURN.Signed-off-by Lai Jiangshan <[email protected]>
Juergen
PATCH 2/11 x86/traps: Move pt_regs only in fixup_bad_iret()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
fixup_bad_iret() and sync_regs() have similar arguments and do similar work that copies full or partial pt_regs to a place and switches stack after return. They are quite the same, but fixup_bad_iret() not only copies the pt_regs but also the return address of error_entry() while sync_regs() copies the pt_regs only and the return address of error_entry() was preserved and handled in ASM code.
This patch makes fixup_bad_iret() work like sync_regs() and the handling of the return address of error_entry() is moved in ASM code.
It removes the need to use the struct bad_iret_stack, simplifies fixup_bad_iret() and makes the ASM error_entry() call fixup_bad_iret() as the same as calling sync_regs() which adds readability because the calling patterns are exactly the same.
It is prepared for later patch to do the stack switch after the error_entry() which simplifies the code further.
--- arch/x86/entry/entry_64.S | 5 ++++- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 17 ++++++----------- 3 files changed, 11 insertions(+), 13 deletions(-)
arch/x86/entry/entry_64.S +4 -1
@@ -1040,9 +1040,12 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - mov %rsp, %rdi + popq %r12 /* save return addr in %12 */ + movq %rsp, %rdi /* arg0 = pt_regs pointer */ call fixup_bad_iret mov %rax, %rsp + ENCODE_FRAME_POINTER + pushq %r12 jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
arch/x86/include/asm/traps.h +1 -1
@@ -13,7 +13,7 @@#ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notrace -struct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s); +struct pt_regs *fixup_bad_iret (struct pt_regs *bad_regs);void __init trap_init (void ); asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);#endif
arch/x86/kernel/traps.c +6 -11
@@ -761,13 +761,8 @@ asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *r }#endif -struct bad_iret_stack { -void *error_entry_ret; -struct pt_regs regs; -}; - asmlinkage __visible noinstr -struct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s) +struct pt_regs *fixup_bad_iret (struct pt_regs *bad_regs) {/* * This is called from entry_64.S early in handling a fault
@@ -777,19 +772,19 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s)* just below the IRET frame) and we want to pretend that the * exception came from the IRET target. */ -struct bad_iret_stack tmp, *new_stack = - (struct bad_iret_stack *)__this_cpu_read (cpu_tss_rw.x86_tss.sp0) -1 ; +struct pt_regs tmp, *new_stack = + (struct pt_regs *)__this_cpu_read (cpu_tss_rw.x86_tss.sp0) -1 ;/* Copy the IRET target to the temporary storage. */ -__memcpy (&tmp.regs.ip, (void *)s->regs.sp,5 *8 ); +__memcpy (&tmp.ip, (void *)bad_regs->sp,5 *8 );/* Copy the remainder of the stack from the current stack. */ - __memcpy(&tmp, s, offsetof(struct bad_iret_stack, regs.ip)); +__memcpy (&tmp, bad_regs, offsetof(struct pt_regs , ip));/* Update the entry stack */ __memcpy (new_stack, &tmp,sizeof (tmp)); -BUG_ON (!user_mode (&new_stack->regs)); +BUG_ON (!user_mode (new_stack)); return new_stack; }#endif
Signature · 1 line
2.19.1.6.gb485710b
PATCH 3/11 x86/entry: Switch the stack after error_entry() returns
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
error_entry() calls sync_regs() to settle/copy the pt_regs and switches the stack directly after sync_regs(). But error_entry() itself is also a function call, the switching has to handle the return address of it together, which causes the work complicated and tangly.
Switching to the stack after error_entry() makes the code simpler and intuitive.
--- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -10
@@ -323,6 +323,8 @@ SYM_CODE_END(ret_from_fork) .macro idtentry_body cfunc has_error_code:req call error_entry + movq %rax, %rsp /* switch stack settled by sync_regs() */ + ENCODE_FRAME_POINTER UNWIND_HINT_REGS movq %rsp, %rdi /* pt_regs pointer into 1st argument*/
@@ -981,14 +983,10 @@ SYM_CODE_START_LOCAL(error_entry) /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax + leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ .Lerror_entry_from_usermode_after_swapgs: /* Put us onto the real thread stack. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 = pt_regs pointer */ call sync_regs - movq %rax, %rsp /* switch stack */ - ENCODE_FRAME_POINTER - pushq %r12 ret /*
@@ -1020,6 +1018,7 @@ SYM_CODE_START_LOCAL(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + leaq 8(%rsp), %rax /* return pt_regs pointer */ ret .Lbstep_iret:
@@ -1040,12 +1039,9 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 = pt_regs pointer */ + leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ call fixup_bad_iret - mov %rax, %rsp - ENCODE_FRAME_POINTER - pushq %r12 + mov %rax, %rdi jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
Signature · 1 line
2.19.1.6.gb485710b
PATCH 4/11 x86/entry: move PUSH_AND_CLEAR_REGS out of error_entry
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
Moving PUSH_AND_CLEAR_REGS out of error_entry doesn't change any functionality. It will enlarge the size:
size arch/x86/entry/entry_64.o.before:
text data bss dec hex filename
17916 384 0 18300 477c arch/x86/entry/entry_64.o
size --format=SysV arch/x86/entry/entry_64.o.before:
.entry.text 5528 0 .orc_unwind 6456 0 .orc_unwind_ip 4304 0
size arch/x86/entry/entry_64.o.after:
text data bss dec hex filename
26868 384 0 27252 6a74 arch/x86/entry/entry_64.o
size --format=SysV arch/x86/entry/entry_64.o.after:
.entry.text 8200 0 .orc_unwind 10224 0 .orc_unwind_ip 6816 0
But .entry.text in x86_64 is 2M aligned, enlarging it to 8.2k doesn't enlarge the final text size.
The tables .orc_unwind[_ip] are enlarged due to it adds many pushes.
It is prepared for not calling error_entry() from XENPV in later patch and for future converting the whole error_entry into C code.
--- arch/x86/entry/entry_64.S | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +3 -2
@@ -322,6 +322,9 @@ SYM_CODE_END(ret_from_fork) */ .macro idtentry_body cfunc has_error_code:req + PUSH_AND_CLEAR_REGS + ENCODE_FRAME_POINTER + call error_entry movq %rax, %rsp /* switch stack settled by sync_regs() */ ENCODE_FRAME_POINTER
@@ -969,8 +972,6 @@ SYM_CODE_END(paranoid_exit) SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC cld - PUSH_AND_CLEAR_REGS save_ret=1 - ENCODE_FRAME_POINTER 8 testb $3, CS+8(%rsp) jz .Lerror_kernelspace
Signature · 1 line
2.19.1.6.gb485710b
PATCH 5/11 x86/entry: Move cld to the start of idtentry
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
Make it next to CLAC
--- arch/x86/entry/entry_64.S | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64.S +5 -3
@@ -356,6 +356,7 @@ SYM_CODE_END(ret_from_fork) SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8 ASM_CLAC + cld .if \has_error_code == 0 pushq $-1 /* ORIG_RAX: no syscall to restart */
@@ -423,6 +424,7 @@ SYM_CODE_END(\asmsym) SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ASM_CLAC + cld pushq $-1 /* ORIG_RAX: no syscall to restart */
@@ -478,6 +480,7 @@ SYM_CODE_END(\asmsym) SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ASM_CLAC + cld /* * If the entry is from userspace, switch stacks and treat it as
@@ -539,6 +542,7 @@ SYM_CODE_END(\asmsym) SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=8 ASM_CLAC + cld /* paranoid_entry returns GS information for paranoid_exit in EBX. */ call paranoid_entry
@@ -853,7 +857,6 @@ SYM_CODE_END(xen_failsafe_callback) */ SYM_CODE_START_LOCAL(paranoid_entry) UNWIND_HINT_FUNC - cld PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -971,7 +974,6 @@ SYM_CODE_END(paranoid_exit) */ SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC - cld testb $3, CS+8(%rsp) jz .Lerror_kernelspace
@@ -1104,6 +1106,7 @@ SYM_CODE_START(asm_exc_nmi) */ ASM_CLAC + cld /* Use %rdx as our temp variable throughout */ pushq %rdx
@@ -1123,7 +1126,6 @@ SYM_CODE_START(asm_exc_nmi) */ swapgs - cld FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=%rdx movq %rsp, %rdx
Signature · 1 line
2.19.1.6.gb485710b
PATCH 6/11 x86/entry: Don't call error_entry for XENPV
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
When in XENPV, it is already in the task stack, and it can't fault for native_iret() nor native_load_gs_index() since XENPV uses its own pvops for iret and load_gs_index(). And it doesn't need to switch CR3. So there is no reason to call error_entry() in XENPV.
--- arch/x86/entry/entry_64.S | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +11 -2
@@ -325,8 +325,17 @@ SYM_CODE_END(ret_from_fork) PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER - call error_entry - movq %rax, %rsp /* switch stack settled by sync_regs() */ + /* + * Call error_entry and switch stack settled by sync_regs(). + * + * When in XENPV, it is already in the task stack, and it can't fault + * for native_iret() nor native_load_gs_index() since XENPV uses its + * own pvops for iret and load_gs_index(). And it doesn't need to + * switch CR3. So it can skip invoking error_entry(). + */ + ALTERNATIVE "call error_entry; movq %rax, %rsp", \ + "", X86_FEATURE_XENPV + ENCODE_FRAME_POINTER UNWIND_HINT_REGS
Signature · 1 line
2.19.1.6.gb485710b
PATCH 7/11 x86/entry: Convert SWAPGS to swapgs in error_entry()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
XENPV doesn't use error_entry() anymore, so the pv-aware SWAPGS can be changed to native swapgs.
--- arch/x86/entry/entry_64.S | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64.S +3 -3
@@ -990,7 +990,7 @@ SYM_CODE_START_LOCAL(error_entry) * We entered from user mode or we're pretending to have entered * from user mode due to an IRET fault. */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
@@ -1022,7 +1022,7 @@ SYM_CODE_START_LOCAL(error_entry) * gsbase and proceed. We'll fix up the exception and land in * .Lgs_change's error handler with kernel gsbase. */ - SWAPGS + swapgs /* * Issue an LFENCE to prevent GS speculation, regardless of whether it is a
@@ -1043,7 +1043,7 @@ SYM_CODE_START_LOCAL(error_entry) * We came from an IRET to user mode, so we have user * gsbase and CR3. Switch to kernel gsbase and CR3: */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
Signature · 1 line
2.19.1.6.gb485710b
PATCH 8/11 x86/entry: Use idtentry macro for entry_INT80_compat
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
entry_INT80_compat is identical to idtentry macro except a special handling for %rax in the prolog.
Add the prolog to idtentry and use idtentry for entry_INT80_compat.
--- arch/x86/entry/entry_64.S | 18 ++++++ arch/x86/entry/entry_64_compat.S | 102 ------------------------------- arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/proto.h | 4 -- 4 files changed, 65 insertions(+), 106 deletions(-)
arch/x86/entry/entry_64.S +18 -0
@@ -371,6 +371,24 @@ SYM_CODE_START(\asmsym) pushq $-1 /* ORIG_RAX: no syscall to restart */ .endif + .if \vector == IA32_SYSCALL_VECTOR + /* + * User tracing code (ptrace or signal handlers) might assume + * that the saved RAX contains a 32-bit number when we're + * invoking a 32-bit syscall. Just in case the high bits are + * nonzero, zero-extend the syscall number. (This could almost + * certainly be deleted with no ill effects.) + */ + movl %eax, %eax + + /* + * do_int80_syscall_32() expects regs->orig_ax to be user ax, + * and regs->ax to be $-ENOSYS. + */ + movq %rax, (%rsp) + movq $-ENOSYS, %rax + .endif + .if \vector == X86_TRAP_BP /* * If coming from kernel space, create a 6-word gap to allow the
arch/x86/entry/entry_64_compat.S +0 -102
105 lines not on this page · raw message
arch/x86/include/asm/idtentry.h +47 -0
65 lines not on this page · raw message
arch/x86/include/asm/proto.h +0 -4
@@ -28,10 +28,6 @@ void entry_SYSENTER_compat(void);void __end_entry_SYSENTER_compat (void );void entry_SYSCALL_compat (void );void entry_SYSCALL_compat_safe_stack (void ); -void entry_INT80_compat (void ); -#ifdef CONFIG_XEN_PV -void xen_entry_INT80_compat (void ); -#endif #endif void x86_configure_nx (void );
Signature · 1 line
2.19.1.6.gb485710b
PATCH 9/11 x86/entry: Convert SWAPGS to swapgs in entry_SYSENTER_compat()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
XENPV has its own entry point for SYSENTER, it doesn't use entry_SYSENTER_compat. So the pv-awared SWAPGS can be changed to swapgs.
--- arch/x86/entry/entry_64_compat.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/entry/entry_64_compat.S +1 -1
@@ -49,7 +49,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY /* Interrupts are off on entry. */ - SWAPGS + swapgs pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
Signature · 1 line
2.19.1.6.gb485710b
PATCH 10/11 x86: Remove the definition of SWAPGS
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
There is no user of the pv-aware SWAPGS anymore.
--- arch/x86/include/asm/irqflags.h | 2 -- 1 file changed, 2 deletions(-)
arch/x86/include/asm/irqflags.h +0 -2
@@ -140,13 +140,11 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)#else #ifdef CONFIG_X86_64 #ifdef CONFIG_XEN_PV -#define SWAPGS ALTERNATIVE "swapgs", "", X86_FEATURE_XENPV#define INTERRUPT_RETURN \ ANNOTATE_RETPOLINE_SAFE; \ ALTERNATIVE_TERNARY("jmp *paravirt_iret(%rip);", \ X86_FEATURE_XENPV, "jmp xen_iret;", "jmp native_iret;")#else -#define SWAPGS swapgs#define INTERRUPT_RETURN jmp native_iret#endif #endif
Signature · 1 line
2.19.1.6.gb485710b
PATCH 11/11 x86/entry: Remove the branch in sync_regs()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
In non-xenpv, the sp0 is the trampoline stack, and sync_regs() is called on non-xenpv only since error_entry is not called on xenpv, so the stack must be the trampoline stack or one of the IST stack and the check in sync_regs() is unneeded.
--- arch/x86/kernel/traps.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/kernel/traps.c +5 -3
@@ -708,13 +708,15 @@ DEFINE_IDTENTRY_RAW(exc_int3)/* * Help handler running on a per-cpu (IST or entry trampoline) stack * to switch to the normal thread stack if the interrupted code was in -* user mode. The actual stack switch is done in entry_64.S +* user mode. The actual stack switch is done in entry_64.S. It is not +* called from XENPV and the caller has ensured it is not on the thread +* stack. */ asmlinkage __visible noinstr struct pt_regs *sync_regs(struct pt_regs *eregs) {struct pt_regs *regs = (struct pt_regs *)this_cpu_read (cpu_current_top_of_stack) -1 ; -if (regs != eregs) - *regs = *eregs; + + *regs = *eregs;return regs; }
Signature · 1 line
2.19.1.6.gb485710b