x86/entry: Clean up entry code
Subsystems (2)
v6 cover x86/entry: Clean up entry code
From: Lai Jiangshan <[email protected]>
This patchset moves the stack-switch code to the place where error_entry() return, unravels error_entry() from XENpv and makes entry_INT80_compat() uses idtentry macro.
This patchset is highly related to XENpv, because it does the extra cleanup to convert SWAPGS to swapgs after major cleanup is done.
The patches are the 6th version to pick patches from the patchset
https://lore.kernel.org/lkml/[email protected]/
which converts ASM code to C code. These patches are prepared for that purpose. But this patchset has it own value: it simplifies the stack switch, avoids leaving the old stack inside a function call, and separates XENpv code with native code without adding new code.
Peter said in V3:
Quoted 4 lines
So AFAICT these patches are indeed correct.
I'd love for some of the other x86 people to also look at this, but a tentative ACK on this.
[V5]: https://lore.kernel.org/lkml/[email protected]/
[V4]: https://lore.kernel.org/lkml/[email protected]/
[V3]: https://lore.kernel.org/lkml/[email protected]/
[V2]: https://lore.kernel.org/lkml/[email protected]/
[V1]: https://lore.kernel.org/lkml/[email protected]/
Changed from V5:
Add a new ASM function to wrap PUSH_AND_CLEAR_REGS rather than inline it in the marco idtentry to reduce text size.
Remove the branch in sync_regs() (pick it from V1 and update the changelog a little)
Changed from V4:
Update changelog largely of patch 1 and patch 2 Update changelog slightly of other patches Unbreak the line of fixup_bad_iret() in patch1
Add Reviewed-by from Juergen Gross <[email protected]> in patch 6 since he gave the Reviewed-by in one of the squashed patches.
Changed from V3:
Only reorder the int80 thing as the last patch to make patches ordering more natural. (Both orders are correct)
Other interactions in V3:
Peter raised several questions and I think I answered them and I don't think the code need to be updated unless I missed some points. (Except reordering the patches)
Josh asked to remove UNWIND_HINT_REGS in patch5, but I think UNWIND_HINT_REGS is old code before this patchset and I don't want to do a cleanup that is not relate to preparing converting ASM code C code in this patchset. He also asked to remove ENCODE_FRAME_POINTER in xenpv case, and I think it just complicates the code for just optimizing out a single assignment to %rbp. I would not always stick to these reasons of mine, but I just keep the code unchanged since he hasn't emphasized it again nor other people has requested it.
Changed from V2:
Make the patch of folding int80 thing as the first patch Add more changelog in "Switch the stack after error_entry() returns"
Changed from V1
Squash cleanup patches converting SWAPGS to swapgs into one patch
Use my official email address (Ant Group). The work is backed by my company and I was incorrectly misunderstood that [email protected] is the only portal for opensource work in the corporate group.
Lai Jiangshan (8):
x86/traps: Move pt_regs only in fixup_bad_iret() x86/entry: Switch the stack after error_entry() returns x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry() x86/entry: Move cld to the start of idtentry macro x86/entry: Don't call error_entry() for XENPV x86/entry: Convert SWAPGS to swapgs and remove the definition of
SWAPGS
x86/entry: Remove the branch in sync_regs() x86/entry: Use idtentry macro for entry_INT80_compat()
arch/x86/entry/entry_64.S | 71 ++++++++++++++++----- arch/x86/entry/entry_64_compat.S | 105 +------------------------------ arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/irqflags.h | 8 --- arch/x86/include/asm/proto.h | 4 -- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 24 +++---- 7 files changed, 112 insertions(+), 149 deletions(-)
Signature · 1 line
2.19.1.6.gb485710b
PATCH v6 1/8 x86/traps: Move pt_regs only in fixup_bad_iret()
From: Lai Jiangshan <[email protected]>
Always stash the address error_entry() is going to return to, in %r12 and get rid of the void *error_entry_ret; slot in struct bad_iret_stack which was supposed to account for it and pt_regs pushed on the stack.
After this, both fixup_bad_iret() and sync_regs() can work on a struct pt_regs pointer directly.
--- arch/x86/entry/entry_64.S | 5 ++++- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 18 ++++++------------ 3 files changed, 11 insertions(+), 14 deletions(-)
arch/x86/entry/entry_64.S +4 -1
@@ -1061,9 +1061,12 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - mov %rsp, %rdi + popq %r12 /* save return addr in %12 */ + movq %rsp, %rdi /* arg0 = pt_regs pointer */ call fixup_bad_iret mov %rax, %rsp + ENCODE_FRAME_POINTER + pushq %r12 jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
arch/x86/include/asm/traps.h +1 -1
@@ -13,7 +13,7 @@#ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notrace -struct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s); +struct pt_regs *fixup_bad_iret (struct pt_regs *bad_regs);void __init trap_init (void ); asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);#endif
arch/x86/kernel/traps.c +6 -12
@@ -898,13 +898,7 @@ asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *r }#endif -struct bad_iret_stack { -void *error_entry_ret; -struct pt_regs regs; -}; - -asmlinkage __visible noinstr -struct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s) +asmlinkage __visible noinstr struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs) {/* * This is called from entry_64.S early in handling a fault
@@ -914,19 +908,19 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s)* just below the IRET frame) and we want to pretend that the * exception came from the IRET target. */ -struct bad_iret_stack tmp, *new_stack = - (struct bad_iret_stack *)__this_cpu_read (cpu_tss_rw.x86_tss.sp0) -1 ; +struct pt_regs tmp, *new_stack = + (struct pt_regs *)__this_cpu_read (cpu_tss_rw.x86_tss.sp0) -1 ;/* Copy the IRET target to the temporary storage. */ -__memcpy (&tmp.regs.ip, (void *)s->regs.sp,5 *8 ); +__memcpy (&tmp.ip, (void *)bad_regs->sp,5 *8 );/* Copy the remainder of the stack from the current stack. */ - __memcpy(&tmp, s, offsetof(struct bad_iret_stack, regs.ip)); +__memcpy (&tmp, bad_regs, offsetof(struct pt_regs , ip));/* Update the entry stack */ __memcpy (new_stack, &tmp,sizeof (tmp)); -BUG_ON (!user_mode (&new_stack->regs)); +BUG_ON (!user_mode (new_stack)); return new_stack; }#endif
Signature · 1 line
2.19.1.6.gb485710b
PATCH v6 2/8 x86/entry: Switch the stack after error_entry() returns
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
error_entry() calls sync_regs(), and fixup_bad_iret() before sync_regs() if it is a fault from bad IRET, to copy the pt_regs to the kernel stack and switches the kernel stack directly after sync_regs().
But error_entry() itself is also a function call, so the code has to stash the address error_entry() is going to return to, in %r12 and makes the work complicated.
Move the code of switching stack after error_entry() and get rid of the need to handle the return address.
--- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -10
@@ -326,6 +326,8 @@ SYM_CODE_END(ret_from_fork) .macro idtentry_body cfunc has_error_code:req call error_entry + movq %rax, %rsp /* switch to the task stack if from userspace */ + ENCODE_FRAME_POINTER UNWIND_HINT_REGS movq %rsp, %rdi /* pt_regs pointer into 1st argument*/
@@ -1002,14 +1004,10 @@ SYM_CODE_START_LOCAL(error_entry) /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax + leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ .Lerror_entry_from_usermode_after_swapgs: /* Put us onto the real thread stack. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 = pt_regs pointer */ call sync_regs - movq %rax, %rsp /* switch stack */ - ENCODE_FRAME_POINTER - pushq %r12 RET /*
@@ -1041,6 +1039,7 @@ SYM_CODE_START_LOCAL(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + leaq 8(%rsp), %rax /* return pt_regs pointer */ RET .Lbstep_iret:
@@ -1061,12 +1060,9 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 = pt_regs pointer */ + leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ call fixup_bad_iret - mov %rax, %rsp - ENCODE_FRAME_POINTER - pushq %r12 + mov %rax, %rdi jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
Signature · 1 line
2.19.1.6.gb485710b
PATCH v6 3/8 x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
The macro idtentry calls error_entry() unconditionally even on XENPV. But the code XENPV needs in error_entry() is PUSH_AND_CLEAR_REGS only. And error_entry() also calls sync_regs() which has to deal with the case of XENPV via an extra branch so that it doesn't copy the pt_regs.
And PUSH_AND_CLEAR_REGS in error_entry() makes the stack not return to its original place when the function returns, which means it is not possible to convert it to a C function.
Move PUSH_AND_CLEAR_REGS out of error_entry(), add a function to wrap PUSH_AND_CLEAR_REGS and call it before error_entry().
The new function call adds two instructions (CALL and RET) for every interrupt or exception. It will allow for error_entry() to be not called on XENPV which allows for sync_regs() to reduce a branch. It will also allow for error_entry() to be converted to C code that can use inlined sync_regs() and save a function call again.
--- arch/x86/entry/entry_64.S | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64.S +12 -3
@@ -318,6 +318,14 @@ SYM_CODE_END(ret_from_fork) #endif .endm +/* Save all registers in pt_regs */ +SYM_CODE_START_LOCAL(push_and_clear_regs) + UNWIND_HINT_FUNC + PUSH_AND_CLEAR_REGS save_ret=1 + ENCODE_FRAME_POINTER 8 + RET +SYM_CODE_END(push_and_clear_regs) + /** * idtentry_body - Macro to emit code calling the C function * @cfunc: C function to be called
@@ -325,6 +333,9 @@ SYM_CODE_END(ret_from_fork) */ .macro idtentry_body cfunc has_error_code:req + call push_and_clear_regs + UNWIND_HINT_REGS + call error_entry movq %rax, %rsp /* switch to the task stack if from userspace */ ENCODE_FRAME_POINTER
@@ -985,13 +996,11 @@ SYM_CODE_START_LOCAL(paranoid_exit) SYM_CODE_END(paranoid_exit) /* - * Save all registers in pt_regs, and switch GS if needed. + * Switch GS and CR3 if needed. */ SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC cld - PUSH_AND_CLEAR_REGS save_ret=1 - ENCODE_FRAME_POINTER 8 testb $3, CS+8(%rsp) jz .Lerror_kernelspace
Signature · 1 line
2.19.1.6.gb485710b
Borislav Petkov
maintainerRe: [PATCH V6 3/8] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 6 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
The macro idtentry calls
error_entry()unconditionally even on XENPV. But the code XENPV needs inerror_entry()isPUSH_AND_CLEAR_REGSonly. Anderror_entry()also callssync_regs()which has to deal with the case of XENPV via an extra branch so that it doesn't copy thept_regs.
What extra branch?
Do you mean the
if (regs != eregs)
test in sync_regs()?
I'm confused. Are you, per chance, aiming to optimize XENPV here or what's up?
Quoted 9 lines
And
PUSH_AND_CLEAR_REGSinerror_entry()makes the stack not return to its original place when the function returns, which means it is not possible to convert it to a C function.Move
PUSH_AND_CLEAR_REGSout oferror_entry(), add a function to wrapPUSH_AND_CLEAR_REGSand call it beforeerror_entry().The new function call adds two instructions (CALL and RET) for every interrupt or exception.
Not only - it pushes all the regs in PUSH_AND_CLEAR_REGS too. I don't understand why that matters here? It was done in error_entry anyway.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH V6 3/8] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 9 lines · Borislav Petkov
Quoted 6 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
The macro idtentry calls
error_entry()unconditionally even on XENPV. But the code XENPV needs inerror_entry()isPUSH_AND_CLEAR_REGSonly. Anderror_entry()also callssync_regs()which has to deal with the case of XENPV via an extra branch so that it doesn't copy thept_regs.
Quoted 7 lines
What extra branch?
Do you mean the
if (regs != eregs)test in
sync_regs()?
Hello, Borislav
Yes.
Quoted 3 lines
I'm confused. Are you, per chance, aiming to optimize XENPV here or what's up?
The branch in sync_regs() can be optimized out for the non-XENPV case since XENPV doesn't call sync_regs() after patch5 which makes XENPV not call error_entry().
The aim of this patch and most of the patchset is to make error_entry() be able to be converted to C. And XENPV cases can also be optimized in the patchset although it is not the major main.
Quoted 5 lines
Quoted 5 lines
And
PUSH_AND_CLEAR_REGSinerror_entry()makes the stack not return to its original place when the function returns, which means it is not possible to convert it to a C function.
Quoted 9 lines
Quoted 5 lines
Move
PUSH_AND_CLEAR_REGSout oferror_entry(), add a function to wrapPUSH_AND_CLEAR_REGSand call it beforeerror_entry().The new function call adds two instructions (CALL and RET) for every interrupt or exception.
Not only - it pushes all the regs in
PUSH_AND_CLEAR_REGStoo. I don't understand why that matters here? It was done inerror_entryanyway.
Compared to the original code, adding the new function call adds two instructions (CALL and RET) for every interrupt or exception.
PUSH_AND_CLEAR_REGS is not extra instructions added here.
Since this patch adds extra overhead (CALL and RET), the changelog has to explain why it is worth it not just for converting ASM to C.
The explanation in the changelog is that it can be offsetted by later reduced overhead.
Thanks
Lai
Quoted 5 lines
--
Regards/Gruss,
Boris.
Borislav Petkov
maintainerRe: [PATCH V6 3/8] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 3 lines · Lai Jiangshan
The branch in
sync_regs()can be optimized out for the non-XENPV case since XENPV doesn't callsync_regs()after patch5 which makes XENPV not callerror_entry().
I find it confusing why you're even mentioning optimizing a simple test.
That thing is basically a *whopping* two fastpath instructions:
# arch/x86/kernel/traps.c:853: if (regs != eregs)
cmpq %rdi, %rax # eregs, <retval> je .L255 #,
Is this such a hot path that it shows in some profiles and removing that test there is so important?
Quoted 3 lines
The aim of this patch and most of the patchset is to make
error_entry()be able to be converted to C. And XENPV cases can also be optimized in the patchset although it is not the major main.
So say just that.
Quoted 2 lines
Since this patch adds extra overhead (CALL and RET), the changelog has to explain why it is worth it not just for converting ASM to C.
What is the practical relevance of this very minor overhead? Does it show even in a microbenchmark?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH V6 3/8] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
In reply to Lai JiangshanQuoted 53 lines · Lai Jiangshan
Quoted 9 lines · Borislav Petkov
Quoted 6 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
The macro idtentry calls
error_entry()unconditionally even on XENPV. But the code XENPV needs inerror_entry()isPUSH_AND_CLEAR_REGSonly. Anderror_entry()also callssync_regs()which has to deal with the case of XENPV via an extra branch so that it doesn't copy thept_regs.Quoted 7 lines
What extra branch?
Do you mean the
if (regs != eregs)test in
sync_regs()?Hello, Borislav
Yes.
Quoted 3 lines
I'm confused. Are you, per chance, aiming to optimize XENPV here or what's up?
The branch in
sync_regs()can be optimized out for the non-XENPV case since XENPV doesn't callsync_regs()after patch5 which makes XENPV not callerror_entry().The aim of this patch and most of the patchset is to make
error_entry()be able to be converted to C. And XENPV cases can also be optimized in the patchset although it is not the major main.Quoted 5 lines
Quoted 5 lines
And
PUSH_AND_CLEAR_REGSinerror_entry()makes the stack not return to its original place when the function returns, which means it is not possible to convert it to a C function.Quoted 9 lines
Quoted 5 lines
Move
PUSH_AND_CLEAR_REGSout oferror_entry(), add a function to wrapPUSH_AND_CLEAR_REGSand call it beforeerror_entry().The new function call adds two instructions (CALL and RET) for every interrupt or exception.
Not only - it pushes all the regs in
PUSH_AND_CLEAR_REGStoo. I don't understand why that matters here? It was done inerror_entryanyway.Compared to the original code, adding the new function call adds two instructions (CALL and RET) for every interrupt or exception.
Quoted 7 lines
PUSH_AND_CLEAR_REGSis not extra instructions added here.Since this patch adds extra overhead (CALL and RET), the changelog has to explain why it is worth it not just for converting ASM to C.
The explanation in the changelog is that it can be offsetted by later reduced overhead.
I think you could avoid the extra call/ret by doing something like:
SYM_CODE_START_LOCAL(error_exit_push_and_save)
UNWIND_HINT_FUNC
PUSH_AND_CLEAR_REGS save_ret=1
ENCODE_FRAME_POINTER 8
jmp error_exitSYM_CODE_END(error_exit_push_and_save)
... and use this instead of patch 5:
ALTERNATIVE "call error_entry_push_and_save; movq %rax, %rsp", \ "call push_and_clear_regs", X86_FEATURE_XENPV
Juergen
Borislav Petkov
maintainerRe: [PATCH V6 3/8] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 9 lines · Juergen Gross
I think you could avoid the extra call/ret by doing something like:
SYM_CODE_START_LOCAL(error_exit_push_and_save)
UNWIND_HINT_FUNC
PUSH_AND_CLEAR_REGSsave_ret=1
ENCODE_FRAME_POINTER8
jmperror_exitSYM_CODE_END(error_exit_push_and_save)
Quoted 4 lines
... and use this instead of patch 5:
ALTERNATIVE "call error_entry_push_and_save; movq %rax, %rsp", \ "call push_and_clear_regs", X86_FEATURE_XENPV
I'm afraid I can't follow - you still need to call error_entry on !XENPV. You have a function error_entry_push_and_save() but above it is called error_*exit*_push_and_save...
Bottomline is, this should be as simple as possible code - an additional CALL/RET doesn't really matter, perf-wise.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v6 4/8 x86/entry: Move cld to the start of idtentry macro
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
Make it next to CLAC
--- arch/x86/entry/entry_64.S | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64.S +5 -3
@@ -371,6 +371,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8 ENDBR ASM_CLAC + cld .if \has_error_code == 0 pushq $-1 /* ORIG_RAX: no syscall to restart */
@@ -439,6 +440,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld pushq $-1 /* ORIG_RAX: no syscall to restart */
@@ -495,6 +497,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld /* * If the entry is from userspace, switch stacks and treat it as
@@ -557,6 +560,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=8 ENDBR ASM_CLAC + cld /* paranoid_entry returns GS information for paranoid_exit in EBX. */ call paranoid_entry
@@ -882,7 +886,6 @@ SYM_CODE_END(xen_failsafe_callback) */ SYM_CODE_START_LOCAL(paranoid_entry) UNWIND_HINT_FUNC - cld PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -1000,7 +1003,6 @@ SYM_CODE_END(paranoid_exit) */ SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC - cld testb $3, CS+8(%rsp) jz .Lerror_kernelspace
@@ -1134,6 +1136,7 @@ SYM_CODE_START(asm_exc_nmi) */ ASM_CLAC + cld /* Use %rdx as our temp variable throughout */ pushq %rdx
@@ -1153,7 +1156,6 @@ SYM_CODE_START(asm_exc_nmi) */ swapgs - cld FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=%rdx movq %rsp, %rdx
Signature · 1 line
2.19.1.6.gb485710b
PATCH v6 5/8 x86/entry: Don't call error_entry() for XENPV
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
When in XENPV, it is already in the task stack, and it can't fault for native_iret() nor native_load_gs_index() since XENPV uses its own pvops for IRET and load_gs_index(). And it doesn't need to switch the CR3.
So there is no reason to call error_entry() in XENPV.
--- arch/x86/entry/entry_64.S | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +11 -2
@@ -336,8 +336,17 @@ SYM_CODE_END(push_and_clear_regs) call push_and_clear_regs UNWIND_HINT_REGS - call error_entry - movq %rax, %rsp /* switch to the task stack if from userspace */ + /* + * Call error_entry() and switch to the task stack if from userspace. + * + * When in XENPV, it is already in the task stack, and it can't fault + * for native_iret() nor native_load_gs_index() since XENPV uses its + * own pvops for IRET and load_gs_index(). And it doesn't need to + * switch the CR3. So it can skip invoking error_entry(). + */ + ALTERNATIVE "call error_entry; movq %rax, %rsp", \ + "", X86_FEATURE_XENPV + ENCODE_FRAME_POINTER UNWIND_HINT_REGS
Signature · 1 line
2.19.1.6.gb485710b
PATCH v6 6/8 x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
XENPV doesn't use swapgs_restore_regs_and_return_to_usermode(), error_entry() and entry_SYSENTER_compat().
Change the PV-compatible SWAPGS to the ASM instruction swapgs in these functions.
Also remove the definition of SWAPGS since there is no user of the SWAPGS anymore.
--- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 2 +- arch/x86/include/asm/irqflags.h | 8 -------- 3 files changed, 4 insertions(+), 12 deletions(-)
arch/x86/entry/entry_64.S +3 -3
@@ -1019,7 +1019,7 @@ SYM_CODE_START_LOCAL(error_entry) * We entered from user mode or we're pretending to have entered * from user mode due to an IRET fault. */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
@@ -1051,7 +1051,7 @@ SYM_CODE_START_LOCAL(error_entry) * gsbase and proceed. We'll fix up the exception and land in * .Lgs_change's error handler with kernel gsbase. */ - SWAPGS + swapgs /* * Issue an LFENCE to prevent GS speculation, regardless of whether it is a
@@ -1072,7 +1072,7 @@ SYM_CODE_START_LOCAL(error_entry) * We came from an IRET to user mode, so we have user * gsbase and CR3. Switch to kernel gsbase and CR3: */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
arch/x86/entry/entry_64_compat.S +1 -1
@@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY ENDBR /* Interrupts are off on entry. */ - SWAPGS + swapgs pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
arch/x86/include/asm/irqflags.h +0 -8
@@ -137,14 +137,6 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)if (!arch_irqs_disabled_flags (flags))arch_local_irq_enable (); } -#else -#ifdef CONFIG_X86_64 -#ifdef CONFIG_XEN_PV -#define SWAPGS ALTERNATIVE "swapgs", "", X86_FEATURE_XENPV -#else -#define SWAPGS swapgs -#endif -#endif #endif /* !__ASSEMBLY__ */ #endif
Signature · 1 line
2.19.1.6.gb485710b
Borislav Petkov
maintainerRe: [PATCH V6 6/8] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
arch/x86/entry/entry_64_compat.S SYM_CODE_START(entry_SYSENTER_compat) · line 56
Quoted 13 lines · Lai Jiangshan
arch/x86/entry/entry_64_compat.S +1
-1@@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY ENDBR /* Interrupts are off on entry. */ - SWAPGS + swapgs pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
I'm not sure about this: why can't XENPV do a 32-bit syscall through the vdso?
Also, looking at this, Jürgen, it looks kinda spaghetti to me:
entry_SYSENTER_compat
...
/* XEN PV guests always use IRET path */
ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \
"jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPVthen at that swapgs_restore_regs_and_return_to_usermode label:
#ifdef CONFIG_XEN_PV
ALTERNATIVE "", "jmp xenpv_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV
#endif
Can we simply jump directly to xenpv_restore_regs_and_return_to_usermode from entry_SYSENTER_compat or is that CONFIG_DEBUG_ENTRY chunk there needed?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH V6 6/8] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
Quoted 31 lines · Borislav Petkov
arch/x86/entry/entry_64_compat.S SYM_CODE_START(entry_SYSENTER_compat) · line 56
Quoted 13 lines · Lai Jiangshan
arch/x86/entry/entry_64_compat.S +1
-1@@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY ENDBR /* Interrupts are off on entry. */ - SWAPGS + swapgs pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=%raxI'm not sure about this: why can't XENPV do a 32-bit syscall through the vdso?
Also, looking at this, Jürgen, it looks kinda spaghetti to me:
entry_SYSENTER_compat...
/* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPVthen at that
swapgs_restore_regs_and_return_to_usermodelabel:
Quoted 8 lines
#ifdef
CONFIG_XEN_PV
ALTERNATIVE "", "jmpxenpv_restore_regs_and_return_to_usermode",X86_FEATURE_XENPV
#endifCan we simply jump directly to
xenpv_restore_regs_and_return_to_usermodefromentry_SYSENTER_compator is thatCONFIG_DEBUG_ENTRYchunk there needed?
Hello
Adding ALTERNATIVE in swapgs_restore_regs_and_return_to_usermode() results a simpler patch which is better served as a bug fix:
https://lore.kernel.org/lkml/163861832551.11128.1645285137833652414.tip-bot2@tip-bot2/
which is also your suggestion:
https://lore.kernel.org/lkml/[email protected]/
While the original patch has sprinkled all those ALTERNATIVE calls everywhere:
https://lore.kernel.org/lkml/[email protected]/
And this original version has a SYM_L_WEAK placeholder which is objected to by some people.
Thanks
Lai
Borislav Petkov
maintainerRe: [PATCH V6 6/8] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
Quoted 1 line · Lai Jiangshan
Adding ALTERNATIVE in
swapgs_restore_regs_and_return_to_usermode()
All I'm saying is to do this:
arch/x86/entry/entry_64_compat.S +1 -1
@@ -140,7 +140,7 @@ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) call do_SYSENTER_32 /* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ - "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV + "jmp xenpv_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV jmp sysret32_from_system_call .Lsysenter_fix_flags:
which eliminates one JMP in the XENPV case if the CONFIG_DEBUG_ENTRY thing is not relevant to XENPV.
But let's see what Jürgen says first...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH V6 6/8] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
In reply to Borislav PetkovQuoted 12 lines · Borislav Petkov
Quoted 11 lines · Lai Jiangshan
arch/x86/entry/entry_64_compat.S +1
-1@@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY ENDBR /* Interrupts are off on entry. */ - SWAPGS + swapgs
Quoted 4 lines
arch/x86/entry/entry_64_compat.S SYM_CODE_START(entry_SYSENTER_compat) · line 56
Quoted 2 lines
arch/x86/entry/entry_64_compat.S +0
-0pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=%raxI'm not sure about this: why can't XENPV do a 32-bit syscall through the vdso?
The syscall will land in the hypervisor, which then will activate the related registered callback (xen_sysenter_target).
Quoted 13 lines
Also, looking at this, Jürgen, it looks kinda spaghetti to me:
entry_SYSENTER_compat...
/* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPVthen at that
swapgs_restore_regs_and_return_to_usermodelabel:
Quoted 8 lines
#ifdef
CONFIG_XEN_PV
ALTERNATIVE "", "jmpxenpv_restore_regs_and_return_to_usermode",X86_FEATURE_XENPV
#endifCan we simply jump directly to
xenpv_restore_regs_and_return_to_usermodefromentry_SYSENTER_compator is thatCONFIG_DEBUG_ENTRYchunk there needed?
I wouldn't insist on the CONFIG_DEBUG_ENTRY chunk.
Juergen
Borislav Petkov
maintainerRe: [PATCH V6 6/8] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
Quoted 2 lines · Juergen Gross
The syscall will land in the hypervisor, which then will activate the related registered callback (
xen_sysenter_target).
Aha, that would do a bit of a fixup and then go to the common label entry_SYSENTER_compat_after_hwframe.
We probably should document this so that it is clear what uses those...
---
arch/x86/entry/entry_64.S +5 -0
@@ -103,6 +103,11 @@ SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) pushq %r11 /* pt_regs->flags */ pushq $__USER_CS /* pt_regs->cs */ pushq %rcx /* pt_regs->ip */ + + /* + * XENPV jumps here after frame fixup in the respective entry points in + * ...xen-asm.S. + */ SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL) pushq %rax /* pt_regs->orig_ax */
arch/x86/entry/entry_64_compat.S +5 -0
@@ -71,6 +71,11 @@ SYM_CODE_START(entry_SYSENTER_compat) pushfq /* pt_regs->flags (except IF = 0) */ pushq $__USER32_CS /* pt_regs->cs */ pushq $0 /* pt_regs->ip = 0 (placeholder) */ + + /* + * XENPV jumps here after frame fixup in the respective entry points in + * ...xen-asm.S. + */ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) /*
Quoted 1 line
I wouldn't insist on the
CONFIG_DEBUG_ENTRYchunk.
Ok, simpler asm, cool, let's do it then.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteBorislav Petkov
maintainerRe: [PATCH V6 6/8] x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
Quoted 1 line · Borislav Petkov
Ok, simpler asm, cool, let's do it then.
Yeah, just tried it, it needs more ifdeffery:
ld: arch/x86/entry/entry_64.o:/home/boris/kernel/linux/arch/x86/entry/entry_64.S:129: undefined reference to `xenpv_restore_regs_and_return_to_usermode'
ld: arch/x86/entry/entry_64_compat.o:(.altinstr_replacement+0x1): undefined reference to `xenpv_restore_regs_and_return_to_usermode'
make: *** [Makefile:1158: vmlinux] Error 1
Oh well.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v6 7/8 x86/entry: Remove the branch in sync_regs()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
In non-XENPV, the tss.sp0 is always the trampoline stack, and sync_regs() is called on non-XENPV only since error_entry() is not called on XENPV, so the stack must be the trampoline stack or one of the IST stack if it is from user mode or bad iret.
Remove the check in sync_regs() and always copy the pt_regs from the stack in the cpu entry area to the task stack.
--- arch/x86/kernel/traps.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/traps.c +3 -3
@@ -845,13 +845,13 @@ DEFINE_IDTENTRY_RAW(exc_int3)/* * Help handler running on a per-cpu (IST or entry trampoline) stack * to switch to the normal thread stack if the interrupted code was in -* user mode. The actual stack switch is done in entry_64.S +* user mode. The actual stack switch is done in entry_64.S. */ asmlinkage __visible noinstr struct pt_regs *sync_regs(struct pt_regs *eregs) {struct pt_regs *regs = (struct pt_regs *)this_cpu_read (cpu_current_top_of_stack) -1 ; -if (regs != eregs) - *regs = *eregs; + + *regs = *eregs;return regs; }
Signature · 1 line
2.19.1.6.gb485710b
PATCH v6 8/8 x86/entry: Use idtentry macro for entry_INT80_compat()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
entry_INT80_compat() is identical to idtentry macro except a special handling for %rax in the prolog.
Add the prolog to idtentry and use idtentry for entry_INT80_compat().
--- arch/x86/entry/entry_64.S | 18 ++++++ arch/x86/entry/entry_64_compat.S | 103 ------------------------------- arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/proto.h | 4 -- 4 files changed, 65 insertions(+), 107 deletions(-)
arch/x86/entry/entry_64.S +18 -0
@@ -386,6 +386,24 @@ SYM_CODE_START(\asmsym) pushq $-1 /* ORIG_RAX: no syscall to restart */ .endif + .if \vector == IA32_SYSCALL_VECTOR + /* + * User tracing code (ptrace or signal handlers) might assume + * that the saved RAX contains a 32-bit number when we're + * invoking a 32-bit syscall. Just in case the high bits are + * nonzero, zero-extend the syscall number. (This could almost + * certainly be deleted with no ill effects.) + */ + movl %eax, %eax + + /* + * do_int80_syscall_32() expects regs->orig_ax to be user ax, + * and regs->ax to be $-ENOSYS. + */ + movq %rax, (%rsp) + movq $-ENOSYS, %rax + .endif + .if \vector == X86_TRAP_BP /* * If coming from kernel space, create a 6-word gap to allow the
arch/x86/entry/entry_64_compat.S +0 -103
106 lines not on this page · raw message
arch/x86/include/asm/idtentry.h +47 -0
65 lines not on this page · raw message
arch/x86/include/asm/proto.h +0 -4
@@ -28,10 +28,6 @@ void entry_SYSENTER_compat(void);void __end_entry_SYSENTER_compat (void );void entry_SYSCALL_compat (void );void entry_SYSCALL_compat_safe_stack (void ); -void entry_INT80_compat (void ); -#ifdef CONFIG_XEN_PV -void xen_entry_INT80_compat (void ); -#endif #endif void x86_configure_nx (void );
Signature · 1 line
2.19.1.6.gb485710b