x86/entry: Clean up entry code
Subsystems (2)
v5 cover x86/entry: Clean up entry code
From: Lai Jiangshan <[email protected]>
This patchset moves the stack-switch code to the place where error_entry() return, unravels error_entry() from XENpv and makes entry_INT80_compat() uses idtentry macro.
This patchset is highly related to XENpv, because it does the extra cleanup to convert SWAPGS to swapgs after major cleanup is done.
The patches are the 5th version to pick patches from the patchset
https://lore.kernel.org/lkml/[email protected]/
which converts ASM code to C code. These patches are prepared for that purpose. But this patchset has it own value: it simplifies the stack switch, avoids leaving the old stack inside a function call, and separates XENpv code with native code without adding new code.
Peter said in V3:
Quoted 4 lines
So AFAICT these patches are indeed correct.
I'd love for some of the other x86 people to also look at this, but a tentative ACK on this.
[V4]: https://lore.kernel.org/lkml/[email protected]/
[V3]: https://lore.kernel.org/lkml/[email protected]/
[V2]: https://lore.kernel.org/lkml/[email protected]/
[V1]: https://lore.kernel.org/lkml/[email protected]/
Changed from V4:
Update changelog largely of patch 1 and patch 2 Update changelog slightly of other patches Unbreak the line of fixup_bad_iret() in patch1
Add Reviewed-by from Juergen Gross <[email protected]> in patch 6 since he gave the Reviewed-by in one of the squashed patches.
Changed from V3:
Only reorder the int80 thing as the last patch to make patches ordering more natural. (Both orders are correct)
Other interactions in V3:
Peter raised several questions and I think I answered them and I don't think the code need to be updated unless I missed some points. (Except reordering the patches)
Josh asked to remove UNWIND_HINT_REGS in patch5, but I think UNWIND_HINT_REGS is old code before this patchset and I don't want to do a cleanup that is not relate to preparing converting ASM code C code in this patchset. He also asked to remove ENCODE_FRAME_POINTER in xenpv case, and I think it just complicates the code for just optimizing out a single assignment to %rbp. I would not always stick to these reasons of mine, but I just keep the code unchanged since he hasn't emphasized it again nor other people has requested it.
Changed from V2:
Make the patch of folding int80 thing as the first patch Add more changelog in "Switch the stack after error_entry() returns"
Changed from V1
Squash cleanup patches converting SWAPGS to swapgs into one patch
Use my official email address (Ant Group). The work is backed by my company and I was incorrectly misunderstood that [email protected] is the only portal for opensource work in the corporate group.
Lai Jiangshan (7):
x86/traps: Move pt_regs only in fixup_bad_iret() x86/entry: Switch the stack after error_entry() returns x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry() x86/entry: Move cld to the start of idtentry macro x86/entry: Don't call error_entry() for XENPV x86/entry: Convert SWAPGS to swapgs and remove the definition of
SWAPGS
x86/entry: Use idtentry macro for entry_INT80_compat
arch/x86/entry/entry_64.S | 61 +++++++++++++----- arch/x86/entry/entry_64_compat.S | 105 +------------------------------ arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/irqflags.h | 8 --- arch/x86/include/asm/proto.h | 4 -- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 18 ++---- 7 files changed, 100 insertions(+), 145 deletions(-)
Signature · 1 line
2.19.1.6.gb485710b
PATCH v5 1/7 x86/traps: Move pt_regs only in fixup_bad_iret()
From: Lai Jiangshan <[email protected]>
Always stash the address error_entry() is going to return to, in %r12 and get rid of the void *error_entry_ret; slot in struct bad_iret_stack which was supposed to account for it and pt_regs pushed on the stack.
After this, both fixup_bad_iret() and sync_regs() can work on a struct pt_regs pointer directly.
--- arch/x86/entry/entry_64.S | 5 ++++- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/traps.c | 18 ++++++------------ 3 files changed, 11 insertions(+), 14 deletions(-)
arch/x86/entry/entry_64.S +4 -1
@@ -1058,9 +1058,12 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - mov %rsp, %rdi + popq %r12 /* save return addr in %12 */ + movq %rsp, %rdi /* arg0 = pt_regs pointer */ call fixup_bad_iret mov %rax, %rsp + ENCODE_FRAME_POINTER + pushq %r12 jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
arch/x86/include/asm/traps.h +1 -1
@@ -13,7 +13,7 @@#ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notrace -struct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s); +struct pt_regs *fixup_bad_iret (struct pt_regs *bad_regs);void __init trap_init (void ); asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);#endif
arch/x86/kernel/traps.c +6 -12
@@ -898,13 +898,7 @@ asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *r }#endif -struct bad_iret_stack { -void *error_entry_ret; -struct pt_regs regs; -}; - -asmlinkage __visible noinstr -struct bad_iret_stack *fixup_bad_iret (struct bad_iret_stack *s) +asmlinkage __visible noinstr struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs) {/* * This is called from entry_64.S early in handling a fault
@@ -914,19 +908,19 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s)* just below the IRET frame) and we want to pretend that the * exception came from the IRET target. */ -struct bad_iret_stack tmp, *new_stack = - (struct bad_iret_stack *)__this_cpu_read (cpu_tss_rw.x86_tss.sp0) -1 ; +struct pt_regs tmp, *new_stack = + (struct pt_regs *)__this_cpu_read (cpu_tss_rw.x86_tss.sp0) -1 ;/* Copy the IRET target to the temporary storage. */ -__memcpy (&tmp.regs.ip, (void *)s->regs.sp,5 *8 ); +__memcpy (&tmp.ip, (void *)bad_regs->sp,5 *8 );/* Copy the remainder of the stack from the current stack. */ - __memcpy(&tmp, s, offsetof(struct bad_iret_stack, regs.ip)); +__memcpy (&tmp, bad_regs, offsetof(struct pt_regs , ip));/* Update the entry stack */ __memcpy (new_stack, &tmp,sizeof (tmp)); -BUG_ON (!user_mode (&new_stack->regs)); +BUG_ON (!user_mode (new_stack)); return new_stack; }#endif
Signature · 1 line
2.19.1.6.gb485710b
PATCH v5 2/7 x86/entry: Switch the stack after error_entry() returns
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
error_entry() calls sync_regs(), and fixup_bad_iret() before sync_regs() if it is a fault from bad IRET, to copy the pt_regs to the kernel stack and switches the kernel stack directly after sync_regs().
But error_entry() itself is also a function call, so the code has to stash the address error_entry() is going to return to, in %r12 and makes the work complicated.
Move the code of switching stack after error_entry() and get rid of the need to handle the return address.
--- arch/x86/entry/entry_64.S | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -10
@@ -326,6 +326,8 @@ SYM_CODE_END(ret_from_fork) .macro idtentry_body cfunc has_error_code:req call error_entry + movq %rax, %rsp /* switch to the task stack if from userspace */ + ENCODE_FRAME_POINTER UNWIND_HINT_REGS movq %rsp, %rdi /* pt_regs pointer into 1st argument*/
@@ -999,14 +1001,10 @@ SYM_CODE_START_LOCAL(error_entry) /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax + leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ .Lerror_entry_from_usermode_after_swapgs: /* Put us onto the real thread stack. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 = pt_regs pointer */ call sync_regs - movq %rax, %rsp /* switch stack */ - ENCODE_FRAME_POINTER - pushq %r12 RET /*
@@ -1038,6 +1036,7 @@ SYM_CODE_START_LOCAL(error_entry) */ .Lerror_entry_done_lfence: FENCE_SWAPGS_KERNEL_ENTRY + leaq 8(%rsp), %rax /* return pt_regs pointer */ RET .Lbstep_iret:
@@ -1058,12 +1057,9 @@ SYM_CODE_START_LOCAL(error_entry) * Pretend that the exception came from user mode: set up pt_regs * as if we faulted immediately after IRET. */ - popq %r12 /* save return addr in %12 */ - movq %rsp, %rdi /* arg0 = pt_regs pointer */ + leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ call fixup_bad_iret - mov %rax, %rsp - ENCODE_FRAME_POINTER - pushq %r12 + mov %rax, %rdi jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
Signature · 1 line
2.19.1.6.gb485710b
PATCH v5 3/7 x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
error_entry() doesn't handle the stack balanced. It includes PUSH_AND_CLEAR_REGS which is commonly needed for all IDT entries and can't pop the regs before it returns.
Move PUSH_AND_CLEAR_REGS out of error_entry() and make error_entry() works on the stack normally.
After this, XENPV doesn't need error_entry() since PUSH_AND_CLEAR_REGS is moved out and error_entry() can be converted to C code in future since it doesn't fiddle the stack.
The text size will be enlarged:
size arch/x86/entry/entry_64.o.before:
text data bss dec hex filename
17916 384 0 18300 477c arch/x86/entry/entry_64.o
size --format=SysV arch/x86/entry/entry_64.o.before:
.entry.text 5528 0 .orc_unwind 6456 0 .orc_unwind_ip 4304 0
size arch/x86/entry/entry_64.o.after:
text data bss dec hex filename
26868 384 0 27252 6a74 arch/x86/entry/entry_64.o
size --format=SysV arch/x86/entry/entry_64.o.after:
.entry.text 8200 0 .orc_unwind 10224 0 .orc_unwind_ip 6816 0
The tables .orc_unwind[_ip] are enlarged due to it adds many pushes.
But .entry.text in x86_64 is 2M aligned, enlarging it to 8.2k doesn't enlarge the final text size.
And it will only increase the footprint when different interrupts and exceptions happen unlikely heavily at the same time.
--- arch/x86/entry/entry_64.S | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +3 -2
@@ -325,6 +325,9 @@ SYM_CODE_END(ret_from_fork) */ .macro idtentry_body cfunc has_error_code:req + PUSH_AND_CLEAR_REGS + ENCODE_FRAME_POINTER + call error_entry movq %rax, %rsp /* switch to the task stack if from userspace */ ENCODE_FRAME_POINTER
@@ -987,8 +990,6 @@ SYM_CODE_END(paranoid_exit) SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC cld - PUSH_AND_CLEAR_REGS save_ret=1 - ENCODE_FRAME_POINTER 8 testb $3, CS+8(%rsp) jz .Lerror_kernelspace
Signature · 1 line
2.19.1.6.gb485710b
Borislav Petkov
maintainerRe: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 3 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
error_entry()doesn't handle the stack balanced.
What does that mean?
Quoted 10 lines
It includes
PUSH_AND_CLEAR_REGSwhich is commonly needed for all IDT entries and can't pop the regs before it returns.Move
PUSH_AND_CLEAR_REGSout oferror_entry()and makeerror_entry()works on the stack normally.After this, XENPV doesn't need
error_entry()sincePUSH_AND_CLEAR_REGSis moved out anderror_entry()can be converted to C code in future since it doesn't fiddle the stack.
This is not a justification for this size increase:
text data bss dec hex filename
16060616 128131358 36384888 180576862 ac3625e vmlinux.before 16065626 128131358 36380792 180577776 ac365f0 vmlinux.after
~5K text increase already with my tailored config.
You can have a asm_error_entry(), written in asm, which does the regs pushing and which calls error_entry() - the latter being the C version. And no need for the size increase.
For example.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
error_entry()doesn't handle the stack balanced.What does that mean?
For a normal function, e.g. a function complied from a C function, the stack will return to its original place when the function returns. The size of memory pushed and popped are the same in a function.
Quoted 15 lines
Quoted 11 lines
It includes
PUSH_AND_CLEAR_REGSwhich is commonly needed for all IDT entries and can't pop the regs before it returns.Move
PUSH_AND_CLEAR_REGSout oferror_entry()and makeerror_entry()works on the stack normally.After this, XENPV doesn't need
error_entry()sincePUSH_AND_CLEAR_REGSis moved out anderror_entry()can be converted to C code in future since it doesn't fiddle the stack.This is not a justification for this size increase:
text data bss dec hex filename
Quoted 8 lines
16060616 128131358 36384888 180576862 ac3625e vmlinux.before 16065626 128131358 36380792 180577776 ac365f0 vmlinux.after~5K text increase already with my tailored config.
You can have a
asm_error_entry(), written in asm, which does the regs pushing and which callserror_entry()- the latter being the C version. And no need for the size increase.
The mapped size for the text is always 2M when the kernel is booted since it is 2M-aligned. So I don't think the size is a concern.
The only concern is the footprint when different interrupts and exceptions happen heavily at the same time. In this case, different copies of PUSH_AND_CLEAR_REGS in the text will be touched.
For example, a heavy page fault and IPI or timer at the same time. I'm not sure if it is a real case.
I'm Okay with asm_error_entry(). And also we can use an ASM function containing PUSH_AND_CLEAR_REGS only.
Thanks
Lai
Borislav Petkov
maintainerRe: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 2 lines · Lai Jiangshan
The mapped size for the text is always 2M when the kernel is booted since it is 2M-aligned. So I don't think the size is a concern.
This is not how this argumentation works: you add 0.01% size increase here, 0.01% slowdown there and it all soon adds up into a bloated kernel. Besides, nowadays the kernel runs a lot as a guest and guest kernel size does matter.
So no, just because you want to turn error_entry() into C code and for that you'll add some bloat doesn't justify it in my book. All changes to core code need to be absolutely needed and justifiable.
Quoted 2 lines
For example, a heavy page fault and IPI or timer at the same time. I'm not sure if it is a real case.
I wouldn't be surprised if it shows in some perf profiles.
Quoted 2 lines
I'm Okay with
asm_error_entry(). And also we can use an ASM function containingPUSH_AND_CLEAR_REGSonly.
Just do the necessary and minimal thing - this is core code and in asm for that matter, so be conservative pls. This is not some random driver.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 9 lines · Borislav Petkov
Quoted 2 lines · Lai Jiangshan
The mapped size for the text is always 2M when the kernel is booted since it is 2M-aligned. So I don't think the size is a concern.
This is not how this argumentation works: you add 0.01% size increase here, 0.01% slowdown there and it all soon adds up into a bloated kernel. Besides, nowadays the kernel runs a lot as a guest and guest kernel size does matter.
I agree with you.
I meant the real .entry.text size is unchanged. No matter what the complied .entry.text size is, 5.5k before the patch, or 8.2k after the patch, the size of .entry.text in the kernel is always 2M.
But the overall size is enlarged which is a concern.
Quoted 8 lines
So no, just because you want to turn
error_entry()into C code and for that you'll add some bloat doesn't justify it in my book. All changes to core code need to be absolutely needed and justifiable.Quoted 2 lines
For example, a heavy page fault and IPI or timer at the same time. I'm not sure if it is a real case.
Quoted 8 lines
I wouldn't be surprised if it shows in some perf profiles.
Quoted 2 lines
I'm Okay with
asm_error_entry(). And also we can use an ASM function containingPUSH_AND_CLEAR_REGSonly.Just do the necessary and minimal thing - this is core code and in asm for that matter, so be conservative pls. This is not some random driver.
I will follow your instructions and add an extra ASM function.
And maybe add this to the changelog:
A function call is added for every interrupt or exception, but it is a low-level asm function call and actually, only a "call" and a "ret" instructions are added. It will enable error_entry() to be C code that can use inlined sync_regs() and save a C function call which will reduce more instructions in return.
I'm also waiting for reviews for other patches and I'm not going to send this new code too soon since I don't want to send a new revision with only this patch updated.
Thanks
Lai
Borislav Petkov
maintainerRe: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()
Quoted 3 lines · Lai Jiangshan
I meant the real .entry.text size is unchanged. No matter what the complied .entry.text size is, 5.5k before the patch, or 8.2k after the patch, the size of .entry.text in the kernel is always 2M.
Yes, unless the bloat I'm talking about above continues and then it becomes 4M. So we need to be careful.
Quoted 1 line
I will follow your instructions and add an extra ASM function.
Thx.
Quoted 3 lines
And maybe add this to the changelog:
A function call is added for every interrupt or exception, but it
I think you need to read submitting-patches.rst again:
"Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."
In your case:
"Add a function call..."
Quoted 2 lines
is a low-level asm function call and actually, only a "call" and a "ret" instructions are added.
CALL, RET - all x86 insns in caps.
Quoted 1 line
It will enable
error_entry()to be
It will allow for sync_regs() to be converted to C...
Quoted 6 lines
C code that can use inlined
sync_regs()and save a C function call which will reduce more instructions in return.I'm also waiting for reviews for other patches and I'm not going to send this new code too soon since I don't want to send a new revision with only this patch updated.
Again, submitting-patches.rst:
"After you have submitted your change, be patient and wait. Reviewers are busy people and may not get to your patch right away.
...
You should receive comments within a week or so; if that does not happen, make sure that you have sent your patches to the right place. Wait for a minimum of one week before resubmitting or pinging reviewers
- possibly longer during busy times like merge windows"
So you send, wait for feedback, incorporate it and resend after a week, but not earlier.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v5 4/7 x86/entry: Move cld to the start of idtentry macro
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
Make it next to CLAC
--- arch/x86/entry/entry_64.S | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64.S +5 -3
@@ -360,6 +360,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8 ENDBR ASM_CLAC + cld .if \has_error_code == 0 pushq $-1 /* ORIG_RAX: no syscall to restart */
@@ -428,6 +429,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld pushq $-1 /* ORIG_RAX: no syscall to restart */
@@ -484,6 +486,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS ENDBR ASM_CLAC + cld /* * If the entry is from userspace, switch stacks and treat it as
@@ -546,6 +549,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=8 ENDBR ASM_CLAC + cld /* paranoid_entry returns GS information for paranoid_exit in EBX. */ call paranoid_entry
@@ -871,7 +875,6 @@ SYM_CODE_END(xen_failsafe_callback) */ SYM_CODE_START_LOCAL(paranoid_entry) UNWIND_HINT_FUNC - cld PUSH_AND_CLEAR_REGS save_ret=1 ENCODE_FRAME_POINTER 8
@@ -989,7 +992,6 @@ SYM_CODE_END(paranoid_exit) */ SYM_CODE_START_LOCAL(error_entry) UNWIND_HINT_FUNC - cld testb $3, CS+8(%rsp) jz .Lerror_kernelspace
@@ -1123,6 +1125,7 @@ SYM_CODE_START(asm_exc_nmi) */ ASM_CLAC + cld /* Use %rdx as our temp variable throughout */ pushq %rdx
@@ -1142,7 +1145,6 @@ SYM_CODE_START(asm_exc_nmi) */ swapgs - cld FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=%rdx movq %rsp, %rdx
Signature · 1 line
2.19.1.6.gb485710b
PATCH v5 5/7 x86/entry: Don't call error_entry() for XENPV
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
When in XENPV, it is already in the task stack, and it can't fault for native_iret() nor native_load_gs_index() since XENPV uses its own pvops for IRET and load_gs_index(). And it doesn't need to switch the CR3.
So there is no reason to call error_entry() in XENPV.
--- arch/x86/entry/entry_64.S | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +11 -2
@@ -328,8 +328,17 @@ SYM_CODE_END(ret_from_fork) PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER - call error_entry - movq %rax, %rsp /* switch to the task stack if from userspace */ + /* + * Call error_entry() and switch to the task stack if from userspace. + * + * When in XENPV, it is already in the task stack, and it can't fault + * for native_iret() nor native_load_gs_index() since XENPV uses its + * own pvops for IRET and load_gs_index(). And it doesn't need to + * switch the CR3. So it can skip invoking error_entry(). + */ + ALTERNATIVE "call error_entry; movq %rax, %rsp", \ + "", X86_FEATURE_XENPV + ENCODE_FRAME_POINTER UNWIND_HINT_REGS
Signature · 1 line
2.19.1.6.gb485710b
Borislav Petkov
maintainerRe: [PATCH V5 5/7] x86/entry: Don't call error_entry() for XENPV
Quoted 28 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
When in XENPV, it is already in the task stack, and it can't fault for
native_iret()nornative_load_gs_index()since XENPV uses its own pvops for IRET andload_gs_index(). And it doesn't need to switch the CR3.So there is no reason to call
error_entry()in XENPV.Signed-off-by Lai Jiangshan <[email protected]>--- arch/x86/entry/entry_64.S | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)arch/x86/entry/entry_64.S +4
-2@@ -328,8 +328,17 @@ SYM_CODE_END(ret_from_fork) PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER - call error_entry - movq %rax, %rsp /* switch to the task stack if from userspace */ + /* + * Call error_entry() and switch to the task stack if from userspace. + * + * When in XENPV, it is already in the task stack, and it can't fault
arch/x86/entry/entry_64.S SYM_CODE_END(ret_from_fork) · line 344
Quoted 10 lines
arch/x86/entry/entry_64.S +7
-0+ * for native_iret() nor native_load_gs_index() since XENPV uses its + * own pvops for IRET and load_gs_index(). And it doesn't need to + * switch the CR3. So it can skip invoking error_entry(). + */ + ALTERNATIVE "call error_entry; movq %rax, %rsp", \ + "", X86_FEATURE_XENPV + ENCODE_FRAME_POINTER UNWIND_HINT_REGS
This one needs Jürgen. CCed.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v5 6/7 x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
XENPV doesn't use swapgs_restore_regs_and_return_to_usermode(), error_entry() and entry_SYSENTER_compat().
Change the PV-compatible SWAPGS to the ASM instruction swapgs in these functions.
Also remove the definition of SWAPGS since there is no user of the SWAPGS anymore.
--- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 2 +- arch/x86/include/asm/irqflags.h | 8 -------- 3 files changed, 4 insertions(+), 12 deletions(-)
arch/x86/entry/entry_64.S +3 -3
@@ -1008,7 +1008,7 @@ SYM_CODE_START_LOCAL(error_entry) * We entered from user mode or we're pretending to have entered * from user mode due to an IRET fault. */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY /* We have user CR3. Change to kernel CR3. */ SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
@@ -1040,7 +1040,7 @@ SYM_CODE_START_LOCAL(error_entry) * gsbase and proceed. We'll fix up the exception and land in * .Lgs_change's error handler with kernel gsbase. */ - SWAPGS + swapgs /* * Issue an LFENCE to prevent GS speculation, regardless of whether it is a
@@ -1061,7 +1061,7 @@ SYM_CODE_START_LOCAL(error_entry) * We came from an IRET to user mode, so we have user * gsbase and CR3. Switch to kernel gsbase and CR3: */ - SWAPGS + swapgs FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
arch/x86/entry/entry_64_compat.S +1 -1
@@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat) UNWIND_HINT_EMPTY ENDBR /* Interrupts are off on entry. */ - SWAPGS + swapgs pushq %rax SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
arch/x86/include/asm/irqflags.h +0 -8
@@ -137,14 +137,6 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)if (!arch_irqs_disabled_flags (flags))arch_local_irq_enable (); } -#else -#ifdef CONFIG_X86_64 -#ifdef CONFIG_XEN_PV -#define SWAPGS ALTERNATIVE "swapgs", "", X86_FEATURE_XENPV -#else -#define SWAPGS swapgs -#endif -#endif #endif /* !__ASSEMBLY__ */ #endif
Signature · 1 line
2.19.1.6.gb485710b
PATCH v5 7/7 x86/entry: Use idtentry macro for entry_INT80_compat
In reply to Lai JiangshanFrom: Lai Jiangshan <[email protected]>
entry_INT80_compat is identical to idtentry macro except a special handling for %rax in the prolog.
Add the prolog to idtentry and use idtentry for entry_INT80_compat.
--- arch/x86/entry/entry_64.S | 18 ++++++ arch/x86/entry/entry_64_compat.S | 103 ------------------------------- arch/x86/include/asm/idtentry.h | 47 ++++++++++++++ arch/x86/include/asm/proto.h | 4 -- 4 files changed, 65 insertions(+), 107 deletions(-)
arch/x86/entry/entry_64.S +18 -0
@@ -375,6 +375,24 @@ SYM_CODE_START(\asmsym) pushq $-1 /* ORIG_RAX: no syscall to restart */ .endif + .if \vector == IA32_SYSCALL_VECTOR + /* + * User tracing code (ptrace or signal handlers) might assume + * that the saved RAX contains a 32-bit number when we're + * invoking a 32-bit syscall. Just in case the high bits are + * nonzero, zero-extend the syscall number. (This could almost + * certainly be deleted with no ill effects.) + */ + movl %eax, %eax + + /* + * do_int80_syscall_32() expects regs->orig_ax to be user ax, + * and regs->ax to be $-ENOSYS. + */ + movq %rax, (%rsp) + movq $-ENOSYS, %rax + .endif + .if \vector == X86_TRAP_BP /* * If coming from kernel space, create a 6-word gap to allow the
arch/x86/entry/entry_64_compat.S +0 -103
106 lines not on this page · raw message
arch/x86/include/asm/idtentry.h +47 -0
65 lines not on this page · raw message
arch/x86/include/asm/proto.h +0 -4
@@ -28,10 +28,6 @@ void entry_SYSENTER_compat(void);void __end_entry_SYSENTER_compat (void );void entry_SYSCALL_compat (void );void entry_SYSCALL_compat_safe_stack (void ); -void entry_INT80_compat (void ); -#ifdef CONFIG_XEN_PV -void xen_entry_INT80_compat (void ); -#endif #endif void x86_configure_nx (void );
Signature · 1 line
2.19.1.6.gb485710b
Re: [PATCH V5 7/7] x86/entry: Use idtentry macro for entry_INT80_compat
Quoted 4 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
entry_INT80_compatis identical to idtentry macro except a special handling for %rax in the prolog.
Seriously?
arch/x86/entry/entry_64_compat.S sysret32_from_system_call: · line 386
Quoted 2 lines
arch/x86/entry/entry_64_compat.S +0
-2- pushq %rsi /* pt_regs->si */ - xorl %esi, %esi /* nospec si */
esi is not cleared in CLEAR_REGS. So much for identical.
Thanks,
tglx
Re: [PATCH V5 7/7] x86/entry: Use idtentry macro for entry_INT80_compat
Quoted 7 lines · Thomas Gleixner
Quoted 4 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
entry_INT80_compatis identical to idtentry macro except a special handling for %rax in the prolog.
Quoted 8 lines
Seriously?
arch/x86/entry/entry_64_compat.S sysret32_from_system_call: · line 386
Quoted 2 lines
arch/x86/entry/entry_64_compat.S +0
-2- pushq %rsi /* pt_regs->si */ - xorl %esi, %esi /* nospec si */esi is not cleared in
CLEAR_REGS. So much for identical.
Hello, Thomas
Thank you for the review.
They (the old entry_INT80_compat() and the new using the macro idtentry) are not identical in ASM code.
The macro idtentry pushes %rsi to the entry stack and then copies it from the entry stack to the kernel stack and then switches the stack.
The original entry_INT80_compat() is much more straightforward and efficient. It switches the stack as soon as possible and then pushes %rsi directly onto the kernel stack.
So they are different in this aspect.
I compared the macro idtentry and the original entry_INT80_compat(), to check if the macro idtentry has all the behaviors that the INT80 thing has and check if what the macro idtentry has but the INT80 thing doesn't is a No-op (like the handling of bad IRET).
In my view, the checks don't fail my expectations except for regs->ax and regs->orig_ax.
As for CLEAR_REGS, I also have reviewed it many times. To me, it equals clearing all registers although it doesn't clear ax, sp, di, si. In the comments, it says
The lower registers are likely clobbered well before they could be put to use in a speculative execution gadget.
When using CLEAR_REGS for the INT80 thing, the %rsi will be cleared explicitly when syscall_enter_from_user_mode() which has 2 arguments is called.
"identical" is overstated. I will change the changelog to say their behaviors are almost similar and the final result are the same when the macro idtentry has the prolog.
Thanks
Lai