Skip to contentSkip to conversation

x86/entry: Clean up entry code

PATCH Superseded Lai Jiangshan 3 participants 16 messages Apr 12, 2022 – Apr 25, 2022

Subsystems (2)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v5 cover x86/entry: Clean up entry code

From: Lai Jiangshan <[email protected]>

This patchset moves the stack-switch code to the place where error_entry() return, unravels error_entry() from XENpv and makes entry_INT80_compat() uses idtentry macro.

This patchset is highly related to XENpv, because it does the extra cleanup to convert SWAPGS to swapgs after major cleanup is done.

The patches are the 5th version to pick patches from the patchset
https://lore.kernel.org/lkml/[email protected]/
which converts ASM code to C code. These patches are prepared for that purpose. But this patchset has it own value: it simplifies the stack switch, avoids leaving the old stack inside a function call, and separates XENpv code with native code without adding new code.

Peter said in V3:

Quoted 4 lines

So AFAICT these patches are indeed correct.

I'd love for some of the other x86 people to also look at this, but a tentative ACK on this.

[V4]: https://lore.kernel.org/lkml/[email protected]/
[V3]: https://lore.kernel.org/lkml/[email protected]/
[V2]: https://lore.kernel.org/lkml/[email protected]/
[V1]: https://lore.kernel.org/lkml/[email protected]/

Changed from V4:
Update changelog largely of patch 1 and patch 2 Update changelog slightly of other patches Unbreak the line of fixup_bad_iret() in patch1

Add Reviewed-by from Juergen Gross <[email protected]> in patch 6 since he gave the Reviewed-by in one of the squashed patches.

Changed from V3:
Only reorder the int80 thing as the last patch to make patches ordering more natural. (Both orders are correct)

Other interactions in V3:
Peter raised several questions and I think I answered them and I don't think the code need to be updated unless I missed some points. (Except reordering the patches)

Josh asked to remove UNWIND_HINT_REGS in patch5, but I think UNWIND_HINT_REGS is old code before this patchset and I don't want to do a cleanup that is not relate to preparing converting ASM code C code in this patchset. He also asked to remove ENCODE_FRAME_POINTER in xenpv case, and I think it just complicates the code for just optimizing out a single assignment to %rbp. I would not always stick to these reasons of mine, but I just keep the code unchanged since he hasn't emphasized it again nor other people has requested it.

Changed from V2:
Make the patch of folding int80 thing as the first patch Add more changelog in "Switch the stack after error_entry() returns"

Changed from V1
Squash cleanup patches converting SWAPGS to swapgs into one patch

Use my official email address (Ant Group). The work is backed by my company and I was incorrectly misunderstood that [email protected] is the only portal for opensource work in the corporate group.

Lai Jiangshan (7):
x86/traps: Move pt_regs only in fixup_bad_iret() x86/entry: Switch the stack after error_entry() returns x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry() x86/entry: Move cld to the start of idtentry macro x86/entry: Don't call error_entry() for XENPV x86/entry: Convert SWAPGS to swapgs and remove the definition of
SWAPGS
x86/entry: Use idtentry macro for entry_INT80_compat

 arch/x86/entry/entry_64.S        |  61 +++++++++++++-----
 arch/x86/entry/entry_64_compat.S | 105 +------------------------------
 arch/x86/include/asm/idtentry.h  |  47 ++++++++++++++
 arch/x86/include/asm/irqflags.h  |   8 ---
 arch/x86/include/asm/proto.h     |   4 --
 arch/x86/include/asm/traps.h     |   2 +-
 arch/x86/kernel/traps.c          |  18 ++----
 7 files changed, 100 insertions(+), 145 deletions(-)
Signature · 1 line
2.19.1.6.gb485710b
#1 lore raw

PATCH v5 1/7 x86/traps: Move pt_regs only in fixup_bad_iret()

From: Lai Jiangshan <[email protected]>

Always stash the address error_entry() is going to return to, in %r12 and get rid of the void *error_entry_ret; slot in struct bad_iret_stack which was supposed to account for it and pt_regs pushed on the stack.

After this, both fixup_bad_iret() and sync_regs() can work on a struct pt_regs pointer directly.

---
 arch/x86/entry/entry_64.S    |  5 ++++-
 arch/x86/include/asm/traps.h |  2 +-
 arch/x86/kernel/traps.c      | 18 ++++++------------
 3 files changed, 11 insertions(+), 14 deletions(-)
arch/x86/entry/entry_64.S +4 -1
index 4faac48ebec5..e9d896717ab4 100644
@@ -1058,9 +1058,12 @@ SYM_CODE_START_LOCAL(error_entry)
 	 * Pretend that the exception came from user mode: set up pt_regs
 	 * as if we faulted immediately after IRET.
 	 */
-	mov	%rsp, %rdi
+	popq	%r12				/* save return addr in %12 */
+	movq	%rsp, %rdi			/* arg0 = pt_regs pointer */
 	call	fixup_bad_iret
 	mov	%rax, %rsp
+	ENCODE_FRAME_POINTER
+	pushq	%r12
 	jmp	.Lerror_entry_from_usermode_after_swapgs
 SYM_CODE_END(error_entry)
 
arch/x86/include/asm/traps.h +1 -1
index 35317c5c551d..47ecfff2c83d 100644
@@ -13,7 +13,7 @@
 #ifdef CONFIG_X86_64
 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
 asmlinkage __visible notrace
-struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s);
+struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs);
 void __init trap_init(void);
 asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);
 #endif
arch/x86/kernel/traps.c +6 -12
index a4e2efde5d1f..111b18d57a54 100644
@@ -898,13 +898,7 @@ asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *r
 }
 #endif
 
-struct bad_iret_stack {
-	void *error_entry_ret;
-	struct pt_regs regs;
-};
-
-asmlinkage __visible noinstr
-struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s)
+asmlinkage __visible noinstr struct pt_regs *fixup_bad_iret(struct pt_regs *bad_regs)
 {
 	/*
 	 * This is called from entry_64.S early in handling a fault
@@ -914,19 +908,19 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s)
 	 * just below the IRET frame) and we want to pretend that the
 	 * exception came from the IRET target.
 	 */
-	struct bad_iret_stack tmp, *new_stack =
-		(struct bad_iret_stack *)__this_cpu_read(cpu_tss_rw.x86_tss.sp0) - 1;
+	struct pt_regs tmp, *new_stack =
+		(struct pt_regs *)__this_cpu_read(cpu_tss_rw.x86_tss.sp0) - 1;
 
 	/* Copy the IRET target to the temporary storage. */
-	__memcpy(&tmp.regs.ip, (void *)s->regs.sp, 5*8);
+	__memcpy(&tmp.ip, (void *)bad_regs->sp, 5*8);
 
 	/* Copy the remainder of the stack from the current stack. */
-	__memcpy(&tmp, s, offsetof(struct bad_iret_stack, regs.ip));
+	__memcpy(&tmp, bad_regs, offsetof(struct pt_regs, ip));
 
 	/* Update the entry stack */
 	__memcpy(new_stack, &tmp, sizeof(tmp));
 
-	BUG_ON(!user_mode(&new_stack->regs));
+	BUG_ON(!user_mode(new_stack));
 	return new_stack;
 }
 #endif
Signature · 1 line
2.19.1.6.gb485710b
#2 lore raw

PATCH v5 2/7 x86/entry: Switch the stack after error_entry() returns

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

error_entry() calls sync_regs(), and fixup_bad_iret() before sync_regs() if it is a fault from bad IRET, to copy the pt_regs to the kernel stack and switches the kernel stack directly after sync_regs().

But error_entry() itself is also a function call, so the code has to stash the address error_entry() is going to return to, in %r12 and makes the work complicated.

Move the code of switching stack after error_entry() and get rid of the need to handle the return address.

---
 arch/x86/entry/entry_64.S | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)
arch/x86/entry/entry_64.S +6 -10
index e9d896717ab4..e1efc56fbcd4 100644
@@ -326,6 +326,8 @@ SYM_CODE_END(ret_from_fork)
 .macro idtentry_body cfunc has_error_code:req
 
 	call	error_entry
+	movq	%rax, %rsp			/* switch to the task stack if from userspace */
+	ENCODE_FRAME_POINTER
 	UNWIND_HINT_REGS
 
 	movq	%rsp, %rdi			/* pt_regs pointer into 1st argument*/
@@ -999,14 +1001,10 @@ SYM_CODE_START_LOCAL(error_entry)
 	/* We have user CR3.  Change to kernel CR3. */
 	SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
 
+	leaq	8(%rsp), %rdi			/* arg0 = pt_regs pointer */
 .Lerror_entry_from_usermode_after_swapgs:
 	/* Put us onto the real thread stack. */
-	popq	%r12				/* save return addr in %12 */
-	movq	%rsp, %rdi			/* arg0 = pt_regs pointer */
 	call	sync_regs
-	movq	%rax, %rsp			/* switch stack */
-	ENCODE_FRAME_POINTER
-	pushq	%r12
 	RET
 
 	/*
@@ -1038,6 +1036,7 @@ SYM_CODE_START_LOCAL(error_entry)
 	 */
 .Lerror_entry_done_lfence:
 	FENCE_SWAPGS_KERNEL_ENTRY
+	leaq	8(%rsp), %rax			/* return pt_regs pointer */
 	RET
 
 .Lbstep_iret:
@@ -1058,12 +1057,9 @@ SYM_CODE_START_LOCAL(error_entry)
 	 * Pretend that the exception came from user mode: set up pt_regs
 	 * as if we faulted immediately after IRET.
 	 */
-	popq	%r12				/* save return addr in %12 */
-	movq	%rsp, %rdi			/* arg0 = pt_regs pointer */
+	leaq	8(%rsp), %rdi			/* arg0 = pt_regs pointer */
 	call	fixup_bad_iret
-	mov	%rax, %rsp
-	ENCODE_FRAME_POINTER
-	pushq	%r12
+	mov	%rax, %rdi
 	jmp	.Lerror_entry_from_usermode_after_swapgs
 SYM_CODE_END(error_entry)
 
Signature · 1 line
2.19.1.6.gb485710b
#3 lore raw

PATCH v5 3/7 x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

error_entry() doesn't handle the stack balanced. It includes PUSH_AND_CLEAR_REGS which is commonly needed for all IDT entries and can't pop the regs before it returns.

Move PUSH_AND_CLEAR_REGS out of error_entry() and make error_entry() works on the stack normally.

After this, XENPV doesn't need error_entry() since PUSH_AND_CLEAR_REGS is moved out and error_entry() can be converted to C code in future since it doesn't fiddle the stack.

The text size will be enlarged:

size arch/x86/entry/entry_64.o.before:
text data bss dec hex filename
17916 384 0 18300 477c arch/x86/entry/entry_64.o

size --format=SysV arch/x86/entry/entry_64.o.before:

.entry.text                      5528      0
.orc_unwind                      6456      0
.orc_unwind_ip                   4304      0

size arch/x86/entry/entry_64.o.after:
text data bss dec hex filename
26868 384 0 27252 6a74 arch/x86/entry/entry_64.o

size --format=SysV arch/x86/entry/entry_64.o.after:

.entry.text                      8200      0
.orc_unwind                     10224      0
.orc_unwind_ip                   6816      0

The tables .orc_unwind[_ip] are enlarged due to it adds many pushes.

But .entry.text in x86_64 is 2M aligned, enlarging it to 8.2k doesn't enlarge the final text size.

And it will only increase the footprint when different interrupts and exceptions happen unlikely heavily at the same time.

---
 arch/x86/entry/entry_64.S | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +3 -2
index e1efc56fbcd4..835b798556fb 100644
@@ -325,6 +325,9 @@ SYM_CODE_END(ret_from_fork)
  */
 .macro idtentry_body cfunc has_error_code:req
 
+	PUSH_AND_CLEAR_REGS
+	ENCODE_FRAME_POINTER
+
 	call	error_entry
 	movq	%rax, %rsp			/* switch to the task stack if from userspace */
 	ENCODE_FRAME_POINTER
@@ -987,8 +990,6 @@ SYM_CODE_END(paranoid_exit)
 SYM_CODE_START_LOCAL(error_entry)
 	UNWIND_HINT_FUNC
 	cld
-	PUSH_AND_CLEAR_REGS save_ret=1
-	ENCODE_FRAME_POINTER 8
 	testb	$3, CS+8(%rsp)
 	jz	.Lerror_kernelspace
 
Signature · 1 line
2.19.1.6.gb485710b
maintainer
#4 lore raw

Re: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()

Quoted 3 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

error_entry() doesn't handle the stack balanced.

What does that mean?

Quoted 10 lines

It includes
PUSH_AND_CLEAR_REGS which is commonly needed for all IDT entries and can't pop the regs before it returns.

Move PUSH_AND_CLEAR_REGS out of error_entry() and make error_entry() works on the stack normally.

After this, XENPV doesn't need error_entry() since PUSH_AND_CLEAR_REGS is moved out and error_entry() can be converted to C code in future since it doesn't fiddle the stack.

This is not a justification for this size increase:

text data bss dec hex filename

16060616        128131358       36384888        180576862       ac3625e vmlinux.before
16065626        128131358       36380792        180577776       ac365f0 vmlinux.after

~5K text increase already with my tailored config.

You can have a asm_error_entry(), written in asm, which does the regs pushing and which calls error_entry() - the latter being the C version. And no need for the size increase.

For example.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#5 lore raw

Re: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()

Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

error_entry() doesn't handle the stack balanced.

What does that mean?

For a normal function, e.g. a function complied from a C function, the stack will return to its original place when the function returns. The size of memory pushed and popped are the same in a function.

Quoted 15 lines
Quoted 11 lines

It includes
PUSH_AND_CLEAR_REGS which is commonly needed for all IDT entries and can't pop the regs before it returns.

Move PUSH_AND_CLEAR_REGS out of error_entry() and make error_entry() works on the stack normally.

After this, XENPV doesn't need error_entry() since PUSH_AND_CLEAR_REGS is moved out and error_entry() can be converted to C code in future since it doesn't fiddle the stack.

This is not a justification for this size increase:

text data bss dec hex filename

Quoted 8 lines
16060616        128131358       36384888        180576862       ac3625e vmlinux.before
16065626        128131358       36380792        180577776       ac365f0 vmlinux.after

~5K text increase already with my tailored config.

You can have a asm_error_entry(), written in asm, which does the regs pushing and which calls error_entry() - the latter being the C version. And no need for the size increase.

The mapped size for the text is always 2M when the kernel is booted since it is 2M-aligned. So I don't think the size is a concern.

The only concern is the footprint when different interrupts and exceptions happen heavily at the same time. In this case, different copies of PUSH_AND_CLEAR_REGS in the text will be touched.

For example, a heavy page fault and IPI or timer at the same time. I'm not sure if it is a real case.

I'm Okay with asm_error_entry(). And also we can use an ASM function containing PUSH_AND_CLEAR_REGS only.

Thanks
Lai

maintainer
#6 lore raw

Re: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()

Quoted 2 lines · Lai Jiangshan

The mapped size for the text is always 2M when the kernel is booted since it is 2M-aligned. So I don't think the size is a concern.

This is not how this argumentation works: you add 0.01% size increase here, 0.01% slowdown there and it all soon adds up into a bloated kernel. Besides, nowadays the kernel runs a lot as a guest and guest kernel size does matter.

So no, just because you want to turn error_entry() into C code and for that you'll add some bloat doesn't justify it in my book. All changes to core code need to be absolutely needed and justifiable.

Quoted 2 lines

For example, a heavy page fault and IPI or timer at the same time. I'm not sure if it is a real case.

I wouldn't be surprised if it shows in some perf profiles.

Quoted 2 lines

I'm Okay with asm_error_entry(). And also we can use an ASM function containing PUSH_AND_CLEAR_REGS only.

Just do the necessary and minimal thing - this is core code and in asm for that matter, so be conservative pls. This is not some random driver.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#7 lore raw

Re: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()

Quoted 9 lines · Borislav Petkov
Quoted 2 lines · Lai Jiangshan

The mapped size for the text is always 2M when the kernel is booted since it is 2M-aligned. So I don't think the size is a concern.

This is not how this argumentation works: you add 0.01% size increase here, 0.01% slowdown there and it all soon adds up into a bloated kernel. Besides, nowadays the kernel runs a lot as a guest and guest kernel size does matter.

I agree with you.

I meant the real .entry.text size is unchanged. No matter what the complied .entry.text size is, 5.5k before the patch, or 8.2k after the patch, the size of .entry.text in the kernel is always 2M.

But the overall size is enlarged which is a concern.

Quoted 8 lines

So no, just because you want to turn error_entry() into C code and for that you'll add some bloat doesn't justify it in my book. All changes to core code need to be absolutely needed and justifiable.

Quoted 2 lines

For example, a heavy page fault and IPI or timer at the same time. I'm not sure if it is a real case.

Quoted 8 lines

I wouldn't be surprised if it shows in some perf profiles.

Quoted 2 lines

I'm Okay with asm_error_entry(). And also we can use an ASM function containing PUSH_AND_CLEAR_REGS only.

Just do the necessary and minimal thing - this is core code and in asm for that matter, so be conservative pls. This is not some random driver.

I will follow your instructions and add an extra ASM function.

And maybe add this to the changelog:

A function call is added for every interrupt or exception, but it is a low-level asm function call and actually, only a "call" and a "ret" instructions are added. It will enable error_entry() to be C code that can use inlined sync_regs() and save a C function call which will reduce more instructions in return.

I'm also waiting for reviews for other patches and I'm not going to send this new code too soon since I don't want to send a new revision with only this patch updated.

Thanks
Lai

maintainer
#8 lore raw

Re: [PATCH V5 3/7] x86/entry: Move PUSH_AND_CLEAR_REGS out of error_entry()

Quoted 3 lines · Lai Jiangshan

I meant the real .entry.text size is unchanged. No matter what the complied .entry.text size is, 5.5k before the patch, or 8.2k after the patch, the size of .entry.text in the kernel is always 2M.

Yes, unless the bloat I'm talking about above continues and then it becomes 4M. So we need to be careful.

Quoted 1 line

I will follow your instructions and add an extra ASM function.

Thx.

Quoted 3 lines

And maybe add this to the changelog:

A function call is added for every interrupt or exception, but it

I think you need to read submitting-patches.rst again:

"Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."

In your case:

"Add a function call..."

Quoted 2 lines

is a low-level asm function call and actually, only a "call" and a "ret" instructions are added.

CALL, RET - all x86 insns in caps.

Quoted 1 line

It will enable error_entry() to be

It will allow for sync_regs() to be converted to C...

Quoted 6 lines

C code that can use inlined sync_regs() and save a C function call which will reduce more instructions in return.

I'm also waiting for reviews for other patches and I'm not going to send this new code too soon since I don't want to send a new revision with only this patch updated.

Again, submitting-patches.rst:

"After you have submitted your change, be patient and wait. Reviewers are busy people and may not get to your patch right away.

...

You should receive comments within a week or so; if that does not happen, make sure that you have sent your patches to the right place. Wait for a minimum of one week before resubmitting or pinging reviewers
- possibly longer during busy times like merge windows"

So you send, wait for feedback, incorporate it and resend after a week, but not earlier.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#9 lore raw

PATCH v5 4/7 x86/entry: Move cld to the start of idtentry macro

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

Make it next to CLAC

---
 arch/x86/entry/entry_64.S | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64.S +5 -3
index 835b798556fb..7b6a0f15bb20 100644
@@ -360,6 +360,7 @@ SYM_CODE_START(\asmsym)
 	UNWIND_HINT_IRET_REGS offset=\has_error_code*8
 	ENDBR
 	ASM_CLAC
+	cld
 
 	.if \has_error_code == 0
 		pushq	$-1			/* ORIG_RAX: no syscall to restart */
@@ -428,6 +429,7 @@ SYM_CODE_START(\asmsym)
 	UNWIND_HINT_IRET_REGS
 	ENDBR
 	ASM_CLAC
+	cld
 
 	pushq	$-1			/* ORIG_RAX: no syscall to restart */
 
@@ -484,6 +486,7 @@ SYM_CODE_START(\asmsym)
 	UNWIND_HINT_IRET_REGS
 	ENDBR
 	ASM_CLAC
+	cld
 
 	/*
 	 * If the entry is from userspace, switch stacks and treat it as
@@ -546,6 +549,7 @@ SYM_CODE_START(\asmsym)
 	UNWIND_HINT_IRET_REGS offset=8
 	ENDBR
 	ASM_CLAC
+	cld
 
 	/* paranoid_entry returns GS information for paranoid_exit in EBX. */
 	call	paranoid_entry
@@ -871,7 +875,6 @@ SYM_CODE_END(xen_failsafe_callback)
  */
 SYM_CODE_START_LOCAL(paranoid_entry)
 	UNWIND_HINT_FUNC
-	cld
 	PUSH_AND_CLEAR_REGS save_ret=1
 	ENCODE_FRAME_POINTER 8
 
@@ -989,7 +992,6 @@ SYM_CODE_END(paranoid_exit)
  */
 SYM_CODE_START_LOCAL(error_entry)
 	UNWIND_HINT_FUNC
-	cld
 	testb	$3, CS+8(%rsp)
 	jz	.Lerror_kernelspace
 
@@ -1123,6 +1125,7 @@ SYM_CODE_START(asm_exc_nmi)
 	 */
 
 	ASM_CLAC
+	cld
 
 	/* Use %rdx as our temp variable throughout */
 	pushq	%rdx
@@ -1142,7 +1145,6 @@ SYM_CODE_START(asm_exc_nmi)
 	 */
 
 	swapgs
-	cld
 	FENCE_SWAPGS_USER_ENTRY
 	SWITCH_TO_KERNEL_CR3 scratch_reg=%rdx
 	movq	%rsp, %rdx
Signature · 1 line
2.19.1.6.gb485710b
#10 lore raw

PATCH v5 5/7 x86/entry: Don't call error_entry() for XENPV

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When in XENPV, it is already in the task stack, and it can't fault for native_iret() nor native_load_gs_index() since XENPV uses its own pvops for IRET and load_gs_index(). And it doesn't need to switch the CR3.

So there is no reason to call error_entry() in XENPV.

---
 arch/x86/entry/entry_64.S | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +11 -2
index 7b6a0f15bb20..3aca7815fe79 100644
@@ -328,8 +328,17 @@ SYM_CODE_END(ret_from_fork)
 	PUSH_AND_CLEAR_REGS
 	ENCODE_FRAME_POINTER
 
-	call	error_entry
-	movq	%rax, %rsp			/* switch to the task stack if from userspace */
+	/*
+	 * Call error_entry() and switch to the task stack if from userspace.
+	 *
+	 * When in XENPV, it is already in the task stack, and it can't fault
+	 * for native_iret() nor native_load_gs_index() since XENPV uses its
+	 * own pvops for IRET and load_gs_index().  And it doesn't need to
+	 * switch the CR3.  So it can skip invoking error_entry().
+	 */
+	ALTERNATIVE "call error_entry; movq %rax, %rsp", \
+		"", X86_FEATURE_XENPV
+
 	ENCODE_FRAME_POINTER
 	UNWIND_HINT_REGS
 
Signature · 1 line
2.19.1.6.gb485710b
maintainer
#11 lore raw

Re: [PATCH V5 5/7] x86/entry: Don't call error_entry() for XENPV

Quoted 28 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

When in XENPV, it is already in the task stack, and it can't fault for native_iret() nor native_load_gs_index() since XENPV uses its own pvops for IRET and load_gs_index(). And it doesn't need to switch the CR3.

So there is no reason to call error_entry() in XENPV.

---
 arch/x86/entry/entry_64.S | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
arch/x86/entry/entry_64.S +4 -2
index 7b6a0f15bb20..3aca7815fe79 100644
@@ -328,8 +328,17 @@ SYM_CODE_END(ret_from_fork)
 	PUSH_AND_CLEAR_REGS
 	ENCODE_FRAME_POINTER
 
-	call	error_entry
-	movq	%rax, %rsp			/* switch to the task stack if from userspace */
+	/*
+	 * Call error_entry() and switch to the task stack if from userspace.
+	 *
+	 * When in XENPV, it is already in the task stack, and it can't fault

arch/x86/entry/entry_64.S SYM_CODE_END(ret_from_fork) · line 344

Quoted 10 lines
arch/x86/entry/entry_64.S +7 -0
+	 * for native_iret() nor native_load_gs_index() since XENPV uses its
+	 * own pvops for IRET and load_gs_index().  And it doesn't need to
+	 * switch the CR3.  So it can skip invoking error_entry().
+	 */
+	ALTERNATIVE "call error_entry; movq %rax, %rsp", \
+		"", X86_FEATURE_XENPV
+
 	ENCODE_FRAME_POINTER
 	UNWIND_HINT_REGS
 

This one needs Jürgen. CCed.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#12 lore raw

PATCH v5 6/7 x86/entry: Convert SWAPGS to swapgs and remove the definition of SWAPGS

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

XENPV doesn't use swapgs_restore_regs_and_return_to_usermode(), error_entry() and entry_SYSENTER_compat().

Change the PV-compatible SWAPGS to the ASM instruction swapgs in these functions.

Also remove the definition of SWAPGS since there is no user of the SWAPGS anymore.

---
 arch/x86/entry/entry_64.S        | 6 +++---
 arch/x86/entry/entry_64_compat.S | 2 +-
 arch/x86/include/asm/irqflags.h  | 8 --------
 3 files changed, 4 insertions(+), 12 deletions(-)
arch/x86/entry/entry_64.S +3 -3
index 3aca7815fe79..6611032979d9 100644
@@ -1008,7 +1008,7 @@ SYM_CODE_START_LOCAL(error_entry)
 	 * We entered from user mode or we're pretending to have entered
 	 * from user mode due to an IRET fault.
 	 */
-	SWAPGS
+	swapgs
 	FENCE_SWAPGS_USER_ENTRY
 	/* We have user CR3.  Change to kernel CR3. */
 	SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
@@ -1040,7 +1040,7 @@ SYM_CODE_START_LOCAL(error_entry)
 	 * gsbase and proceed.  We'll fix up the exception and land in
 	 * .Lgs_change's error handler with kernel gsbase.
 	 */
-	SWAPGS
+	swapgs
 
 	/*
 	 * Issue an LFENCE to prevent GS speculation, regardless of whether it is a
@@ -1061,7 +1061,7 @@ SYM_CODE_START_LOCAL(error_entry)
 	 * We came from an IRET to user mode, so we have user
 	 * gsbase and CR3.  Switch to kernel gsbase and CR3:
 	 */
-	SWAPGS
+	swapgs
 	FENCE_SWAPGS_USER_ENTRY
 	SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
 
arch/x86/entry/entry_64_compat.S +1 -1
index 4fdb007cddbd..c5aeb0819707 100644
@@ -50,7 +50,7 @@ SYM_CODE_START(entry_SYSENTER_compat)
 	UNWIND_HINT_EMPTY
 	ENDBR
 	/* Interrupts are off on entry. */
-	SWAPGS
+	swapgs
 
 	pushq	%rax
 	SWITCH_TO_KERNEL_CR3 scratch_reg=%rax
arch/x86/include/asm/irqflags.h +0 -8
index 111104d1c2cd..7793e52d6237 100644
@@ -137,14 +137,6 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)
 	if (!arch_irqs_disabled_flags(flags))
 		arch_local_irq_enable();
 }
-#else
-#ifdef CONFIG_X86_64
-#ifdef CONFIG_XEN_PV
-#define SWAPGS	ALTERNATIVE "swapgs", "", X86_FEATURE_XENPV
-#else
-#define SWAPGS	swapgs
-#endif
-#endif
 #endif /* !__ASSEMBLY__ */
 
 #endif
Signature · 1 line
2.19.1.6.gb485710b
#13 lore raw

PATCH v5 7/7 x86/entry: Use idtentry macro for entry_INT80_compat

In reply to Lai Jiangshan

From: Lai Jiangshan <[email protected]>

entry_INT80_compat is identical to idtentry macro except a special handling for %rax in the prolog.

Add the prolog to idtentry and use idtentry for entry_INT80_compat.

---
 arch/x86/entry/entry_64.S        |  18 ++++++
 arch/x86/entry/entry_64_compat.S | 103 -------------------------------
 arch/x86/include/asm/idtentry.h  |  47 ++++++++++++++
 arch/x86/include/asm/proto.h     |   4 --
 4 files changed, 65 insertions(+), 107 deletions(-)
arch/x86/entry/entry_64.S +18 -0
index 6611032979d9..d49b93f494df 100644
@@ -375,6 +375,24 @@ SYM_CODE_START(\asmsym)
 		pushq	$-1			/* ORIG_RAX: no syscall to restart */
 	.endif
 
+	.if \vector == IA32_SYSCALL_VECTOR
+		/*
+		 * User tracing code (ptrace or signal handlers) might assume
+		 * that the saved RAX contains a 32-bit number when we're
+		 * invoking a 32-bit syscall.  Just in case the high bits are
+		 * nonzero, zero-extend the syscall number.  (This could almost
+		 * certainly be deleted with no ill effects.)
+		 */
+		movl	%eax, %eax
+
+		/*
+		 * do_int80_syscall_32() expects regs->orig_ax to be user ax,
+		 * and regs->ax to be $-ENOSYS.
+		 */
+		movq	%rax, (%rsp)
+		movq	$-ENOSYS, %rax
+	.endif
+
 	.if \vector == X86_TRAP_BP
 		/*
 		 * If coming from kernel space, create a 6-word gap to allow the
arch/x86/entry/entry_64_compat.S +0 -103

106 lines not on this page · raw message

arch/x86/include/asm/idtentry.h +47 -0

65 lines not on this page · raw message

arch/x86/include/asm/proto.h +0 -4
index 0f899c8d7a4e..4700d1650410 100644
@@ -28,10 +28,6 @@ void entry_SYSENTER_compat(void);
 void __end_entry_SYSENTER_compat(void);
 void entry_SYSCALL_compat(void);
 void entry_SYSCALL_compat_safe_stack(void);
-void entry_INT80_compat(void);
-#ifdef CONFIG_XEN_PV
-void xen_entry_INT80_compat(void);
-#endif
 #endif
 
 void x86_configure_nx(void);
Signature · 1 line
2.19.1.6.gb485710b
#14 lore raw

Re: [PATCH V5 7/7] x86/entry: Use idtentry macro for entry_INT80_compat

Quoted 4 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

entry_INT80_compat is identical to idtentry macro except a special handling for %rax in the prolog.

Seriously?

arch/x86/entry/entry_64_compat.S sysret32_from_system_call: · line 386

Quoted 2 lines
arch/x86/entry/entry_64_compat.S +0 -2
-	pushq	%rsi			/* pt_regs->si */
-	xorl	%esi, %esi		/* nospec   si */

esi is not cleared in CLEAR_REGS. So much for identical.

Thanks,

tglx

#15 lore raw

Re: [PATCH V5 7/7] x86/entry: Use idtentry macro for entry_INT80_compat

Quoted 7 lines · Thomas Gleixner
Quoted 4 lines · Lai Jiangshan

From: Lai Jiangshan <[email protected]>

entry_INT80_compat is identical to idtentry macro except a special handling for %rax in the prolog.

Quoted 8 lines

Seriously?

arch/x86/entry/entry_64_compat.S sysret32_from_system_call: · line 386

Quoted 2 lines
arch/x86/entry/entry_64_compat.S +0 -2
-     pushq   %rsi                    /* pt_regs->si */
-     xorl    %esi, %esi              /* nospec   si */

esi is not cleared in CLEAR_REGS. So much for identical.

Hello, Thomas

Thank you for the review.

They (the old entry_INT80_compat() and the new using the macro idtentry) are not identical in ASM code.

The macro idtentry pushes %rsi to the entry stack and then copies it from the entry stack to the kernel stack and then switches the stack.

The original entry_INT80_compat() is much more straightforward and efficient. It switches the stack as soon as possible and then pushes %rsi directly onto the kernel stack.

So they are different in this aspect.

I compared the macro idtentry and the original entry_INT80_compat(), to check if the macro idtentry has all the behaviors that the INT80 thing has and check if what the macro idtentry has but the INT80 thing doesn't is a No-op (like the handling of bad IRET).

In my view, the checks don't fail my expectations except for regs->ax and regs->orig_ax.

As for CLEAR_REGS, I also have reviewed it many times. To me, it equals clearing all registers although it doesn't clear ax, sp, di, si. In the comments, it says

The lower registers are likely clobbered well before they could be put to use in a speculative execution gadget.

When using CLEAR_REGS for the INT80 thing, the %rsi will be cleared explicitly when syscall_enter_from_user_mode() which has 2 arguments is called.

"identical" is overstated. I will change the changelog to say their behaviors are almost similar and the final result are the same when the macro idtentry has the prolog.

Thanks
Lai