x86: Rewrite all syscall entries except native 64-bit
Subsystems (7)
Andy Lutomirski
maintainercover x86: Rewrite all syscall entries except native 64-bit
Here's a monster series that I'm working on. I think it's in decent shape now.
The first couple patches are tests and some old stuff. There's a test that validates the vDSO AT_SYSINFO annotations (which fails on 32-bit Debian systems for some reason that I can't yet fathom because fast syscalls simply don't happen on my VM for unknown reasons presumably related to glibc bugs or misconfiguration, and I need to do something about the test). There's also a test that exercises some assumptions that signal handling and ptracers make about syscalls that currently do *not* hold on 64-bit AMD using 32-bit AT_SYSINFO.
The next few patches are the NT stuff. Ingo, feel free to pretend you don't see it until the merge window closes :)
The rest is basically a rewrite of syscalls for all cases except 64-bit native. With these patches applied, there is a single 32-bit vDSO and it uses SYSCALL, SYSENTER, and INT80 almost interchangeably via alternatives. The semantics of SYSENTER and SYSCALL are defined as:
1. If SYSCALL, ESP = ECX
2. ECX = *ESP
3. IP = INT80 landing pad
4. Opportunistic SYSRET/SYSEXIT is enabled on return
The vDSO is rearranged so that these semantics work. Anything that backs IP up by 2 ends up pointing at a bona fide int $0x80 instruction with the expected regs.
In the process, the vDSO CFI annotations (which are actually used) get rewritten using normal CFI directives.
Opportunistic SYSRET/SYSEXIT only happens on return when CS and SS are as expected, IP points to the INT80 landing pad, and flags are in good shape.
Other than that, the system call entries are simplified to the bare minimum prologue and a call to a C function. Amusingly, SYSENTER and SYSCALL32 use the same C function.
To make that work, I had to remove all the 32-bit syscall stubs except the clone argument hack. This is because, for C code to call through the system call table, the system call table entries need to be real function pointers with C-compatible ABIs.
There is nothing at all anymore that requires that x86_32 syscalls be asmlinkage. That could be removed in a subsequent patch.
The upshot appears to be a ~25 cycle performance hit on 32-bit fast path syscalls. The slow path is probably faster under most circumstances and, if the exit slow path gets hit, it'll be much faster because (as we already do in the 64-bit native case) we can still use SYSEXIT/SYSRET.
The patchset is structured as a removal of the old fast syscall code, then the change that makes syscalls into real functions, then a clean re-implementation of fast syscalls.
If we want some of the 25 cycles back, we could consider open-coding a new C fast path.
When reading the diffstat, keep in mind that 544 lines are new tests and ~187 are reinstatement of the CFI macros for compatability with old binutils. There are only ~444 lines of real new code in here, which I think is pretty good. The asm diffstat is:
arch/x86/entry/entry_32.S | 184 +++---- arch/x86/entry/entry_64_compat.S | 541 +++++---------------- arch/x86/entry/vdso/vdso32/int80.S | 56 --- arch/x86/entry/vdso/vdso32/syscall.S | 75 --- arch/x86/entry/vdso/vdso32/sysenter.S | 116 ----- arch/x86/entry/vdso/vdso32/system_call.S | 57 +++ 6 files changed, 246 insertions(+), 783 deletions(-)
Andy Lutomirski (30):
selftests/x86: Add a test for vDSO unwinding selftests/x86: Add a test for syscall restart and arg modification x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access x86/entry: Move lockdep_sys_exit to prepare_exit_to_usermode x86/entry/64/compat: After SYSENTER, move STI after the NT fixup x86/sched/64: Don't save flags on context switch (reinstated) x86/vdso: Remove runtime 32-bit vDSO selection
x86/asm: Re-add manual CFI infrastructure
x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm x86/vdso: Replace hex int80 CFI annotations with gas directives x86/vdso/32: Save extra registers in the INT80 vsyscall path x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries x86/entry/64/compat: Remove audit optimizations x86/entry/64/compat: Remove most of the fast system call machinery x86/entry/64/compat: Set up full pt_regs for all compat syscalls x86/entry/syscalls: Move syscall table declarations into
asm/syscalls.h
x86/syscalls: Give sys_call_ptr_t a useful type
x86/entry: Add do_syscall_32, a C function to do 32-bit syscalls x86/entry/64/compat: Migrate the body of the syscall entry to C x86/entry: Add C code for fast system call entries x86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls x86/entry/32: Open-code return tracking from fork and kthreads x86/entry/32: Switch INT80 to the new C syscall path x86/entry/32: Re-implement SYSENTER using the new C path x86/asm: Remove thread_info.sysenter_return
x86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls x86/entry: Make irqs_disabled checks in exit code depend on lockdep x86/entry: Force inlining of 32-bit syscall code x86/entry: Micro-optimize compat fast syscall arg fetch
arch/x86/Makefile | 10 +- arch/x86/entry/common.c | 138 +++++- arch/x86/entry/entry_32.S | 184 +++---- arch/x86/entry/entry_64.S | 3 +- arch/x86/entry/entry_64_compat.S | 541 +++++---------------- arch/x86/entry/syscall_32.c | 9 +- arch/x86/entry/syscall_64.c | 4 +- arch/x86/entry/syscalls/syscall_32.tbl | 8 +- arch/x86/entry/vdso/Makefile | 39 +- arch/x86/entry/vdso/vdso2c.c | 2 +- arch/x86/entry/vdso/vdso32-setup.c | 28 +- arch/x86/entry/vdso/vdso32/int80.S | 56 --- arch/x86/entry/vdso/vdso32/syscall.S | 75 --- arch/x86/entry/vdso/vdso32/sysenter.S | 116 ----- arch/x86/entry/vdso/vdso32/system_call.S | 57 +++ arch/x86/entry/vdso/vma.c | 13 +- arch/x86/ia32/ia32_signal.c | 4 +- arch/x86/include/asm/dwarf2.h | 177 +++++++ arch/x86/include/asm/elf.h | 2 +- arch/x86/include/asm/switch_to.h | 12 +- arch/x86/include/asm/syscall.h | 14 +- arch/x86/include/asm/thread_info.h | 1 - arch/x86/include/asm/vdso.h | 10 +- arch/x86/kernel/asm-offsets.c | 3 - arch/x86/kernel/signal.c | 4 +- arch/x86/um/sys_call_table_32.c | 7 +- arch/x86/um/sys_call_table_64.c | 7 +- arch/x86/xen/setup.c | 13 +- tools/testing/selftests/x86/Makefile | 5 +- tools/testing/selftests/x86/ptrace_syscall.c | 294 +++++++++++ .../testing/selftests/x86/raw_syscall_helper_32.S | 46 ++ tools/testing/selftests/x86/unwind_vdso.c | 204 ++++++++ 32 files changed, 1175 insertions(+), 911 deletions(-) delete mode 100644 arch/x86/entry/vdso/vdso32/int80.S delete mode 100644 arch/x86/entry/vdso/vdso32/syscall.S delete mode 100644 arch/x86/entry/vdso/vdso32/sysenter.S create mode 100644 arch/x86/entry/vdso/vdso32/system_call.S create mode 100644 arch/x86/include/asm/dwarf2.h create mode 100644 tools/testing/selftests/x86/ptrace_syscall.c create mode 100644 tools/testing/selftests/x86/raw_syscall_helper_32.S create mode 100644 tools/testing/selftests/x86/unwind_vdso.c
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 1/30 selftests/x86: Add a test for vDSO unwinding
While the kernel itself doesn't use DWARF unwinding, user code expects to be able to unwind the vDSO. The vsyscall (AT_SYSINFO) entry is manually CFI-annotated, and this tests that it unwinds correctly.
I tested the test by incorrectly annotating __kernel_vsyscall, and the test indeed fails if I do that.
--- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/unwind_vdso.c | 204 ++++++++++++++++++++++++++++++ 2 files changed, 205 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/unwind_vdso.c
tools/testing/selftests/x86/Makefile +1 -1
@@ -5,7 +5,7 @@ include ../lib.mk .PHONY: all all_32 all_64 warn_32bit_failure clean TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt -TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn +TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn unwind_vdso TARGETS_C_32BIT_ALL := $(TARGETS_C_BOTHBITS) $(TARGETS_C_32BIT_ONLY) BINARIES_32 := $(TARGETS_C_32BIT_ALL:%=%_32)
tools/testing/selftests/x86/unwind_vdso.c new file +204 -0
204 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 2/30 selftests/x86: Add a test for syscall restart and arg modification
In reply to Andy LutomirskiThis tests assumptions about how fast syscall works wrt pt_regs and, in particular, what happens if IP is decremented by 2 during a syscall.
--- tools/testing/selftests/x86/Makefile | 3 +- tools/testing/selftests/x86/ptrace_syscall.c | 294 +++++++++++++++++++++ .../testing/selftests/x86/raw_syscall_helper_32.S | 46 ++++ 3 files changed, 342 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/ptrace_syscall.c create mode 100644 tools/testing/selftests/x86/raw_syscall_helper_32.S
tools/testing/selftests/x86/Makefile +2 -1
@@ -4,7 +4,7 @@ include ../lib.mk .PHONY: all all_32 all_64 warn_32bit_failure clean -TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt +TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt ptrace_syscall TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn unwind_vdso TARGETS_C_32BIT_ALL := $(TARGETS_C_BOTHBITS) $(TARGETS_C_32BIT_ONLY)
@@ -60,3 +60,4 @@ endif # Some tests have additional dependencies. sysret_ss_attrs_64: thunks.S +ptrace_syscall_32: raw_syscall_helper_32.S
tools/testing/selftests/x86/ptrace_syscall.c new file +294 -0
294 lines not on this page · raw message
tools/testing/selftests/x86/raw_syscall_helper_32.S new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 3/30 x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access
In reply to Andy LutomirskiClearing NT is part of the prologue, whereas loading up arg6 makes more sense to think about as part of syscall processing. Reorder them.
--- arch/x86/entry/entry_64_compat.S | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
arch/x86/entry/entry_64_compat.S +9 -9
@@ -89,15 +89,6 @@ ENTRY(entry_SYSENTER_compat) sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* - * no need to do an access_ok check here because rbp has been - * 32-bit zero extended - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - /* * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether * NT was set instead of doing an unconditional popfq.
@@ -106,6 +97,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: + /* + * no need to do an access_ok check here because rbp has been + * 32-bit zero extended + */ + ASM_STAC +1: movl (%rbp), %ebp + _ASM_EXTABLE(1b, ia32_badarg) + ASM_CLAC + orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) jnz sysenter_tracesys
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 4/30 x86/entry: Move lockdep_sys_exit to prepare_exit_to_usermode
In reply to Andy LutomirskiRather than worrying about exactly where LOCKDEP_SYS_EXIT should go in the asm code, add it to prepare_exit_from_usermode and remove all of the asm calls that are followed by prepare_exit_to_usermode.
LOCKDEP_SYS_EXIT now appears only in the syscall fast paths.
--- arch/x86/entry/common.c | 2 ++ arch/x86/entry/entry_32.S | 2 -- arch/x86/entry/entry_64.S | 1 - 3 files changed, 2 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +2 -0
@@ -220,6 +220,8 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)if (WARN_ON (!irqs_disabled ()))local_irq_disable (); +lockdep_sys_exit (); +/* * In order to return to user mode, we need to have IRQs off with * none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY,
arch/x86/entry/entry_32.S +0 -2
@@ -255,7 +255,6 @@ ret_from_intr: jb resume_kernel # not returning to v8086 or userspace ENTRY(resume_userspace) - LOCKDEP_SYS_EXIT DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF movl %esp, %eax
@@ -372,7 +371,6 @@ syscall_call: syscall_after_call: movl %eax, PT_EAX(%esp) # store the return value syscall_exit: - LOCKDEP_SYS_EXIT jmp syscall_exit_work restore_all:
arch/x86/entry/entry_64.S +0 -1
@@ -557,7 +557,6 @@ ret_from_intr: jz retint_kernel /* Interrupt came from user space */ - LOCKDEP_SYS_EXIT_IRQ GLOBAL(retint_user) mov %rsp,%rdi call prepare_exit_to_usermode
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 5/30 x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
In reply to Andy LutomirskiWe eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts.
--- arch/x86/entry/entry_64_compat.S | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
arch/x86/entry/entry_64_compat.S +22 -8
@@ -58,14 +58,9 @@ ENDPROC(native_usergs_sysret32) * with the int 0x80 path. */ ENTRY(entry_SYSENTER_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %ebp, %ebp
@@ -76,7 +71,16 @@ ENTRY(entry_SYSENTER_compat) /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ pushq %rbp /* pt_regs->sp */ - pushfq /* pt_regs->flags */ + + /* + * Push flags. This is nasty. First, interrupts are currently + * off, but we need pt_regs->flags to have IF set. Second, even + * if TF was set when SYSENTER started, it's clear by now. We fix + * that later using TIF_SINGLESTEP. + */ + pushfq /* pt_regs->flags (except IF = 0) */ + orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */ + pushq $__USER32_CS /* pt_regs->cs */ pushq %r10 /* pt_regs->ip = thread_info->sysenter_return */ pushq %rax /* pt_regs->orig_ax */
@@ -92,12 +96,22 @@ ENTRY(entry_SYSENTER_compat) * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether * NT was set instead of doing an unconditional popfq. + * This needs to happen before enabling interrupts so that + * we don't get preempted with NT set. */ testl $X86_EFLAGS_NT, EFLAGS(%rsp) jnz sysenter_fix_flags sysenter_flags_fixed: /* + * Re-enable interrupts. IRQ tracing already thinks that IRQs are + * on (since we treat user mode as having IRQs on), and the + * prologue above is too short for it to be worth adding a + * tracing round trip. + */ + ENABLE_INTERRUPTS(CLBR_NONE) + + /* * no need to do an access_ok check here because rbp has been * 32-bit zero extended */
@@ -244,7 +258,7 @@ sysexit_audit: #endif sysenter_fix_flags: - pushq $(X86_EFLAGS_IF|X86_EFLAGS_FIXED) + pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 6/30 x86/sched/64: Don't save flags on context switch (reinstated)
In reply to Andy LutomirskiThis reinstates 2c7577a75837 ("sched/x86_64: Don't save flags on context switch"), which was reverted in 512255a2ad2c.
Historically, Linux has always saved and restored EFLAGS across context switches. As far as I know, the only reason to do this is because of the NT flag. In particular, if something calls switch_to with the NT flag set, then we don't want to leak the NT flag into a different task that might try to IRET and fail because NT is set.
Before 8c7aa698baca ("x86_64, entry: Filter RFLAGS.NT on entry from userspace"), we could run system call bodies with NT set. This would be a DoS or possibly privilege escalation hole if scheduling in such a system call would leak NT into a different task.
Importantly, we don't need to worry about NT being set while preemptible or across page faults. The only way we can schedule due to preemption or a page fault is in an interrupt entry that nests inside the SYSENTER prologue. The CPU will clear NT when entering through an interrupt gate, so we won't schedule with NT set.
The only other interesting flags are IOPL and AC. Allowing switch_to to change IOPL has no effect, as the value loaded during kernel execution doesn't matter at all except between a SYSENTER entry and the subsequent PUSHF, and anythign that interrupts in that window will restore IOPL on return.
If we call __switch_to with AC set, we have bigger problems.
--- arch/x86/include/asm/switch_to.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
arch/x86/include/asm/switch_to.h +8 -4
@@ -79,12 +79,12 @@ do { \#else /* CONFIG_X86_32 */ /* frame pointer must be last for get_wchan */ -#define SAVE_CONTEXT "pushf ; pushq %%rbp ; movq %%rsi,%%rbp\n\t" -#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp ; popf\t" +#define SAVE_CONTEXT "pushq %%rbp ; movq %%rsi,%%rbp\n\t" +#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp\t"#define __EXTRA_CLOBBER \ , "rcx", "rbx", "rdx", "r8", "r9", "r10", "r11", \ - "r12", "r13", "r14", "r15" + "r12", "r13", "r14", "r15", "flags"#ifdef CONFIG_CC_STACKPROTECTOR #define __switch_canary \
@@ -100,7 +100,11 @@ do { \#define __switch_canary_iparam#endif /* CC_STACKPROTECTOR */ -/* Save restore flags to clear handle leaking NT */ +/* +* There is no need to save or restore flags, because flags are always +* clean in kernel mode, with the possible exception of IOPL. Kernel IOPL +* has no effect. +*/ #define switch_to (prev, next, last) \ asm volatile(SAVE_CONTEXT \ "movq %%rsp,%P[threadrsp](%[prev])\n\t"/* save RSP */ \
Signature · 1 line
2.4.3
Re: [RFC 06/30] x86/sched/64: Don't save flags on context switch (reinstated)
Quoted 2 lines · Andy Lutomirski
This reinstates
2c7577a75837("sched/x86_64: Don't save flags on context switch"), which was reverted in512255a2ad2c.
Hi Ingo and Thomas-
I just realized that there's no good reason that this patch belongs with the rest of the entry series -- it's totally independent. Should I resend it by itself, or would you rather just apply it as is?
I'll send an updated entry series soon.
--Andy
Quoted 67 lines
Historically, Linux has always saved and restored EFLAGS across context switches. As far as I know, the only reason to do this is because of the NT flag. In particular, if something calls
switch_towith the NT flag set, then we don't want to leak the NT flag into a different task that might try to IRET and fail because NT is set.Before
8c7aa698baca("x86_64, entry: Filter RFLAGS.NT on entry from userspace"), we could run system call bodies with NT set. This would be a DoS or possibly privilege escalation hole if scheduling in such a system call would leak NT into a different task.Importantly, we don't need to worry about NT being set while preemptible or across page faults. The only way we can schedule due to preemption or a page fault is in an interrupt entry that nests inside the SYSENTER prologue. The CPU will clear NT when entering through an interrupt gate, so we won't schedule with NT set.
The only other interesting flags are IOPL and AC. Allowing
switch_toto change IOPL has no effect, as the value loaded during kernel execution doesn't matter at all except between a SYSENTER entry and the subsequent PUSHF, and anythign that interrupts in that window will restore IOPL on return.If we call
__switch_towith AC set, we have bigger problems.Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/include/asm/switch_to.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)arch/x86/include/asm/switch_to.h +8
-5@@ -79,12 +79,12 @@ do { \#else /* CONFIG_X86_32 */ /* frame pointer must be last for get_wchan */ -#define SAVE_CONTEXT "pushf ; pushq %%rbp ; movq %%rsi,%%rbp\n\t" -#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp ; popf\t" +#define SAVE_CONTEXT "pushq %%rbp ; movq %%rsi,%%rbp\n\t" +#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp\t"#define __EXTRA_CLOBBER \ , "rcx", "rbx", "rdx", "r8", "r9", "r10", "r11", \ - "r12", "r13", "r14", "r15" + "r12", "r13", "r14", "r15", "flags"#ifdef CONFIG_CC_STACKPROTECTOR #define __switch_canary \@@ -100,7 +100,11 @@ do { \#define __switch_canary_iparam#endif /* CC_STACKPROTECTOR */ -/* Save restore flags to clear handle leaking NT */ +/* +* There is no need to save or restore flags, because flags are always +* clean in kernel mode, with the possible exception of IOPL. Kernel IOPL +* has no effect. +*/ #define switch_to (prev, next, last) \ asm volatile(SAVE_CONTEXT \ "movq %%rsp,%P[threadrsp](%[prev])\n\t"/* save RSP */ \ --2.4.3
Signature · 2 lines
Andy Lutomirski AMA Capital Management, LLC
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/sched/64: Don't save flags on context switch ( reinstated)
In reply to Andy Lutomirskix86/sched/64: Don't save flags on context switch (reinstated)
Commit-ID: 3f2c5085ed99b6ad233cf77009c2f4f898b2f7c8 Gitweb: http://git.kernel.org/tip/3f2c5085ed99b6ad233cf77009c2f4f898b2f7c8 Author: Andy Lutomirski <[email protected]> AuthorDate: Tue, 1 Sep 2015 15:41:06 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 25 Sep 2015 09:29:17 +0200
x86/sched/64: Don't save flags on context switch (reinstated)
This reinstates the following commit:
2c7577a75837 ("sched/x86_64: Don't save flags on context switch")
which was reverted in:
512255a2ad2c ("Revert 'sched/x86_64: Don't save flags on context switch'")
Historically, Linux has always saved and restored EFLAGS across context switches. As far as I know, the only reason to do this is because of the NT flag. In particular, if something calls switch_to() with the NT flag set, then we don't want to leak the NT flag into a different task that might try to IRET and fail because NT is set.
Before this commit:
8c7aa698baca ("x86_64, entry: Filter RFLAGS.NT on entry from userspace")
we could run system call bodies with NT set. This would be a DoS or possibly privilege escalation hole if scheduling in such a system call would leak NT into a different task.
Importantly, we don't need to worry about NT being set while preemptible or across page faults. The only way we can schedule due to preemption or a page fault is in an interrupt entry that nests inside the SYSENTER prologue. The CPU will clear NT when entering through an interrupt gate, so we won't schedule with NT set.
The only other interesting flags are IOPL and AC. Allowing switch_to() to change IOPL has no effect, as the value loaded during kernel execution doesn't matter at all except between a SYSENTER entry and the subsequent PUSHF, and anythign that interrupts in that window will restore IOPL on return.
If we call __switch_to() with AC set, we have bigger problems.
--- arch/x86/include/asm/switch_to.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
arch/x86/include/asm/switch_to.h +8 -4
@@ -79,12 +79,12 @@ do { \#else /* CONFIG_X86_32 */ /* frame pointer must be last for get_wchan */ -#define SAVE_CONTEXT "pushf ; pushq %%rbp ; movq %%rsi,%%rbp\n\t" -#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp ; popf\t" +#define SAVE_CONTEXT "pushq %%rbp ; movq %%rsi,%%rbp\n\t" +#define RESTORE_CONTEXT "movq %%rbp,%%rsi ; popq %%rbp\t"#define __EXTRA_CLOBBER \ , "rcx", "rbx", "rdx", "r8", "r9", "r10", "r11", \ - "r12", "r13", "r14", "r15" + "r12", "r13", "r14", "r15", "flags"#ifdef CONFIG_CC_STACKPROTECTOR #define __switch_canary \
@@ -100,7 +100,11 @@ do { \#define __switch_canary_iparam#endif /* CC_STACKPROTECTOR */ -/* Save restore flags to clear handle leaking NT */ +/* +* There is no need to save or restore flags, because flags are always +* clean in kernel mode, with the possible exception of IOPL. Kernel IOPL +* has no effect. +*/ #define switch_to (prev, next, last) \ asm volatile(SAVE_CONTEXT \ "movq %%rsp,%P[threadrsp](%[prev])\n\t"/* save RSP */ \
Andy Lutomirski
maintainerRFC 7/30 x86/vdso: Remove runtime 32-bit vDSO selection
In reply to Andy Lutomirski32-bit userspace will now always see the same vDSO, which is exactly what used to be the int80 vDSO. Subsequent patches will clean it up and make it support SYSENTER and SYSCALL using alternatives.
--- arch/x86/entry/vdso/Makefile | 35 +++------- arch/x86/entry/vdso/vdso2c.c | 1 - arch/x86/entry/vdso/vdso32-setup.c | 28 +------- arch/x86/entry/vdso/vdso32/int80.S | 56 --------------- arch/x86/entry/vdso/vdso32/syscall.S | 75 -------------------- arch/x86/entry/vdso/vdso32/sysenter.S | 116 ------------------------------- arch/x86/entry/vdso/vdso32/system_call.S | 56 +++++++++++++++ arch/x86/entry/vdso/vma.c | 13 +--- arch/x86/ia32/ia32_signal.c | 4 +- arch/x86/include/asm/elf.h | 2 +- arch/x86/include/asm/vdso.h | 9 +-- arch/x86/kernel/signal.c | 4 +- arch/x86/xen/setup.c | 13 +--- 13 files changed, 77 insertions(+), 335 deletions(-) delete mode 100644 arch/x86/entry/vdso/vdso32/int80.S delete mode 100644 arch/x86/entry/vdso/vdso32/syscall.S delete mode 100644 arch/x86/entry/vdso/vdso32/sysenter.S create mode 100644 arch/x86/entry/vdso/vdso32/system_call.S
arch/x86/entry/vdso/Makefile +11 -24
@@ -19,9 +19,7 @@ obj-y += vma.o # vDSO images to build vdso_img-$(VDSO64-y) += 64 vdso_img-$(VDSOX32-y) += x32 -vdso_img-$(VDSO32-y) += 32-int80 -vdso_img-$(CONFIG_IA32_EMULATION) += 32-syscall -vdso_img-$(VDSO32-y) += 32-sysenter +vdso_img-$(VDSO32-y) += 32 obj-$(VDSO32-y) += vdso32-setup.o
@@ -122,15 +120,6 @@ $(obj)/%.so: $(obj)/%.so.dbg $(obj)/vdsox32.so.dbg: $(src)/vdsox32.lds $(vobjx32s) FORCE $(call if_changed,vdso) -# -# Build multiple 32-bit vDSO images to choose from at boot time. -# -vdso32.so-$(VDSO32-y) += int80 -vdso32.so-$(CONFIG_IA32_EMULATION) += syscall -vdso32.so-$(VDSO32-y) += sysenter - -vdso32-images = $(vdso32.so-y:%=vdso32-%.so) - CPPFLAGS_vdso32.lds = $(CPPFLAGS_vdso.lds) VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1
@@ -139,14 +128,12 @@ VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1 override obj-dirs = $(dir $(obj)) $(obj)/vdso32/ targets += vdso32/vdso32.lds -targets += vdso32/note.o vdso32/vclock_gettime.o $(vdso32.so-y:%=vdso32/%.o) +targets += vdso32/note.o vdso32/vclock_gettime.o vdso32/system_call.o targets += vdso32/vclock_gettime.o -$(obj)/vdso32.o: $(vdso32-images:%=$(obj)/%) - KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) -$(vdso32-images:%=$(obj)/%.dbg): KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) -$(vdso32-images:%=$(obj)/%.dbg): asflags-$(CONFIG_X86_64) += -m32 +$(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) +$(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32 KBUILD_CFLAGS_32 := $(filter-out -m64,$(KBUILD_CFLAGS)) KBUILD_CFLAGS_32 := $(filter-out -mcmodel=kernel,$(KBUILD_CFLAGS_32))
@@ -157,13 +144,13 @@ KBUILD_CFLAGS_32 += $(call cc-option, -fno-stack-protector) KBUILD_CFLAGS_32 += $(call cc-option, -foptimize-sibling-calls) KBUILD_CFLAGS_32 += -fno-omit-frame-pointer KBUILD_CFLAGS_32 += -DDISABLE_BRANCH_PROFILING -$(vdso32-images:%=$(obj)/%.dbg): KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) +$(obj)/vdso32.so.dbg: KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) -$(vdso32-images:%=$(obj)/%.dbg): $(obj)/vdso32-%.so.dbg: FORCE \ - $(obj)/vdso32/vdso32.lds \ - $(obj)/vdso32/vclock_gettime.o \ - $(obj)/vdso32/note.o \ - $(obj)/vdso32/%.o +$(obj)/vdso32.so.dbg: FORCE \ + $(obj)/vdso32/vdso32.lds \ + $(obj)/vdso32/vclock_gettime.o \ + $(obj)/vdso32/note.o \ + $(obj)/vdso32/system_call.o $(call if_changed,vdso) #
@@ -206,4 +193,4 @@ $(vdso_img_insttargets): install_%: $(obj)/%.dbg $(MODLIB)/vdso FORCE PHONY += vdso_install $(vdso_img_insttargets) vdso_install: $(vdso_img_insttargets) FORCE -clean-files := vdso32-syscall* vdso32-sysenter* vdso32-int80* vdso64* vdso-image-*.c vdsox32.so* +clean-files := vdso32.so vdso32.so.dbg vdso64* vdso-image-*.c vdsox32.so*
arch/x86/entry/vdso/vdso2c.c +0 -1
@@ -98,7 +98,6 @@ struct vdso_sym required_syms[] = {"VDSO_FAKE_SECTION_TABLE_END" , false }, {"VDSO32_NOTE_MASK", true}, - {"VDSO32_SYSENTER_RETURN", true}, {"__kernel_vsyscall", true}, {"__kernel_sigreturn", true}, {"__kernel_rt_sigreturn", true},
arch/x86/entry/vdso/vdso32-setup.c +1 -27
@@ -48,35 +48,9 @@ __setup("vdso32=", vdso32_setup);__setup_param ("vdso=" , vdso_setup, vdso32_setup,0 );#endif -#ifdef CONFIG_X86_64 - -#define vdso32_sysenter () (boot_cpu_has(X86_FEATURE_SYSENTER32)) -#define vdso32_syscall () (boot_cpu_has(X86_FEATURE_SYSCALL32)) - -#else /* CONFIG_X86_32 */ - -#define vdso32_sysenter () (boot_cpu_has(X86_FEATURE_SEP)) -#define vdso32_syscall () (0) - -#endif /* CONFIG_X86_64 */ - -#if defined(CONFIG_X86_32 ) || defined(CONFIG_COMPAT ) -const struct vdso_image *selected_vdso32; -#endif -int __init sysenter_setup (void ) { -#ifdef CONFIG_COMPAT -if (vdso32_syscall ()) - selected_vdso32 = &vdso_image_32_syscall; - else -#endif -if (vdso32_sysenter ()) - selected_vdso32 = &vdso_image_32_sysenter; -else - selected_vdso32 = &vdso_image_32_int80; - -init_vdso_image (selected_vdso32); +init_vdso_image (&vdso_image_32);return 0 ; }
arch/x86/entry/vdso/vdso32/int80.S deleted +0 -56
56 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/syscall.S deleted +0 -75
75 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/sysenter.S deleted +0 -116
116 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/system_call.S new file +56 -0
56 lines not on this page · raw message
arch/x86/entry/vdso/vma.c +1 -12
@@ -180,21 +180,10 @@ up_fail:#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )static int load_vdso32 (void ) { -int ret; -if (vdso32_enabled !=1 )/* Other values all mean "disabled" */ return 0 ; - ret =map_vdso (selected_vdso32, false); -if (ret) -return ret; - -if (selected_vdso32->sym_VDSO32_SYSENTER_RETURN) -current_thread_info ()->sysenter_return = - current->mm->context.vdso + - selected_vdso32->sym_VDSO32_SYSENTER_RETURN; - -return 0 ; +return map_vdso (&vdso_image_32, false); }#endif
arch/x86/ia32/ia32_signal.c +2 -2
@@ -289,7 +289,7 @@ int ia32_setup_frame(int sig, struct ksignal *ksig,/* Return stub is in 32bit vsyscall page */ if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode; }
@@ -368,7 +368,7 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, restorer = ksig->ka.sa.sa_restorer; else restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;put_user_ex (ptr_to_compat (restorer), &frame->pretcode);/*
arch/x86/include/asm/elf.h +1 -1
@@ -328,7 +328,7 @@ else \#define VDSO_ENTRY \ ((unsigned long)current->mm->context.vdso + \ - selected_vdso32->sym___kernel_vsyscall) + vdso_image_32.sym___kernel_vsyscall)struct linux_binprm ;
arch/x86/include/asm/vdso.h +1 -8
@@ -26,7 +26,6 @@ struct vdso_image {long sym___kernel_sigreturn;long sym___kernel_rt_sigreturn;long sym___kernel_vsyscall; -long sym_VDSO32_SYSENTER_RETURN; };#ifdef CONFIG_X86_64
@@ -38,13 +37,7 @@ extern const struct vdso_image vdso_image_x32;#endif #if definedCONFIG_X86_32 || definedCONFIG_COMPAT -extern const struct vdso_image vdso_image_32_int80; -#ifdef CONFIG_COMPAT -extern const struct vdso_image vdso_image_32_syscall; -#endif -extern const struct vdso_image vdso_image_32_sysenter; - -extern const struct vdso_image *selected_vdso32; +extern const struct vdso_image vdso_image_32;#endif extern void __init init_vdso_image (const struct vdso_image *image);
arch/x86/kernel/signal.c +2 -2
@@ -299,7 +299,7 @@ __setup_frame(int sig, struct ksignal *ksig, sigset_t *set,if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode;if (ksig->ka.sa.sa_flags &SA_RESTORER )
@@ -363,7 +363,7 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig,/* Set up to return from userspace. */ restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;if (ksig->ka.sa.sa_flags &SA_RESTORER ) restorer = ksig->ka.sa.sa_restorer;put_user_ex (restorer, &frame->pretcode);
arch/x86/xen/setup.c +2 -11
@@ -753,17 +753,8 @@ char * __init xen_auto_xlated_memory_setup(void)static void __init fiddle_vdso (void ) {#ifdef CONFIG_X86_32 -/* -* This could be called before selected_vdso32 is initialized, so -* just fiddle with both possible images. vdso_image_32_syscall -* can't be selected, since it only exists on 64-bit systems. -*/ -u32 *mask; - mask = vdso_image_32_int80.data + - vdso_image_32_int80.sym_VDSO32_NOTE_MASK; - *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ; - mask = vdso_image_32_sysenter.data + - vdso_image_32_sysenter.sym_VDSO32_NOTE_MASK; +u32 *mask = vdso_image_32.data + + vdso_image_32.sym_VDSO32_NOTE_MASK; *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ;#endif }
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 8/30 x86/asm: Re-add manual CFI infrastructure
In reply to Andy LutomirskiCommit 131484c8da97 ("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations") removed all the manual DWARF annotations outside the vDSO. It also removed the macros we used for the manual annotations.
Re-add the macros so that we can clean up the vDSO annotations.
--- arch/x86/Makefile | 10 ++- arch/x86/include/asm/dwarf2.h | 170 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/dwarf2.h
arch/x86/Makefile +8 -2
@@ -159,6 +159,12 @@ endif sp-$(CONFIG_X86_32) := esp sp-$(CONFIG_X86_64) := rsp +# do binutils support CFI? +cfi := $(call as-instr,.cfi_startproc\n.cfi_rel_offset $(sp-y)$(comma)0\n.cfi_endproc,-DCONFIG_AS_CFI=1) +# is .cfi_signal_frame supported too? +cfi-sigframe := $(call as-instr,.cfi_startproc\n.cfi_signal_frame\n.cfi_endproc,-DCONFIG_AS_CFI_SIGNAL_FRAME=1) +cfi-sections := $(call as-instr,.cfi_sections .debug_frame,-DCONFIG_AS_CFI_SECTIONS=1) + # does binutils support specific instructions? asinstr := $(call as-instr,fxsaveq (%rax),-DCONFIG_AS_FXSAVEQ=1) asinstr += $(call as-instr,pshufb %xmm0$(comma)%xmm0,-DCONFIG_AS_SSSE3=1)
@@ -166,8 +172,8 @@ asinstr += $(call as-instr,crc32l %eax$(comma)%eax,-DCONFIG_AS_CRC32=1) avx_instr := $(call as-instr,vxorps %ymm0$(comma)%ymm1$(comma)%ymm2,-DCONFIG_AS_AVX=1) avx2_instr :=$(call as-instr,vpbroadcastb %xmm0$(comma)%ymm1,-DCONFIG_AS_AVX2=1) -KBUILD_AFLAGS += $(asinstr) $(avx_instr) $(avx2_instr) -KBUILD_CFLAGS += $(asinstr) $(avx_instr) $(avx2_instr) +KBUILD_AFLAGS += $(cfi) $(cfi-sigframe) $(cfi-sections) $(asinstr) $(avx_instr) $(avx2_instr) +KBUILD_CFLAGS += $(cfi) $(cfi-sigframe) $(cfi-sections) $(asinstr) $(avx_instr) $(avx2_instr) LDFLAGS := -m elf_$(UTS_MACHINE)
arch/x86/include/asm/dwarf2.h new file +170 -0
170 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 9/30 x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm
In reply to Andy LutomirskiFor the vDSO, user code wants runtime unwind info. Make sure that, if we use .cfi directives, we generate it.
--- arch/x86/entry/vdso/Makefile | 4 ++-- arch/x86/include/asm/dwarf2.h | 13 ++++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)
arch/x86/entry/vdso/Makefile +2 -2
@@ -67,7 +67,7 @@ $(obj)/vdso-image-%.c: $(obj)/vdso%.so.dbg $(obj)/vdso%.so $(obj)/vdso2c FORCE CFL := $(PROFILING) -mcmodel=small -fPIC -O2 -fasynchronous-unwind-tables -m64 \ $(filter -g%,$(KBUILD_CFLAGS)) $(call cc-option, -fno-stack-protector) \ -fno-omit-frame-pointer -foptimize-sibling-calls \ - -DDISABLE_BRANCH_PROFILING + -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO $(vobjs): KBUILD_CFLAGS += $(CFL)
@@ -131,7 +131,7 @@ targets += vdso32/vdso32.lds targets += vdso32/note.o vdso32/vclock_gettime.o vdso32/system_call.o targets += vdso32/vclock_gettime.o -KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) +KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) -DBUILD_VDSO $(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) $(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32
arch/x86/include/asm/dwarf2.h +10 -3
@@ -36,15 +36,22 @@#endif #if defined(CONFIG_AS_CFI_SECTIONS ) && defined(__ASSEMBLY__) +#ifndef BUILD_VDSO /* * Emit CFI data in .debug_frame sections, not .eh_frame sections. * The latter we currently just discard since we don't do DWARF * unwinding at runtime. So only the offline DWARF information is -* useful to anyone. Note we should not use this directive if this -* file is used in the vDSO assembly, or if vmlinux.lds.S gets -* changed so it doesn't discard .eh_frame. +* useful to anyone. Note we should not use this directive if +* vmlinux.lds.S gets changed so it doesn't discard .eh_frame. */ .cfi_sections .debug_frame +#else +/* +* For the vDSO, emit both runtime unwind information and debug +* symbols for the .dbg file. +*/ + .cfi_sections .eh_frame, .debug_frame +#endif #endif #else
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 10/30 x86/vdso: Replace hex int80 CFI annotations with gas directives
In reply to Andy LutomirskiMaintaining the current CFI annotations written in R'lyehian is difficult for most of us. Translate them to something a little closer to English.
This will remove the CFI data for kernels built with extremely old versions of binutils. I think this is a fair tradeoff for the ability for mortals to edit the asm.
--- arch/x86/entry/vdso/vdso32/system_call.S | 48 ++++++-------------------------- 1 file changed, 8 insertions(+), 40 deletions(-)
arch/x86/entry/vdso/vdso32/system_call.S +8 -40
@@ -1,6 +1,10 @@ /* * Code for the vDSO. This version uses the old int $0x80 method. - * +*/ + +#include <asm/dwarf2.h> + +/* * First get the common code for the sigreturn entry points. * This must come first. */
@@ -11,46 +15,10 @@ .type __kernel_vsyscall,@function ALIGN __kernel_vsyscall: -.LSTART_vsyscall: + CFI_STARTPROC int $0x80 ret -.LEND_vsyscall: - .size __kernel_vsyscall,.-.LSTART_vsyscall - .previous - - .section .eh_frame,"a",@progbits -.LSTARTFRAMEDLSI: - .long .LENDCIEDLSI-.LSTARTCIEDLSI -.LSTARTCIEDLSI: - .long 0 /* CIE ID */ - .byte 1 /* Version number */ - .string "zR" /* NUL-terminated augmentation string */ - .uleb128 1 /* Code alignment factor */ - .sleb128 -4 /* Data alignment factor */ - .byte 8 /* Return address register column */ - .uleb128 1 /* Augmentation value length */ - .byte 0x1b /* DW_EH_PE_pcrel|DW_EH_PE_sdata4. */ - .byte 0x0c /* DW_CFA_def_cfa */ - .uleb128 4 - .uleb128 4 - .byte 0x88 /* DW_CFA_offset, column 0x8 */ - .uleb128 1 - .align 4 -.LENDCIEDLSI: - .long .LENDFDEDLSI-.LSTARTFDEDLSI /* Length FDE */ -.LSTARTFDEDLSI: - .long .LSTARTFDEDLSI-.LSTARTFRAMEDLSI /* CIE pointer */ - .long .LSTART_vsyscall-. /* PC-relative start address */ - .long .LEND_vsyscall-.LSTART_vsyscall - .uleb128 0 - .align 4 -.LENDFDEDLSI: - .previous + CFI_ENDPROC - /* - * Pad out the segment to match the size of the sysenter.S version. - */ -VDSO32_vsyscall_eh_frame_size = 0x40 - .section .data,"aw",@progbits - .space VDSO32_vsyscall_eh_frame_size-(.LENDFDEDLSI-.LSTARTFRAMEDLSI), 0 + .size __kernel_vsyscall,.-__kernel_vsyscall .previous
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 11/30 x86/vdso/32: Save extra registers in the INT80 vsyscall path
In reply to Andy LutomirskiThe goal is to integrate the SYSENTER and SYSCALL32 entry paths with the INT80 path. SYSENTER clobbers ESP and EIP. SYSCALL32 clobbers ECX (and, invisibly, R11). SYSRETL (long mode to compat mode) clobbers ECX and, invisibly, R11. SYSEXIT (which we only need for native 32-bit) clobbers ECX and EDX.
This means that we'll need to provide ESP to the kernel in a register (I chose ECX, since it's only needed for SYSENTER) and we need to provide the args that normally live in ECX and EDX in memory.
The epilogue needs to restore ECX and EDX, since user code relies on regs being preserved.
We don't need to do anything special about EIP, since the kernel already knows where we are. The kernel will eventually need to know where int $0x80 lands, so add a vdso_image entry for it.
The only user-visible effect of this code is that ptrace-induced changes to ECX and EDX during fast syscalls will be lost. This is already the case for the SYSENTER path.
--- arch/x86/entry/vdso/vdso2c.c | 1 + arch/x86/entry/vdso/vdso32/system_call.S | 25 ++++++++++++++++++++++++- arch/x86/include/asm/vdso.h | 1 + 3 files changed, 26 insertions(+), 1 deletion(-)
arch/x86/entry/vdso/vdso2c.c +1 -0
@@ -101,6 +101,7 @@ struct vdso_sym required_syms[] = { {"__kernel_vsyscall", true}, {"__kernel_sigreturn", true}, {"__kernel_rt_sigreturn", true}, + {"int80_landing_pad", true}, }; __attribute__((format(printf, 1, 2))) __attribute__((noreturn))
arch/x86/entry/vdso/vdso32/system_call.S +24 -1
@@ -16,7 +16,30 @@ ALIGN __kernel_vsyscall: CFI_STARTPROC - int $0x80 + /* + * Reshuffle regs so that all of any of the entry instructions + * will preserve enough state. + */ + pushl %edx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET edx, 0 + pushl %ecx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET ecx, 0 + movl %esp, %ecx + + /* Enter using int $0x80 */ + movl (%esp), %ecx + int $0x80 +GLOBAL(int80_landing_pad) + + /* Restore ECX and EDX in case they were clobbered. */ + popl %ecx + CFI_RESTORE ecx + CFI_ADJUST_CFA_OFFSET -4 + popl %edx + CFI_RESTORE edx + CFI_ADJUST_CFA_OFFSET -4 ret CFI_ENDPROC
arch/x86/include/asm/vdso.h +1 -0
@@ -26,6 +26,7 @@ struct vdso_image {long sym___kernel_sigreturn;long sym___kernel_rt_sigreturn;long sym___kernel_vsyscall; +long sym_int80_landing_pad; };#ifdef CONFIG_X86_64
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 12/30 x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries
In reply to Andy LutomirskiWe've disabled the vDSO helpers to call them, so turn off the entries entirely (temporarily) in preparation for cleaning them up.
--- arch/x86/entry/entry_64_compat.S | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/x86/entry/entry_64_compat.S +13 -0
@@ -103,6 +103,14 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: + /* Temporary: SYSENTER is disabled. */ +#ifdef CONFIG_CONTEXT_TRACKING + call enter_from_user_mode +#endif + ENABLE_INTERRUPTS(CLBR_NONE) + movl $11, %edi + call do_exit + /* * Re-enable interrupts. IRQ tracing already thinks that IRQs are * on (since we treat user mode as having IRQs on), and the
@@ -324,6 +332,11 @@ ENTRY(entry_SYSCALL_compat) * it is too small to ever cause noticeable irq latency. */ SWAPGS_UNSAFE_STACK + + /* Temporary: SYSCALL32 is disabled. */ + movl $-ENOSYS, %eax + USERGS_SYSRET32 + movl %esp, %r8d movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp ENABLE_INTERRUPTS(CLBR_NONE)
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 13/30 x86/entry/64/compat: Remove audit optimizations
In reply to Andy LutomirskiThese audit optimizations are messy and hard to maintain. We'll get a similar effect from opportunistic sysret when fast compat system calls are re-implemented.
--- arch/x86/entry/entry_64_compat.S | 98 +--------------------------------------- 1 file changed, 2 insertions(+), 96 deletions(-)
arch/x86/entry/entry_64_compat.S +2 -96
@@ -16,16 +16,6 @@ #include <linux/linkage.h> #include <linux/err.h> -/* Avoid __ASSEMBLER__'ifying <linux/audit.h> just for this. */ -#include <linux/elf-em.h> -#define AUDIT_ARCH_I386 (EM_386|__AUDIT_ARCH_LE) -#define __AUDIT_ARCH_LE 0x40000000 - -#ifndef CONFIG_AUDITSYSCALL -# define sysexit_audit ia32_ret_from_sys_call_irqs_off -# define sysretl_audit ia32_ret_from_sys_call_irqs_off -#endif - .section .entry.text, "ax" #ifdef CONFIG_PARAVIRT
@@ -148,7 +138,7 @@ sysenter_dispatch: DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysexit_audit + jnz ia32_ret_from_sys_call_irqs_off sysexit_from_sys_call: /* * NB: SYSEXIT is not obviously safe for 64-bit kernels -- an
@@ -200,81 +190,12 @@ sysexit_from_sys_call: */ USERGS_SYSRET32 -#ifdef CONFIG_AUDITSYSCALL - .macro auditsys_entry_common - /* - * At this point, registers hold syscall args in the 32-bit syscall ABI: - * EAX is syscall number, the 6 args are in EBX,ECX,EDX,ESI,EDI,EBP. - * - * We want to pass them to __audit_syscall_entry(), which is a 64-bit - * C function with 5 parameters, so shuffle them to match what - * the function expects: RDI,RSI,RDX,RCX,R8. - */ - movl %esi, %r8d /* arg5 (R8 ) <= 4th syscall arg (ESI) */ - xchg %ecx, %edx /* arg4 (RCX) <= 3rd syscall arg (EDX) */ - /* arg3 (RDX) <= 2nd syscall arg (ECX) */ - movl %ebx, %esi /* arg2 (RSI) <= 1st syscall arg (EBX) */ - movl %eax, %edi /* arg1 (RDI) <= syscall number (EAX) */ - call __audit_syscall_entry - - /* - * We are going to jump back to the syscall dispatch code. - * Prepare syscall args as required by the 64-bit C ABI. - * Registers clobbered by __audit_syscall_entry() are - * loaded from pt_regs on stack: - */ - movl ORIG_RAX(%rsp), %eax /* syscall number */ - movl %ebx, %edi /* arg1 */ - movl RCX(%rsp), %esi /* arg2 */ - movl RDX(%rsp), %edx /* arg3 */ - movl RSI(%rsp), %ecx /* arg4 */ - movl RDI(%rsp), %r8d /* arg5 */ - .endm - - .macro auditsys_exit exit - TRACE_IRQS_ON - ENABLE_INTERRUPTS(CLBR_NONE) - testl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call - movl %eax, %esi /* second arg, syscall return value */ - cmpl $-MAX_ERRNO, %eax /* is it an error ? */ - jbe 1f - movslq %eax, %rsi /* if error sign extend to 64 bits */ -1: setbe %al /* 1 if error, 0 if not */ - movzbl %al, %edi /* zero-extend that into %edi */ - call __audit_syscall_exit - movl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT), %edi - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl %edi, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz \exit - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - jmp int_ret_from_sys_call_irqs_off - .endm - -sysenter_auditsys: - auditsys_entry_common - movl %ebp, %r9d /* reload 6th syscall arg */ - jmp sysenter_dispatch - -sysexit_audit: - auditsys_exit sysexit_from_sys_call -#endif - sysenter_fix_flags: pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed sysenter_tracesys: -#ifdef CONFIG_AUDITSYSCALL - testl $(_TIF_WORK_SYSCALL_ENTRY & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz sysenter_auditsys -#endif SAVE_EXTRA_REGS xorl %eax, %eax /* Do not leak kernel information */ movq %rax, R11(%rsp)
@@ -389,7 +310,7 @@ cstar_dispatch: DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysretl_audit + jnz ia32_ret_from_sys_call_irqs_off sysretl_from_sys_call: andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS)
@@ -420,22 +341,7 @@ sysretl_from_sys_call: */ USERGS_SYSRET32 -#ifdef CONFIG_AUDITSYSCALL -cstar_auditsys: - movl %r9d, R9(%rsp) /* register to be clobbered by call */ - auditsys_entry_common - movl R9(%rsp), %r9d /* reload 6th syscall arg */ - jmp cstar_dispatch - -sysretl_audit: - auditsys_exit sysretl_from_sys_call -#endif - cstar_tracesys: -#ifdef CONFIG_AUDITSYSCALL - testl $(_TIF_WORK_SYSCALL_ENTRY & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz cstar_auditsys -#endif xchgl %r9d, %ebp SAVE_EXTRA_REGS xorl %eax, %eax /* Do not leak kernel information */
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 14/30 x86/entry/64/compat: Remove most of the fast system call machinery
In reply to Andy LutomirskiWe now have only one code path that calls through the compat syscall table. This will make it much more pleasant to change the pt_regs vs register calling convention, which we need to do to move the call into C.
--- arch/x86/entry/entry_64_compat.S | 246 +-------------------------------------- 1 file changed, 4 insertions(+), 242 deletions(-)
arch/x86/entry/entry_64_compat.S +4 -242
@@ -101,119 +101,13 @@ sysenter_flags_fixed: movl $11, %edi call do_exit - /* - * Re-enable interrupts. IRQ tracing already thinks that IRQs are - * on (since we treat user mode as having IRQs on), and the - * prologue above is too short for it to be worth adding a - * tracing round trip. - */ - ENABLE_INTERRUPTS(CLBR_NONE) - - /* - * no need to do an access_ok check here because rbp has been - * 32-bit zero extended - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysenter_tracesys - -sysenter_do_call: - /* 32-bit syscall -> 64-bit C ABI argument conversion */ - movl %edi, %r8d /* arg5 */ - movl %ebp, %r9d /* arg6 */ - xchg %ecx, %esi /* rsi:arg2, rcx:arg4 */ - movl %ebx, %edi /* arg1 */ - movl %edx, %edx /* arg3 (zero extension) */ -sysenter_dispatch: - cmpq $(IA32_NR_syscalls-1), %rax - ja 1f - call *ia32_sys_call_table(, %rax, 8) - movq %rax, RAX(%rsp) -1: - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call_irqs_off -sysexit_from_sys_call: - /* - * NB: SYSEXIT is not obviously safe for 64-bit kernels -- an - * NMI between STI and SYSEXIT has poorly specified behavior, - * and and NMI followed by an IRQ with usergs is fatal. So - * we just pretend we're using SYSEXIT but we really use - * SYSRETL instead. - * - * This code path is still called 'sysexit' because it pairs - * with 'sysenter' and it uses the SYSENTER calling convention. - */ - andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - movl RIP(%rsp), %ecx /* User %eip */ - movq RAX(%rsp), %rax - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - xorl %edx, %edx /* Do not leak kernel information */ - xorq %r8, %r8 - xorq %r9, %r9 - xorq %r10, %r10 - movl EFLAGS(%rsp), %r11d /* User eflags */ - TRACE_IRQS_ON - - /* - * SYSRETL works even on Intel CPUs. Use it in preference to SYSEXIT, - * since it avoids a dicey window with interrupts enabled. - */ - movl RSP(%rsp), %esp - - /* - * USERGS_SYSRET32 does: - * gsbase = user's gs base - * eip = ecx - * rflags = r11 - * cs = __USER32_CS - * ss = __USER_DS - * - * The prologue set RIP(%rsp) to VDSO32_SYSENTER_RETURN, which does: - * - * pop %ebp - * pop %edx - * pop %ecx - * - * Therefore, we invoke SYSRETL with EDX and R8-R10 zeroed to - * avoid info leaks. R11 ends up with VDSO32_SYSENTER_RETURN's - * address (already known to user code), and R12-R15 are - * callee-saved and therefore don't contain any interesting - * kernel data. - */ - USERGS_SYSRET32 + /* Unreachable. */ + ud2 sysenter_fix_flags: pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed - -sysenter_tracesys: - SAVE_EXTRA_REGS - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - movq %rsp, %rdi /* &pt_regs -> arg1 */ - call syscall_trace_enter - - /* Reload arg registers from stack. (see sysenter_tracesys) */ - movl RCX(%rsp), %ecx - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl %eax, %eax /* zero extension */ - - RESTORE_EXTRA_REGS - jmp sysenter_do_call ENDPROC(entry_SYSENTER_compat) /*
@@ -280,142 +174,10 @@ ENTRY(entry_SYSCALL_compat) pushq $-ENOSYS /* pt_regs->ax */ sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ - /* - * No need to do an access_ok check here because r8 has been - * 32-bit zero extended: - */ - ASM_STAC -1: movl (%r8), %r9d - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz cstar_tracesys - -cstar_do_call: - /* 32-bit syscall -> 64-bit C ABI argument conversion */ - movl %edi, %r8d /* arg5 */ - /* r9 already loaded */ /* arg6 */ - xchg %ecx, %esi /* rsi:arg2, rcx:arg4 */ - movl %ebx, %edi /* arg1 */ - movl %edx, %edx /* arg3 (zero extension) */ - -cstar_dispatch: - cmpq $(IA32_NR_syscalls-1), %rax - ja 1f - - call *ia32_sys_call_table(, %rax, 8) - movq %rax, RAX(%rsp) -1: - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call_irqs_off - -sysretl_from_sys_call: - andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl RIP(%rsp), %ecx - movl EFLAGS(%rsp), %r11d - movq RAX(%rsp), %rax - xorq %r10, %r10 - xorq %r9, %r9 - xorq %r8, %r8 - TRACE_IRQS_ON - movl RSP(%rsp), %esp - /* - * 64-bit->32-bit SYSRET restores eip from ecx, - * eflags from r11 (but RF and VM bits are forced to 0), - * cs and ss are loaded from MSRs. - * (Note: 32-bit->32-bit SYSRET is different: since r11 - * does not exist, it merely sets eflags.IF=1). - * - * NB: On AMD CPUs with the X86_BUG_SYSRET_SS_ATTRS bug, the ss - * descriptor is not reinitialized. This means that we must - * avoid SYSRET with SS == NULL, which could happen if we schedule, - * exit the kernel, and re-enter using an interrupt vector. (All - * interrupt entries on x86_64 set SS to NULL.) We prevent that - * from happening by reloading SS in __switch_to. - */ - USERGS_SYSRET32 - -cstar_tracesys: - xchgl %r9d, %ebp - SAVE_EXTRA_REGS - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %r9, R9(%rsp) - movq %rax, R8(%rsp) - movq %rsp, %rdi /* &pt_regs -> arg1 */ - call syscall_trace_enter - movl R9(%rsp), %r9d - - /* Reload arg registers from stack. (see sysenter_tracesys) */ - movl RCX(%rsp), %ecx - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl %eax, %eax /* zero extension */ - - RESTORE_EXTRA_REGS - xchgl %ebp, %r9d - jmp cstar_do_call + /* Unreachable. */ + ud2 END(entry_SYSCALL_compat) -ia32_badarg: - /* - * So far, we've entered kernel mode, set AC, turned on IRQs, and - * saved C regs except r8-r11. We haven't done any of the other - * standard entry work, though. We want to bail, but we shouldn't - * treat this as a syscall entry since we don't even know what the - * args are. Instead, treat this as a non-syscall entry, finish - * the entry work, and immediately exit after setting AX = -EFAULT. - * - * We're really just being polite here. Killing the task outright - * would be a reasonable action, too. Given that the only valid - * way to have gotten here is through the vDSO, and we already know - * that the stack pointer is bad, the task isn't going to survive - * for long no matter what we do. - */ - - ASM_CLAC /* undo STAC */ - movq $-EFAULT, RAX(%rsp) /* return -EFAULT if possible */ - - /* Fill in the rest of pt_regs */ - xorl %eax, %eax - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - SAVE_EXTRA_REGS - - /* Turn IRQs back off. */ - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - - /* Now finish entering normal kernel mode. */ -#ifdef CONFIG_CONTEXT_TRACKING - call enter_from_user_mode -#endif - - /* And exit again. */ - jmp retint_user - -ia32_ret_from_sys_call_irqs_off: - TRACE_IRQS_ON - ENABLE_INTERRUPTS(CLBR_NONE) - -ia32_ret_from_sys_call: - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - jmp int_ret_from_sys_call - /* * Emulated IA32 system calls via int 0x80. *
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 15/30 x86/entry/64/compat: Set up full pt_regs for all compat syscalls
In reply to Andy LutomirskiThis is conceptually simpler. More importantly, it eliminates the PTREGSCALL stubs, which were not compatible with the C ABI. This means that C code can call through the compat syscall table.
--- arch/x86/entry/entry_64_compat.S | 42 +++++++++++----------------------- arch/x86/entry/syscalls/syscall_32.tbl | 8 +++---- 2 files changed, 17 insertions(+), 33 deletions(-)
arch/x86/entry/entry_64_compat.S +13 -29
@@ -219,12 +219,18 @@ ENTRY(entry_INT80_compat) pushq %rdx /* pt_regs->dx */ pushq %rcx /* pt_regs->cx */ pushq $-ENOSYS /* pt_regs->ax */ - pushq $0 /* pt_regs->r8 */ - pushq $0 /* pt_regs->r9 */ - pushq $0 /* pt_regs->r10 */ - pushq $0 /* pt_regs->r11 */ + xorq %r8,%r8 + pushq %r8 /* pt_regs->r8 = 0 */ + pushq %r8 /* pt_regs->r9 = 0 */ + pushq %r8 /* pt_regs->r10 = 0 */ + pushq %r8 /* pt_regs->r11 = 0 */ + pushq %rbx /* pt_regs->rbx */ + pushq %rbp /* pt_regs->rbp */ + pushq %r8 /* pt_regs->r12 = 0 */ + pushq %r8 /* pt_regs->r13 = 0 */ + pushq %r8 /* pt_regs->r14 = 0 */ + pushq %r8 /* pt_regs->r15 = 0 */ cld - sub $(6*8), %rsp /* pt_regs->bp, bx, r12-15 not saved */ orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS)
@@ -243,10 +249,10 @@ ia32_do_call: call *ia32_sys_call_table(, %rax, 8) movq %rax, RAX(%rsp) 1: + RESTORE_EXTRA_REGS jmp int_ret_from_sys_call ia32_tracesys: - SAVE_EXTRA_REGS movq %rsp, %rdi /* &pt_regs -> arg1 */ call syscall_trace_enter /*
@@ -261,25 +267,11 @@ ia32_tracesys: movl RSI(%rsp), %esi movl RDI(%rsp), %edi movl %eax, %eax /* zero extension */ - RESTORE_EXTRA_REGS jmp ia32_do_call END(entry_INT80_compat) - .macro PTREGSCALL label, func - ALIGN -GLOBAL(\label) - leaq \func(%rip), %rax - jmp ia32_ptregs_common - .endm - - PTREGSCALL stub32_rt_sigreturn, sys32_rt_sigreturn - PTREGSCALL stub32_sigreturn, sys32_sigreturn - PTREGSCALL stub32_fork, sys_fork - PTREGSCALL stub32_vfork, sys_vfork - ALIGN GLOBAL(stub32_clone) - leaq sys_clone(%rip), %rax /* * The 32-bit clone ABI is: clone(..., int tls_val, int *child_tidptr). * The 64-bit clone ABI is: clone(..., int *child_tidptr, int tls_val).
@@ -288,12 +280,4 @@ GLOBAL(stub32_clone) * so we need to swap arguments here before calling it: */ xchg %r8, %rcx - jmp ia32_ptregs_common - - ALIGN -ia32_ptregs_common: - SAVE_EXTRA_REGS 8 - call *%rax - RESTORE_EXTRA_REGS 8 - ret -END(ia32_ptregs_common) + jmp sys_clone
arch/x86/entry/syscalls/syscall_32.tbl +4 -4
@@ -8,7 +8,7 @@ # 0 i386 restart_syscall sys_restart_syscall 1 i386 exit sys_exit -2 i386 fork sys_fork stub32_fork +2 i386 fork sys_fork sys_fork 3 i386 read sys_read 4 i386 write sys_write 5 i386 open sys_open compat_sys_open
@@ -125,7 +125,7 @@ 116 i386 sysinfo sys_sysinfo compat_sys_sysinfo 117 i386 ipc sys_ipc compat_sys_ipc 118 i386 fsync sys_fsync -119 i386 sigreturn sys_sigreturn stub32_sigreturn +119 i386 sigreturn sys_sigreturn sys32_sigreturn 120 i386 clone sys_clone stub32_clone 121 i386 setdomainname sys_setdomainname 122 i386 uname sys_newuname
@@ -179,7 +179,7 @@ 170 i386 setresgid sys_setresgid16 171 i386 getresgid sys_getresgid16 172 i386 prctl sys_prctl -173 i386 rt_sigreturn sys_rt_sigreturn stub32_rt_sigreturn +173 i386 rt_sigreturn sys_rt_sigreturn sys32_rt_sigreturn 174 i386 rt_sigaction sys_rt_sigaction compat_sys_rt_sigaction 175 i386 rt_sigprocmask sys_rt_sigprocmask 176 i386 rt_sigpending sys_rt_sigpending compat_sys_rt_sigpending
@@ -196,7 +196,7 @@ 187 i386 sendfile sys_sendfile compat_sys_sendfile 188 i386 getpmsg 189 i386 putpmsg -190 i386 vfork sys_vfork stub32_vfork +190 i386 vfork sys_vfork sys_vfork 191 i386 ugetrlimit sys_getrlimit compat_sys_getrlimit 192 i386 mmap2 sys_mmap_pgoff 193 i386 truncate64 sys_truncate64 sys32_truncate64
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 16/30 x86/entry/syscalls: Move syscall table declarations into asm/syscalls.h
In reply to Andy LutomirskiThe header was missing some compat declarations.
Also make sys_call_ptr_t have a consistent type.
--- arch/x86/entry/syscall_32.c | 5 +---- arch/x86/include/asm/syscall.h | 12 +++++++++++- arch/x86/um/sys_call_table_32.c | 3 +-- arch/x86/um/sys_call_table_64.c | 3 +-- 4 files changed, 14 insertions(+), 9 deletions(-)
arch/x86/entry/syscall_32.c +1 -4
@@ -4,13 +4,12 @@#include <linux/sys.h> #include <linux/cache.h> #include <asm/asm-offsets.h> +#include <asm/syscall.h> #ifdef CONFIG_IA32_EMULATION #define SYM (sym, compat) compat#else #define SYM (sym, compat) sym -#define ia32_sys_call_table sys_call_table -#define __NR_syscall_compat_max __NR_syscall_max#endif #define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void SYM(sym, compat)(void) ;
@@ -19,8 +18,6 @@#define __SYSCALL_I386 (nr, sym, compat) [nr] = SYM(sym, compat), -typedef asmlinkage void (*sys_call_ptr_t)(void); - extern asmlinkage void sys_ni_syscall(void); __visible const sys_call_ptr_t ia32_sys_call_table[__NR_syscall_compat_max+1] = {
arch/x86/include/asm/syscall.h +11 -1
@@ -20,9 +20,19 @@#include <asm/thread_info.h> /* for TS_COMPAT */ #include <asm/unistd.h> -typedef void (*sys_call_ptr_t )(void ); +typedef asmlinkage void (*sys_call_ptr_t)(void);extern const sys_call_ptr_t sys_call_table[]; +#if defined(CONFIG_X86_32 ) +#define ia32_sys_call_table sys_call_table +#define __NR_syscall_compat_max __NR_syscall_max +#define IA32_NR_syscalls NR_syscalls +#endif + +#if defined(CONFIG_IA32_EMULATION ) +extern const sys_call_ptr_t ia32_sys_call_table[]; +#endif +/* * Only the low 32 bits of orig_ax are meaningful, so we return int. * This importantly ignores the high bits on 64-bit, so comparisons
arch/x86/um/sys_call_table_32.c +1 -2
@@ -7,6 +7,7 @@#include <linux/sys.h> #include <linux/cache.h> #include <generated/user_constants.h> +#include <asm/syscall.h> #define __NO_STUBS
@@ -30,8 +31,6 @@#undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [ nr ] = sym, -typedef asmlinkage void (*sys_call_ptr_t)(void); - extern asmlinkage void sys_ni_syscall(void);const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {
arch/x86/um/sys_call_table_64.c +1 -2
@@ -7,6 +7,7 @@#include <linux/sys.h> #include <linux/cache.h> #include <generated/user_constants.h> +#include <asm/syscall.h> #define __NO_STUBS
@@ -43,8 +44,6 @@#undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [ nr ] = sym, -typedef void (*sys_call_ptr_t )(void ); -extern void sys_ni_syscall (void );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 17/30 x86/syscalls: Give sys_call_ptr_t a useful type
In reply to Andy LutomirskiSyscalls are asmlinkage functions (on 32-bit kernels), take six args of type unsigned long, and return long. Note that uml could probably be slightly cleaned up on top of this patch.
--- arch/x86/entry/syscall_32.c | 4 ++-- arch/x86/entry/syscall_64.c | 4 ++-- arch/x86/include/asm/syscall.h | 4 +++- arch/x86/um/sys_call_table_32.c | 4 ++-- arch/x86/um/sys_call_table_64.c | 4 ++-- 5 files changed, 11 insertions(+), 9 deletions(-)
arch/x86/entry/syscall_32.c +2 -2
@@ -12,13 +12,13 @@#define SYM (sym, compat) sym#endif -#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void SYM(sym, compat)(void) ; +#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage long SYM(sym, compat)(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_32.h> #undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [nr] = SYM(sym, compat), -extern asmlinkage void sys_ni_syscall(void); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ); __visible const sys_call_ptr_t ia32_sys_call_table[__NR_syscall_compat_max+1] = {/*
arch/x86/entry/syscall_64.c +2 -2
@@ -14,13 +14,13 @@# define __SYSCALL_X32 (nr, sym, compat)/* nothing */ #endif -#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_64.h> #undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [nr] = sym, -extern void sys_ni_syscall (void ); +extern long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ); asmlinkage const sys_call_ptr_t sys_call_table[__NR_syscall_max+1] = {/*
arch/x86/include/asm/syscall.h +3 -1
@@ -20,7 +20,9 @@#include <asm/thread_info.h> /* for TS_COMPAT */ #include <asm/unistd.h> -typedef asmlinkage void (*sys_call_ptr_t)(void); +typedef asmlinkage long (*sys_call_ptr_t )(unsigned long ,unsigned long , +unsigned long ,unsigned long , +unsigned long ,unsigned long );extern const sys_call_ptr_t sys_call_table[];#if defined(CONFIG_X86_32 )
arch/x86/um/sys_call_table_32.c +2 -2
@@ -25,13 +25,13 @@#define old_mmap sys_old_mmap -#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_32.h> #undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [ nr ] = sym, -extern asmlinkage void sys_ni_syscall(void); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {/*
arch/x86/um/sys_call_table_64.c +2 -2
@@ -38,13 +38,13 @@#define __SYSCALL_COMMON (nr, sym, compat) __SYSCALL_64(nr, sym, compat)#define __SYSCALL_X32 (nr, sym, compat)/* Not supported */ -#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_64.h> #undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [ nr ] = sym, -extern void sys_ni_syscall (void ); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {/*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 18/30 x86/entry: Add do_syscall_32, a C function to do 32-bit syscalls
In reply to Andy LutomirskiSystem calls are really quite simple. Add a helper to call a 32-bit system call.
--- arch/x86/entry/common.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/entry/common.c +43 -0
@@ -318,3 +318,46 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)local_irq_disable ();prepare_exit_to_usermode (regs); } + +#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION ) +/* +* Does a 32-bit syscall. Called with IRQs off and does all entry and +* exit work. +*/ +__visible void do_int80_syscall_32(struct pt_regs *regs) +{ +struct thread_info *ti =pt_regs_to_thread_info (regs); +unsigned int nr = (unsigned int )regs->orig_ax; + +#ifdef CONFIG_IA32_EMULATION + ti->status |=TS_COMPAT ; +#endif + +local_irq_enable (); + +if (READ_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY) { +/* +* Subtlety here: if ptrace pokes something larger than +* 2^32-1 into orig_ax, this truncates it. This may or +* may not be necessary, but it matches the old asm +* behavior. +*/ + nr =syscall_trace_enter (regs); + } + +if (nr < IA32_NR_syscalls) { +/* +* It's possible that a 32-bit syscall implementation +* takes a 64-bit parameter but nonetheless assumes that +* the high bits are zero. Make sure we zero-extend all +* of the args. +*/ + regs->ax = ia32_sys_call_table[nr]( + (unsigned int )regs->bx, (unsigned int )regs->cx, + (unsigned int )regs->dx, (unsigned int )regs->si, + (unsigned int )regs->di, (unsigned int )regs->bp); + } + +syscall_return_slowpath (regs); +} +#endif
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 19/30 x86/entry/64/compat: Migrate the body of the syscall entry to C
In reply to Andy Lutomirski--- arch/x86/entry/entry_64.S | 2 +- arch/x86/entry/entry_64_compat.S | 56 +++++++++++++--------------------------- 2 files changed, 19 insertions(+), 39 deletions(-)
arch/x86/entry/entry_64.S +1 -1
@@ -586,7 +586,7 @@ retint_kernel: * At this label, code paths which return to kernel and to user, * which come from interrupts/exception and from syscalls, merge. */ -restore_regs_and_iret: +GLOBAL(restore_regs_and_iret) RESTORE_EXTRA_REGS restore_c_regs_and_iret: RESTORE_C_REGS
arch/x86/entry/entry_64_compat.S +18 -38
74 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 20/30 x86/entry: Add C code for fast system call entries
In reply to Andy LutomirskiThis handles both SYSENTER and SYSCALL. The asm glue will take care of the differences.
--- arch/x86/entry/common.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
arch/x86/entry/common.c +39 -0
@@ -24,6 +24,8 @@#include <asm/desc.h> #include <asm/traps.h> +#include <asm/vdso.h> +#include <asm/uaccess.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -360,4 +362,41 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } + +__visible void do_fast_syscall_32(struct pt_regs *regs) +{ +/* +* Called using the internal vDSO SYSENTER calling convention. +* Adjust regs so it looks like we entered using int80. +*/ + +unsigned long landing_pad = (unsigned long )current->mm->context.vdso + + vdso_image_32.sym_int80_landing_pad; + +/* SYSENTER loses EIP. Fix it up. */ + regs->ip = landing_pad; + +/* +* Fetch ECX from where the vDSO stashed it. +* +* WARNING: We are in CONTEXT_USER and RCU isn't paying attention! +*/ +local_irq_enable (); +if (get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long )(u32 )regs->sp)) { +/* User code screwed up. */ +local_irq_disable (); + regs->ax = -EFAULT ; +#ifdef CONFIG_CONTEXT_TRACKING +enter_from_user_mode (); +#endif +prepare_exit_to_usermode (regs); +return ; + } +local_irq_disable (); + +/* Now this is just like a normal syscall. */ +do_int80_syscall_32 (regs); +return ; +}#endif
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 21/30 x86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace
In reply to Andy LutomirskiWhat, you didn't realize that SYSENTER and SYSCALL were actually the same thing? :)
Unlike the old code, this actually passes the ptrace_syscall_32 test on AMD systems.
--- arch/x86/entry/entry_64_compat.S | 90 +++++++++++++++++++------------- arch/x86/entry/vdso/vdso32/system_call.S | 8 +++ 2 files changed, 62 insertions(+), 36 deletions(-)
arch/x86/entry/entry_64_compat.S +54 -36
@@ -52,15 +52,18 @@ ENTRY(entry_SYSENTER_compat) SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - /* Zero-extending 32-bit regs, do not remove */ - movl %ebp, %ebp + /* + * User tracing code (ptrace or signal handlers) might assume that + * the saved RAX contains a 32-bit number when we're invoking a 32-bit + * syscall. Just in case the high bits are nonzero, zero-extend + * the syscall number. (This could almost certainly be deleted + * with no ill effects.) + */ movl %eax, %eax - movl ASM_THREAD_INFO(TI_sysenter_return, %rsp, 0), %r10d - /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ - pushq %rbp /* pt_regs->sp */ + pushq %rcx /* pt_regs->sp */ /* * Push flags. This is nasty. First, interrupts are currently
@@ -70,17 +73,28 @@ ENTRY(entry_SYSENTER_compat) */ pushfq /* pt_regs->flags (except IF = 0) */ orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */ + ASM_CLAC /* Clear AC after saving FLAGS */ pushq $__USER32_CS /* pt_regs->cs */ - pushq %r10 /* pt_regs->ip = thread_info->sysenter_return */ + xorq %r8,%r8 + pushq %r8 /* pt_regs->ip = 0 (placeholder) */ pushq %rax /* pt_regs->orig_ax */ pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */ pushq %rdx /* pt_regs->dx */ - pushq %rcx /* pt_regs->cx */ + pushq %rcx /* pt_regs->cx (will be overwritten) */ pushq $-ENOSYS /* pt_regs->ax */ + pushq %r8 /* pt_regs->r8 = 0 */ + pushq %r8 /* pt_regs->r9 = 0 */ + pushq %r8 /* pt_regs->r10 = 0 */ + pushq %r8 /* pt_regs->r11 = 0 */ + pushq %rbx /* pt_regs->rbx */ + pushq %rbp /* pt_regs->rbp */ + pushq %r8 /* pt_regs->r12 = 0 */ + pushq %r8 /* pt_regs->r13 = 0 */ + pushq %r8 /* pt_regs->r14 = 0 */ + pushq %r8 /* pt_regs->r15 = 0 */ cld - sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* * Sysenter doesn't filter flags, so we need to clear NT
@@ -93,16 +107,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: - /* Temporary: SYSENTER is disabled. */ -#ifdef CONFIG_CONTEXT_TRACKING - call enter_from_user_mode -#endif - ENABLE_INTERRUPTS(CLBR_NONE) - movl $11, %edi - call do_exit + /* + * User mode is traced as though IRQs are on, and SYSENTER + * turned them off. + */ + TRACE_IRQS_OFF - /* Unreachable. */ - ud2 + movq %rsp, %rdi + call do_fast_syscall_32 + jmp .Lsyscall_32_done sysenter_fix_flags: pushq $X86_EFLAGS_FIXED
@@ -135,26 +148,14 @@ ENDPROC(entry_SYSENTER_compat) * edi arg5 * esp user stack * 0(%esp) arg6 - * - * This is purely a fast path. For anything complicated we use the int 0x80 - * path below. We set up a complete hardware stack frame to share code - * with the int 0x80 path. */ ENTRY(entry_SYSCALL_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK - /* Temporary: SYSCALL32 is disabled. */ - movl $-ENOSYS, %eax - USERGS_SYSRET32 - + /* Stash user ESP and switch to the kernel stack. */ movl %esp, %r8d movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %eax, %eax
@@ -169,13 +170,29 @@ ENTRY(entry_SYSCALL_compat) pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */ pushq %rdx /* pt_regs->dx */ - pushq %rbp /* pt_regs->cx */ - movl %ebp, %ecx + pushq %rcx /* pt_regs->cx (will be overwritten) */ pushq $-ENOSYS /* pt_regs->ax */ - sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ + xorq %r8,%r8 + pushq %r8 /* pt_regs->r8 = 0 */ + pushq %r8 /* pt_regs->r9 = 0 */ + pushq %r8 /* pt_regs->r10 = 0 */ + pushq %r8 /* pt_regs->r11 = 0 */ + pushq %rbx /* pt_regs->rbx */ + pushq %rbp /* pt_regs->rbp */ + pushq %r8 /* pt_regs->r12 = 0 */ + pushq %r8 /* pt_regs->r13 = 0 */ + pushq %r8 /* pt_regs->r14 = 0 */ + pushq %r8 /* pt_regs->r15 = 0 */ + + /* + * User mode is traced as though IRQs are on, and SYSENTER + * turned them off. + */ + TRACE_IRQS_OFF - /* Unreachable. */ - ud2 + movq %rsp, %rdi + call do_fast_syscall_32 + jmp .Lsyscall_32_done END(entry_SYSCALL_compat) /*
@@ -243,6 +260,7 @@ ENTRY(entry_INT80_compat) movq %rsp, %rdi call do_int80_syscall_32 +.Lsyscall_32_done: /* Go back to user mode. */ TRACE_IRQS_ON
arch/x86/entry/vdso/vdso32/system_call.S +8 -0
@@ -3,6 +3,8 @@ */ #include <asm/dwarf2.h> +#include <asm/cpufeature.h> +#include <asm/alternative-asm.h> /* * First get the common code for the sigreturn entry points.
@@ -28,6 +30,12 @@ __kernel_vsyscall: CFI_REL_OFFSET ecx, 0 movl %esp, %ecx +#ifdef CONFIG_X86_64 + /* If SYSENTER (Intel) or SYSCALL32 (AMD) is available, use it. */ + ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ + "syscall", X86_FEATURE_SYSCALL32 +#endif + /* Enter using int $0x80 */ movl (%esp), %ecx int $0x80
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 22/30 x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
In reply to Andy LutomirskiIf CS, SS and IP are as expected and FLAGS is compatible with SYSRETL, then return from fast compat syscalls (both SYSCALL and SYSENTER) using SYSRETL.
Unlike native 64-bit opportunistic SYSRET, this is not invisible to user code: RCX and R8-R15 end up in a different state than shown saved in pt_regs. To compensate, we only do this when returning to the vDSO fast syscall return path. This won't interfere with syscall restart, as we won't use SYSRETL when returning to the INT80 restart instruction.
--- arch/x86/entry/common.c | 23 +++++++++++++++++++--- arch/x86/entry/entry_64_compat.S | 42 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 5 deletions(-)
arch/x86/entry/common.c +20 -3
@@ -363,7 +363,8 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } -__visible void do_fast_syscall_32(struct pt_regs *regs) +/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ +__visible long do_fast_syscall_32 (struct pt_regs *regs) {/* * Called using the internal vDSO SYSENTER calling convention.
@@ -391,12 +392,28 @@ __visible void do_fast_syscall_32(struct pt_regs *regs)enter_from_user_mode ();#endif prepare_exit_to_usermode (regs); - return; + return 0; /* Keep it simple: use IRET. */ }local_irq_disable ();/* Now this is just like a normal syscall. */ do_int80_syscall_32 (regs); -return ; + +#ifdef CONFIG_X86_64 +/* +* Opportunistic SYSRETL: if possible, try to return using SYSRETL. +* SYSRETL is available on all 64-bit CPUs, so we don't need to +* bother with SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +*/ +return regs->cs == __USER32_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ; +#else +return 0 ; +#endif }#endif
arch/x86/entry/entry_64_compat.S +40 -2
54 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 23/30 x86/entry/32: Open-code return tracking from fork and kthreads
In reply to Andy Lutomirskisyscall_exit is going away, and return tracing is just a function call now, so open-code the two non-syscall 32-bit users.
While we're at it, update the big register layout comment.
--- arch/x86/entry/entry_32.S | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
arch/x86/entry/entry_32.S +15 -3
@@ -3,7 +3,7 @@ * * entry_32.S contains the system-call and low-level fault and trap handling routines. * - * Stack layout in 'syscall_exit': + * Stack layout while running C code: * ptrace needs to have all registers on the stack. * If the order here is changed, it needs to be * updated in fork.c:copy_process(), signal.c:do_signal(),
@@ -211,7 +211,11 @@ ENTRY(ret_from_fork) popl %eax pushl $0x0202 # Reset kernel eflags popfl - jmp syscall_exit + + /* When we fork, we trace the syscall return in the child, too. */ + movl %esp, %eax + call syscall_return_slowpath + jmp restore_all END(ret_from_fork) ENTRY(ret_from_kernel_thread)
@@ -224,7 +228,15 @@ ENTRY(ret_from_kernel_thread) movl PT_EBP(%esp), %eax call *PT_EBX(%esp) movl $0, PT_EAX(%esp) - jmp syscall_exit + + /* + * Kernel threads return to userspace as if returning from a syscall. + * We should check whether anything actually uses this path and, if so, + * consider switching it over to ret_from_fork. + */ + movl %esp, %eax + call syscall_return_slowpath + jmp restore_all ENDPROC(ret_from_kernel_thread) /*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 24/30 x86/entry/32: Switch INT80 to the new C syscall path
In reply to Andy Lutomirski--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)
arch/x86/entry/entry_32.S +13 -21
@@ -153,13 +153,13 @@ #endif /* CONFIG_X86_32_LAZY_GS */ -.macro SAVE_ALL +.macro SAVE_ALL pt_regs_ax=%eax cld PUSH_GS pushl %fs pushl %es pushl %ds - pushl %eax + pushl \pt_regs_ax pushl %ebp pushl %edi pushl %esi
@@ -370,20 +370,17 @@ ENDPROC(entry_SYSENTER_32) # system call handler stub ENTRY(entry_INT80_32) ASM_CLAC - pushl %eax # save orig_eax - SAVE_ALL - GET_THREAD_INFO(%ebp) - # system call tracing in operation / emulation - testl $_TIF_WORK_SYSCALL_ENTRY, TI_flags(%ebp) - jnz syscall_trace_entry - cmpl $(NR_syscalls), %eax - jae syscall_badsys -syscall_call: - call *sys_call_table(, %eax, 4) -syscall_after_call: - movl %eax, PT_EAX(%esp) # store the return value -syscall_exit: - jmp syscall_exit_work + pushl %eax /* pt_regs->orig_ax */ + SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest, load -ENOSYS into ax */ + + /* + * User mode is traced as though IRQs are on, and the interrupt gate + * turned them off. + */ + TRACE_IRQS_OFF + + movl %esp, %eax + call do_int80_syscall_32 restore_all: TRACE_IRQS_IRET
@@ -491,11 +488,6 @@ syscall_fault: jmp resume_userspace END(syscall_fault) -syscall_badsys: - movl $-ENOSYS, %eax - jmp syscall_after_call -END(syscall_badsys) - sysenter_badsys: movl $-ENOSYS, %eax jmp sysenter_after_call
Signature · 1 line
2.4.3
Re: [RFC 24/30] x86/entry/32: Switch INT80 to the new C syscall path
Quoted 70 lines · Andy Lutomirski
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)arch/x86/entry/entry_32.S +13
-22@@ -153,13 +153,13 @@ #endif /* CONFIG_X86_32_LAZY_GS */ -.macro SAVE_ALL +.macro SAVE_ALL pt_regs_ax=%eax cld PUSH_GS pushl %fs pushl %es pushl %ds - pushl %eax + pushl \pt_regs_ax pushl %ebp pushl %edi pushl %esi@@ -370,20 +370,17 @@ ENDPROC(entry_SYSENTER_32) # system call handler stub ENTRY(entry_INT80_32) ASM_CLAC - pushl %eax # save orig_eax - SAVE_ALL - GET_THREAD_INFO(%ebp) - # system call tracing in operation / emulation - testl $_TIF_WORK_SYSCALL_ENTRY, TI_flags(%ebp) - jnz syscall_trace_entry - cmpl $(NR_syscalls), %eax - jae syscall_badsys -syscall_call: - call *sys_call_table(, %eax, 4) -syscall_after_call: - movl %eax, PT_EAX(%esp) # store the return value -syscall_exit: - jmp syscall_exit_work + pushl %eax /* pt_regs->orig_ax */ + SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest, load -ENOSYS into ax */ + + /* + * User mode is traced as though IRQs are on, and the interrupt gate + * turned them off. + */ + TRACE_IRQS_OFF + + movl %esp, %eax + call do_int80_syscall_32 restore_all: TRACE_IRQS_IRET@@ -491,11 +488,6 @@ syscall_fault: jmp resume_userspace END(syscall_fault) -syscall_badsys: - movl $-ENOSYS, %eax - jmp syscall_after_call -END(syscall_badsys) - sysenter_badsys: movl $-ENOSYS, %eax jmp sysenter_after_call --
Quoted 2 lines
2.4.3
While bisecting a boot failure on the 32-bit native kernel I came across this build error:arch/x86/built-in.o: In function `syscall_trace_entry':/home/bgerst/kernel/linux/arch/x86/entry/entry_32.S:468: undefined reference to syscall_call' /home/bgerst/kernel/linux/arch/x86/entry/entry_32.S:469: undefined reference to syscall_exit'
--
Brian Gerst
Re: [RFC 24/30] x86/entry/32: Switch INT80 to the new C syscall path
Quoted 74 lines · Brian Gerst
Quoted 70 lines · Andy Lutomirski
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)arch/x86/entry/entry_32.S +13
-22@@ -153,13 +153,13 @@ #endif /* CONFIG_X86_32_LAZY_GS */ -.macro SAVE_ALL +.macro SAVE_ALL pt_regs_ax=%eax cld PUSH_GS pushl %fs pushl %es pushl %ds - pushl %eax + pushl \pt_regs_ax pushl %ebp pushl %edi pushl %esi@@ -370,20 +370,17 @@ ENDPROC(entry_SYSENTER_32) # system call handler stub ENTRY(entry_INT80_32) ASM_CLAC - pushl %eax # save orig_eax - SAVE_ALL - GET_THREAD_INFO(%ebp) - # system call tracing in operation / emulation - testl $_TIF_WORK_SYSCALL_ENTRY, TI_flags(%ebp) - jnz syscall_trace_entry - cmpl $(NR_syscalls), %eax - jae syscall_badsys -syscall_call: - call *sys_call_table(, %eax, 4) -syscall_after_call: - movl %eax, PT_EAX(%esp) # store the return value -syscall_exit: - jmp syscall_exit_work + pushl %eax /* pt_regs->orig_ax */ + SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest, load -ENOSYS into ax */ + + /* + * User mode is traced as though IRQs are on, and the interrupt gate + * turned them off. + */ + TRACE_IRQS_OFF + + movl %esp, %eax + call do_int80_syscall_32 restore_all: TRACE_IRQS_IRET@@ -491,11 +488,6 @@ syscall_fault: jmp resume_userspace END(syscall_fault) -syscall_badsys: - movl $-ENOSYS, %eax - jmp syscall_after_call -END(syscall_badsys) - sysenter_badsys: movl $-ENOSYS, %eax jmp sysenter_after_call --Quoted 2 lines
2.4.3
Quoted 8 lines
While bisecting a boot failure on the 32-bit native kernel I came across this build error:
arch/x86/built-in.o: In function `syscall_trace_entry':/home/bgerst/kernel/linux/arch/x86/entry/entry_32.S:468: undefined reference tosyscall_call' /home/bgerst/kernel/linux/arch/x86/entry/entry_32.S:469: undefined reference tosyscall_exit'
Whoops! I'll fix that up for v2.
--Andy
Andy Lutomirski
maintainerRFC 25/30 x86/entry/32: Re-implement SYSENTER using the new C path
In reply to Andy Lutomirski--- arch/x86/entry/common.c | 15 +++- arch/x86/entry/entry_32.S | 132 ++++++++----------------------- arch/x86/entry/vdso/vdso32/system_call.S | 2 + 3 files changed, 50 insertions(+), 99 deletions(-)
arch/x86/entry/common.c +14 -1
@@ -413,7 +413,20 @@ __visible long do_fast_syscall_32(struct pt_regs *regs) regs->ip == landing_pad && (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ;#else -return 0 ; +/* +* Opportunistic SYSEXIT: if possible, try to return using SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +* +* We don't allow syscalls at all from vm86 mode, but we still +* need to check VM, becuase we might be returning from sys_vm86. +*/ +return static_cpu_has (X86_FEATURE_SEP ) && + regs->cs == __USER_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF |X86_EFLAGS_VM )) ==0 ; #endif }#endif
arch/x86/entry/entry_32.S +34 -98
165 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/system_call.S +2 -0
@@ -34,6 +34,8 @@ __kernel_vsyscall: /* If SYSENTER (Intel) or SYSCALL32 (AMD) is available, use it. */ ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ "syscall", X86_FEATURE_SYSCALL32 +#else + ALTERNATIVE "", "sysenter", X86_FEATURE_SEP #endif /* Enter using int $0x80 */
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 26/30 x86/asm: Remove thread_info.sysenter_return
In reply to Andy LutomirskiIt's no longer needed.
We could reinstate something like it as an optimization, which would remove two cachelines from the fast syscall entry working set. I benchmarked it, and it makes no difference whatsoever to the performance of cache-hot compat syscalls on Sandy Bridge.
--- arch/x86/include/asm/thread_info.h | 1 - arch/x86/kernel/asm-offsets.c | 3 --- 2 files changed, 4 deletions(-)
arch/x86/include/asm/thread_info.h +0 -1
@@ -59,7 +59,6 @@ struct thread_info {__u32 cpu;/* current CPU */ int saved_preempt_count;mm_segment_t addr_limit; -void __user *sysenter_return; unsigned int sig_on_uaccess_error:1; unsigned int uaccess_err:1; /* uaccess failed */ };
arch/x86/kernel/asm-offsets.c +0 -3
@@ -54,9 +54,6 @@ void common(void) {OFFSET (IA32_SIGCONTEXT_ip, sigcontext_ia32, ip);BLANK (); -OFFSET (TI_sysenter_return, thread_info, sysenter_return); - -BLANK ();OFFSET (IA32_RT_SIGFRAME_sigcontext, rt_sigframe_ia32, uc.uc_mcontext);#endif
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 27/30 x86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls
In reply to Andy LutomirskiThis is slightly messy, but it eliminates an unnecessary cli;sti pair.
--- arch/x86/entry/common.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
arch/x86/entry/common.c +11 -7
@@ -323,10 +323,10 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )/* -* Does a 32-bit syscall. Called with IRQs off and does all entry and -* exit work. +* Does a 32-bit syscall. Called with IRQs on and does all entry and +* exit work and returns with IRQs off. */ -__visible void do_int80_syscall_32(struct pt_regs *regs) +static void do_syscall_32_irqs_on (struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned int nr = (unsigned int )regs->orig_ax;
@@ -335,8 +335,6 @@ __visible void do_int80_syscall_32(struct pt_regs *regs) ti->status |=TS_COMPAT ;#endif -local_irq_enable (); -if (READ_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY) {/* * Subtlety here: if ptrace pokes something larger than
@@ -363,6 +361,13 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } +/* Handles int $0x80 */ +__visible void do_int80_syscall_32(struct pt_regs *regs) +{ +local_irq_enable (); +do_syscall_32_irqs_on (regs); +} +/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ __visible long do_fast_syscall_32 (struct pt_regs *regs) {
@@ -394,10 +399,9 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)prepare_exit_to_usermode (regs);return 0 ;/* Keep it simple: use IRET. */ } -local_irq_disable ();/* Now this is just like a normal syscall. */ -do_int80_syscall_32 (regs); +do_syscall_32_irqs_on (regs);#ifdef CONFIG_X86_64 /*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 28/30 x86/entry: Make irqs_disabled checks in exit code depend on lockdep
In reply to Andy LutomirskiThese checks are quite slow. Disable them in non-lockdep kernels to reduce the performance hit.
--- arch/x86/entry/common.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +3 -3
@@ -219,7 +219,7 @@ static struct thread_info *pt_regs_to_thread_info(struct pt_regs *regs)/* Called with IRQs disabled. */ __visible void prepare_exit_to_usermode(struct pt_regs *regs) { -if (WARN_ON (!irqs_disabled ())) +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) &&WARN_ON (!irqs_disabled ()))local_irq_disable ();lockdep_sys_exit ();
@@ -281,8 +281,8 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)CT_WARN_ON (ct_state () !=CONTEXT_KERNEL ); -if (WARN (irqs_disabled (),"syscall %ld left IRQs disabled" , - regs->orig_ax)) +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) && +WARN (irqs_disabled (),"syscall %ld left IRQs disabled" , regs->orig_ax))local_irq_enable ();/*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 29/30 x86/entry: Force inlining of 32-bit syscall code
In reply to Andy LutomirskiOn systems that support fast syscalls, we only really care about the performance of the fast syscall path. Forcibly inline it and add a likely annotation.
This saves 4-6 cycles.
--- arch/x86/entry/common.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +5 -3
@@ -324,9 +324,11 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )/* * Does a 32-bit syscall. Called with IRQs on and does all entry and -* exit work and returns with IRQs off. +* exit work and returns with IRQs off. This function is extremely hot +* in workloads that use it, and it's usually called from +* do_fast_syscall_32, so forcibly inline it to improve performance. */ -static void do_syscall_32_irqs_on (struct pt_regs *regs) +static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned int nr = (unsigned int )regs->orig_ax;
@@ -345,7 +347,7 @@ static void do_syscall_32_irqs_on(struct pt_regs *regs) nr =syscall_trace_enter (regs); } -if (nr < IA32_NR_syscalls) { +if (likely (nr < IA32_NR_syscalls)) {/* * It's possible that a 32-bit syscall implementation * takes a 64-bit parameter but nonetheless assumes that
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerRFC 30/30 x86/entry: Micro-optimize compat fast syscall arg fetch
In reply to Andy Lutomirskiwe're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the access_ok check isn't needed.
This saves about three cycles on a cache-hot fast syscall.
--- arch/x86/entry/common.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
arch/x86/entry/common.c +14 -2
@@ -390,8 +390,20 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)* WARNING: We are in CONTEXT_USER and RCU isn't paying attention! */ local_irq_enable (); -if (get_user (*(u32 *)®s->cx, - (u32 __user __force *)(unsigned long )(u32 )regs->sp)) { + if ( +#ifdef CONFIG_X86_64 +/* +* Micro-optimization: the pointer we're following is explicitly +* 32 bits, so it can't be out of range. +*/ +__get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long)(u32)regs->sp) +#else + get_user(*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long)(u32)regs->sp) +#endif + ) { +/* User code screwed up. */ local_irq_disable (); regs->ax = -EFAULT ;
Signature · 1 line
2.4.3
Quoted 31 lines · Andy Lutomirski
Here's a monster series that I'm working on. I think it's in decent shape now.
The first couple patches are tests and some old stuff. There's a test that validates the vDSO
AT_SYSINFOannotations (which fails on 32-bit Debian systems for some reason that I can't yet fathom because fast syscalls simply don't happen on my VM for unknown reasons presumably related to glibc bugs or misconfiguration, and I need to do something about the test). There's also a test that exercises some assumptions that signal handling and ptracers make about syscalls that currently do *not* hold on 64-bit AMD using 32-bitAT_SYSINFO.The next few patches are the NT stuff. Ingo, feel free to pretend you don't see it until the merge window closes :)
The rest is basically a rewrite of syscalls for all cases except 64-bit native. With these patches applied, there is a single 32-bit vDSO and it uses SYSCALL, SYSENTER, and INT80 almost interchangeably via alternatives. The semantics of SYSENTER and SYSCALL are defined as:
1. If SYSCALL, ESP = ECX
2. ECX = *ESP
3. IP = INT80 landing pad
4. Opportunistic SYSRET/SYSEXIT is enabled on returnThe vDSO is rearranged so that these semantics work. Anything that backs IP up by 2 ends up pointing at a bona fide int $0x80 instruction with the expected regs.
Quoted 6 lines
In the process, the vDSO CFI annotations (which are actually used) get rewritten using normal CFI directives.
Opportunistic SYSRET/SYSEXIT only happens on return when CS and SS are as expected, IP points to the INT80 landing pad, and flags are in good shape.
I think the opportunistic exit code could be improved a bit more. The checks are only be necessary if force_iret() was called meaning registers were changed. One possibility is to add a ti->status flag TS_FASTSYSCALL. Then we could move the tests to force_iret(), which would clear the flag if the registers fail validation. The syscall exit path then would check the flag and exit via IRET if it's clear. That would reduce the impact of the tests on the fast path where no regs were changed.
Quoted 11 lines
Other than that, the system call entries are simplified to the bare minimum prologue and a call to a C function. Amusingly, SYSENTER and SYSCALL32 use the same C function.
To make that work, I had to remove all the 32-bit syscall stubs except the clone argument hack. This is because, for C code to call through the system call table, the system call table entries need to be real function pointers with C-compatible ABIs.
There is nothing at all anymore that requires that
x86_32syscalls be asmlinkage. That could be removed in a subsequent patch.
Other arches (at least IA-64) still need asmlinkage or something equivalent for their syscalls.
asmlinkage_protect() can also be removed.
Quoted 6 lines
The upshot appears to be a ~25 cycle performance hit on 32-bit fast path syscalls. The slow path is probably faster under most circumstances and, if the exit slow path gets hit, it'll be much faster because (as we already do in the 64-bit native case) we can still use SYSEXIT/SYSRET.
Quoted 6 lines
The patchset is structured as a removal of the old fast syscall code, then the change that makes syscalls into real functions, then a clean re-implementation of fast syscalls.
If we want some of the 25 cycles back, we could consider open-coding a new C fast path.
Is the 25 cycles for the compat or native case? I'd expect the native case to be hit harder because of register pressure.
--
Brian Gerst
Quoted 39 lines · Brian Gerst
Quoted 31 lines · Andy Lutomirski
Here's a monster series that I'm working on. I think it's in decent shape now.
The first couple patches are tests and some old stuff. There's a test that validates the vDSO
AT_SYSINFOannotations (which fails on 32-bit Debian systems for some reason that I can't yet fathom because fast syscalls simply don't happen on my VM for unknown reasons presumably related to glibc bugs or misconfiguration, and I need to do something about the test). There's also a test that exercises some assumptions that signal handling and ptracers make about syscalls that currently do *not* hold on 64-bit AMD using 32-bitAT_SYSINFO.The next few patches are the NT stuff. Ingo, feel free to pretend you don't see it until the merge window closes :)
The rest is basically a rewrite of syscalls for all cases except 64-bit native. With these patches applied, there is a single 32-bit vDSO and it uses SYSCALL, SYSENTER, and INT80 almost interchangeably via alternatives. The semantics of SYSENTER and SYSCALL are defined as:
1. If SYSCALL, ESP = ECX
2. ECX = *ESP
3. IP = INT80 landing pad
4. Opportunistic SYSRET/SYSEXIT is enabled on returnThe vDSO is rearranged so that these semantics work. Anything that backs IP up by 2 ends up pointing at a bona fide int $0x80 instruction with the expected regs.
Quoted 6 lines
In the process, the vDSO CFI annotations (which are actually used) get rewritten using normal CFI directives.
Opportunistic SYSRET/SYSEXIT only happens on return when CS and SS are as expected, IP points to the INT80 landing pad, and flags are in good shape.
Quoted 8 lines
I think the opportunistic exit code could be improved a bit more. The checks are only be necessary if
force_iret()was called meaning registers were changed. One possibility is to add ati->statusflagTS_FASTSYSCALL. Then we could move the tests toforce_iret(), which would clear the flag if the registers fail validation. The syscall exit path then would check the flag and exit via IRET if it's clear. That would reduce the impact of the tests on the fast path where no regs were changed.
Historically, it's not just force_iret() (which is quite new) but anything that triggers the slow path. If we want to go that route, I'd be more comfortable doing something more like:
if (!(ti->flags & _TIF_SYSCALL_EXIT_WORK))
return true;
i.e. just bypassing the slow path exit and the check. This might get some more of those cycles back for the full fast path, albeit at the cost of more complexity in the C code.
Maybe some day we should add better accessors for pt_regs that warn if misused and set flags if used for write. For example, const struct pt_regs *syscall_pt_regs_read() and struct pt_regs *syscall_pt_regs_write(). The latter could set a flag.
Quoted 10 lines
Quoted 10 lines
Other than that, the system call entries are simplified to the bare minimum prologue and a call to a C function. Amusingly, SYSENTER and SYSCALL32 use the same C function.
To make that work, I had to remove all the 32-bit syscall stubs except the clone argument hack. This is because, for C code to call through the system call table, the system call table entries need to be real function pointers with C-compatible ABIs.
Quoted 5 lines
Quoted 2 lines
There is nothing at all anymore that requires that
x86_32syscalls be asmlinkage. That could be removed in a subsequent patch.Other arches (at least IA-64) still need asmlinkage or something equivalent for their syscalls.
We should probably add a macro syscall_abi that expands to nothing on x86 and to asmlinkage on IA-64. (Why asm "linkage"? It has nothing to do with linkage.)
Quoted 2 lines
asmlinkage_protect()can also be removed.
Wow, that's gross. I'm a bit surprised that no new compiler has clever enough to break that hack.
Quoted 11 lines
Quoted 11 lines
The upshot appears to be a ~25 cycle performance hit on 32-bit fast path syscalls. The slow path is probably faster under most circumstances and, if the exit slow path gets hit, it'll be much faster because (as we already do in the 64-bit native case) we can still use SYSEXIT/SYSRET.
The patchset is structured as a removal of the old fast syscall code, then the change that makes syscalls into real functions, then a clean re-implementation of fast syscalls.
Quoted 5 lines
Quoted 2 lines
If we want some of the 25 cycles back, we could consider open-coding a new C fast path.
Is the 25 cycles for the compat or native case? I'd expect the native case to be hit harder because of register pressure.
Compat, which I find easier to benchmark because my 32-bit VM steadfastly refuses to issue syscalls via AT_SYSINFO. (I can do it manually, and static binaries built elsewhere work fine, but something's wrong with its glibc.)
I'll benchmark native 32-bit soon.
--Andy