x86: Rewrite all syscall entries except native 64-bit
Subsystems (7)
Andy Lutomirski
maintainerv2 cover x86: Rewrite all syscall entries except native 64-bit
The first two patches are optimizations that I'm surprised we didn't already have. I noticed them when I was looking at the generated asm.
The next two patches are tests and some old stuff. There's a test that validates the vDSO AT_SYSINFO annotations. There's also a test that exercises some assumptions that signal handling and ptracers make about syscalls that currently do *not* hold on 64-bit AMD using 32-bit AT_SYSINFO.
The next three patches are NT cleanups and a lockdep cleanup.
It may pay to apply the beginning of the series (at most through "x86/entry/64/compat: After SYSENTER, move STI after the NT fixup") without waiting for everyone to wrap their heads around the rest.
The rest is basically a rewrite of syscalls for all cases except 64-bit native. With these patches applied, there is a single 32-bit vDSO and it uses SYSCALL, SYSENTER, and INT80 almost interchangeably via alternatives. The semantics of SYSENTER and SYSCALL are defined as:
1. If SYSCALL, ESP = ECX
2. ECX = *ESP
3. IP = INT80 landing pad
4. Opportunistic SYSRET/SYSEXIT is enabled on return
The vDSO is rearranged so that these semantics work. Anything that backs IP up by 2 ends up pointing at a bona fide int $0x80 instruction with the expected regs.
In the process, the vDSO CFI annotations (which are actually used) get rewritten using normal CFI directives.
Opportunistic SYSRET/SYSEXIT only happens on return when CS and SS are as expected, IP points to the INT80 landing pad, and flags are in good shape. (There is no longer any assumption that full fast-path 32-bit syscalls don't muck with the registers that matter for fast exits -- I played with maintaining an optimization like that with poor results. I may try again if it saves a few cycles.)
Other than that, the system call entries are simplified to the bare minimum prologue and a call to a C function. Amusingly, SYSENTER and SYSCALL32 use the same C function.
To make that work, I had to remove all the 32-bit syscall stubs except the clone argument hack. This is because, for C code to call through the system call table, the system call table entries need to be real function pointers with C-compatible ABIs.
There is nothing at all anymore that requires that x86_32 syscalls be asmlinkage. That could be removed in a subsequent patch.
The upshot appears to be a ~16 cycle performance hit on 32-bit fast path syscalls. (On my system, my little prctl test takes 172 cycles before and 188 cycles with these patches applied.)
The slow path is probably faster under most circumstances and, if the exit slow path gets hit, it'll be much faster because (as we already do in the 64-bit native case) we can still use SYSEXIT/SYSRET.
The patchset is structured as a removal of the old fast syscall code, then the change that makes syscalls into real functions, then a clean re-implementation of fast syscalls.
If we want some of the 25 cycles back, we could consider open-coding a new C fast path.
Changes from v1:
- The unwind_vdso_32 test now warns on broken Debian installations instead of failing. The problem is now fully understood, will be fixed by Debian and possibly also fixed by upstream glibc.
- execve was rather broken in v1.
- It's quite a bit faster now (the optimizations at the end are mostly new).
- int80 on 64-bit no longer clobbers extra regs (thanks Denys!).
- The uaccess stuff is new.
- Lots of other things that I forgot, I'm sure.
Andy Lutomirski (36):
x86/uaccess: Tell the compiler that uaccess is unlikely to fault x86/uaccess: __chk_range_not_ok is unlikely to return true selftests/x86: Add a test for vDSO unwinding selftests/x86: Add a test for syscall restart and arg modification x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access x86/entry: Move lockdep_sys_exit to prepare_exit_to_usermode x86/entry/64/compat: After SYSENTER, move STI after the NT fixup x86/vdso: Remove runtime 32-bit vDSO selection
x86/asm: Re-add manual CFI infrastructure
x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm x86/vdso: Replace hex int80 CFI annotations with gas directives x86/elf/64: Clear more registers in elf_common_init x86/vdso/32: Save extra registers in the INT80 vsyscall path x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries x86/entry/64/compat: Remove audit optimizations x86/entry/64/compat: Remove most of the fast system call machinery x86/entry/64/compat: Set up full pt_regs for all compat syscalls x86/entry/syscalls: Move syscall table declarations into
asm/syscalls.h
x86/syscalls: Give sys_call_ptr_t a useful type
x86/entry: Add do_syscall_32, a C function to do 32-bit syscalls x86/entry/64/compat: Migrate the body of the syscall entry to C x86/entry: Add C code for fast system call entries x86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls x86/entry/32: Open-code return tracking from fork and kthreads x86/entry/32: Switch INT80 to the new C syscall path x86/entry/32: Re-implement SYSENTER using the new C path x86/asm: Remove thread_info.sysenter_return
x86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls x86/entry: Make irqs_disabled checks in exit code depend on lockdep x86/entry: Force inlining of 32-bit syscall code x86/entry: Micro-optimize compat fast syscall arg fetch x86/entry: Hide two syscall entry assertions behind CONFIG_DEBUG_ENTRY x86/entry: Use pt_regs_to_thread_info() in syscall entry tracing x86/entry: Split and inline prepare_exit_to_usermode x86/entry: Split and inline syscall_return_slowpath
arch/x86/Makefile | 10 +- arch/x86/entry/common.c | 255 ++++++++-- arch/x86/entry/entry_32.S | 184 +++---- arch/x86/entry/entry_64.S | 9 +- arch/x86/entry/entry_64_compat.S | 541 +++++---------------- arch/x86/entry/syscall_32.c | 9 +- arch/x86/entry/syscall_64.c | 4 +- arch/x86/entry/syscalls/syscall_32.tbl | 12 +- arch/x86/entry/vdso/Makefile | 39 +- arch/x86/entry/vdso/vdso2c.c | 2 +- arch/x86/entry/vdso/vdso32-setup.c | 28 +- arch/x86/entry/vdso/vdso32/int80.S | 56 --- arch/x86/entry/vdso/vdso32/syscall.S | 75 --- arch/x86/entry/vdso/vdso32/sysenter.S | 116 ----- arch/x86/entry/vdso/vdso32/system_call.S | 57 +++ arch/x86/entry/vdso/vma.c | 13 +- arch/x86/ia32/ia32_signal.c | 4 +- arch/x86/include/asm/dwarf2.h | 177 +++++++ arch/x86/include/asm/elf.h | 10 +- arch/x86/include/asm/syscall.h | 14 +- arch/x86/include/asm/thread_info.h | 1 - arch/x86/include/asm/uaccess.h | 14 +- arch/x86/include/asm/vdso.h | 10 +- arch/x86/kernel/asm-offsets.c | 3 - arch/x86/kernel/signal.c | 4 +- arch/x86/um/sys_call_table_32.c | 7 +- arch/x86/um/sys_call_table_64.c | 7 +- arch/x86/xen/setup.c | 13 +- tools/testing/selftests/x86/Makefile | 5 +- tools/testing/selftests/x86/ptrace_syscall.c | 294 +++++++++++ .../testing/selftests/x86/raw_syscall_helper_32.S | 46 ++ tools/testing/selftests/x86/unwind_vdso.c | 209 ++++++++ 32 files changed, 1258 insertions(+), 970 deletions(-) delete mode 100644 arch/x86/entry/vdso/vdso32/int80.S delete mode 100644 arch/x86/entry/vdso/vdso32/syscall.S delete mode 100644 arch/x86/entry/vdso/vdso32/sysenter.S create mode 100644 arch/x86/entry/vdso/vdso32/system_call.S create mode 100644 arch/x86/include/asm/dwarf2.h create mode 100644 tools/testing/selftests/x86/ptrace_syscall.c create mode 100644 tools/testing/selftests/x86/raw_syscall_helper_32.S create mode 100644 tools/testing/selftests/x86/unwind_vdso.c
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 1/36 x86/uaccess: Tell the compiler that uaccess is unlikely to fault
GCC doesn't realize that get_user, put_user, and their __ variants are unlikely to fail. Tell it.
I noticed this while playing with the C entry code.
Before:
text data bss dec hex filename
21828763 5194760 1277952 28301475 1afd8a3 baseline
After:
text data bss dec hex filename
21828379 5194760 1277952 28301091 1afd723 vmlinux
--- arch/x86/include/asm/uaccess.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/uaccess.h +4 -4
@@ -182,7 +182,7 @@ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL)) : "=a" (__ret_gu), "=r" (__val_gu) \ : "0" (ptr), "i" (sizeof(*(ptr)))); \ (x) = (__force __typeof__(*(ptr))) __val_gu; \ - __ret_gu; \ +__builtin_expect (__ret_gu,0 ); \ })#define __put_user_x (size, x, ptr, __ret_pu) \
@@ -278,7 +278,7 @@ extern void __put_user_8(void);__put_user_x (X , __pu_val, ptr, __ret_pu); \break ; \ } \ - __ret_pu; \ +__builtin_expect (__ret_pu,0 ); \ })#define __put_user_size (x, ptr, size, retval, errret) \
@@ -401,7 +401,7 @@ do { \ ({ \int __pu_err; \__put_user_size ((x), (ptr), (size), __pu_err, -EFAULT ); \ - __pu_err; \ +__builtin_expect (__pu_err,0 ); \ })#define __get_user_nocheck (x, ptr, size) \
@@ -410,7 +410,7 @@ do { \unsigned long __gu_val; \__get_user_size (__gu_val, (ptr), (size), __gu_err, -EFAULT ); \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ - __gu_err; \ +__builtin_expect (__gu_err,0 ); \ })/* FIXME: this hack is definitely wrong -AK */
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 3/36 selftests/x86: Add a test for vDSO unwinding
In reply to Andy LutomirskiWhile the kernel itself doesn't use DWARF unwinding, user code expects to be able to unwind the vDSO. The vsyscall (AT_SYSINFO) entry is manually CFI-annotated, and this tests that it unwinds correctly.
I tested the test by incorrectly annotating __kernel_vsyscall, and the test indeed fails if I do that.
--- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/unwind_vdso.c | 209 ++++++++++++++++++++++++++++++ 2 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/unwind_vdso.c
tools/testing/selftests/x86/Makefile +1 -1
@@ -5,7 +5,7 @@ include ../lib.mk .PHONY: all all_32 all_64 warn_32bit_failure clean TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt -TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn test_syscall_vdso +TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn test_syscall_vdso unwind_vdso TARGETS_C_32BIT_ALL := $(TARGETS_C_BOTHBITS) $(TARGETS_C_32BIT_ONLY) BINARIES_32 := $(TARGETS_C_32BIT_ALL:%=%_32)
tools/testing/selftests/x86/unwind_vdso.c new file +209 -0
209 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 4/36 selftests/x86: Add a test for syscall restart and arg modification
In reply to Andy LutomirskiThis tests assumptions about how fast syscall works wrt pt_regs and, in particular, what happens if IP is decremented by 2 during a syscall.
--- tools/testing/selftests/x86/Makefile | 3 +- tools/testing/selftests/x86/ptrace_syscall.c | 294 +++++++++++++++++++++ .../testing/selftests/x86/raw_syscall_helper_32.S | 46 ++++ 3 files changed, 342 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/ptrace_syscall.c create mode 100644 tools/testing/selftests/x86/raw_syscall_helper_32.S
tools/testing/selftests/x86/Makefile +2 -1
@@ -4,7 +4,7 @@ include ../lib.mk .PHONY: all all_32 all_64 warn_32bit_failure clean -TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt +TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt ptrace_syscall TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn test_syscall_vdso unwind_vdso TARGETS_C_32BIT_ALL := $(TARGETS_C_BOTHBITS) $(TARGETS_C_32BIT_ONLY)
@@ -60,4 +60,5 @@ endif # Some tests have additional dependencies. sysret_ss_attrs_64: thunks.S +ptrace_syscall_32: raw_syscall_helper_32.S test_syscall_vdso_32: thunks_32.S
tools/testing/selftests/x86/ptrace_syscall.c new file +294 -0
294 lines not on this page · raw message
tools/testing/selftests/x86/raw_syscall_helper_32.S new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 7/36 x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
In reply to Andy LutomirskiWe eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts.
--- arch/x86/entry/entry_64_compat.S | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
arch/x86/entry/entry_64_compat.S +22 -8
@@ -58,14 +58,9 @@ ENDPROC(native_usergs_sysret32) * with the int 0x80 path. */ ENTRY(entry_SYSENTER_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %ebp, %ebp
@@ -76,7 +71,16 @@ ENTRY(entry_SYSENTER_compat) /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ pushq %rbp /* pt_regs->sp */ - pushfq /* pt_regs->flags */ + + /* + * Push flags. This is nasty. First, interrupts are currently + * off, but we need pt_regs->flags to have IF set. Second, even + * if TF was set when SYSENTER started, it's clear by now. We fix + * that later using TIF_SINGLESTEP. + */ + pushfq /* pt_regs->flags (except IF = 0) */ + orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */ + pushq $__USER32_CS /* pt_regs->cs */ pushq %r10 /* pt_regs->ip = thread_info->sysenter_return */ pushq %rax /* pt_regs->orig_ax */
@@ -92,12 +96,22 @@ ENTRY(entry_SYSENTER_compat) * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether * NT was set instead of doing an unconditional popfq. + * This needs to happen before enabling interrupts so that + * we don't get preempted with NT set. */ testl $X86_EFLAGS_NT, EFLAGS(%rsp) jnz sysenter_fix_flags sysenter_flags_fixed: /* + * Re-enable interrupts. IRQ tracing already thinks that IRQs are + * on (since we treat user mode as having IRQs on), and the + * prologue above is too short for it to be worth adding a + * tracing round trip. + */ + ENABLE_INTERRUPTS(CLBR_NONE) + + /* * no need to do an access_ok check here because rbp has been * 32-bit zero extended */
@@ -244,7 +258,7 @@ sysexit_audit: #endif sysenter_fix_flags: - pushq $(X86_EFLAGS_IF|X86_EFLAGS_FIXED) + pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 8/36 x86/vdso: Remove runtime 32-bit vDSO selection
In reply to Andy Lutomirski32-bit userspace will now always see the same vDSO, which is exactly what used to be the int80 vDSO. Subsequent patches will clean it up and make it support SYSENTER and SYSCALL using alternatives.
--- arch/x86/entry/vdso/Makefile | 35 +++------- arch/x86/entry/vdso/vdso2c.c | 1 - arch/x86/entry/vdso/vdso32-setup.c | 28 +------- arch/x86/entry/vdso/vdso32/int80.S | 56 --------------- arch/x86/entry/vdso/vdso32/syscall.S | 75 -------------------- arch/x86/entry/vdso/vdso32/sysenter.S | 116 ------------------------------- arch/x86/entry/vdso/vdso32/system_call.S | 56 +++++++++++++++ arch/x86/entry/vdso/vma.c | 13 +--- arch/x86/ia32/ia32_signal.c | 4 +- arch/x86/include/asm/elf.h | 2 +- arch/x86/include/asm/vdso.h | 9 +-- arch/x86/kernel/signal.c | 4 +- arch/x86/xen/setup.c | 13 +--- 13 files changed, 77 insertions(+), 335 deletions(-) delete mode 100644 arch/x86/entry/vdso/vdso32/int80.S delete mode 100644 arch/x86/entry/vdso/vdso32/syscall.S delete mode 100644 arch/x86/entry/vdso/vdso32/sysenter.S create mode 100644 arch/x86/entry/vdso/vdso32/system_call.S
arch/x86/entry/vdso/Makefile +11 -24
@@ -19,9 +19,7 @@ obj-y += vma.o # vDSO images to build vdso_img-$(VDSO64-y) += 64 vdso_img-$(VDSOX32-y) += x32 -vdso_img-$(VDSO32-y) += 32-int80 -vdso_img-$(CONFIG_IA32_EMULATION) += 32-syscall -vdso_img-$(VDSO32-y) += 32-sysenter +vdso_img-$(VDSO32-y) += 32 obj-$(VDSO32-y) += vdso32-setup.o
@@ -122,15 +120,6 @@ $(obj)/%.so: $(obj)/%.so.dbg $(obj)/vdsox32.so.dbg: $(src)/vdsox32.lds $(vobjx32s) FORCE $(call if_changed,vdso) -# -# Build multiple 32-bit vDSO images to choose from at boot time. -# -vdso32.so-$(VDSO32-y) += int80 -vdso32.so-$(CONFIG_IA32_EMULATION) += syscall -vdso32.so-$(VDSO32-y) += sysenter - -vdso32-images = $(vdso32.so-y:%=vdso32-%.so) - CPPFLAGS_vdso32.lds = $(CPPFLAGS_vdso.lds) VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1
@@ -139,14 +128,12 @@ VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1 override obj-dirs = $(dir $(obj)) $(obj)/vdso32/ targets += vdso32/vdso32.lds -targets += vdso32/note.o vdso32/vclock_gettime.o $(vdso32.so-y:%=vdso32/%.o) +targets += vdso32/note.o vdso32/vclock_gettime.o vdso32/system_call.o targets += vdso32/vclock_gettime.o -$(obj)/vdso32.o: $(vdso32-images:%=$(obj)/%) - KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) -$(vdso32-images:%=$(obj)/%.dbg): KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) -$(vdso32-images:%=$(obj)/%.dbg): asflags-$(CONFIG_X86_64) += -m32 +$(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) +$(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32 KBUILD_CFLAGS_32 := $(filter-out -m64,$(KBUILD_CFLAGS)) KBUILD_CFLAGS_32 := $(filter-out -mcmodel=kernel,$(KBUILD_CFLAGS_32))
@@ -157,13 +144,13 @@ KBUILD_CFLAGS_32 += $(call cc-option, -fno-stack-protector) KBUILD_CFLAGS_32 += $(call cc-option, -foptimize-sibling-calls) KBUILD_CFLAGS_32 += -fno-omit-frame-pointer KBUILD_CFLAGS_32 += -DDISABLE_BRANCH_PROFILING -$(vdso32-images:%=$(obj)/%.dbg): KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) +$(obj)/vdso32.so.dbg: KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) -$(vdso32-images:%=$(obj)/%.dbg): $(obj)/vdso32-%.so.dbg: FORCE \ - $(obj)/vdso32/vdso32.lds \ - $(obj)/vdso32/vclock_gettime.o \ - $(obj)/vdso32/note.o \ - $(obj)/vdso32/%.o +$(obj)/vdso32.so.dbg: FORCE \ + $(obj)/vdso32/vdso32.lds \ + $(obj)/vdso32/vclock_gettime.o \ + $(obj)/vdso32/note.o \ + $(obj)/vdso32/system_call.o $(call if_changed,vdso) #
@@ -206,4 +193,4 @@ $(vdso_img_insttargets): install_%: $(obj)/%.dbg $(MODLIB)/vdso FORCE PHONY += vdso_install $(vdso_img_insttargets) vdso_install: $(vdso_img_insttargets) FORCE -clean-files := vdso32-syscall* vdso32-sysenter* vdso32-int80* vdso64* vdso-image-*.c vdsox32.so* +clean-files := vdso32.so vdso32.so.dbg vdso64* vdso-image-*.c vdsox32.so*
arch/x86/entry/vdso/vdso2c.c +0 -1
@@ -98,7 +98,6 @@ struct vdso_sym required_syms[] = {"VDSO_FAKE_SECTION_TABLE_END" , false }, {"VDSO32_NOTE_MASK", true}, - {"VDSO32_SYSENTER_RETURN", true}, {"__kernel_vsyscall", true}, {"__kernel_sigreturn", true}, {"__kernel_rt_sigreturn", true},
arch/x86/entry/vdso/vdso32-setup.c +1 -27
@@ -48,35 +48,9 @@ __setup("vdso32=", vdso32_setup);__setup_param ("vdso=" , vdso_setup, vdso32_setup,0 );#endif -#ifdef CONFIG_X86_64 - -#define vdso32_sysenter () (boot_cpu_has(X86_FEATURE_SYSENTER32)) -#define vdso32_syscall () (boot_cpu_has(X86_FEATURE_SYSCALL32)) - -#else /* CONFIG_X86_32 */ - -#define vdso32_sysenter () (boot_cpu_has(X86_FEATURE_SEP)) -#define vdso32_syscall () (0) - -#endif /* CONFIG_X86_64 */ - -#if defined(CONFIG_X86_32 ) || defined(CONFIG_COMPAT ) -const struct vdso_image *selected_vdso32; -#endif -int __init sysenter_setup (void ) { -#ifdef CONFIG_COMPAT -if (vdso32_syscall ()) - selected_vdso32 = &vdso_image_32_syscall; - else -#endif -if (vdso32_sysenter ()) - selected_vdso32 = &vdso_image_32_sysenter; -else - selected_vdso32 = &vdso_image_32_int80; - -init_vdso_image (selected_vdso32); +init_vdso_image (&vdso_image_32);return 0 ; }
arch/x86/entry/vdso/vdso32/int80.S deleted +0 -56
56 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/syscall.S deleted +0 -75
75 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/sysenter.S deleted +0 -116
116 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/system_call.S new file +56 -0
56 lines not on this page · raw message
arch/x86/entry/vdso/vma.c +1 -12
@@ -180,21 +180,10 @@ up_fail:#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )static int load_vdso32 (void ) { -int ret; -if (vdso32_enabled !=1 )/* Other values all mean "disabled" */ return 0 ; - ret =map_vdso (selected_vdso32, false); -if (ret) -return ret; - -if (selected_vdso32->sym_VDSO32_SYSENTER_RETURN) -current_thread_info ()->sysenter_return = - current->mm->context.vdso + - selected_vdso32->sym_VDSO32_SYSENTER_RETURN; - -return 0 ; +return map_vdso (&vdso_image_32, false); }#endif
arch/x86/ia32/ia32_signal.c +2 -2
@@ -289,7 +289,7 @@ int ia32_setup_frame(int sig, struct ksignal *ksig,/* Return stub is in 32bit vsyscall page */ if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode; }
@@ -368,7 +368,7 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, restorer = ksig->ka.sa.sa_restorer; else restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;put_user_ex (ptr_to_compat (restorer), &frame->pretcode);/*
arch/x86/include/asm/elf.h +1 -1
@@ -328,7 +328,7 @@ else \#define VDSO_ENTRY \ ((unsigned long)current->mm->context.vdso + \ - selected_vdso32->sym___kernel_vsyscall) + vdso_image_32.sym___kernel_vsyscall)struct linux_binprm ;
arch/x86/include/asm/vdso.h +1 -8
@@ -26,7 +26,6 @@ struct vdso_image {long sym___kernel_sigreturn;long sym___kernel_rt_sigreturn;long sym___kernel_vsyscall; -long sym_VDSO32_SYSENTER_RETURN; };#ifdef CONFIG_X86_64
@@ -38,13 +37,7 @@ extern const struct vdso_image vdso_image_x32;#endif #if definedCONFIG_X86_32 || definedCONFIG_COMPAT -extern const struct vdso_image vdso_image_32_int80; -#ifdef CONFIG_COMPAT -extern const struct vdso_image vdso_image_32_syscall; -#endif -extern const struct vdso_image vdso_image_32_sysenter; - -extern const struct vdso_image *selected_vdso32; +extern const struct vdso_image vdso_image_32;#endif extern void __init init_vdso_image (const struct vdso_image *image);
arch/x86/kernel/signal.c +2 -2
@@ -299,7 +299,7 @@ __setup_frame(int sig, struct ksignal *ksig, sigset_t *set,if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode;if (ksig->ka.sa.sa_flags &SA_RESTORER )
@@ -363,7 +363,7 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig,/* Set up to return from userspace. */ restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;if (ksig->ka.sa.sa_flags &SA_RESTORER ) restorer = ksig->ka.sa.sa_restorer;put_user_ex (restorer, &frame->pretcode);
arch/x86/xen/setup.c +2 -11
@@ -965,17 +965,8 @@ char * __init xen_auto_xlated_memory_setup(void)static void __init fiddle_vdso (void ) {#ifdef CONFIG_X86_32 -/* -* This could be called before selected_vdso32 is initialized, so -* just fiddle with both possible images. vdso_image_32_syscall -* can't be selected, since it only exists on 64-bit systems. -*/ -u32 *mask; - mask = vdso_image_32_int80.data + - vdso_image_32_int80.sym_VDSO32_NOTE_MASK; - *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ; - mask = vdso_image_32_sysenter.data + - vdso_image_32_sysenter.sym_VDSO32_NOTE_MASK; +u32 *mask = vdso_image_32.data + + vdso_image_32.sym_VDSO32_NOTE_MASK; *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ;#endif }
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 10/36 x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm
In reply to Andy LutomirskiFor the vDSO, user code wants runtime unwind info. Make sure that, if we use .cfi directives, we generate it.
--- arch/x86/entry/vdso/Makefile | 4 ++-- arch/x86/include/asm/dwarf2.h | 13 ++++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)
arch/x86/entry/vdso/Makefile +2 -2
@@ -67,7 +67,7 @@ $(obj)/vdso-image-%.c: $(obj)/vdso%.so.dbg $(obj)/vdso%.so $(obj)/vdso2c FORCE CFL := $(PROFILING) -mcmodel=small -fPIC -O2 -fasynchronous-unwind-tables -m64 \ $(filter -g%,$(KBUILD_CFLAGS)) $(call cc-option, -fno-stack-protector) \ -fno-omit-frame-pointer -foptimize-sibling-calls \ - -DDISABLE_BRANCH_PROFILING + -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO $(vobjs): KBUILD_CFLAGS += $(CFL)
@@ -131,7 +131,7 @@ targets += vdso32/vdso32.lds targets += vdso32/note.o vdso32/vclock_gettime.o vdso32/system_call.o targets += vdso32/vclock_gettime.o -KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) +KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) -DBUILD_VDSO $(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) $(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32
arch/x86/include/asm/dwarf2.h +10 -3
@@ -36,15 +36,22 @@#endif #if defined(CONFIG_AS_CFI_SECTIONS ) && defined(__ASSEMBLY__) +#ifndef BUILD_VDSO /* * Emit CFI data in .debug_frame sections, not .eh_frame sections. * The latter we currently just discard since we don't do DWARF * unwinding at runtime. So only the offline DWARF information is -* useful to anyone. Note we should not use this directive if this -* file is used in the vDSO assembly, or if vmlinux.lds.S gets -* changed so it doesn't discard .eh_frame. +* useful to anyone. Note we should not use this directive if +* vmlinux.lds.S gets changed so it doesn't discard .eh_frame. */ .cfi_sections .debug_frame +#else +/* +* For the vDSO, emit both runtime unwind information and debug +* symbols for the .dbg file. +*/ + .cfi_sections .eh_frame, .debug_frame +#endif #endif #else
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 9/36 x86/asm: Re-add manual CFI infrastructure
In reply to Andy LutomirskiCommit 131484c8da97 ("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations") removed all the manual DWARF annotations outside the vDSO. It also removed the macros we used for the manual annotations.
Re-add the macros so that we can clean up the vDSO annotations.
--- arch/x86/Makefile | 10 ++- arch/x86/include/asm/dwarf2.h | 170 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/dwarf2.h
arch/x86/Makefile +8 -2
@@ -159,6 +159,12 @@ endif sp-$(CONFIG_X86_32) := esp sp-$(CONFIG_X86_64) := rsp +# do binutils support CFI? +cfi := $(call as-instr,.cfi_startproc\n.cfi_rel_offset $(sp-y)$(comma)0\n.cfi_endproc,-DCONFIG_AS_CFI=1) +# is .cfi_signal_frame supported too? +cfi-sigframe := $(call as-instr,.cfi_startproc\n.cfi_signal_frame\n.cfi_endproc,-DCONFIG_AS_CFI_SIGNAL_FRAME=1) +cfi-sections := $(call as-instr,.cfi_sections .debug_frame,-DCONFIG_AS_CFI_SECTIONS=1) + # does binutils support specific instructions? asinstr := $(call as-instr,fxsaveq (%rax),-DCONFIG_AS_FXSAVEQ=1) asinstr += $(call as-instr,pshufb %xmm0$(comma)%xmm0,-DCONFIG_AS_SSSE3=1)
@@ -166,8 +172,8 @@ asinstr += $(call as-instr,crc32l %eax$(comma)%eax,-DCONFIG_AS_CRC32=1) avx_instr := $(call as-instr,vxorps %ymm0$(comma)%ymm1$(comma)%ymm2,-DCONFIG_AS_AVX=1) avx2_instr :=$(call as-instr,vpbroadcastb %xmm0$(comma)%ymm1,-DCONFIG_AS_AVX2=1) -KBUILD_AFLAGS += $(asinstr) $(avx_instr) $(avx2_instr) -KBUILD_CFLAGS += $(asinstr) $(avx_instr) $(avx2_instr) +KBUILD_AFLAGS += $(cfi) $(cfi-sigframe) $(cfi-sections) $(asinstr) $(avx_instr) $(avx2_instr) +KBUILD_CFLAGS += $(cfi) $(cfi-sigframe) $(cfi-sections) $(asinstr) $(avx_instr) $(avx2_instr) LDFLAGS := -m elf_$(UTS_MACHINE)
arch/x86/include/asm/dwarf2.h new file +170 -0
170 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 11/36 x86/vdso: Replace hex int80 CFI annotations with gas directives
In reply to Andy LutomirskiMaintaining the current CFI annotations written in R'lyehian is difficult for most of us. Translate them to something a little closer to English.
This will remove the CFI data for kernels built with extremely old versions of binutils. I think this is a fair tradeoff for the ability for mortals to edit the asm.
--- arch/x86/entry/vdso/vdso32/system_call.S | 48 ++++++-------------------------- 1 file changed, 8 insertions(+), 40 deletions(-)
arch/x86/entry/vdso/vdso32/system_call.S +8 -40
@@ -1,6 +1,10 @@ /* * Code for the vDSO. This version uses the old int $0x80 method. - * +*/ + +#include <asm/dwarf2.h> + +/* * First get the common code for the sigreturn entry points. * This must come first. */
@@ -11,46 +15,10 @@ .type __kernel_vsyscall,@function ALIGN __kernel_vsyscall: -.LSTART_vsyscall: + CFI_STARTPROC int $0x80 ret -.LEND_vsyscall: - .size __kernel_vsyscall,.-.LSTART_vsyscall - .previous - - .section .eh_frame,"a",@progbits -.LSTARTFRAMEDLSI: - .long .LENDCIEDLSI-.LSTARTCIEDLSI -.LSTARTCIEDLSI: - .long 0 /* CIE ID */ - .byte 1 /* Version number */ - .string "zR" /* NUL-terminated augmentation string */ - .uleb128 1 /* Code alignment factor */ - .sleb128 -4 /* Data alignment factor */ - .byte 8 /* Return address register column */ - .uleb128 1 /* Augmentation value length */ - .byte 0x1b /* DW_EH_PE_pcrel|DW_EH_PE_sdata4. */ - .byte 0x0c /* DW_CFA_def_cfa */ - .uleb128 4 - .uleb128 4 - .byte 0x88 /* DW_CFA_offset, column 0x8 */ - .uleb128 1 - .align 4 -.LENDCIEDLSI: - .long .LENDFDEDLSI-.LSTARTFDEDLSI /* Length FDE */ -.LSTARTFDEDLSI: - .long .LSTARTFDEDLSI-.LSTARTFRAMEDLSI /* CIE pointer */ - .long .LSTART_vsyscall-. /* PC-relative start address */ - .long .LEND_vsyscall-.LSTART_vsyscall - .uleb128 0 - .align 4 -.LENDFDEDLSI: - .previous + CFI_ENDPROC - /* - * Pad out the segment to match the size of the sysenter.S version. - */ -VDSO32_vsyscall_eh_frame_size = 0x40 - .section .data,"aw",@progbits - .space VDSO32_vsyscall_eh_frame_size-(.LENDFDEDLSI-.LSTARTFRAMEDLSI), 0 + .size __kernel_vsyscall,.-__kernel_vsyscall .previous
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 17/36 x86/entry/64/compat: Set up full pt_regs for all compat syscalls
In reply to Andy LutomirskiThis is conceptually simpler. More importantly, it eliminates the PTREGSCALL and execve stubs, which were not compatible with the C ABI. This means that C code can call through the compat syscall table.
The execve stubs are a bit subtle. They did two things: they cleared some registers and they forced slow-path return. Neither is necessary any more: elf_common_init clears the extra registers and start_thread calls force_iret().
--- arch/x86/entry/entry_64.S | 6 +---- arch/x86/entry/entry_64_compat.S | 42 +++++++++++----------------------- arch/x86/entry/syscalls/syscall_32.tbl | 12 +++++----- 3 files changed, 20 insertions(+), 40 deletions(-)
arch/x86/entry/entry_64.S +1 -5
@@ -391,20 +391,16 @@ GLOBAL(stub_execveat) jmp return_from_execve END(stub_execveat) -#if defined(CONFIG_X86_X32_ABI) || defined(CONFIG_IA32_EMULATION) +#if defined(CONFIG_X86_X32_ABI) .align 8 GLOBAL(stub_x32_execve) -GLOBAL(stub32_execve) call compat_sys_execve jmp return_from_execve -END(stub32_execve) END(stub_x32_execve) .align 8 GLOBAL(stub_x32_execveat) -GLOBAL(stub32_execveat) call compat_sys_execveat jmp return_from_execve -END(stub32_execveat) END(stub_x32_execveat) #endif
arch/x86/entry/entry_64_compat.S +13 -29
72 lines not on this page · raw message
arch/x86/entry/syscalls/syscall_32.tbl +6 -6
@@ -8,7 +8,7 @@ # 0 i386 restart_syscall sys_restart_syscall 1 i386 exit sys_exit -2 i386 fork sys_fork stub32_fork +2 i386 fork sys_fork sys_fork 3 i386 read sys_read 4 i386 write sys_write 5 i386 open sys_open compat_sys_open
@@ -17,7 +17,7 @@ 8 i386 creat sys_creat 9 i386 link sys_link 10 i386 unlink sys_unlink -11 i386 execve sys_execve stub32_execve +11 i386 execve sys_execve compat_sys_execve 12 i386 chdir sys_chdir 13 i386 time sys_time compat_sys_time 14 i386 mknod sys_mknod
@@ -125,7 +125,7 @@ 116 i386 sysinfo sys_sysinfo compat_sys_sysinfo 117 i386 ipc sys_ipc compat_sys_ipc 118 i386 fsync sys_fsync -119 i386 sigreturn sys_sigreturn stub32_sigreturn +119 i386 sigreturn sys_sigreturn sys32_sigreturn 120 i386 clone sys_clone stub32_clone 121 i386 setdomainname sys_setdomainname 122 i386 uname sys_newuname
@@ -179,7 +179,7 @@ 170 i386 setresgid sys_setresgid16 171 i386 getresgid sys_getresgid16 172 i386 prctl sys_prctl -173 i386 rt_sigreturn sys_rt_sigreturn stub32_rt_sigreturn +173 i386 rt_sigreturn sys_rt_sigreturn sys32_rt_sigreturn 174 i386 rt_sigaction sys_rt_sigaction compat_sys_rt_sigaction 175 i386 rt_sigprocmask sys_rt_sigprocmask 176 i386 rt_sigpending sys_rt_sigpending compat_sys_rt_sigpending
@@ -196,7 +196,7 @@ 187 i386 sendfile sys_sendfile compat_sys_sendfile 188 i386 getpmsg 189 i386 putpmsg -190 i386 vfork sys_vfork stub32_vfork +190 i386 vfork sys_vfork sys_vfork 191 i386 ugetrlimit sys_getrlimit compat_sys_getrlimit 192 i386 mmap2 sys_mmap_pgoff 193 i386 truncate64 sys_truncate64 sys32_truncate64
@@ -364,7 +364,7 @@ 355 i386 getrandom sys_getrandom 356 i386 memfd_create sys_memfd_create 357 i386 bpf sys_bpf -358 i386 execveat sys_execveat stub32_execveat +358 i386 execveat sys_execveat compat_sys_execveat 359 i386 socket sys_socket 360 i386 socketpair sys_socketpair 361 i386 bind sys_bind
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 19/36 x86/syscalls: Give sys_call_ptr_t a useful type
In reply to Andy LutomirskiSyscalls are asmlinkage functions (on 32-bit kernels), take six args of type unsigned long, and return long. Note that uml could probably be slightly cleaned up on top of this patch.
--- arch/x86/entry/syscall_32.c | 4 ++-- arch/x86/entry/syscall_64.c | 4 ++-- arch/x86/include/asm/syscall.h | 4 +++- arch/x86/um/sys_call_table_32.c | 4 ++-- arch/x86/um/sys_call_table_64.c | 4 ++-- 5 files changed, 11 insertions(+), 9 deletions(-)
arch/x86/entry/syscall_32.c +2 -2
@@ -12,13 +12,13 @@#define SYM (sym, compat) sym#endif -#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void SYM(sym, compat)(void) ; +#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage long SYM(sym, compat)(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_32.h> #undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [nr] = SYM(sym, compat), -extern asmlinkage void sys_ni_syscall(void); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ); __visible const sys_call_ptr_t ia32_sys_call_table[__NR_syscall_compat_max+1] = {/*
arch/x86/entry/syscall_64.c +2 -2
@@ -14,13 +14,13 @@# define __SYSCALL_X32 (nr, sym, compat)/* nothing */ #endif -#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_64.h> #undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [nr] = sym, -extern void sys_ni_syscall (void ); +extern long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ); asmlinkage const sys_call_ptr_t sys_call_table[__NR_syscall_max+1] = {/*
arch/x86/include/asm/syscall.h +3 -1
@@ -20,7 +20,9 @@#include <asm/thread_info.h> /* for TS_COMPAT */ #include <asm/unistd.h> -typedef asmlinkage void (*sys_call_ptr_t)(void); +typedef asmlinkage long (*sys_call_ptr_t )(unsigned long ,unsigned long , +unsigned long ,unsigned long , +unsigned long ,unsigned long );extern const sys_call_ptr_t sys_call_table[];#if defined(CONFIG_X86_32 )
arch/x86/um/sys_call_table_32.c +2 -2
@@ -25,13 +25,13 @@#define old_mmap sys_old_mmap -#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_32.h> #undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [ nr ] = sym, -extern asmlinkage void sys_ni_syscall(void); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {/*
arch/x86/um/sys_call_table_64.c +2 -2
@@ -38,13 +38,13 @@#define __SYSCALL_COMMON (nr, sym, compat) __SYSCALL_64(nr, sym, compat)#define __SYSCALL_X32 (nr, sym, compat)/* Not supported */ -#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_64.h> #undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [ nr ] = sym, -extern void sys_ni_syscall (void ); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {/*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 21/36 x86/entry/64/compat: Migrate the body of the syscall entry to C
In reply to Andy Lutomirski--- arch/x86/entry/entry_64.S | 2 +- arch/x86/entry/entry_64_compat.S | 56 +++++++++++++--------------------------- 2 files changed, 19 insertions(+), 39 deletions(-)
arch/x86/entry/entry_64.S +1 -1
@@ -582,7 +582,7 @@ retint_kernel: * At this label, code paths which return to kernel and to user, * which come from interrupts/exception and from syscalls, merge. */ -restore_regs_and_iret: +GLOBAL(restore_regs_and_iret) RESTORE_EXTRA_REGS restore_c_regs_and_iret: RESTORE_C_REGS
arch/x86/entry/entry_64_compat.S +18 -38
74 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 24/36 x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
In reply to Andy LutomirskiIf CS, SS and IP are as expected and FLAGS is compatible with SYSRETL, then return from fast compat syscalls (both SYSCALL and SYSENTER) using SYSRETL.
Unlike native 64-bit opportunistic SYSRET, this is not invisible to user code: RCX and R8-R15 end up in a different state than shown saved in pt_regs. To compensate, we only do this when returning to the vDSO fast syscall return path. This won't interfere with syscall restart, as we won't use SYSRETL when returning to the INT80 restart instruction.
--- arch/x86/entry/common.c | 23 +++++++++++++++++++--- arch/x86/entry/entry_64_compat.S | 42 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 5 deletions(-)
arch/x86/entry/common.c +20 -3
@@ -363,7 +363,8 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } -__visible void do_fast_syscall_32(struct pt_regs *regs) +/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ +__visible long do_fast_syscall_32 (struct pt_regs *regs) {/* * Called using the internal vDSO SYSENTER/SYSCALL32 calling
@@ -395,12 +396,28 @@ __visible void do_fast_syscall_32(struct pt_regs *regs)enter_from_user_mode ();#endif prepare_exit_to_usermode (regs); - return; + return 0; /* Keep it simple: use IRET. */ }local_irq_disable ();/* Now this is just like a normal syscall. */ do_int80_syscall_32 (regs); -return ; + +#ifdef CONFIG_X86_64 +/* +* Opportunistic SYSRETL: if possible, try to return using SYSRETL. +* SYSRETL is available on all 64-bit CPUs, so we don't need to +* bother with SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +*/ +return regs->cs == __USER32_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ; +#else +return 0 ; +#endif }#endif
arch/x86/entry/entry_64_compat.S +40 -2
54 lines not on this page · raw message
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 29/36 x86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls
In reply to Andy LutomirskiThis is slightly messy, but it eliminates an unnecessary cli;sti pair.
--- arch/x86/entry/common.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
arch/x86/entry/common.c +11 -7
@@ -323,10 +323,10 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )/* -* Does a 32-bit syscall. Called with IRQs off and does all entry and -* exit work. +* Does a 32-bit syscall. Called with IRQs on and does all entry and +* exit work and returns with IRQs off. */ -__visible void do_int80_syscall_32(struct pt_regs *regs) +static void do_syscall_32_irqs_on (struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned int nr = (unsigned int )regs->orig_ax;
@@ -335,8 +335,6 @@ __visible void do_int80_syscall_32(struct pt_regs *regs) ti->status |=TS_COMPAT ;#endif -local_irq_enable (); -if (READ_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY) {/* * Subtlety here: if ptrace pokes something larger than
@@ -363,6 +361,13 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } +/* Handles int $0x80 */ +__visible void do_int80_syscall_32(struct pt_regs *regs) +{ +local_irq_enable (); +do_syscall_32_irqs_on (regs); +} +/* Returns 0 to return using IRET or 1 to return using SYSEXIT/SYSRETL. */ __visible long do_fast_syscall_32 (struct pt_regs *regs) {
@@ -398,10 +403,9 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)prepare_exit_to_usermode (regs);return 0 ;/* Keep it simple: use IRET. */ } -local_irq_disable ();/* Now this is just like a normal syscall. */ -do_int80_syscall_32 (regs); +do_syscall_32_irqs_on (regs);#ifdef CONFIG_X86_64 /*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 28/36 x86/asm: Remove thread_info.sysenter_return
In reply to Andy LutomirskiIt's no longer needed.
We could reinstate something like it as an optimization, which would remove two cachelines from the fast syscall entry working set. I benchmarked it, and it makes no difference whatsoever to the performance of cache-hot compat syscalls on Sandy Bridge.
--- arch/x86/include/asm/thread_info.h | 1 - arch/x86/kernel/asm-offsets.c | 3 --- 2 files changed, 4 deletions(-)
arch/x86/include/asm/thread_info.h +0 -1
@@ -59,7 +59,6 @@ struct thread_info {__u32 cpu;/* current CPU */ int saved_preempt_count;mm_segment_t addr_limit; -void __user *sysenter_return; unsigned int sig_on_uaccess_error:1; unsigned int uaccess_err:1; /* uaccess failed */ };
arch/x86/kernel/asm-offsets.c +0 -3
@@ -54,9 +54,6 @@ void common(void) {OFFSET (IA32_SIGCONTEXT_ip, sigcontext_ia32, ip);BLANK (); -OFFSET (TI_sysenter_return, thread_info, sysenter_return); - -BLANK ();OFFSET (IA32_RT_SIGFRAME_sigcontext, rt_sigframe_ia32, uc.uc_mcontext);#endif
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 27/36 x86/entry/32: Re-implement SYSENTER using the new C path
In reply to Andy Lutomirski--- arch/x86/entry/common.c | 17 +++- arch/x86/entry/entry_32.S | 132 ++++++++----------------------- arch/x86/entry/vdso/vdso32/system_call.S | 2 + 3 files changed, 51 insertions(+), 100 deletions(-)
arch/x86/entry/common.c +15 -2
@@ -363,7 +363,7 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } -/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ +/* Returns 0 to return using IRET or 1 to return using SYSEXIT/SYSRETL. */ __visible long do_fast_syscall_32 (struct pt_regs *regs) {/*
@@ -417,7 +417,20 @@ __visible long do_fast_syscall_32(struct pt_regs *regs) regs->ip == landing_pad && (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ;#else -return 0 ; +/* +* Opportunistic SYSEXIT: if possible, try to return using SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +* +* We don't allow syscalls at all from vm86 mode, but we still +* need to check VM, becuase we might be returning from sys_vm86. +*/ +return static_cpu_has (X86_FEATURE_SEP ) && + regs->cs == __USER_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF |X86_EFLAGS_VM )) ==0 ; #endif }#endif
arch/x86/entry/entry_32.S +34 -98
165 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/system_call.S +2 -0
@@ -34,6 +34,8 @@ __kernel_vsyscall: /* If SYSENTER (Intel) or SYSCALL32 (AMD) is available, use it. */ ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ "syscall", X86_FEATURE_SYSCALL32 +#else + ALTERNATIVE "", "sysenter", X86_FEATURE_SEP #endif /* Enter using int $0x80 */
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 31/36 x86/entry: Force inlining of 32-bit syscall code
In reply to Andy LutomirskiOn systems that support fast syscalls, we only really care about the performance of the fast syscall path. Forcibly inline it and add a likely annotation.
This saves 4-6 cycles.
--- arch/x86/entry/common.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +5 -3
@@ -324,9 +324,11 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )/* * Does a 32-bit syscall. Called with IRQs on and does all entry and -* exit work and returns with IRQs off. +* exit work and returns with IRQs off. This function is extremely hot +* in workloads that use it, and it's usually called from +* do_fast_syscall_32, so forcibly inline it to improve performance. */ -static void do_syscall_32_irqs_on (struct pt_regs *regs) +static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned int nr = (unsigned int )regs->orig_ax;
@@ -345,7 +347,7 @@ static void do_syscall_32_irqs_on(struct pt_regs *regs) nr =syscall_trace_enter (regs); } -if (nr < IA32_NR_syscalls) { +if (likely (nr < IA32_NR_syscalls)) {/* * It's possible that a 32-bit syscall implementation * takes a 64-bit parameter but nonetheless assumes that
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 36/36 x86/entry: Split and inline syscall_return_slowpath
In reply to Andy LutomirskiGCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part.
Help it out with syscall_return_slowpath by splitting it into two parts and inline the hot part.
Saves 6 cycles for compat syscalls.
--- arch/x86/entry/common.c | 50 ++++++++++++++++++++++++++++--------------------- 1 file changed, 29 insertions(+), 21 deletions(-)
arch/x86/entry/common.c +29 -21
@@ -284,15 +284,40 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs)user_enter (); } +#define SYSCALL_EXIT_WORK_FLAGS \ + (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ + _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT) + +static void syscall_slow_exit_work (struct pt_regs *regs,u32 cached_flags) +{ +bool step; + +audit_syscall_exit (regs); + +if (cached_flags & _TIF_SYSCALL_TRACEPOINT) +trace_sys_exit (regs, regs->ax); + +/* +* If TIF_SYSCALL_EMU is set, we only get here because of +* TIF_SINGLESTEP (i.e. this is PTRACE_SYSEMU_SINGLESTEP). +* We already reported this syscall instruction in +* syscall_trace_enter(). +*/ + step =unlikely ( + (cached_flags & (_TIF_SINGLESTEP | _TIF_SYSCALL_EMU)) + == _TIF_SINGLESTEP); +if (step || cached_flags & _TIF_SYSCALL_TRACE) +tracehook_report_syscall_exit (regs, step); +} +/* * Called with IRQs on and fully valid regs. Returns with IRQs off in a * state such that we can immediately switch to user mode. */ -__visible void syscall_return_slowpath(struct pt_regs *regs) +__visible inline void syscall_return_slowpath(struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);u32 cached_flags =READ_ONCE (ti->flags); -bool step;CT_WARN_ON (ct_state () !=CONTEXT_KERNEL );
@@ -304,25 +329,8 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)* First do one-time work. If these work items are enabled, we * want to run them exactly once per syscall exit with IRQs on. */ -if (cached_flags & (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | - _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT)) { -audit_syscall_exit (regs); - -if (cached_flags & _TIF_SYSCALL_TRACEPOINT) -trace_sys_exit (regs, regs->ax); - -/* -* If TIF_SYSCALL_EMU is set, we only get here because of -* TIF_SINGLESTEP (i.e. this is PTRACE_SYSEMU_SINGLESTEP). -* We already reported this syscall instruction in -* syscall_trace_enter(). -*/ - step =unlikely ( - (cached_flags & (_TIF_SINGLESTEP | _TIF_SYSCALL_EMU)) - == _TIF_SINGLESTEP); -if (step || cached_flags & _TIF_SYSCALL_TRACE) -tracehook_report_syscall_exit (regs, step); - } +if (unlikely (cached_flags &SYSCALL_EXIT_WORK_FLAGS )) +syscall_slow_exit_work (regs, cached_flags);#ifdef CONFIG_COMPAT /*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 35/36 x86/entry: Split and inline prepare_exit_to_usermode
In reply to Andy LutomirskiGCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part.
Help it out with prepare_exit_to_usermode by splitting it into two parts and inline the hot part.
Saves 6-8 cycles for compat syscalls.
--- arch/x86/entry/common.c | 43 ++++++++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 15 deletions(-)
arch/x86/entry/common.c +28 -15
@@ -218,14 +218,12 @@ long syscall_trace_enter(struct pt_regs *regs)return syscall_trace_enter_phase2 (regs, arch, phase1_result); } -/* Called with IRQs disabled. */ -__visible void prepare_exit_to_usermode(struct pt_regs *regs) -{ -if (IS_ENABLED (CONFIG_PROVE_LOCKING ) &&WARN_ON (!irqs_disabled ())) -local_irq_disable (); - -lockdep_sys_exit (); +#define EXIT_TO_USERMODE_LOOP_FLAGS \ + (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ + _TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY) +static void exit_to_usermode_loop (struct pt_regs *regs,u32 cached_flags) +{/* * In order to return to user mode, we need to have IRQs off with * none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY,
@@ -235,14 +233,6 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)* work to clear some of the flags can sleep. */ while (true) { -u32 cached_flags = -READ_ONCE (pt_regs_to_thread_info (regs)->flags); - -if (!(cached_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | - _TIF_UPROBE | _TIF_NEED_RESCHED | - _TIF_USER_RETURN_NOTIFY))) -break ; -/* We have work to do. */ local_irq_enable ();
@@ -266,7 +256,30 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)/* Disable IRQs and retry */ local_irq_disable (); + + cached_flags =READ_ONCE (pt_regs_to_thread_info (regs)->flags); + +if (!(cached_flags &EXIT_TO_USERMODE_LOOP_FLAGS )) +break ; + } +} + +/* Called with IRQs disabled. */ +__visible inline void prepare_exit_to_usermode(struct pt_regs *regs) +{ +u32 cached_flags; + +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) &&WARN_ON (!irqs_disabled ())) +local_irq_disable (); + +lockdep_sys_exit (); + + cached_flags = +READ_ONCE (pt_regs_to_thread_info (regs)->flags); + +if (unlikely (cached_flags &EXIT_TO_USERMODE_LOOP_FLAGS )) +exit_to_usermode_loop (regs, cached_flags);user_enter (); }
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 34/36 x86/entry: Use pt_regs_to_thread_info() in syscall entry tracing
In reply to Andy LutomirskiIt generates simpler and faster code than current_thread_info().
--- arch/x86/entry/common.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
arch/x86/entry/common.c +11 -11
@@ -30,6 +30,13 @@#define CREATE_TRACE_POINTS #include <trace/events/syscalls.h> +static struct thread_info *pt_regs_to_thread_info (struct pt_regs *regs) +{ +unsigned long top_of_stack = + (unsigned long )(regs +1 ) +TOP_OF_KERNEL_STACK_PADDING ; +return (struct thread_info *)(top_of_stack -THREAD_SIZE ); +} +#ifdef CONFIG_CONTEXT_TRACKING /* Called on entry from user mode with IRQs off. */ __visible void enter_from_user_mode(void)
@@ -68,14 +75,14 @@ static void do_audit_syscall_entry(struct pt_regs *regs, u32 arch)*/ unsigned long syscall_trace_enter_phase1 (struct pt_regs *regs,u32 arch) { +struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned long ret =0 ;u32 work;if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))BUG_ON (regs !=task_pt_regs (current)); - work =ACCESS_ONCE (current_thread_info ()->flags) & - _TIF_WORK_SYSCALL_ENTRY; + work =ACCESS_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY;#ifdef CONFIG_CONTEXT_TRACKING /*
@@ -157,9 +164,9 @@ unsigned long syscall_trace_enter_phase1(struct pt_regs *regs, u32 arch)long syscall_trace_enter_phase2 (struct pt_regs *regs,u32 arch,unsigned long phase1_result) { +struct thread_info *ti =pt_regs_to_thread_info (regs);long ret =0 ; -u32 work =ACCESS_ONCE (current_thread_info ()->flags) & - _TIF_WORK_SYSCALL_ENTRY; +u32 work =ACCESS_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY;if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))BUG_ON (regs !=task_pt_regs (current));
@@ -211,13 +218,6 @@ long syscall_trace_enter(struct pt_regs *regs)return syscall_trace_enter_phase2 (regs, arch, phase1_result); } -static struct thread_info *pt_regs_to_thread_info (struct pt_regs *regs) -{ -unsigned long top_of_stack = - (unsigned long )(regs +1 ) +TOP_OF_KERNEL_STACK_PADDING ; -return (struct thread_info *)(top_of_stack -THREAD_SIZE ); -} -/* Called with IRQs disabled. */ __visible void prepare_exit_to_usermode(struct pt_regs *regs) {
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 33/36 x86/entry: Hide two syscall entry assertions behind CONFIG_DEBUG_ENTRY
In reply to Andy LutomirskiThis shaves a few cycles off the slow paths.
--- arch/x86/entry/common.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/entry/common.c +4 -2
@@ -71,7 +71,8 @@ unsigned long syscall_trace_enter_phase1(struct pt_regs *regs, u32 arch)unsigned long ret =0 ;u32 work; -BUG_ON (regs !=task_pt_regs (current)); +if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) +BUG_ON (regs !=task_pt_regs (current)); work =ACCESS_ONCE (current_thread_info ()->flags) & _TIF_WORK_SYSCALL_ENTRY;
@@ -160,7 +161,8 @@ long syscall_trace_enter_phase2(struct pt_regs *regs, u32 arch,u32 work =ACCESS_ONCE (current_thread_info ()->flags) & _TIF_WORK_SYSCALL_ENTRY; -BUG_ON (regs !=task_pt_regs (current)); +if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) +BUG_ON (regs !=task_pt_regs (current));/* * If we stepped into a sysenter/syscall insn, it trapped in
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 32/36 x86/entry: Micro-optimize compat fast syscall arg fetch
In reply to Andy Lutomirskiwe're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the access_ok check isn't needed.
This saves about three cycles on a cache-hot fast syscall.
--- arch/x86/entry/common.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
arch/x86/entry/common.c +14 -2
@@ -394,8 +394,20 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)* WARNING: We are in CONTEXT_USER and RCU isn't paying attention! */ local_irq_enable (); -if (get_user (*(u32 *)®s->cx, - (u32 __user __force *)(unsigned long )(u32 )regs->sp)) { + if ( +#ifdef CONFIG_X86_64 +/* +* Micro-optimization: the pointer we're following is explicitly +* 32 bits, so it can't be out of range. +*/ +__get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long)(u32)regs->sp) +#else + get_user(*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long)(u32)regs->sp) +#endif + ) { +/* User code screwed up. */ local_irq_disable (); regs->ax = -EFAULT ;
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 30/36 x86/entry: Make irqs_disabled checks in exit code depend on lockdep
In reply to Andy LutomirskiThese checks are quite slow. Disable them in non-lockdep kernels to reduce the performance hit.
--- arch/x86/entry/common.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +3 -3
@@ -219,7 +219,7 @@ static struct thread_info *pt_regs_to_thread_info(struct pt_regs *regs)/* Called with IRQs disabled. */ __visible void prepare_exit_to_usermode(struct pt_regs *regs) { -if (WARN_ON (!irqs_disabled ())) +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) &&WARN_ON (!irqs_disabled ()))local_irq_disable ();lockdep_sys_exit ();
@@ -281,8 +281,8 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)CT_WARN_ON (ct_state () !=CONTEXT_KERNEL ); -if (WARN (irqs_disabled (),"syscall %ld left IRQs disabled" , - regs->orig_ax)) +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) && +WARN (irqs_disabled (),"syscall %ld left IRQs disabled" , regs->orig_ax))local_irq_enable ();/*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 26/36 x86/entry/32: Switch INT80 to the new C syscall path
In reply to Andy Lutomirski--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)
arch/x86/entry/entry_32.S +13 -21
@@ -153,13 +153,13 @@ #endif /* CONFIG_X86_32_LAZY_GS */ -.macro SAVE_ALL +.macro SAVE_ALL pt_regs_ax=%eax cld PUSH_GS pushl %fs pushl %es pushl %ds - pushl %eax + pushl \pt_regs_ax pushl %ebp pushl %edi pushl %esi
@@ -370,20 +370,17 @@ ENDPROC(entry_SYSENTER_32) # system call handler stub ENTRY(entry_INT80_32) ASM_CLAC - pushl %eax # save orig_eax - SAVE_ALL - GET_THREAD_INFO(%ebp) - # system call tracing in operation / emulation - testl $_TIF_WORK_SYSCALL_ENTRY, TI_flags(%ebp) - jnz syscall_trace_entry - cmpl $(NR_syscalls), %eax - jae syscall_badsys -syscall_call: - call *sys_call_table(, %eax, 4) -syscall_after_call: - movl %eax, PT_EAX(%esp) # store the return value -syscall_exit: - jmp syscall_exit_work + pushl %eax /* pt_regs->orig_ax */ + SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest, load -ENOSYS into ax */ + + /* + * User mode is traced as though IRQs are on, and the interrupt gate + * turned them off. + */ + TRACE_IRQS_OFF + + movl %esp, %eax + call do_int80_syscall_32 restore_all: TRACE_IRQS_IRET
@@ -491,11 +488,6 @@ syscall_fault: jmp resume_userspace END(syscall_fault) -syscall_badsys: - movl $-ENOSYS, %eax - jmp syscall_after_call -END(syscall_badsys) - sysenter_badsys: movl $-ENOSYS, %eax jmp sysenter_after_call
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 25/36 x86/entry/32: Open-code return tracking from fork and kthreads
In reply to Andy Lutomirskisyscall_exit is going away, and return tracing is just a function call now, so open-code the two non-syscall 32-bit users.
While we're at it, update the big register layout comment.
--- arch/x86/entry/entry_32.S | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
arch/x86/entry/entry_32.S +15 -3
@@ -3,7 +3,7 @@ * * entry_32.S contains the system-call and low-level fault and trap handling routines. * - * Stack layout in 'syscall_exit': + * Stack layout while running C code: * ptrace needs to have all registers on the stack. * If the order here is changed, it needs to be * updated in fork.c:copy_process(), signal.c:do_signal(),
@@ -211,7 +211,11 @@ ENTRY(ret_from_fork) popl %eax pushl $0x0202 # Reset kernel eflags popfl - jmp syscall_exit + + /* When we fork, we trace the syscall return in the child, too. */ + movl %esp, %eax + call syscall_return_slowpath + jmp restore_all END(ret_from_fork) ENTRY(ret_from_kernel_thread)
@@ -224,7 +228,15 @@ ENTRY(ret_from_kernel_thread) movl PT_EBP(%esp), %eax call *PT_EBX(%esp) movl $0, PT_EAX(%esp) - jmp syscall_exit + + /* + * Kernel threads return to userspace as if returning from a syscall. + * We should check whether anything actually uses this path and, if so, + * consider switching it over to ret_from_fork. + */ + movl %esp, %eax + call syscall_return_slowpath + jmp restore_all ENDPROC(ret_from_kernel_thread) /*
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 23/36 x86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace
In reply to Andy LutomirskiWhat, you didn't realize that SYSENTER and SYSCALL were actually the same thing? :)
Unlike the old code, this actually passes the ptrace_syscall_32 test on AMD systems.
--- arch/x86/entry/entry_64_compat.S | 90 +++++++++++++++++++------------- arch/x86/entry/vdso/vdso32/system_call.S | 8 +++ 2 files changed, 62 insertions(+), 36 deletions(-)
arch/x86/entry/entry_64_compat.S +54 -36
@@ -52,15 +52,18 @@ ENTRY(entry_SYSENTER_compat) SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - /* Zero-extending 32-bit regs, do not remove */ - movl %ebp, %ebp + /* + * User tracing code (ptrace or signal handlers) might assume that + * the saved RAX contains a 32-bit number when we're invoking a 32-bit + * syscall. Just in case the high bits are nonzero, zero-extend + * the syscall number. (This could almost certainly be deleted + * with no ill effects.) + */ movl %eax, %eax - movl ASM_THREAD_INFO(TI_sysenter_return, %rsp, 0), %r10d - /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ - pushq %rbp /* pt_regs->sp */ + pushq %rcx /* pt_regs->sp */ /* * Push flags. This is nasty. First, interrupts are currently
@@ -70,17 +73,28 @@ ENTRY(entry_SYSENTER_compat) */ pushfq /* pt_regs->flags (except IF = 0) */ orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */ + ASM_CLAC /* Clear AC after saving FLAGS */ pushq $__USER32_CS /* pt_regs->cs */ - pushq %r10 /* pt_regs->ip = thread_info->sysenter_return */ + xorq %r8,%r8 + pushq %r8 /* pt_regs->ip = 0 (placeholder) */ pushq %rax /* pt_regs->orig_ax */ pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */ pushq %rdx /* pt_regs->dx */ - pushq %rcx /* pt_regs->cx */ + pushq %rcx /* pt_regs->cx (will be overwritten) */ pushq $-ENOSYS /* pt_regs->ax */ + pushq %r8 /* pt_regs->r8 = 0 */ + pushq %r8 /* pt_regs->r9 = 0 */ + pushq %r8 /* pt_regs->r10 = 0 */ + pushq %r8 /* pt_regs->r11 = 0 */ + pushq %rbx /* pt_regs->rbx */ + pushq %rbp /* pt_regs->rbp */ + pushq %r8 /* pt_regs->r12 = 0 */ + pushq %r8 /* pt_regs->r13 = 0 */ + pushq %r8 /* pt_regs->r14 = 0 */ + pushq %r8 /* pt_regs->r15 = 0 */ cld - sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* * Sysenter doesn't filter flags, so we need to clear NT
@@ -93,16 +107,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: - /* Temporary: SYSENTER is disabled. */ -#ifdef CONFIG_CONTEXT_TRACKING - call enter_from_user_mode -#endif - ENABLE_INTERRUPTS(CLBR_NONE) - movl $11, %edi - call do_exit + /* + * User mode is traced as though IRQs are on, and SYSENTER + * turned them off. + */ + TRACE_IRQS_OFF - /* Unreachable. */ - ud2 + movq %rsp, %rdi + call do_fast_syscall_32 + jmp .Lsyscall_32_done sysenter_fix_flags: pushq $X86_EFLAGS_FIXED
@@ -135,26 +148,14 @@ ENDPROC(entry_SYSENTER_compat) * edi arg5 * esp user stack * 0(%esp) arg6 - * - * This is purely a fast path. For anything complicated we use the int 0x80 - * path below. We set up a complete hardware stack frame to share code - * with the int 0x80 path. */ ENTRY(entry_SYSCALL_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK - /* Temporary: SYSCALL32 is disabled. */ - movl $-ENOSYS, %eax - USERGS_SYSRET32 - + /* Stash user ESP and switch to the kernel stack. */ movl %esp, %r8d movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %eax, %eax
@@ -169,13 +170,29 @@ ENTRY(entry_SYSCALL_compat) pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */ pushq %rdx /* pt_regs->dx */ - pushq %rbp /* pt_regs->cx */ - movl %ebp, %ecx + pushq %rcx /* pt_regs->cx (will be overwritten) */ pushq $-ENOSYS /* pt_regs->ax */ - sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ + xorq %r8,%r8 + pushq %r8 /* pt_regs->r8 = 0 */ + pushq %r8 /* pt_regs->r9 = 0 */ + pushq %r8 /* pt_regs->r10 = 0 */ + pushq %r8 /* pt_regs->r11 = 0 */ + pushq %rbx /* pt_regs->rbx */ + pushq %rbp /* pt_regs->rbp */ + pushq %r8 /* pt_regs->r12 = 0 */ + pushq %r8 /* pt_regs->r13 = 0 */ + pushq %r8 /* pt_regs->r14 = 0 */ + pushq %r8 /* pt_regs->r15 = 0 */ - /* Unreachable. */ - ud2 + /* + * User mode is traced as though IRQs are on, and SYSENTER + * turned them off. + */ + TRACE_IRQS_OFF + + movq %rsp, %rdi + call do_fast_syscall_32 + jmp .Lsyscall_32_done END(entry_SYSCALL_compat) /*
@@ -243,6 +260,7 @@ ENTRY(entry_INT80_compat) movq %rsp, %rdi call do_int80_syscall_32 +.Lsyscall_32_done: /* Go back to user mode. */ TRACE_IRQS_ON
arch/x86/entry/vdso/vdso32/system_call.S +8 -0
@@ -3,6 +3,8 @@ */ #include <asm/dwarf2.h> +#include <asm/cpufeature.h> +#include <asm/alternative-asm.h> /* * First get the common code for the sigreturn entry points.
@@ -28,6 +30,12 @@ __kernel_vsyscall: CFI_REL_OFFSET ecx, 0 movl %esp, %ecx +#ifdef CONFIG_X86_64 + /* If SYSENTER (Intel) or SYSCALL32 (AMD) is available, use it. */ + ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ + "syscall", X86_FEATURE_SYSCALL32 +#endif + /* Enter using int $0x80 */ movl (%esp), %ecx int $0x80
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 22/36 x86/entry: Add C code for fast system call entries
In reply to Andy LutomirskiThis handles both SYSENTER and SYSCALL. The asm glue will take care of the differences.
--- arch/x86/entry/common.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/entry/common.c +43 -0
@@ -24,6 +24,8 @@#include <asm/desc.h> #include <asm/traps.h> +#include <asm/vdso.h> +#include <asm/uaccess.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -360,4 +362,45 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } + +__visible void do_fast_syscall_32(struct pt_regs *regs) +{ +/* +* Called using the internal vDSO SYSENTER/SYSCALL32 calling +* convention. Adjust regs so it looks like we entered using int80. +*/ + +unsigned long landing_pad = (unsigned long )current->mm->context.vdso + + vdso_image_32.sym_int80_landing_pad; + +/* +* SYSENTER loses EIP, and even SYSCALL32 needs us to skip forward +* so that 'regs->ip -= 2' lands back on an int $0x80 instruction. +* Fix it up. +*/ + regs->ip = landing_pad; + +/* +* Fetch ECX from where the vDSO stashed it. +* +* WARNING: We are in CONTEXT_USER and RCU isn't paying attention! +*/ +local_irq_enable (); +if (get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long )(u32 )regs->sp)) { +/* User code screwed up. */ +local_irq_disable (); + regs->ax = -EFAULT ; +#ifdef CONFIG_CONTEXT_TRACKING +enter_from_user_mode (); +#endif +prepare_exit_to_usermode (regs); +return ; + } +local_irq_disable (); + +/* Now this is just like a normal syscall. */ +do_int80_syscall_32 (regs); +return ; +}#endif
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 20/36 x86/entry: Add do_syscall_32, a C function to do 32-bit syscalls
In reply to Andy LutomirskiSystem calls are really quite simple. Add a helper to call a 32-bit system call.
--- arch/x86/entry/common.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/entry/common.c +43 -0
@@ -318,3 +318,46 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)local_irq_disable ();prepare_exit_to_usermode (regs); } + +#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION ) +/* +* Does a 32-bit syscall. Called with IRQs off and does all entry and +* exit work. +*/ +__visible void do_int80_syscall_32(struct pt_regs *regs) +{ +struct thread_info *ti =pt_regs_to_thread_info (regs); +unsigned int nr = (unsigned int )regs->orig_ax; + +#ifdef CONFIG_IA32_EMULATION + ti->status |=TS_COMPAT ; +#endif + +local_irq_enable (); + +if (READ_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY) { +/* +* Subtlety here: if ptrace pokes something larger than +* 2^32-1 into orig_ax, this truncates it. This may or +* may not be necessary, but it matches the old asm +* behavior. +*/ + nr =syscall_trace_enter (regs); + } + +if (nr < IA32_NR_syscalls) { +/* +* It's possible that a 32-bit syscall implementation +* takes a 64-bit parameter but nonetheless assumes that +* the high bits are zero. Make sure we zero-extend all +* of the args. +*/ + regs->ax = ia32_sys_call_table[nr]( + (unsigned int )regs->bx, (unsigned int )regs->cx, + (unsigned int )regs->dx, (unsigned int )regs->si, + (unsigned int )regs->di, (unsigned int )regs->bp); + } + +syscall_return_slowpath (regs); +} +#endif
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 18/36 x86/entry/syscalls: Move syscall table declarations into asm/syscalls.h
In reply to Andy LutomirskiThe header was missing some compat declarations.
Also make sys_call_ptr_t have a consistent type.
--- arch/x86/entry/syscall_32.c | 5 +---- arch/x86/include/asm/syscall.h | 12 +++++++++++- arch/x86/um/sys_call_table_32.c | 3 +-- arch/x86/um/sys_call_table_64.c | 3 +-- 4 files changed, 14 insertions(+), 9 deletions(-)
arch/x86/entry/syscall_32.c +1 -4
@@ -4,13 +4,12 @@#include <linux/sys.h> #include <linux/cache.h> #include <asm/asm-offsets.h> +#include <asm/syscall.h> #ifdef CONFIG_IA32_EMULATION #define SYM (sym, compat) compat#else #define SYM (sym, compat) sym -#define ia32_sys_call_table sys_call_table -#define __NR_syscall_compat_max __NR_syscall_max#endif #define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void SYM(sym, compat)(void) ;
@@ -19,8 +18,6 @@#define __SYSCALL_I386 (nr, sym, compat) [nr] = SYM(sym, compat), -typedef asmlinkage void (*sys_call_ptr_t)(void); - extern asmlinkage void sys_ni_syscall(void); __visible const sys_call_ptr_t ia32_sys_call_table[__NR_syscall_compat_max+1] = {
arch/x86/include/asm/syscall.h +11 -1
@@ -20,9 +20,19 @@#include <asm/thread_info.h> /* for TS_COMPAT */ #include <asm/unistd.h> -typedef void (*sys_call_ptr_t )(void ); +typedef asmlinkage void (*sys_call_ptr_t)(void);extern const sys_call_ptr_t sys_call_table[]; +#if defined(CONFIG_X86_32 ) +#define ia32_sys_call_table sys_call_table +#define __NR_syscall_compat_max __NR_syscall_max +#define IA32_NR_syscalls NR_syscalls +#endif + +#if defined(CONFIG_IA32_EMULATION ) +extern const sys_call_ptr_t ia32_sys_call_table[]; +#endif +/* * Only the low 32 bits of orig_ax are meaningful, so we return int. * This importantly ignores the high bits on 64-bit, so comparisons
arch/x86/um/sys_call_table_32.c +1 -2
@@ -7,6 +7,7 @@#include <linux/sys.h> #include <linux/cache.h> #include <generated/user_constants.h> +#include <asm/syscall.h> #define __NO_STUBS
@@ -30,8 +31,6 @@#undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [ nr ] = sym, -typedef asmlinkage void (*sys_call_ptr_t)(void); - extern asmlinkage void sys_ni_syscall(void);const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {
arch/x86/um/sys_call_table_64.c +1 -2
@@ -7,6 +7,7 @@#include <linux/sys.h> #include <linux/cache.h> #include <generated/user_constants.h> +#include <asm/syscall.h> #define __NO_STUBS
@@ -43,8 +44,6 @@#undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [ nr ] = sym, -typedef void (*sys_call_ptr_t )(void ); -extern void sys_ni_syscall (void );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 16/36 x86/entry/64/compat: Remove most of the fast system call machinery
In reply to Andy LutomirskiWe now have only one code path that calls through the compat syscall table. This will make it much more pleasant to change the pt_regs vs register calling convention, which we need to do to move the call into C.
--- arch/x86/entry/entry_64_compat.S | 246 +-------------------------------------- 1 file changed, 4 insertions(+), 242 deletions(-)
arch/x86/entry/entry_64_compat.S +4 -242
@@ -101,119 +101,13 @@ sysenter_flags_fixed: movl $11, %edi call do_exit - /* - * Re-enable interrupts. IRQ tracing already thinks that IRQs are - * on (since we treat user mode as having IRQs on), and the - * prologue above is too short for it to be worth adding a - * tracing round trip. - */ - ENABLE_INTERRUPTS(CLBR_NONE) - - /* - * no need to do an access_ok check here because rbp has been - * 32-bit zero extended - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysenter_tracesys - -sysenter_do_call: - /* 32-bit syscall -> 64-bit C ABI argument conversion */ - movl %edi, %r8d /* arg5 */ - movl %ebp, %r9d /* arg6 */ - xchg %ecx, %esi /* rsi:arg2, rcx:arg4 */ - movl %ebx, %edi /* arg1 */ - movl %edx, %edx /* arg3 (zero extension) */ -sysenter_dispatch: - cmpq $(IA32_NR_syscalls-1), %rax - ja 1f - call *ia32_sys_call_table(, %rax, 8) - movq %rax, RAX(%rsp) -1: - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call_irqs_off -sysexit_from_sys_call: - /* - * NB: SYSEXIT is not obviously safe for 64-bit kernels -- an - * NMI between STI and SYSEXIT has poorly specified behavior, - * and and NMI followed by an IRQ with usergs is fatal. So - * we just pretend we're using SYSEXIT but we really use - * SYSRETL instead. - * - * This code path is still called 'sysexit' because it pairs - * with 'sysenter' and it uses the SYSENTER calling convention. - */ - andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - movl RIP(%rsp), %ecx /* User %eip */ - movq RAX(%rsp), %rax - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - xorl %edx, %edx /* Do not leak kernel information */ - xorq %r8, %r8 - xorq %r9, %r9 - xorq %r10, %r10 - movl EFLAGS(%rsp), %r11d /* User eflags */ - TRACE_IRQS_ON - - /* - * SYSRETL works even on Intel CPUs. Use it in preference to SYSEXIT, - * since it avoids a dicey window with interrupts enabled. - */ - movl RSP(%rsp), %esp - - /* - * USERGS_SYSRET32 does: - * gsbase = user's gs base - * eip = ecx - * rflags = r11 - * cs = __USER32_CS - * ss = __USER_DS - * - * The prologue set RIP(%rsp) to VDSO32_SYSENTER_RETURN, which does: - * - * pop %ebp - * pop %edx - * pop %ecx - * - * Therefore, we invoke SYSRETL with EDX and R8-R10 zeroed to - * avoid info leaks. R11 ends up with VDSO32_SYSENTER_RETURN's - * address (already known to user code), and R12-R15 are - * callee-saved and therefore don't contain any interesting - * kernel data. - */ - USERGS_SYSRET32 + /* Unreachable. */ + ud2 sysenter_fix_flags: pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed - -sysenter_tracesys: - SAVE_EXTRA_REGS - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - movq %rsp, %rdi /* &pt_regs -> arg1 */ - call syscall_trace_enter - - /* Reload arg registers from stack. (see sysenter_tracesys) */ - movl RCX(%rsp), %ecx - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl %eax, %eax /* zero extension */ - - RESTORE_EXTRA_REGS - jmp sysenter_do_call ENDPROC(entry_SYSENTER_compat) /*
@@ -280,142 +174,10 @@ ENTRY(entry_SYSCALL_compat) pushq $-ENOSYS /* pt_regs->ax */ sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ - /* - * No need to do an access_ok check here because r8 has been - * 32-bit zero extended: - */ - ASM_STAC -1: movl (%r8), %r9d - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz cstar_tracesys - -cstar_do_call: - /* 32-bit syscall -> 64-bit C ABI argument conversion */ - movl %edi, %r8d /* arg5 */ - /* r9 already loaded */ /* arg6 */ - xchg %ecx, %esi /* rsi:arg2, rcx:arg4 */ - movl %ebx, %edi /* arg1 */ - movl %edx, %edx /* arg3 (zero extension) */ - -cstar_dispatch: - cmpq $(IA32_NR_syscalls-1), %rax - ja 1f - - call *ia32_sys_call_table(, %rax, 8) - movq %rax, RAX(%rsp) -1: - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call_irqs_off - -sysretl_from_sys_call: - andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl RIP(%rsp), %ecx - movl EFLAGS(%rsp), %r11d - movq RAX(%rsp), %rax - xorq %r10, %r10 - xorq %r9, %r9 - xorq %r8, %r8 - TRACE_IRQS_ON - movl RSP(%rsp), %esp - /* - * 64-bit->32-bit SYSRET restores eip from ecx, - * eflags from r11 (but RF and VM bits are forced to 0), - * cs and ss are loaded from MSRs. - * (Note: 32-bit->32-bit SYSRET is different: since r11 - * does not exist, it merely sets eflags.IF=1). - * - * NB: On AMD CPUs with the X86_BUG_SYSRET_SS_ATTRS bug, the ss - * descriptor is not reinitialized. This means that we must - * avoid SYSRET with SS == NULL, which could happen if we schedule, - * exit the kernel, and re-enter using an interrupt vector. (All - * interrupt entries on x86_64 set SS to NULL.) We prevent that - * from happening by reloading SS in __switch_to. - */ - USERGS_SYSRET32 - -cstar_tracesys: - xchgl %r9d, %ebp - SAVE_EXTRA_REGS - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %r9, R9(%rsp) - movq %rax, R8(%rsp) - movq %rsp, %rdi /* &pt_regs -> arg1 */ - call syscall_trace_enter - movl R9(%rsp), %r9d - - /* Reload arg registers from stack. (see sysenter_tracesys) */ - movl RCX(%rsp), %ecx - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl %eax, %eax /* zero extension */ - - RESTORE_EXTRA_REGS - xchgl %ebp, %r9d - jmp cstar_do_call + /* Unreachable. */ + ud2 END(entry_SYSCALL_compat) -ia32_badarg: - /* - * So far, we've entered kernel mode, set AC, turned on IRQs, and - * saved C regs except r8-r11. We haven't done any of the other - * standard entry work, though. We want to bail, but we shouldn't - * treat this as a syscall entry since we don't even know what the - * args are. Instead, treat this as a non-syscall entry, finish - * the entry work, and immediately exit after setting AX = -EFAULT. - * - * We're really just being polite here. Killing the task outright - * would be a reasonable action, too. Given that the only valid - * way to have gotten here is through the vDSO, and we already know - * that the stack pointer is bad, the task isn't going to survive - * for long no matter what we do. - */ - - ASM_CLAC /* undo STAC */ - movq $-EFAULT, RAX(%rsp) /* return -EFAULT if possible */ - - /* Fill in the rest of pt_regs */ - xorl %eax, %eax - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - SAVE_EXTRA_REGS - - /* Turn IRQs back off. */ - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - - /* Now finish entering normal kernel mode. */ -#ifdef CONFIG_CONTEXT_TRACKING - call enter_from_user_mode -#endif - - /* And exit again. */ - jmp retint_user - -ia32_ret_from_sys_call_irqs_off: - TRACE_IRQS_ON - ENABLE_INTERRUPTS(CLBR_NONE) - -ia32_ret_from_sys_call: - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - jmp int_ret_from_sys_call - /* * Emulated IA32 system calls via int 0x80. *
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 15/36 x86/entry/64/compat: Remove audit optimizations
In reply to Andy LutomirskiThese audit optimizations are messy and hard to maintain. We'll get a similar effect from opportunistic sysret when fast compat system calls are re-implemented.
--- arch/x86/entry/entry_64_compat.S | 98 +--------------------------------------- 1 file changed, 2 insertions(+), 96 deletions(-)
arch/x86/entry/entry_64_compat.S +2 -96
@@ -16,16 +16,6 @@ #include <linux/linkage.h> #include <linux/err.h> -/* Avoid __ASSEMBLER__'ifying <linux/audit.h> just for this. */ -#include <linux/elf-em.h> -#define AUDIT_ARCH_I386 (EM_386|__AUDIT_ARCH_LE) -#define __AUDIT_ARCH_LE 0x40000000 - -#ifndef CONFIG_AUDITSYSCALL -# define sysexit_audit ia32_ret_from_sys_call_irqs_off -# define sysretl_audit ia32_ret_from_sys_call_irqs_off -#endif - .section .entry.text, "ax" #ifdef CONFIG_PARAVIRT
@@ -148,7 +138,7 @@ sysenter_dispatch: DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysexit_audit + jnz ia32_ret_from_sys_call_irqs_off sysexit_from_sys_call: /* * NB: SYSEXIT is not obviously safe for 64-bit kernels -- an
@@ -200,81 +190,12 @@ sysexit_from_sys_call: */ USERGS_SYSRET32 -#ifdef CONFIG_AUDITSYSCALL - .macro auditsys_entry_common - /* - * At this point, registers hold syscall args in the 32-bit syscall ABI: - * EAX is syscall number, the 6 args are in EBX,ECX,EDX,ESI,EDI,EBP. - * - * We want to pass them to __audit_syscall_entry(), which is a 64-bit - * C function with 5 parameters, so shuffle them to match what - * the function expects: RDI,RSI,RDX,RCX,R8. - */ - movl %esi, %r8d /* arg5 (R8 ) <= 4th syscall arg (ESI) */ - xchg %ecx, %edx /* arg4 (RCX) <= 3rd syscall arg (EDX) */ - /* arg3 (RDX) <= 2nd syscall arg (ECX) */ - movl %ebx, %esi /* arg2 (RSI) <= 1st syscall arg (EBX) */ - movl %eax, %edi /* arg1 (RDI) <= syscall number (EAX) */ - call __audit_syscall_entry - - /* - * We are going to jump back to the syscall dispatch code. - * Prepare syscall args as required by the 64-bit C ABI. - * Registers clobbered by __audit_syscall_entry() are - * loaded from pt_regs on stack: - */ - movl ORIG_RAX(%rsp), %eax /* syscall number */ - movl %ebx, %edi /* arg1 */ - movl RCX(%rsp), %esi /* arg2 */ - movl RDX(%rsp), %edx /* arg3 */ - movl RSI(%rsp), %ecx /* arg4 */ - movl RDI(%rsp), %r8d /* arg5 */ - .endm - - .macro auditsys_exit exit - TRACE_IRQS_ON - ENABLE_INTERRUPTS(CLBR_NONE) - testl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call - movl %eax, %esi /* second arg, syscall return value */ - cmpl $-MAX_ERRNO, %eax /* is it an error ? */ - jbe 1f - movslq %eax, %rsi /* if error sign extend to 64 bits */ -1: setbe %al /* 1 if error, 0 if not */ - movzbl %al, %edi /* zero-extend that into %edi */ - call __audit_syscall_exit - movl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT), %edi - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl %edi, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz \exit - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - jmp int_ret_from_sys_call_irqs_off - .endm - -sysenter_auditsys: - auditsys_entry_common - movl %ebp, %r9d /* reload 6th syscall arg */ - jmp sysenter_dispatch - -sysexit_audit: - auditsys_exit sysexit_from_sys_call -#endif - sysenter_fix_flags: pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed sysenter_tracesys: -#ifdef CONFIG_AUDITSYSCALL - testl $(_TIF_WORK_SYSCALL_ENTRY & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz sysenter_auditsys -#endif SAVE_EXTRA_REGS xorl %eax, %eax /* Do not leak kernel information */ movq %rax, R11(%rsp)
@@ -389,7 +310,7 @@ cstar_dispatch: DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysretl_audit + jnz ia32_ret_from_sys_call_irqs_off sysretl_from_sys_call: andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS)
@@ -420,22 +341,7 @@ sysretl_from_sys_call: */ USERGS_SYSRET32 -#ifdef CONFIG_AUDITSYSCALL -cstar_auditsys: - movl %r9d, R9(%rsp) /* register to be clobbered by call */ - auditsys_entry_common - movl R9(%rsp), %r9d /* reload 6th syscall arg */ - jmp cstar_dispatch - -sysretl_audit: - auditsys_exit sysretl_from_sys_call -#endif - cstar_tracesys: -#ifdef CONFIG_AUDITSYSCALL - testl $(_TIF_WORK_SYSCALL_ENTRY & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz cstar_auditsys -#endif xchgl %r9d, %ebp SAVE_EXTRA_REGS xorl %eax, %eax /* Do not leak kernel information */
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 14/36 x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries
In reply to Andy LutomirskiWe've disabled the vDSO helpers to call them, so turn off the entries entirely (temporarily) in preparation for cleaning them up.
--- arch/x86/entry/entry_64_compat.S | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/x86/entry/entry_64_compat.S +13 -0
@@ -103,6 +103,14 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: + /* Temporary: SYSENTER is disabled. */ +#ifdef CONFIG_CONTEXT_TRACKING + call enter_from_user_mode +#endif + ENABLE_INTERRUPTS(CLBR_NONE) + movl $11, %edi + call do_exit + /* * Re-enable interrupts. IRQ tracing already thinks that IRQs are * on (since we treat user mode as having IRQs on), and the
@@ -324,6 +332,11 @@ ENTRY(entry_SYSCALL_compat) * it is too small to ever cause noticeable irq latency. */ SWAPGS_UNSAFE_STACK + + /* Temporary: SYSCALL32 is disabled. */ + movl $-ENOSYS, %eax + USERGS_SYSRET32 + movl %esp, %r8d movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp ENABLE_INTERRUPTS(CLBR_NONE)
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 13/36 x86/vdso/32: Save extra registers in the INT80 vsyscall path
In reply to Andy LutomirskiThe goal is to integrate the SYSENTER and SYSCALL32 entry paths with the INT80 path. SYSENTER clobbers ESP and EIP. SYSCALL32 clobbers ECX (and, invisibly, R11). SYSRETL (long mode to compat mode) clobbers ECX and, invisibly, R11. SYSEXIT (which we only need for native 32-bit) clobbers ECX and EDX.
This means that we'll need to provide ESP to the kernel in a register (I chose ECX, since it's only needed for SYSENTER) and we need to provide the args that normally live in ECX and EDX in memory.
The epilogue needs to restore ECX and EDX, since user code relies on regs being preserved.
We don't need to do anything special about EIP, since the kernel already knows where we are. The kernel will eventually need to know where int $0x80 lands, so add a vdso_image entry for it.
The only user-visible effect of this code is that ptrace-induced changes to ECX and EDX during fast syscalls will be lost. This is already the case for the SYSENTER path.
--- arch/x86/entry/vdso/vdso2c.c | 1 + arch/x86/entry/vdso/vdso32/system_call.S | 25 ++++++++++++++++++++++++- arch/x86/include/asm/vdso.h | 1 + 3 files changed, 26 insertions(+), 1 deletion(-)
arch/x86/entry/vdso/vdso2c.c +1 -0
@@ -101,6 +101,7 @@ struct vdso_sym required_syms[] = { {"__kernel_vsyscall", true}, {"__kernel_sigreturn", true}, {"__kernel_rt_sigreturn", true}, + {"int80_landing_pad", true}, }; __attribute__((format(printf, 1, 2))) __attribute__((noreturn))
arch/x86/entry/vdso/vdso32/system_call.S +24 -1
@@ -16,7 +16,30 @@ ALIGN __kernel_vsyscall: CFI_STARTPROC - int $0x80 + /* + * Reshuffle regs so that all of any of the entry instructions + * will preserve enough state. + */ + pushl %edx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET edx, 0 + pushl %ecx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET ecx, 0 + movl %esp, %ecx + + /* Enter using int $0x80 */ + movl (%esp), %ecx + int $0x80 +GLOBAL(int80_landing_pad) + + /* Restore ECX and EDX in case they were clobbered. */ + popl %ecx + CFI_RESTORE ecx + CFI_ADJUST_CFA_OFFSET -4 + popl %edx + CFI_RESTORE edx + CFI_ADJUST_CFA_OFFSET -4 ret CFI_ENDPROC
arch/x86/include/asm/vdso.h +1 -0
@@ -26,6 +26,7 @@ struct vdso_image {long sym___kernel_sigreturn;long sym___kernel_rt_sigreturn;long sym___kernel_vsyscall; +long sym_int80_landing_pad; };#ifdef CONFIG_X86_64
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 12/36 x86/elf/64: Clear more registers in elf_common_init
In reply to Andy LutomirskiBefore we start calling execve in contexts that honor the full pt_regs, we need to teach it to initialize all registers.
--- arch/x86/include/asm/elf.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/elf.h +4 -4
@@ -171,11 +171,11 @@ do { \static inline void elf_common_init (struct thread_struct *t,struct pt_regs *regs,const u16 ds) { -/* Commented-out registers are cleared in stub_execve */ -/*regs->ax = regs->bx =*/ regs->cx = regs->dx =0 ; - regs->si = regs->di/*= regs->bp*/ =0 ; +/* ax gets execve's return value. */ +/*regs->ax = */ regs->bx = regs->cx = regs->dx =0 ; + regs->si = regs->di = regs->bp =0 ; regs->r8 = regs->r9 = regs->r10 = regs->r11 =0 ; -/*regs->r12 = regs->r13 = regs->r14 = regs->r15 = 0;*/ + regs->r12 = regs->r13 = regs->r14 = regs->r15 =0 ; t->fs = t->gs =0 ; t->fsindex = t->gsindex =0 ; t->ds = t->es = ds;
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 6/36 x86/entry: Move lockdep_sys_exit to prepare_exit_to_usermode
In reply to Andy LutomirskiRather than worrying about exactly where LOCKDEP_SYS_EXIT should go in the asm code, add it to prepare_exit_from_usermode and remove all of the asm calls that are followed by prepare_exit_to_usermode.
LOCKDEP_SYS_EXIT now appears only in the syscall fast paths.
--- arch/x86/entry/common.c | 2 ++ arch/x86/entry/entry_32.S | 2 -- arch/x86/entry/entry_64.S | 1 - 3 files changed, 2 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +2 -0
@@ -220,6 +220,8 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)if (WARN_ON (!irqs_disabled ()))local_irq_disable (); +lockdep_sys_exit (); +/* * In order to return to user mode, we need to have IRQs off with * none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY,
arch/x86/entry/entry_32.S +0 -2
@@ -255,7 +255,6 @@ ret_from_intr: jb resume_kernel # not returning to v8086 or userspace ENTRY(resume_userspace) - LOCKDEP_SYS_EXIT DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF movl %esp, %eax
@@ -372,7 +371,6 @@ syscall_call: syscall_after_call: movl %eax, PT_EAX(%esp) # store the return value syscall_exit: - LOCKDEP_SYS_EXIT jmp syscall_exit_work restore_all:
arch/x86/entry/entry_64.S +0 -1
@@ -557,7 +557,6 @@ ret_from_intr: jz retint_kernel /* Interrupt came from user space */ - LOCKDEP_SYS_EXIT_IRQ GLOBAL(retint_user) mov %rsp,%rdi call prepare_exit_to_usermode
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 5/36 x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access
In reply to Andy LutomirskiClearing NT is part of the prologue, whereas loading up arg6 makes more sense to think about as part of syscall processing. Reorder them.
--- arch/x86/entry/entry_64_compat.S | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
arch/x86/entry/entry_64_compat.S +9 -9
@@ -89,15 +89,6 @@ ENTRY(entry_SYSENTER_compat) sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* - * no need to do an access_ok check here because rbp has been - * 32-bit zero extended - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - /* * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether * NT was set instead of doing an unconditional popfq.
@@ -106,6 +97,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: + /* + * no need to do an access_ok check here because rbp has been + * 32-bit zero extended + */ + ASM_STAC +1: movl (%rbp), %ebp + _ASM_EXTABLE(1b, ia32_badarg) + ASM_CLAC + orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) jnz sysenter_tracesys
Signature · 1 line
2.4.3
Andy Lutomirski
maintainerPATCH v2 2/36 x86/uaccess: __chk_range_not_ok is unlikely to return true
In reply to Andy LutomirskiThis should improve code quality a bit. It also shrinks the kernel text.
Before:
text data bss dec hex filename
21828379 5194760 1277952 28301091 1afd723 vmlinux
text data bss dec hex filename
21827997 5194760 1277952 28300709 1afd5a5 vmlinux
--- arch/x86/include/asm/uaccess.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/uaccess.h +3 -3
@@ -51,13 +51,13 @@ static inline bool __chk_range_not_ok(unsigned long addr, unsigned long size, un* limit, not add it to the address). */ if (__builtin_constant_p (size)) -return addr > limit - size; +return unlikely (addr > limit - size);/* Arbitrary sizes? Be careful about overflow */ addr += size; -if (addr < size) +if (unlikely (addr < size))return true; -return addr > limit; +return unlikely (addr > limit); }#define __range_not_ok (addr, size, limit) \
Signature · 1 line
2.4.3
Re: [PATCH v2 09/36] x86/asm: Re-add manual CFI infrastructure
In reply to Andy LutomirskiQuoted 5 lines · Andy Lutomirski
Commit
131484c8da97("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations") removed all the manual DWARF annotations outside the vDSO. It also removed the macros we used for the manual annotations.
Quoted 8 lines
Re-add the macros so that we can clean up the vDSO annotations.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/Makefile | 10 ++- arch/x86/include/asm/dwarf2.h | 170 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/dwarf2.h
Please don't add all this gunk back in!
AFAICS the later patches use just a very small subset - add only the minimal subset absolutely needed.
Thanks,
Ingo
Re: [PATCH v2 22/36] x86/entry: Add C code for fast system call entries
In reply to Andy LutomirskiQuoted 2 lines · Andy Lutomirski
This handles both SYSENTER and SYSCALL. The asm glue will take care of the differences.
This seems non-optimal.
Quoted 1 line
+__visible void
do_fast_syscall_32(struct pt_regs *regs)
.. so this is the fast case on 32-bit, but:
Quoted 3 lines
arch/x86/entry/common.c +3
-0+local_irq_enable (); +if (get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long )(u32 )regs->sp)) {
...
Quoted 1 line
+
local_irq_disable();
this is expensive. Since we now do it in C code and can easily do this, why does the code not do this all with interrupts disabled, which is valid for user accesses but disables page faults, and then in the unlikely situation where that fails, we do it the slow and careful way?
It would be nice to see this patch series not just get rid of asm code, but also do things to optimize cases that used to be annoyingly hard to do in user space. No?
Linus
Re: [PATCH v2 22/36] x86/entry: Add C code for fast system call entries
Quoted 4 lines · Linus Torvalds
arch/x86/entry/common.c __visible void do_int80_syscall_32(struct pt_regs *regs) · line 390
Quoted 4 lines
arch/x86/entry/common.c +3
-0+local_irq_enable (); +if (get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long )(u32 )regs->sp)) {
Quoted 8 lines
...
Quoted 1 line
+
local_irq_disable();this is expensive. Since we now do it in C code and can easily do this, why does the code not do this all with interrupts disabled, which is valid for user accesses but disables page faults, and then in the unlikely situation where that fails, we do it the slow and careful way?
Ok. I notice that then a later patch removes the local_irq_disable() and calls do_syscall_32_irqs_on().
So I guess that "just run get_user with interrupts disabled" optimization is pointless, because we'll just end up enabling interrupts at some point anyway, and it can just be done before the get_user().
So never mind.
Linus
Quoted 7 lines · Andy Lutomirski
The patchset is structured as a removal of the old fast syscall code, then the change that makes syscalls into real functions, then a clean re-implementation of fast syscalls.
If we want some of the 25 cycles back, we could consider open-coding a new C fast path.
Ok, the 25 cycles are apparently just 12 cycles. Did you forget to update that after doing some of the optimizations - or is it 25 cycles on 64-bit and 12 cycles on 32-bit?
But regardless, I've read through the series twice, and I don't see anything wrong. The one thing I thought I noticed ended up being mooted by a patch later in the series.
Of course, this is the kind of code where reading the patch doesn't necessarily catch any of the really subtle problems at all, but you seem to have tested it fairly heavily too.
So ACK on the whole series. I don't mind a few cycles for the fast case if it means it's more maintainable, and sadly I suspect that with so many distributions using audit functionality, the fast case is almost unheard of anyway, and the fact that you sped up the slow case is what matters more. Snif.
I'd like to see this as a git tree, with your longish explanation as the pull request for the x86 merge, so that that gets saved for posterity too in the kernel commit logs. But that's between you and Ingo or whoever ends up being the person who picks this up.
Linus
Quoted 6 lines · Linus Torvalds
[...]
I'd like to see this as a git tree, with your longish explanation as the pull request for the x86 merge, so that that gets saved for posterity too in the kernel commit logs. But that's between you and Ingo or whoever ends up being the person who picks this up.
Yeah, will make sure that description gets preserved.
Will wait a days or so for reviews to trickle in before applying the patches, but this looks really good so far, kudos Andy!
Thanks,
Ingo
Re: [PATCH v2 09/36] x86/asm: Re-add manual CFI infrastructure
In reply to Ingo MolnarQuoted 17 lines · Ingo Molnar
Quoted 5 lines · Andy Lutomirski
Commit
131484c8da97("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations") removed all the manual DWARF annotations outside the vDSO. It also removed the macros we used for the manual annotations.Quoted 8 lines
Re-add the macros so that we can clean up the vDSO annotations.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/Makefile | 10 ++- arch/x86/include/asm/dwarf2.h | 170 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/dwarf2.h
Quoted 4 lines
Please don't add all this gunk back in!
AFAICS the later patches use just a very small subset - add only the minimal subset absolutely needed.
OK, I'll strip out the push, movq, etc stuff for v3. I'll leave all of the CFI_XYZ macros, even the unused one, because that's only a couple of lines of code and they all have very simple and obvious meanings (or as simple as the underlying directives, anyway).
--Andy
Re: [PATCH v2 22/36] x86/entry: Add C code for fast system call entries
In reply to Linus TorvaldsQuoted 18 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds
arch/x86/entry/common.c __visible void do_int80_syscall_32(struct pt_regs *regs) · line 390
Quoted 4 lines
arch/x86/entry/common.c +3
-0+local_irq_enable (); +if (get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long )(u32 )regs->sp)) {Quoted 8 lines
...
Quoted 1 line
+
local_irq_disable();this is expensive. Since we now do it in C code and can easily do this, why does the code not do this all with interrupts disabled, which is valid for user accesses but disables page faults, and then in the unlikely situation where that fails, we do it the slow and careful way?
Ok. I notice that then a later patch removes the
local_irq_disable()and callsdo_syscall_32_irqs_on().
Quoted 7 lines
So I guess that "just run
get_userwith interrupts disabled" optimization is pointless, because we'll just end up enabling interrupts at some point anyway, and it can just be done before theget_user().So never mind.
I'll improve the changelog. This is all clear in my head, but I could certainly describe it better.
I tried to structure this as much as possible as simple unoptimized changes that would be easyish to understand from a correctness perspective and then to optimize at the end.
--Andy
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Andy Lutomirski
The patchset is structured as a removal of the old fast syscall code, then the change that makes syscalls into real functions, then a clean re-implementation of fast syscalls.
Quoted 6 lines
Quoted 2 lines
If we want some of the 25 cycles back, we could consider open-coding a new C fast path.
Ok, the 25 cycles are apparently just 12 cycles. Did you forget to update that after doing some of the optimizations - or is it 25 cycles on 64-bit and 12 cycles on 32-bit?
Forgot to update. I'll fix this in v3. I'll also re-benchmark on native 32-bit, which is a bit easier now that I figured out why my Debian VM couldn't do fast syscalls. (Hint: if you have a 32-bit Debian or Debian-derived setup without libc6-i686 installed, install it and everything works better.)
--Andy
Borislav Petkov
maintainerRe: [PATCH v2 02/36] x86/uaccess: __chk_range_not_ok is unlikely to return true
In reply to Andy LutomirskiQuoted 22 lines · Andy Lutomirski
This should improve code quality a bit. It also shrinks the kernel text.
Before:
text data bss dec hex filename
21828379 5194760 1277952 28301091 1afd723 vmlinux
text data bss dec hex filename
21827997 5194760 1277952 28300709 1afd5a5 vmlinuxSigned-off-by Andy Lutomirski <[email protected]>--- arch/x86/include/asm/uaccess.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)arch/x86/include/asm/uaccess.h +0
-0@@ -51,13 +51,13 @@ static inline bool __chk_range_not_ok(unsigned long addr, unsigned long size, un* limit, not add it to the address). */ if (__builtin_constant_p (size))
arch/x86/include/asm/uaccess.h static inline bool __chk_range_not_ok(unsigned long addr, unsigned long size, un · line 60
Quoted 10 lines
arch/x86/include/asm/uaccess.h +3
-3-return addr > limit - size; +return unlikely (addr > limit - size);/* Arbitrary sizes? Be careful about overflow */ addr += size; -if (addr < size) +if (unlikely (addr < size))return true; -return addr > limit; +return unlikely (addr > limit);
It certainly uglifies it though. Are the wins worth the (un-)readability hit?
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Borislav Petkov
maintainerRe: [PATCH v2 05/36] x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access
In reply to Andy LutomirskiQuoted 30 lines · Andy Lutomirski
Clearing NT is part of the prologue, whereas loading up arg6 makes more sense to think about as part of syscall processing. Reorder them.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_64_compat.S | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)arch/x86/entry/entry_64_compat.S +0
-9@@ -89,15 +89,6 @@ ENTRY(entry_SYSENTER_compat) sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* - * no need to do an access_ok check here because rbp has been - * 32-bit zero extended - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - /* * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether * NT was set instead of doing an unconditional popfq.
arch/x86/entry/entry_64_compat.S ENTRY(entry_SYSENTER_compat) · line 98
Quoted 2 lines
arch/x86/entry/entry_64_compat.S +0
-0@@ -106,6 +97,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed:
Btw, do I see it correctly that we can save us this jumping to sysenter_fix_flags and back to sysenter_flags_fixed? Label is jumped to only once and the couple of insns there can be behind a JZ...
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH v2 05/36] x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access
Quoted 29 lines · Borislav Petkov
Quoted 28 lines · Andy Lutomirski
Clearing NT is part of the prologue, whereas loading up arg6 makes more sense to think about as part of syscall processing. Reorder them.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_64_compat.S | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)arch/x86/entry/entry_64_compat.S +0
-9@@ -89,15 +89,6 @@ ENTRY(entry_SYSENTER_compat) sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* - * no need to do an access_ok check here because rbp has been - * 32-bit zero extended - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - /* * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether
Quoted 8 lines
arch/x86/entry/entry_64_compat.S ENTRY(entry_SYSENTER_compat) · line 98
Quoted 4 lines
arch/x86/entry/entry_64_compat.S +0
-0* NT was set instead of doing an unconditional popfq.@@ -106,6 +97,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed:Btw, do I see it correctly that we can save us this jumping to
sysenter_fix_flagsand back tosysenter_flags_fixed? Label is jumped to only once and the couple of insns there can be behind a JZ...
This is an optimization, since NT is unlikely to be set. The forward branch is usually predicted as not taken, and it keeps these rarely used instructions out of the hot cache.
--
Brian Gerst
Borislav Petkov
maintainerRe: [PATCH v2 05/36] x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access
Quoted 3 lines · Brian Gerst
This is an optimization, since NT is unlikely to be set. The forward branch is usually predicted as not taken, and it keeps these rarely used instructions out of the hot cache.
Ah, slick. Because branch predictors mark new branches as not taken and this particular one will remain non-taken in the majority of the cases. I think this warrants a mention in the comment above it though...
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/uaccess: Add unlikely() to __chk_range_not_ok() failure paths
In reply to Andy Lutomirskix86/uaccess: Add unlikely() to __chk_range_not_ok() failure paths
Commit-ID: 7e0f51cb445be8d3aee80e433ed8da4a33ad0157 Gitweb: http://git.kernel.org/tip/7e0f51cb445be8d3aee80e433ed8da4a33ad0157 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:50 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:06 +0200
x86/uaccess: Add unlikely() to __chk_range_not_ok() failure paths
This should improve code quality a bit. It also shrinks the kernel text:
Before:
text data bss dec filename
21828379 5194760 1277952 28301091 vmlinux
After:
text data bss dec filename
21827997 5194760 1277952 28300709 vmlinux
... by 382 bytes.
--- arch/x86/include/asm/uaccess.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/uaccess.h +3 -3
@@ -51,13 +51,13 @@ static inline bool __chk_range_not_ok(unsigned long addr, unsigned long size, un* limit, not add it to the address). */ if (__builtin_constant_p (size)) -return addr > limit - size; +return unlikely (addr > limit - size);/* Arbitrary sizes? Be careful about overflow */ addr += size; -if (addr < size) +if (unlikely (addr < size))return true; -return addr > limit; +return unlikely (addr > limit); }#define __range_not_ok (addr, size, limit) \
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/uaccess: Tell the compiler that uaccess is unlikely to fault
In reply to Andy Lutomirskix86/uaccess: Tell the compiler that uaccess is unlikely to fault
Commit-ID: a76cf66e948afbaeda8e3ecc861f29c47a026c27 Gitweb: http://git.kernel.org/tip/a76cf66e948afbaeda8e3ecc861f29c47a026c27 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:49 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:06 +0200
x86/uaccess: Tell the compiler that uaccess is unlikely to fault
GCC doesn't realize that get_user(), put_user(), and their __ variants are unlikely to fail. Tell it.
I noticed this while playing with the C entry code.
Before:
text data bss dec filename
21828763 5194760 1277952 28301475 vmlinux.baseline
After:
text data bss dec filename
21828379 5194760 1277952 28301091 vmlinux.new
The generated code shrunk by 384 bytes.
--- arch/x86/include/asm/uaccess.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/uaccess.h +4 -4
@@ -182,7 +182,7 @@ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL)) : "=a" (__ret_gu), "=r" (__val_gu) \ : "0" (ptr), "i" (sizeof(*(ptr)))); \ (x) = (__force __typeof__(*(ptr))) __val_gu; \ - __ret_gu; \ +__builtin_expect (__ret_gu,0 ); \ })#define __put_user_x (size, x, ptr, __ret_pu) \
@@ -278,7 +278,7 @@ extern void __put_user_8(void);__put_user_x (X , __pu_val, ptr, __ret_pu); \break ; \ } \ - __ret_pu; \ +__builtin_expect (__ret_pu,0 ); \ })#define __put_user_size (x, ptr, size, retval, errret) \
@@ -401,7 +401,7 @@ do { \ ({ \int __pu_err; \__put_user_size ((x), (ptr), (size), __pu_err, -EFAULT ); \ - __pu_err; \ +__builtin_expect (__pu_err,0 ); \ })#define __get_user_nocheck (x, ptr, size) \
@@ -410,7 +410,7 @@ do { \unsigned long __gu_val; \__get_user_size (__gu_val, (ptr), (size), __gu_err, -EFAULT ); \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ - __gu_err; \ +__builtin_expect (__gu_err,0 ); \ })/* FIXME: this hack is definitely wrong -AK */
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] selftests/x86: Add a test for vDSO unwinding
In reply to Andy Lutomirskiselftests/x86: Add a test for vDSO unwinding
Commit-ID: 3b56aae34bc695638b8673fc8459be1837c18730 Gitweb: http://git.kernel.org/tip/3b56aae34bc695638b8673fc8459be1837c18730 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:51 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:06 +0200
selftests/x86: Add a test for vDSO unwinding
While the kernel itself doesn't use DWARF unwinding, user code expects to be able to unwind the vDSO. The vsyscall (AT_SYSINFO) entry is manually CFI-annotated, and this tests that it unwinds correctly.
I tested the test by incorrectly annotating __kernel_vsyscall, and the test indeed fails if I do that.
--- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/unwind_vdso.c | 209 ++++++++++++++++++++++++++++++ 2 files changed, 210 insertions(+), 1 deletion(-)
tools/testing/selftests/x86/Makefile +1 -1
@@ -5,7 +5,7 @@ include ../lib.mk .PHONY: all all_32 all_64 warn_32bit_failure clean TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt -TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn test_syscall_vdso +TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn test_syscall_vdso unwind_vdso TARGETS_C_32BIT_ALL := $(TARGETS_C_BOTHBITS) $(TARGETS_C_32BIT_ONLY) BINARIES_32 := $(TARGETS_C_32BIT_ALL:%=%_32)
tools/testing/selftests/x86/unwind_vdso.c new file +209 -0
209 lines not on this page · raw message
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] selftests/x86: Add a test for ptrace syscall restart and arg modification
In reply to Andy Lutomirskiselftests/x86: Add a test for ptrace syscall restart and arg modification
Commit-ID: 04235c00b6bb72b589e99efcc18883378ee76f1b Gitweb: http://git.kernel.org/tip/04235c00b6bb72b589e99efcc18883378ee76f1b Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:52 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:07 +0200
selftests/x86: Add a test for ptrace syscall restart and arg modification
This tests assumptions about how fast syscall works wrt pt_regs and, in particular, what happens if IP is decremented by 2 during a syscall.
--- tools/testing/selftests/x86/Makefile | 3 +- tools/testing/selftests/x86/ptrace_syscall.c | 294 +++++++++++++++++++++ .../testing/selftests/x86/raw_syscall_helper_32.S | 46 ++++ 3 files changed, 342 insertions(+), 1 deletion(-)
tools/testing/selftests/x86/Makefile +2 -1
@@ -4,7 +4,7 @@ include ../lib.mk .PHONY: all all_32 all_64 warn_32bit_failure clean -TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt +TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs ldt_gdt syscall_nt ptrace_syscall TARGETS_C_32BIT_ONLY := entry_from_vm86 syscall_arg_fault sigreturn test_syscall_vdso unwind_vdso TARGETS_C_32BIT_ALL := $(TARGETS_C_BOTHBITS) $(TARGETS_C_32BIT_ONLY)
@@ -60,4 +60,5 @@ endif # Some tests have additional dependencies. sysret_ss_attrs_64: thunks.S +ptrace_syscall_32: raw_syscall_helper_32.S test_syscall_vdso_32: thunks_32.S
tools/testing/selftests/x86/ptrace_syscall.c new file +294 -0
294 lines not on this page · raw message
tools/testing/selftests/x86/raw_syscall_helper_32.S new file +46 -0
46 lines not on this page · raw message
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/64/compat: Fix SYSENTER' s NT flag before user memory access
In reply to Andy Lutomirskix86/entry/64/compat: Fix SYSENTER's NT flag before user memory access
Commit-ID: dd27f998f0ed3c797032a82033fa191be7c61e4c Gitweb: http://git.kernel.org/tip/dd27f998f0ed3c797032a82033fa191be7c61e4c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:53 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:07 +0200
x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access
Clearing NT is part of the prologue, whereas loading up arg6 makes more sense to think about as part of syscall processing. Reorder them.
--- arch/x86/entry/entry_64_compat.S | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
arch/x86/entry/entry_64_compat.S +9 -9
@@ -89,15 +89,6 @@ ENTRY(entry_SYSENTER_compat) sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* - * no need to do an access_ok check here because rbp has been - * 32-bit zero extended - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - /* * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether * NT was set instead of doing an unconditional popfq.
@@ -106,6 +97,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: + /* + * No need to do an access_ok() check here because RBP has been + * 32-bit zero extended: + */ + ASM_STAC +1: movl (%rbp), %ebp + _ASM_EXTABLE(1b, ia32_badarg) + ASM_CLAC + orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) jnz sysenter_tracesys
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry, locking/lockdep: Move lockdep_sys_exit() to prepare_exit_to_usermode()
In reply to Andy Lutomirskix86/entry, locking/lockdep: Move lockdep_sys_exit() to prepare_exit_to_usermode()
Commit-ID: 72f924783b8a87e4454516520ffb5f35e4930371 Gitweb: http://git.kernel.org/tip/72f924783b8a87e4454516520ffb5f35e4930371 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:54 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:07 +0200
x86/entry, locking/lockdep: Move lockdep_sys_exit() to prepare_exit_to_usermode()
Rather than worrying about exactly where LOCKDEP_SYS_EXIT should go in the asm code, add it to prepare_exit_from_usermode() and remove all of the asm calls that are followed by prepare_exit_to_usermode().
LOCKDEP_SYS_EXIT now appears only in the syscall fast paths.
--- arch/x86/entry/common.c | 2 ++ arch/x86/entry/entry_32.S | 2 -- arch/x86/entry/entry_64.S | 1 - 3 files changed, 2 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +2 -0
@@ -220,6 +220,8 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)if (WARN_ON (!irqs_disabled ()))local_irq_disable (); +lockdep_sys_exit (); +/* * In order to return to user mode, we need to have IRQs off with * none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY,
arch/x86/entry/entry_32.S +0 -2
@@ -255,7 +255,6 @@ ret_from_intr: jb resume_kernel # not returning to v8086 or userspace ENTRY(resume_userspace) - LOCKDEP_SYS_EXIT DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF movl %esp, %eax
@@ -372,7 +371,6 @@ syscall_call: syscall_after_call: movl %eax, PT_EAX(%esp) # store the return value syscall_exit: - LOCKDEP_SYS_EXIT jmp syscall_exit_work restore_all:
arch/x86/entry/entry_64.S +0 -1
@@ -557,7 +557,6 @@ ret_from_intr: jz retint_kernel /* Interrupt came from user space */ - LOCKDEP_SYS_EXIT_IRQ GLOBAL(retint_user) mov %rsp,%rdi call prepare_exit_to_usermode
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
In reply to Andy Lutomirskix86/entry/64/compat: After SYSENTER, move STI after the NT fixup
Commit-ID: b611acf4736b5b00c89dcc238f640337832abcb4 Gitweb: http://git.kernel.org/tip/b611acf4736b5b00c89dcc238f640337832abcb4 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:55 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:08 +0200
x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
We eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts.
--- arch/x86/entry/entry_64_compat.S | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
arch/x86/entry/entry_64_compat.S +22 -8
@@ -58,14 +58,9 @@ ENDPROC(native_usergs_sysret32) * with the int 0x80 path. */ ENTRY(entry_SYSENTER_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %ebp, %ebp
@@ -76,7 +71,16 @@ ENTRY(entry_SYSENTER_compat) /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ pushq %rbp /* pt_regs->sp */ - pushfq /* pt_regs->flags */ + + /* + * Push flags. This is nasty. First, interrupts are currently + * off, but we need pt_regs->flags to have IF set. Second, even + * if TF was set when SYSENTER started, it's clear by now. We fix + * that later using TIF_SINGLESTEP. + */ + pushfq /* pt_regs->flags (except IF = 0) */ + orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */ + pushq $__USER32_CS /* pt_regs->cs */ pushq %r10 /* pt_regs->ip = thread_info->sysenter_return */ pushq %rax /* pt_regs->orig_ax */
@@ -92,12 +96,22 @@ ENTRY(entry_SYSENTER_compat) * Sysenter doesn't filter flags, so we need to clear NT * ourselves. To save a few cycles, we can check whether * NT was set instead of doing an unconditional popfq. + * This needs to happen before enabling interrupts so that + * we don't get preempted with NT set. */ testl $X86_EFLAGS_NT, EFLAGS(%rsp) jnz sysenter_fix_flags sysenter_flags_fixed: /* + * Re-enable interrupts. IRQ tracing already thinks that IRQs are + * on (since we treat user mode as having IRQs on), and the + * prologue above is too short for it to be worth adding a + * tracing round trip. + */ + ENABLE_INTERRUPTS(CLBR_NONE) + + /* * No need to do an access_ok() check here because RBP has been * 32-bit zero extended: */
@@ -244,7 +258,7 @@ sysexit_audit: #endif sysenter_fix_flags: - pushq $(X86_EFLAGS_IF|X86_EFLAGS_FIXED) + pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/vdso: Remove runtime 32-bit vDSO selection
In reply to Andy Lutomirskix86/vdso: Remove runtime 32-bit vDSO selection
Commit-ID: 0a6d1fa0d2b48fbae444e46e7f37a4832b2f8bdf Gitweb: http://git.kernel.org/tip/0a6d1fa0d2b48fbae444e46e7f37a4832b2f8bdf Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:56 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:08 +0200
x86/vdso: Remove runtime 32-bit vDSO selection
32-bit userspace will now always see the same vDSO, which is exactly what used to be the int80 vDSO. Subsequent patches will clean it up and make it support SYSENTER and SYSCALL using alternatives.
---
arch/x86/entry/vdso/Makefile | 35 ++-----
arch/x86/entry/vdso/vdso2c.c | 1 -
arch/x86/entry/vdso/vdso32-setup.c | 28 +----
arch/x86/entry/vdso/vdso32/syscall.S | 75 -------------
arch/x86/entry/vdso/vdso32/sysenter.S | 116 ---------------------
.../entry/vdso/vdso32/{int80.S => system_call.S} | 0
arch/x86/entry/vdso/vma.c | 13 +--
arch/x86/ia32/ia32_signal.c | 4 +-
arch/x86/include/asm/elf.h | 2 +-
arch/x86/include/asm/vdso.h | 9 +-
arch/x86/kernel/signal.c | 4 +-
arch/x86/xen/setup.c | 13 +--
12 files changed, 21 insertions(+), 279 deletions(-)arch/x86/entry/vdso/Makefile +11 -24
@@ -19,9 +19,7 @@ obj-y += vma.o # vDSO images to build vdso_img-$(VDSO64-y) += 64 vdso_img-$(VDSOX32-y) += x32 -vdso_img-$(VDSO32-y) += 32-int80 -vdso_img-$(CONFIG_IA32_EMULATION) += 32-syscall -vdso_img-$(VDSO32-y) += 32-sysenter +vdso_img-$(VDSO32-y) += 32 obj-$(VDSO32-y) += vdso32-setup.o
@@ -122,15 +120,6 @@ $(obj)/%.so: $(obj)/%.so.dbg $(obj)/vdsox32.so.dbg: $(src)/vdsox32.lds $(vobjx32s) FORCE $(call if_changed,vdso) -# -# Build multiple 32-bit vDSO images to choose from at boot time. -# -vdso32.so-$(VDSO32-y) += int80 -vdso32.so-$(CONFIG_IA32_EMULATION) += syscall -vdso32.so-$(VDSO32-y) += sysenter - -vdso32-images = $(vdso32.so-y:%=vdso32-%.so) - CPPFLAGS_vdso32.lds = $(CPPFLAGS_vdso.lds) VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1
@@ -139,14 +128,12 @@ VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1 override obj-dirs = $(dir $(obj)) $(obj)/vdso32/ targets += vdso32/vdso32.lds -targets += vdso32/note.o vdso32/vclock_gettime.o $(vdso32.so-y:%=vdso32/%.o) +targets += vdso32/note.o vdso32/vclock_gettime.o vdso32/system_call.o targets += vdso32/vclock_gettime.o -$(obj)/vdso32.o: $(vdso32-images:%=$(obj)/%) - KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) -$(vdso32-images:%=$(obj)/%.dbg): KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) -$(vdso32-images:%=$(obj)/%.dbg): asflags-$(CONFIG_X86_64) += -m32 +$(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) +$(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32 KBUILD_CFLAGS_32 := $(filter-out -m64,$(KBUILD_CFLAGS)) KBUILD_CFLAGS_32 := $(filter-out -mcmodel=kernel,$(KBUILD_CFLAGS_32))
@@ -157,13 +144,13 @@ KBUILD_CFLAGS_32 += $(call cc-option, -fno-stack-protector) KBUILD_CFLAGS_32 += $(call cc-option, -foptimize-sibling-calls) KBUILD_CFLAGS_32 += -fno-omit-frame-pointer KBUILD_CFLAGS_32 += -DDISABLE_BRANCH_PROFILING -$(vdso32-images:%=$(obj)/%.dbg): KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) +$(obj)/vdso32.so.dbg: KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) -$(vdso32-images:%=$(obj)/%.dbg): $(obj)/vdso32-%.so.dbg: FORCE \ - $(obj)/vdso32/vdso32.lds \ - $(obj)/vdso32/vclock_gettime.o \ - $(obj)/vdso32/note.o \ - $(obj)/vdso32/%.o +$(obj)/vdso32.so.dbg: FORCE \ + $(obj)/vdso32/vdso32.lds \ + $(obj)/vdso32/vclock_gettime.o \ + $(obj)/vdso32/note.o \ + $(obj)/vdso32/system_call.o $(call if_changed,vdso) #
@@ -206,4 +193,4 @@ $(vdso_img_insttargets): install_%: $(obj)/%.dbg $(MODLIB)/vdso FORCE PHONY += vdso_install $(vdso_img_insttargets) vdso_install: $(vdso_img_insttargets) FORCE -clean-files := vdso32-syscall* vdso32-sysenter* vdso32-int80* vdso64* vdso-image-*.c vdsox32.so* +clean-files := vdso32.so vdso32.so.dbg vdso64* vdso-image-*.c vdsox32.so*
arch/x86/entry/vdso/vdso2c.c +0 -1
@@ -98,7 +98,6 @@ struct vdso_sym required_syms[] = {"VDSO_FAKE_SECTION_TABLE_END" , false }, {"VDSO32_NOTE_MASK", true}, - {"VDSO32_SYSENTER_RETURN", true}, {"__kernel_vsyscall", true}, {"__kernel_sigreturn", true}, {"__kernel_rt_sigreturn", true},
arch/x86/entry/vdso/vdso32-setup.c +1 -27
@@ -48,35 +48,9 @@ __setup("vdso32=", vdso32_setup);__setup_param ("vdso=" , vdso_setup, vdso32_setup,0 );#endif -#ifdef CONFIG_X86_64 - -#define vdso32_sysenter () (boot_cpu_has(X86_FEATURE_SYSENTER32)) -#define vdso32_syscall () (boot_cpu_has(X86_FEATURE_SYSCALL32)) - -#else /* CONFIG_X86_32 */ - -#define vdso32_sysenter () (boot_cpu_has(X86_FEATURE_SEP)) -#define vdso32_syscall () (0) - -#endif /* CONFIG_X86_64 */ - -#if defined(CONFIG_X86_32 ) || defined(CONFIG_COMPAT ) -const struct vdso_image *selected_vdso32; -#endif -int __init sysenter_setup (void ) { -#ifdef CONFIG_COMPAT -if (vdso32_syscall ()) - selected_vdso32 = &vdso_image_32_syscall; - else -#endif -if (vdso32_sysenter ()) - selected_vdso32 = &vdso_image_32_sysenter; -else - selected_vdso32 = &vdso_image_32_int80; - -init_vdso_image (selected_vdso32); +init_vdso_image (&vdso_image_32);return 0 ; }
arch/x86/entry/vdso/vdso32/syscall.S deleted +0 -75
75 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/sysenter.S deleted +0 -116
116 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/system_call.S renamed from arch/x86/entry/vdso/vdso32/int80.S +0 -0
arch/x86/entry/vdso/vma.c +1 -12
@@ -180,21 +180,10 @@ up_fail:#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )static int load_vdso32 (void ) { -int ret; -if (vdso32_enabled !=1 )/* Other values all mean "disabled" */ return 0 ; - ret =map_vdso (selected_vdso32, false); -if (ret) -return ret; - -if (selected_vdso32->sym_VDSO32_SYSENTER_RETURN) -current_thread_info ()->sysenter_return = - current->mm->context.vdso + - selected_vdso32->sym_VDSO32_SYSENTER_RETURN; - -return 0 ; +return map_vdso (&vdso_image_32, false); }#endif
arch/x86/ia32/ia32_signal.c +2 -2
@@ -289,7 +289,7 @@ int ia32_setup_frame(int sig, struct ksignal *ksig,/* Return stub is in 32bit vsyscall page */ if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode; }
@@ -368,7 +368,7 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, restorer = ksig->ka.sa.sa_restorer; else restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;put_user_ex (ptr_to_compat (restorer), &frame->pretcode);/*
arch/x86/include/asm/elf.h +1 -1
@@ -328,7 +328,7 @@ else \#define VDSO_ENTRY \ ((unsigned long)current->mm->context.vdso + \ - selected_vdso32->sym___kernel_vsyscall) + vdso_image_32.sym___kernel_vsyscall)struct linux_binprm ;
arch/x86/include/asm/vdso.h +1 -8
@@ -26,7 +26,6 @@ struct vdso_image {long sym___kernel_sigreturn;long sym___kernel_rt_sigreturn;long sym___kernel_vsyscall; -long sym_VDSO32_SYSENTER_RETURN; };#ifdef CONFIG_X86_64
@@ -38,13 +37,7 @@ extern const struct vdso_image vdso_image_x32;#endif #if definedCONFIG_X86_32 || definedCONFIG_COMPAT -extern const struct vdso_image vdso_image_32_int80; -#ifdef CONFIG_COMPAT -extern const struct vdso_image vdso_image_32_syscall; -#endif -extern const struct vdso_image vdso_image_32_sysenter; - -extern const struct vdso_image *selected_vdso32; +extern const struct vdso_image vdso_image_32;#endif extern void __init init_vdso_image (const struct vdso_image *image);
arch/x86/kernel/signal.c +2 -2
@@ -299,7 +299,7 @@ __setup_frame(int sig, struct ksignal *ksig, sigset_t *set,if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode;if (ksig->ka.sa.sa_flags &SA_RESTORER )
@@ -363,7 +363,7 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig,/* Set up to return from userspace. */ restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;if (ksig->ka.sa.sa_flags &SA_RESTORER ) restorer = ksig->ka.sa.sa_restorer;put_user_ex (restorer, &frame->pretcode);
arch/x86/xen/setup.c +2 -11
@@ -965,17 +965,8 @@ char * __init xen_auto_xlated_memory_setup(void)static void __init fiddle_vdso (void ) {#ifdef CONFIG_X86_32 -/* -* This could be called before selected_vdso32 is initialized, so -* just fiddle with both possible images. vdso_image_32_syscall -* can't be selected, since it only exists on 64-bit systems. -*/ -u32 *mask; - mask = vdso_image_32_int80.data + - vdso_image_32_int80.sym_VDSO32_NOTE_MASK; - *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ; - mask = vdso_image_32_sysenter.data + - vdso_image_32_sysenter.sym_VDSO32_NOTE_MASK; +u32 *mask = vdso_image_32.data + + vdso_image_32.sym_VDSO32_NOTE_MASK; *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ;#endif }
Re: [PATCH v2 02/36] x86/uaccess: __chk_range_not_ok is unlikely to return true
In reply to Borislav PetkovQuoted 26 lines · Borislav Petkov
Quoted 25 lines · Andy Lutomirski
This should improve code quality a bit. It also shrinks the kernel text.
Before:
text data bss dec hex filename
21828379 5194760 1277952 28301091 1afd723 vmlinux
text data bss dec hex filename
21827997 5194760 1277952 28300709 1afd5a5 vmlinuxSigned-off-by Andy Lutomirski <[email protected]>--- arch/x86/include/asm/uaccess.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)arch/x86/include/asm/uaccess.h +1
-1@@ -51,13 +51,13 @@ static inline bool __chk_range_not_ok(unsigned long addr, unsigned long size, un* limit, not add it to the address). */ if (__builtin_constant_p (size)) -return addr > limit - size; +return unlikely (addr > limit - size);
Quoted 10 lines
arch/x86/include/asm/uaccess.h static inline bool __chk_range_not_ok(unsigned long addr, unsigned long size, un · line 60
Quoted 7 lines
arch/x86/include/asm/uaccess.h +2
-2/* Arbitrary sizes? Be careful about overflow */ addr += size; -if (addr < size) +if (unlikely (addr < size))return true; -return addr > limit; +return unlikely (addr > limit);It certainly uglifies it though. Are the wins worth the (un-)readability hit?
Well, adding likely/unlikely hints in headers is OK I think, especially since these are performance sensitive user-access routines.
Thanks,
Ingo
Re: [tip:x86/asm] x86/vdso: Remove runtime 32-bit vDSO selection
In reply to tip-bot for Andy LutomirskiQuoted 7 lines · tip-bot for Andy Lutomirski
Commit-ID: 0a6d1fa0d2b48fbae444e46e7f37a4832b2f8bdf Gitweb: http://git.kernel.org/tip/0a6d1fa0d2b48fbae444e46e7f37a4832b2f8bdf Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:56 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:08 +0200
Quoted 6 lines
x86/vdso: Remove runtime 32-bit vDSO selection
32-bit userspace will now always see the same vDSO, which is exactly what used to be the int80 vDSO. Subsequent patches will clean it up and make it support SYSENTER and SYSCALL using alternatives.
This doesn't make sense to apply without the rest of the series.
--
Brian Gerst
Re: [PATCH v2 07/36] x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
In reply to Andy LutomirskiQuoted 33 lines · Andy Lutomirski
We eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_64_compat.S | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)arch/x86/entry/entry_64_compat.S +1
-6@@ -58,14 +58,9 @@ ENDPROC(native_usergs_sysret32) * with the int 0x80 path. */ ENTRY(entry_SYSENTER_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %ebp, %ebp@@ -76,7 +71,16 @@ ENTRY(entry_SYSENTER_compat) /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ pushq %rbp /* pt_regs->sp */
arch/x86/entry/entry_64_compat.S ENTRY(entry_SYSENTER_compat) · line 82
Quoted 10 lines
arch/x86/entry/entry_64_compat.S +9
-1- pushfq /* pt_regs->flags */ + + /* + * Push flags. This is nasty. First, interrupts are currently + * off, but we need pt_regs->flags to have IF set. Second, even + * if TF was set when SYSENTER started, it's clear by now. We fix + * that later using TIF_SINGLESTEP. + */ + pushfq /* pt_regs->flags (except IF = 0) */ + orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */
The sequence of "push + insn_using_rsp" is a bit slow on most CPUs because stack engine (the machinery which makes consecutive pushes fast) needs syncronizing with register file.
It may be better to move the ORL insn here:
push, push, push
cld
sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */+ orl $X86_EFLAGS_IF, EFLAGS(%rsp) /* Fix saved flags to have .IF = 1 */
where we already eat that penalty.
Re: [PATCH v2 27/36] x86/entry/32: Re-implement SYSENTER using the new C path
In reply to Andy LutomirskiQuoted 75 lines · Andy Lutomirski
arch/x86/entry/entry_32.S +13
-55# SYSENTER call handler stub ENTRY(entry_SYSENTER_32) movl TSS_sysenter_sp0(%esp), %esp sysenter_past_esp: + pushl $__USER_DS /* pt_regs->ss */ + pushl %ecx /* pt_regs->cx */ + pushfl /* pt_regs->flags (except IF = 0) */ + orl $X86_EFLAGS_IF, (%esp) /* Fix IF */ + pushl $__USER_CS /* pt_regs->cs */ + pushl $0 /* pt_regs->ip = 0 (placeholder) */ + pushl %eax /* pt_regs->orig_ax */ + SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest */ + /* - * Interrupts are disabled here, but we can't trace it until - * enough kernel state to call TRACE_IRQS_OFF can be called - but - * we immediately enable interrupts at that point anyway. - */ - pushl $__USER_DS - pushl %ebp - pushfl - orl $X86_EFLAGS_IF, (%esp) - pushl $__USER_CS - /* - * Push current_thread_info()->sysenter_return to the stack. - * A tiny bit of offset fixup is necessary: TI_sysenter_return - * is relative to thread_info, which is at the bottom of the - * kernel stack page. 4*4 means the 4 words pushed above; - * TOP_OF_KERNEL_STACK_PADDING takes us to the top of the stack; - * and THREAD_SIZE takes us to the bottom. + * User mode is traced as though IRQs are on, and SYSENTER + * turned them off. */ - pushl ((TI_sysenter_return) - THREAD_SIZE + TOP_OF_KERNEL_STACK_PADDING + 4*4)(%esp) - - pushl %eax - SAVE_ALL - ENABLE_INTERRUPTS(CLBR_NONE) - -/* - * Load the potential sixth argument from user stack. - * Careful about security. - */ - cmpl $__PAGE_OFFSET-3, %ebp - jae syscall_fault - ASM_STAC -1: movl (%ebp), %ebp - ASM_CLAC - movl %ebp, PT_EBP(%esp) - _ASM_EXTABLE(1b, syscall_fault) - - GET_THREAD_INFO(%ebp) - - testl $_TIF_WORK_SYSCALL_ENTRY, TI_flags(%ebp) - jnz syscall_trace_entry -sysenter_do_call: - cmpl $(NR_syscalls), %eax - jae sysenter_badsys - call *sys_call_table(, %eax, 4) -sysenter_after_call: - movl %eax, PT_EAX(%esp) - LOCKDEP_SYS_EXIT - DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF - movl TI_flags(%ebp), %ecx - testl $_TIF_ALLWORK_MASK, %ecx - jnz syscall_exit_work_irqs_off -sysenter_exit: -/* if something modifies registers it must also disable sysexit */ - movl PT_EIP(%esp), %edx - movl PT_OLDESP(%esp), %ecx - xorl %ebp, %ebp - TRACE_IRQS_ON + + movl %esp, %eax
arch/x86/entry/entry_32.S need_resched: · line 319
Quoted 10 lines
arch/x86/entry/entry_32.S +10
-0+ call do_fast_syscall_32 + testl %eax, %eax + jz .Lsyscall_32_done + +/* Opportunistic SYSEXIT */ + TRACE_IRQS_ON /* User mode traces as IRQs on. */ + movl PT_EIP(%esp), %edx /* pt_regs->ip */ + movl PT_OLDESP(%esp), %ecx /* pt_regs->sp */ + popl %ebx /* pt_regs->bx */ + addl $2*4, %esp /* skip pt_regs->cx and pt_regs->dx */
Here stack engine and ALUs operating on ESP conflict, potentially adding a stall both before and after ADD.
It might be faster to just pop twice into an unused register, say,
popl %eax
popl %eax
Quoted 6 lines
arch/x86/entry/entry_32.S +4
-0+ popl %esi /* pt_regs->si */ + popl %edi /* pt_regs->di */ + popl %ebp /* pt_regs->bp */ + popl %eax /* pt_regs->ax */ 1: mov PT_FS(%esp), %fs PTGS_TO_GS
Re: [PATCH v2 07/36] x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
In reply to Denys VlasenkoQuoted 49 lines · Denys Vlasenko
Quoted 33 lines · Andy Lutomirski
We eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_64_compat.S | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)arch/x86/entry/entry_64_compat.S +1
-6@@ -58,14 +58,9 @@ ENDPROC(native_usergs_sysret32) * with the int 0x80 path. */ ENTRY(entry_SYSENTER_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %ebp, %ebp@@ -76,7 +71,16 @@ ENTRY(entry_SYSENTER_compat) /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ pushq %rbp /* pt_regs->sp */arch/x86/entry/entry_64_compat.S ENTRY(entry_SYSENTER_compat) · line 82
Quoted 10 lines
arch/x86/entry/entry_64_compat.S +9
-1- pushfq /* pt_regs->flags */ + + /* + * Push flags. This is nasty. First, interrupts are currently + * off, but we need pt_regs->flags to have IF set. Second, even + * if TF was set when SYSENTER started, it's clear by now. We fix + * that later using TIF_SINGLESTEP. + */ + pushfq /* pt_regs->flags (except IF = 0) */ + orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */The sequence of "push +
insn_using_rsp" is a bit slow on most CPUs because stack engine (the machinery which makes consecutive pushes fast) needs syncronizing with register file.
Quoted 10 lines
It may be better to move the ORL insn here:
push, push, push
cld
sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */
+ orl $X86_EFLAGS_IF, EFLAGS(%rsp) /* Fix saved flags to have .IF = 1 */where we already eat that penalty.
I'll benchmark this and, if it's a win, I'll tack it on to the end of the series.
Signature · 2 lines
Andy Lutomirski AMA Capital Management, LLC
Re: [PATCH v2 27/36] x86/entry/32: Re-implement SYSENTER using the new C path
In reply to Denys VlasenkoQuoted 12 lines · Denys Vlasenko
arch/x86/entry/entry_32.S need_resched: · line 319
Quoted 7 lines · Andy Lutomirski
arch/x86/entry/entry_32.S +7
-0+ +/* Opportunistic SYSEXIT */ + TRACE_IRQS_ON /* User mode traces as IRQs on. */ + movl PT_EIP(%esp), %edx /* pt_regs->ip */ + movl PT_OLDESP(%esp), %ecx /* pt_regs->sp */ + popl %ebx /* pt_regs->bx */ + addl $2*4, %esp /* skip pt_regs->cx and pt_regs->dx */Here stack engine and ALUs operating on ESP conflict, potentially adding a stall both before and after ADD.
Quoted 11 lines
It might be faster to just pop twice into an unused register, say,
popl %eax
popl %eaxQuoted 6 lines
arch/x86/entry/entry_32.S +4
-0+ popl %esi /* pt_regs->si */ + popl %edi /* pt_regs->di */ + popl %ebp /* pt_regs->bp */ + popl %eax /* pt_regs->ax */ 1: mov PT_FS(%esp), %fs PTGS_TO_GS
I'll benchmark it and possibly add it at the end of the series.
--Andy
Re: [tip:x86/asm] x86/entry, locking/lockdep: Move lockdep_sys_exit() to prepare_exit_to_usermode()
In reply to tip-bot for Andy LutomirskiQuoted 7 lines · tip-bot for Andy Lutomirski
Commit-ID: 72f924783b8a87e4454516520ffb5f35e4930371 Gitweb: http://git.kernel.org/tip/72f924783b8a87e4454516520ffb5f35e4930371 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:54 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:07 +0200
Quoted 8 lines
x86/entry, locking/lockdep: Move
lockdep_sys_exit()toprepare_exit_to_usermode()Rather than worrying about exactly where
LOCKDEP_SYS_EXITshould go in the asm code, add it toprepare_exit_from_usermode()and remove all of the asm calls that are followed byprepare_exit_to_usermode().
LOCKDEP_SYS_EXITnow appears only in the syscall fast paths.
Does that not pose a risk that something that always takes the slow path (signals? tracing?) will leak a lock to userspace?
Borislav Petkov
maintainerRe: [PATCH v2 09/36] x86/asm: Re-add manual CFI infrastructure
In reply to Andy LutomirskiQuoted 4 lines · Andy Lutomirski
OK, I'll strip out the push, movq, etc stuff for v3. I'll leave all of the
CFI_XYZmacros, even the unused one, because that's only a couple of lines of code and they all have very simple and obvious meanings (or as simple as the underlying directives, anyway).
If they're going to be used by the vDSO only, maybe they all should go into a header which is private to the vDSO only so that people don't get any ideas and the gunk starts spreading again...
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH v2 09/36] x86/asm: Re-add manual CFI infrastructure
Quoted 10 lines · Borislav Petkov
Quoted 4 lines · Andy Lutomirski
OK, I'll strip out the push, movq, etc stuff for v3. I'll leave all of the
CFI_XYZmacros, even the unused one, because that's only a couple of lines of code and they all have very simple and obvious meanings (or as simple as the underlying directives, anyway).If they're going to be used by the vDSO only, maybe they all should go into a header which is private to the vDSO only so that people don't get any ideas and the gunk starts spreading again...
Yeah. So for the vDSO code itself it's an obvious step forward: replacing black magic hexa encodings with symbolic directives.
Thanks,
Ingo
Borislav Petkov
maintainerRe: [PATCH v2 14/36] x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries
In reply to Andy LutomirskiQuoted 7 lines · Andy Lutomirski
We've disabled the vDSO helpers to call them, so turn off the entries entirely (temporarily) in preparation for cleaning them up.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_64_compat.S | 13 +++++++++++++ 1 file changed, 13 insertions(+)
What do we say for the case when someone bisects, lands at this patch and, as Murphy would generally do it, has a reproducer which is calling SYSENTER/SYSCALL32?
Tough luck?
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH v2 10/36] x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm
In reply to Andy LutomirskiQuoted 49 lines · Andy Lutomirski
For the vDSO, user code wants runtime unwind info. Make sure that, if we use .cfi directives, we generate it.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/vdso/Makefile | 4 ++-- arch/x86/include/asm/dwarf2.h | 13 ++++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)arch/x86/entry/vdso/Makefile +2
-2@@ -67,7 +67,7 @@ $(obj)/vdso-image-%.c: $(obj)/vdso%.so.dbg $(obj)/vdso%.so $(obj)/vdso2c FORCE CFL := $(PROFILING) -mcmodel=small -fPIC -O2 -fasynchronous-unwind-tables -m64 \ $(filter -g%,$(KBUILD_CFLAGS)) $(call cc-option, -fno-stack-protector) \ -fno-omit-frame-pointer -foptimize-sibling-calls \ - -DDISABLE_BRANCH_PROFILING + -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO $(vobjs): KBUILD_CFLAGS += $(CFL)@@ -131,7 +131,7 @@ targets += vdso32/vdso32.lds targets += vdso32/note.o vdso32/vclock_gettime.o vdso32/system_call.o targets += vdso32/vclock_gettime.o -KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) +KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) -DBUILD_VDSO $(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) $(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32arch/x86/include/asm/dwarf2.h +3
-3@@ -36,15 +36,22 @@#endif #if defined(CONFIG_AS_CFI_SECTIONS ) && defined(__ASSEMBLY__) +#ifndef BUILD_VDSO /* * Emit CFI data in .debug_frame sections, not .eh_frame sections. * The latter we currently just discard since we don't do DWARF * unwinding at runtime. So only the offline DWARF information is -* useful to anyone. Note we should not use this directive if this -* file is used in the vDSO assembly, or if vmlinux.lds.S gets -* changed so it doesn't discard .eh_frame. +* useful to anyone. Note we should not use this directive if +* vmlinux.lds.S gets changed so it doesn't discard .eh_frame.
arch/x86/include/asm/dwarf2.h line 55
Quoted 10 lines
arch/x86/include/asm/dwarf2.h +7
-0*/ .cfi_sections .debug_frame +#else +/* +* For the vDSO, emit both runtime unwind information and debug +* symbols for the .dbg file. +*/ + .cfi_sections .eh_frame, .debug_frame +#endif #endif
So it's exactly such hacks why I dislike the old CFI code:
1)
Emitting assembler directives from .h files is just disgusting. What's wrong with adding it to the .c file, so that it becomes obvious to everyone what's going on? Or if it's possible, do it in the build command line - the vDSO is built in a different way anyway.
2)
Could we require that to build the vDSO you need CFI capable tooling? All other tooling is probably on very ancient systems which can live with only having the vsyscall and the regular int80 entry methods, or something like that.
So could you please clean this all up instead of reintroducing old hacks? I'll apply the patches and see how well they work to move things forward, but this part is disgusting and needs to be resolved.
Thanks,
Ingo
Re: [PATCH v2 32/36] x86/entry: Micro-optimize compat fast syscall arg fetch
In reply to Andy LutomirskiQuoted 4 lines · Andy Lutomirski
we're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the
access_okcheck isn't needed.This saves about three cycles on a cache-hot fast syscall.
Another request: could you please stick the benchmarking code of the various x86 system call variants into 'perf bench' - under tools/perf/bench/, so that measurements can be done on more hardware and can be reproduced easily?
I'd suggest we dedicate an entirely new benchmark family to it: 'perf bench x86' and then have:
perf bench x86 syscall vdso
perf bench x86 syscall int80
perf bench x86 syscall vdso-compat
or so?
( I have some perf bench cleanups in -tip, so if you do this please base it on top
of that. )
Thanks,
Ingo
Quoted 151 lines · Andy Lutomirski
The first two patches are optimizations that I'm surprised we didn't already have. I noticed them when I was looking at the generated asm.
The next two patches are tests and some old stuff. There's a test that validates the vDSO
AT_SYSINFOannotations. There's also a test that exercises some assumptions that signal handling and ptracers make about syscalls that currently do *not* hold on 64-bit AMD using 32-bitAT_SYSINFO.The next three patches are NT cleanups and a lockdep cleanup.
It may pay to apply the beginning of the series (at most through "x86/entry/64/compat: After SYSENTER, move STI after the NT fixup") without waiting for everyone to wrap their heads around the rest.
The rest is basically a rewrite of syscalls for all cases except 64-bit native. With these patches applied, there is a single 32-bit vDSO and it uses SYSCALL, SYSENTER, and INT80 almost interchangeably via alternatives. The semantics of SYSENTER and SYSCALL are defined as:
1. If SYSCALL, ESP = ECX
2. ECX = *ESP
3. IP = INT80 landing pad
4. Opportunistic SYSRET/SYSEXIT is enabled on returnThe vDSO is rearranged so that these semantics work. Anything that backs IP up by 2 ends up pointing at a bona fide int $0x80 instruction with the expected regs.
In the process, the vDSO CFI annotations (which are actually used) get rewritten using normal CFI directives.
Opportunistic SYSRET/SYSEXIT only happens on return when CS and SS are as expected, IP points to the INT80 landing pad, and flags are in good shape. (There is no longer any assumption that full fast-path 32-bit syscalls don't muck with the registers that matter for fast exits -- I played with maintaining an optimization like that with poor results. I may try again if it saves a few cycles.)
Other than that, the system call entries are simplified to the bare minimum prologue and a call to a C function. Amusingly, SYSENTER and SYSCALL32 use the same C function.
To make that work, I had to remove all the 32-bit syscall stubs except the clone argument hack. This is because, for C code to call through the system call table, the system call table entries need to be real function pointers with C-compatible ABIs.
There is nothing at all anymore that requires that
x86_32syscalls be asmlinkage. That could be removed in a subsequent patch.The upshot appears to be a ~16 cycle performance hit on 32-bit fast path syscalls. (On my system, my little prctl test takes 172 cycles before and 188 cycles with these patches applied.)
The slow path is probably faster under most circumstances and, if the exit slow path gets hit, it'll be much faster because (as we already do in the 64-bit native case) we can still use SYSEXIT/SYSRET.
The patchset is structured as a removal of the old fast syscall code, then the change that makes syscalls into real functions, then a clean re-implementation of fast syscalls.
If we want some of the 25 cycles back, we could consider open-coding a new C fast path.
Changes from v1:
- Theunwind_vdso_32test now warns on broken Debian installations instead of failing. The problem is now fully understood, will be fixed by Debian and possibly also fixed by upstream glibc.
- execve was rather broken in v1.
- It's quite a bit faster now (the optimizations at the end are mostly new).
- int80 on 64-bit no longer clobbers extra regs (thanks Denys!).
- The uaccess stuff is new.
- Lots of other things that I forgot, I'm sure.Andy Lutomirski (36):
x86/uaccess: Tell the compiler that uaccess is unlikely to fault x86/uaccess:__chk_range_not_okis unlikely to return true selftests/x86: Add a test for vDSO unwinding selftests/x86: Add a test for syscall restart and arg modification x86/entry/64/compat: Fix SYSENTER's NT flag before user memory access x86/entry: Movelockdep_sys_exittoprepare_exit_to_usermodex86/entry/64/compat: After SYSENTER, move STI after the NT fixup x86/vdso: Remove runtime 32-bit vDSO selection
x86/asm: Re-add manual CFI infrastructure
x86/vdso: DefineBUILD_VDSOwhile building and emit .eh_frame in asm x86/vdso: Replace hex int80 CFI annotations with gas directives x86/elf/64: Clear more registers inelf_common_initx86/vdso/32: Save extra registers in the INT80 vsyscall path x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries x86/entry/64/compat: Remove audit optimizations x86/entry/64/compat: Remove most of the fast system call machinery x86/entry/64/compat: Set up fullpt_regsfor all compat syscalls x86/entry/syscalls: Move syscall table declarations into
asm/syscalls.h
x86/syscalls: Givesys_call_ptr_ta useful type
x86/entry: Adddo_syscall_32, a C function to do 32-bit syscalls x86/entry/64/compat: Migrate the body of the syscall entry to C x86/entry: Add C code for fast system call entries x86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls x86/entry/32: Open-code return tracking from fork and kthreads x86/entry/32: Switch INT80 to the new C syscall path x86/entry/32: Re-implement SYSENTER using the new C path x86/asm: Remove thread_info.sysenter_return
x86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls x86/entry: Makeirqs_disabledchecks in exit code depend on lockdep x86/entry: Force inlining of 32-bit syscall code x86/entry: Micro-optimize compat fast syscall arg fetch x86/entry: Hide two syscall entry assertions behindCONFIG_DEBUG_ENTRYx86/entry: Usept_regs_to_thread_info()in syscall entry tracing x86/entry: Split and inlineprepare_exit_to_usermodex86/entry: Split and inlinesyscall_return_slowpatharch/x86/Makefile | 10 +- arch/x86/entry/common.c | 255 ++++++++-- arch/x86/entry/entry_32.S | 184 +++---- arch/x86/entry/entry_64.S | 9 +- arch/x86/entry/entry_64_compat.S | 541 +++++---------------- arch/x86/entry/syscall_32.c | 9 +- arch/x86/entry/syscall_64.c | 4 +- arch/x86/entry/syscalls/syscall_32.tbl | 12 +- arch/x86/entry/vdso/Makefile | 39 +- arch/x86/entry/vdso/vdso2c.c | 2 +- arch/x86/entry/vdso/vdso32-setup.c | 28 +- arch/x86/entry/vdso/vdso32/int80.S | 56 --- arch/x86/entry/vdso/vdso32/syscall.S | 75 --- arch/x86/entry/vdso/vdso32/sysenter.S | 116 ----- arch/x86/entry/vdso/vdso32/system_call.S | 57 +++ arch/x86/entry/vdso/vma.c | 13 +- arch/x86/ia32/ia32_signal.c | 4 +- arch/x86/include/asm/dwarf2.h | 177 +++++++ arch/x86/include/asm/elf.h | 10 +- arch/x86/include/asm/syscall.h | 14 +- arch/x86/include/asm/thread_info.h | 1 - arch/x86/include/asm/uaccess.h | 14 +- arch/x86/include/asm/vdso.h | 10 +- arch/x86/kernel/asm-offsets.c | 3 - arch/x86/kernel/signal.c | 4 +- arch/x86/um/sys_call_table_32.c | 7 +- arch/x86/um/sys_call_table_64.c | 7 +- arch/x86/xen/setup.c | 13 +- tools/testing/selftests/x86/Makefile | 5 +- tools/testing/selftests/x86/ptrace_syscall.c | 294 +++++++++++ .../testing/selftests/x86/raw_syscall_helper_32.S | 46 ++ tools/testing/selftests/x86/unwind_vdso.c | 209 ++++++++ 32 files changed, 1258 insertions(+), 970 deletions(-)
Quoted 8 lines
delete mode 100644 arch/x86/entry/vdso/vdso32/int80.S delete mode 100644 arch/x86/entry/vdso/vdso32/syscall.S delete mode 100644 arch/x86/entry/vdso/vdso32/sysenter.S create mode 100644 arch/x86/entry/vdso/vdso32/system_call.S create mode 100644 arch/x86/include/asm/dwarf2.h create mode 100644 tools/testing/selftests/x86/ptrace_syscall.c create mode 100644 tools/testing/selftests/x86/raw_syscall_helper_32.S create mode 100644 tools/testing/selftests/x86/unwind_vdso.c
Ok, so I applied all of them to tip:x86/asm, in two phases, with small (stylistic) edits - it all seems to work fine for me so far, so I pushed it all out to -tip and linux-next.
Thanks,
Ingo
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/asm: Re-add parts of the manual CFI infrastructure
In reply to Andy Lutomirskix86/asm: Re-add parts of the manual CFI infrastructure
Commit-ID: 7b956f035a9ef8bd3ef5490f49fc1bd834d8a70a Gitweb: http://git.kernel.org/tip/7b956f035a9ef8bd3ef5490f49fc1bd834d8a70a Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:57 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:05 +0200
x86/asm: Re-add parts of the manual CFI infrastructure
Commit:
131484c8da97 ("x86/debug: Remove perpetually broken, unmaintainable dwarf annotations")
removed all the manual DWARF annotations outside the vDSO. It also removed the macros we used for the manual annotations.
Re-add these macros so that we can clean up the vDSO annotations.
--- arch/x86/Makefile | 10 ++- arch/x86/include/asm/dwarf2.h | 170 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 2 deletions(-)
arch/x86/Makefile +8 -2
@@ -159,6 +159,12 @@ endif sp-$(CONFIG_X86_32) := esp sp-$(CONFIG_X86_64) := rsp +# do binutils support CFI? +cfi := $(call as-instr,.cfi_startproc\n.cfi_rel_offset $(sp-y)$(comma)0\n.cfi_endproc,-DCONFIG_AS_CFI=1) +# is .cfi_signal_frame supported too? +cfi-sigframe := $(call as-instr,.cfi_startproc\n.cfi_signal_frame\n.cfi_endproc,-DCONFIG_AS_CFI_SIGNAL_FRAME=1) +cfi-sections := $(call as-instr,.cfi_sections .debug_frame,-DCONFIG_AS_CFI_SECTIONS=1) + # does binutils support specific instructions? asinstr := $(call as-instr,fxsaveq (%rax),-DCONFIG_AS_FXSAVEQ=1) asinstr += $(call as-instr,pshufb %xmm0$(comma)%xmm0,-DCONFIG_AS_SSSE3=1)
@@ -166,8 +172,8 @@ asinstr += $(call as-instr,crc32l %eax$(comma)%eax,-DCONFIG_AS_CRC32=1) avx_instr := $(call as-instr,vxorps %ymm0$(comma)%ymm1$(comma)%ymm2,-DCONFIG_AS_AVX=1) avx2_instr :=$(call as-instr,vpbroadcastb %xmm0$(comma)%ymm1,-DCONFIG_AS_AVX2=1) -KBUILD_AFLAGS += $(asinstr) $(avx_instr) $(avx2_instr) -KBUILD_CFLAGS += $(asinstr) $(avx_instr) $(avx2_instr) +KBUILD_AFLAGS += $(cfi) $(cfi-sigframe) $(cfi-sections) $(asinstr) $(avx_instr) $(avx2_instr) +KBUILD_CFLAGS += $(cfi) $(cfi-sigframe) $(cfi-sections) $(asinstr) $(avx_instr) $(avx2_instr) LDFLAGS := -m elf_$(UTS_MACHINE)
arch/x86/include/asm/dwarf2.h new file +170 -0
170 lines not on this page · raw message
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm
In reply to Andy Lutomirskix86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm
Commit-ID: f24f910884277aa6824bbc2dda4b5d0418d45c28 Gitweb: http://git.kernel.org/tip/f24f910884277aa6824bbc2dda4b5d0418d45c28 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:58 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:05 +0200
x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm
For the vDSO, user code wants runtime unwind info. Make sure that, if we use .cfi directives, we generate it.
--- arch/x86/entry/vdso/Makefile | 4 ++-- arch/x86/include/asm/dwarf2.h | 13 ++++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)
arch/x86/entry/vdso/Makefile +2 -2
@@ -67,7 +67,7 @@ $(obj)/vdso-image-%.c: $(obj)/vdso%.so.dbg $(obj)/vdso%.so $(obj)/vdso2c FORCE CFL := $(PROFILING) -mcmodel=small -fPIC -O2 -fasynchronous-unwind-tables -m64 \ $(filter -g%,$(KBUILD_CFLAGS)) $(call cc-option, -fno-stack-protector) \ -fno-omit-frame-pointer -foptimize-sibling-calls \ - -DDISABLE_BRANCH_PROFILING + -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO $(vobjs): KBUILD_CFLAGS += $(CFL)
@@ -131,7 +131,7 @@ targets += vdso32/vdso32.lds targets += vdso32/note.o vdso32/vclock_gettime.o vdso32/system_call.o targets += vdso32/vclock_gettime.o -KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) +KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) -DBUILD_VDSO $(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) $(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32
arch/x86/include/asm/dwarf2.h +10 -3
@@ -36,15 +36,22 @@#endif #if defined(CONFIG_AS_CFI_SECTIONS ) && defined(__ASSEMBLY__) +#ifndef BUILD_VDSO /* * Emit CFI data in .debug_frame sections, not .eh_frame sections. * The latter we currently just discard since we don't do DWARF * unwinding at runtime. So only the offline DWARF information is -* useful to anyone. Note we should not use this directive if this -* file is used in the vDSO assembly, or if vmlinux.lds.S gets -* changed so it doesn't discard .eh_frame. +* useful to anyone. Note we should not use this directive if +* vmlinux.lds.S gets changed so it doesn't discard .eh_frame. */ .cfi_sections .debug_frame +#else +/* +* For the vDSO, emit both runtime unwind information and debug +* symbols for the .dbg file. +*/ + .cfi_sections .eh_frame, .debug_frame +#endif #endif #else
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/vdso: Replace hex int80 CFI annotations with GAS directives
In reply to Andy Lutomirskix86/vdso: Replace hex int80 CFI annotations with GAS directives
Commit-ID: 29c0ce9508458ed31a9db2ed425f64c0d6d3ddfb Gitweb: http://git.kernel.org/tip/29c0ce9508458ed31a9db2ed425f64c0d6d3ddfb Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:59 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:06 +0200
x86/vdso: Replace hex int80 CFI annotations with GAS directives
Maintaining the current CFI annotations written in R'lyehian is difficult for most of us. Translate them to something a little closer to English.
This will remove the CFI data for kernels built with extremely old versions of binutils. I think this is a fair tradeoff for the ability for mortals to edit the asm.
--- arch/x86/entry/vdso/vdso32/system_call.S | 48 ++++++-------------------------- 1 file changed, 8 insertions(+), 40 deletions(-)
arch/x86/entry/vdso/vdso32/system_call.S +8 -40
@@ -1,6 +1,10 @@ /* * Code for the vDSO. This version uses the old int $0x80 method. - * +*/ + +#include <asm/dwarf2.h> + +/* * First get the common code for the sigreturn entry points. * This must come first. */
@@ -11,46 +15,10 @@ .type __kernel_vsyscall,@function ALIGN __kernel_vsyscall: -.LSTART_vsyscall: + CFI_STARTPROC int $0x80 ret -.LEND_vsyscall: - .size __kernel_vsyscall,.-.LSTART_vsyscall - .previous - - .section .eh_frame,"a",@progbits -.LSTARTFRAMEDLSI: - .long .LENDCIEDLSI-.LSTARTCIEDLSI -.LSTARTCIEDLSI: - .long 0 /* CIE ID */ - .byte 1 /* Version number */ - .string "zR" /* NUL-terminated augmentation string */ - .uleb128 1 /* Code alignment factor */ - .sleb128 -4 /* Data alignment factor */ - .byte 8 /* Return address register column */ - .uleb128 1 /* Augmentation value length */ - .byte 0x1b /* DW_EH_PE_pcrel|DW_EH_PE_sdata4. */ - .byte 0x0c /* DW_CFA_def_cfa */ - .uleb128 4 - .uleb128 4 - .byte 0x88 /* DW_CFA_offset, column 0x8 */ - .uleb128 1 - .align 4 -.LENDCIEDLSI: - .long .LENDFDEDLSI-.LSTARTFDEDLSI /* Length FDE */ -.LSTARTFDEDLSI: - .long .LSTARTFDEDLSI-.LSTARTFRAMEDLSI /* CIE pointer */ - .long .LSTART_vsyscall-. /* PC-relative start address */ - .long .LEND_vsyscall-.LSTART_vsyscall - .uleb128 0 - .align 4 -.LENDFDEDLSI: - .previous + CFI_ENDPROC - /* - * Pad out the segment to match the size of the sysenter.S version. - */ -VDSO32_vsyscall_eh_frame_size = 0x40 - .section .data,"aw",@progbits - .space VDSO32_vsyscall_eh_frame_size-(.LENDFDEDLSI-.LSTARTFRAMEDLSI), 0 + .size __kernel_vsyscall,.-__kernel_vsyscall .previous
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/elf/64: Clear more registers in elf_common_init ()
In reply to Andy Lutomirskix86/elf/64: Clear more registers in elf_common_init()
Commit-ID: 7bcdea4d050cbe4912854a68b93494203eec8b24 Gitweb: http://git.kernel.org/tip/7bcdea4d050cbe4912854a68b93494203eec8b24 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:00 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:06 +0200
x86/elf/64: Clear more registers in elf_common_init()
Before we start calling execve in contexts that honor the full pt_regs, we need to teach it to initialize all registers.
--- arch/x86/include/asm/elf.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/elf.h +4 -4
@@ -171,11 +171,11 @@ do { \static inline void elf_common_init (struct thread_struct *t,struct pt_regs *regs,const u16 ds) { -/* Commented-out registers are cleared in stub_execve */ -/*regs->ax = regs->bx =*/ regs->cx = regs->dx =0 ; - regs->si = regs->di/*= regs->bp*/ =0 ; +/* ax gets execve's return value. */ +/*regs->ax = */ regs->bx = regs->cx = regs->dx =0 ; + regs->si = regs->di = regs->bp =0 ; regs->r8 = regs->r9 = regs->r10 = regs->r11 =0 ; -/*regs->r12 = regs->r13 = regs->r14 = regs->r15 = 0;*/ + regs->r12 = regs->r13 = regs->r14 = regs->r15 =0 ; t->fs = t->gs =0 ; t->fsindex = t->gsindex =0 ; t->ds = t->es = ds;
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/vdso/32: Save extra registers in the INT80 vsyscall path
In reply to Andy Lutomirskix86/vdso/32: Save extra registers in the INT80 vsyscall path
Commit-ID: 8242c6c84a644e5f0f721e4ae2bd542f640c89f9 Gitweb: http://git.kernel.org/tip/8242c6c84a644e5f0f721e4ae2bd542f640c89f9 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:01 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:06 +0200
x86/vdso/32: Save extra registers in the INT80 vsyscall path
The goal is to integrate the SYSENTER and SYSCALL32 entry paths with the INT80 path. SYSENTER clobbers ESP and EIP. SYSCALL32 clobbers ECX (and, invisibly, R11). SYSRETL (long mode to compat mode) clobbers ECX and, invisibly, R11. SYSEXIT (which we only need for native 32-bit) clobbers ECX and EDX.
This means that we'll need to provide ESP to the kernel in a register (I chose ECX, since it's only needed for SYSENTER) and we need to provide the args that normally live in ECX and EDX in memory.
The epilogue needs to restore ECX and EDX, since user code relies on regs being preserved.
We don't need to do anything special about EIP, since the kernel already knows where we are. The kernel will eventually need to know where int $0x80 lands, so add a vdso_image entry for it.
The only user-visible effect of this code is that ptrace-induced changes to ECX and EDX during fast syscalls will be lost. This is already the case for the SYSENTER path.
--- arch/x86/entry/vdso/vdso2c.c | 1 + arch/x86/entry/vdso/vdso32/system_call.S | 25 ++++++++++++++++++++++++- arch/x86/include/asm/vdso.h | 1 + 3 files changed, 26 insertions(+), 1 deletion(-)
arch/x86/entry/vdso/vdso2c.c +1 -0
@@ -101,6 +101,7 @@ struct vdso_sym required_syms[] = { {"__kernel_vsyscall", true}, {"__kernel_sigreturn", true}, {"__kernel_rt_sigreturn", true}, + {"int80_landing_pad", true}, }; __attribute__((format(printf, 1, 2))) __attribute__((noreturn))
arch/x86/entry/vdso/vdso32/system_call.S +24 -1
@@ -16,7 +16,30 @@ ALIGN __kernel_vsyscall: CFI_STARTPROC - int $0x80 + /* + * Reshuffle regs so that all of any of the entry instructions + * will preserve enough state. + */ + pushl %edx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET edx, 0 + pushl %ecx + CFI_ADJUST_CFA_OFFSET 4 + CFI_REL_OFFSET ecx, 0 + movl %esp, %ecx + + /* Enter using int $0x80 */ + movl (%esp), %ecx + int $0x80 +GLOBAL(int80_landing_pad) + + /* Restore ECX and EDX in case they were clobbered. */ + popl %ecx + CFI_RESTORE ecx + CFI_ADJUST_CFA_OFFSET -4 + popl %edx + CFI_RESTORE edx + CFI_ADJUST_CFA_OFFSET -4 ret CFI_ENDPROC
arch/x86/include/asm/vdso.h +1 -0
@@ -26,6 +26,7 @@ struct vdso_image {long sym___kernel_sigreturn;long sym___kernel_rt_sigreturn;long sym___kernel_vsyscall; +long sym_int80_landing_pad; };#ifdef CONFIG_X86_64
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries
In reply to Andy Lutomirskix86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries
Commit-ID: e62a254a1f93fcc7299497a5c7231639400b8c3c Gitweb: http://git.kernel.org/tip/e62a254a1f93fcc7299497a5c7231639400b8c3c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:02 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:07 +0200
x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries
We've disabled the vDSO helpers to call them, so turn off the entries entirely (temporarily) in preparation for cleaning them up.
--- arch/x86/entry/entry_64_compat.S | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/x86/entry/entry_64_compat.S +13 -0
@@ -103,6 +103,14 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: + /* Temporary: SYSENTER is disabled. */ +#ifdef CONFIG_CONTEXT_TRACKING + call enter_from_user_mode +#endif + ENABLE_INTERRUPTS(CLBR_NONE) + movl $11, %edi + call do_exit + /* * Re-enable interrupts. IRQ tracing already thinks that IRQs are * on (since we treat user mode as having IRQs on), and the
@@ -324,6 +332,11 @@ ENTRY(entry_SYSCALL_compat) * it is too small to ever cause noticeable irq latency. */ SWAPGS_UNSAFE_STACK + + /* Temporary: SYSCALL32 is disabled. */ + movl $-ENOSYS, %eax + USERGS_SYSRET32 + movl %esp, %r8d movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp ENABLE_INTERRUPTS(CLBR_NONE)
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/64/compat: Remove audit optimizations
In reply to Andy Lutomirskix86/entry/64/compat: Remove audit optimizations
Commit-ID: c5f638ac90d514202155c87aa58730e86d484d9e Gitweb: http://git.kernel.org/tip/c5f638ac90d514202155c87aa58730e86d484d9e Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:03 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:07 +0200
x86/entry/64/compat: Remove audit optimizations
These audit optimizations are messy and hard to maintain. We'll get a similar effect from opportunistic sysret when fast compat system calls are re-implemented.
--- arch/x86/entry/entry_64_compat.S | 98 +--------------------------------------- 1 file changed, 2 insertions(+), 96 deletions(-)
arch/x86/entry/entry_64_compat.S +2 -96
@@ -16,16 +16,6 @@ #include <linux/linkage.h> #include <linux/err.h> -/* Avoid __ASSEMBLER__'ifying <linux/audit.h> just for this. */ -#include <linux/elf-em.h> -#define AUDIT_ARCH_I386 (EM_386|__AUDIT_ARCH_LE) -#define __AUDIT_ARCH_LE 0x40000000 - -#ifndef CONFIG_AUDITSYSCALL -# define sysexit_audit ia32_ret_from_sys_call_irqs_off -# define sysretl_audit ia32_ret_from_sys_call_irqs_off -#endif - .section .entry.text, "ax" #ifdef CONFIG_PARAVIRT
@@ -148,7 +138,7 @@ sysenter_dispatch: DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysexit_audit + jnz ia32_ret_from_sys_call_irqs_off sysexit_from_sys_call: /* * NB: SYSEXIT is not obviously safe for 64-bit kernels -- an
@@ -200,81 +190,12 @@ sysexit_from_sys_call: */ USERGS_SYSRET32 -#ifdef CONFIG_AUDITSYSCALL - .macro auditsys_entry_common - /* - * At this point, registers hold syscall args in the 32-bit syscall ABI: - * EAX is syscall number, the 6 args are in EBX,ECX,EDX,ESI,EDI,EBP. - * - * We want to pass them to __audit_syscall_entry(), which is a 64-bit - * C function with 5 parameters, so shuffle them to match what - * the function expects: RDI,RSI,RDX,RCX,R8. - */ - movl %esi, %r8d /* arg5 (R8 ) <= 4th syscall arg (ESI) */ - xchg %ecx, %edx /* arg4 (RCX) <= 3rd syscall arg (EDX) */ - /* arg3 (RDX) <= 2nd syscall arg (ECX) */ - movl %ebx, %esi /* arg2 (RSI) <= 1st syscall arg (EBX) */ - movl %eax, %edi /* arg1 (RDI) <= syscall number (EAX) */ - call __audit_syscall_entry - - /* - * We are going to jump back to the syscall dispatch code. - * Prepare syscall args as required by the 64-bit C ABI. - * Registers clobbered by __audit_syscall_entry() are - * loaded from pt_regs on stack: - */ - movl ORIG_RAX(%rsp), %eax /* syscall number */ - movl %ebx, %edi /* arg1 */ - movl RCX(%rsp), %esi /* arg2 */ - movl RDX(%rsp), %edx /* arg3 */ - movl RSI(%rsp), %ecx /* arg4 */ - movl RDI(%rsp), %r8d /* arg5 */ - .endm - - .macro auditsys_exit exit - TRACE_IRQS_ON - ENABLE_INTERRUPTS(CLBR_NONE) - testl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call - movl %eax, %esi /* second arg, syscall return value */ - cmpl $-MAX_ERRNO, %eax /* is it an error ? */ - jbe 1f - movslq %eax, %rsi /* if error sign extend to 64 bits */ -1: setbe %al /* 1 if error, 0 if not */ - movzbl %al, %edi /* zero-extend that into %edi */ - call __audit_syscall_exit - movl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT), %edi - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl %edi, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz \exit - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - jmp int_ret_from_sys_call_irqs_off - .endm - -sysenter_auditsys: - auditsys_entry_common - movl %ebp, %r9d /* reload 6th syscall arg */ - jmp sysenter_dispatch - -sysexit_audit: - auditsys_exit sysexit_from_sys_call -#endif - sysenter_fix_flags: pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed sysenter_tracesys: -#ifdef CONFIG_AUDITSYSCALL - testl $(_TIF_WORK_SYSCALL_ENTRY & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz sysenter_auditsys -#endif SAVE_EXTRA_REGS xorl %eax, %eax /* Do not leak kernel information */ movq %rax, R11(%rsp)
@@ -389,7 +310,7 @@ cstar_dispatch: DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysretl_audit + jnz ia32_ret_from_sys_call_irqs_off sysretl_from_sys_call: andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS)
@@ -420,22 +341,7 @@ sysretl_from_sys_call: */ USERGS_SYSRET32 -#ifdef CONFIG_AUDITSYSCALL -cstar_auditsys: - movl %r9d, R9(%rsp) /* register to be clobbered by call */ - auditsys_entry_common - movl R9(%rsp), %r9d /* reload 6th syscall arg */ - jmp cstar_dispatch - -sysretl_audit: - auditsys_exit sysretl_from_sys_call -#endif - cstar_tracesys: -#ifdef CONFIG_AUDITSYSCALL - testl $(_TIF_WORK_SYSCALL_ENTRY & ~_TIF_SYSCALL_AUDIT), ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jz cstar_auditsys -#endif xchgl %r9d, %ebp SAVE_EXTRA_REGS xorl %eax, %eax /* Do not leak kernel information */
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/64/compat: Remove most of the fast system call machinery
In reply to Andy Lutomirskix86/entry/64/compat: Remove most of the fast system call machinery
Commit-ID: 2ec67971faccc21ff18878552ccfe4409088c4c8 Gitweb: http://git.kernel.org/tip/2ec67971faccc21ff18878552ccfe4409088c4c8 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:04 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:07 +0200
x86/entry/64/compat: Remove most of the fast system call machinery
We now have only one code path that calls through the compat syscall table. This will make it much more pleasant to change the pt_regs vs register calling convention, which we need to do to move the call into C.
--- arch/x86/entry/entry_64_compat.S | 246 +-------------------------------------- 1 file changed, 4 insertions(+), 242 deletions(-)
arch/x86/entry/entry_64_compat.S +4 -242
@@ -101,119 +101,13 @@ sysenter_flags_fixed: movl $11, %edi call do_exit - /* - * Re-enable interrupts. IRQ tracing already thinks that IRQs are - * on (since we treat user mode as having IRQs on), and the - * prologue above is too short for it to be worth adding a - * tracing round trip. - */ - ENABLE_INTERRUPTS(CLBR_NONE) - - /* - * No need to do an access_ok() check here because RBP has been - * 32-bit zero extended: - */ - ASM_STAC -1: movl (%rbp), %ebp - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - - orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz sysenter_tracesys - -sysenter_do_call: - /* 32-bit syscall -> 64-bit C ABI argument conversion */ - movl %edi, %r8d /* arg5 */ - movl %ebp, %r9d /* arg6 */ - xchg %ecx, %esi /* rsi:arg2, rcx:arg4 */ - movl %ebx, %edi /* arg1 */ - movl %edx, %edx /* arg3 (zero extension) */ -sysenter_dispatch: - cmpq $(IA32_NR_syscalls-1), %rax - ja 1f - call *ia32_sys_call_table(, %rax, 8) - movq %rax, RAX(%rsp) -1: - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call_irqs_off -sysexit_from_sys_call: - /* - * NB: SYSEXIT is not obviously safe for 64-bit kernels -- an - * NMI between STI and SYSEXIT has poorly specified behavior, - * and and NMI followed by an IRQ with usergs is fatal. So - * we just pretend we're using SYSEXIT but we really use - * SYSRETL instead. - * - * This code path is still called 'sysexit' because it pairs - * with 'sysenter' and it uses the SYSENTER calling convention. - */ - andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - movl RIP(%rsp), %ecx /* User %eip */ - movq RAX(%rsp), %rax - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - xorl %edx, %edx /* Do not leak kernel information */ - xorq %r8, %r8 - xorq %r9, %r9 - xorq %r10, %r10 - movl EFLAGS(%rsp), %r11d /* User eflags */ - TRACE_IRQS_ON - - /* - * SYSRETL works even on Intel CPUs. Use it in preference to SYSEXIT, - * since it avoids a dicey window with interrupts enabled. - */ - movl RSP(%rsp), %esp - - /* - * USERGS_SYSRET32 does: - * gsbase = user's gs base - * eip = ecx - * rflags = r11 - * cs = __USER32_CS - * ss = __USER_DS - * - * The prologue set RIP(%rsp) to VDSO32_SYSENTER_RETURN, which does: - * - * pop %ebp - * pop %edx - * pop %ecx - * - * Therefore, we invoke SYSRETL with EDX and R8-R10 zeroed to - * avoid info leaks. R11 ends up with VDSO32_SYSENTER_RETURN's - * address (already known to user code), and R12-R15 are - * callee-saved and therefore don't contain any interesting - * kernel data. - */ - USERGS_SYSRET32 + /* Unreachable. */ + ud2 sysenter_fix_flags: pushq $X86_EFLAGS_FIXED popfq jmp sysenter_flags_fixed - -sysenter_tracesys: - SAVE_EXTRA_REGS - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - movq %rsp, %rdi /* &pt_regs -> arg1 */ - call syscall_trace_enter - - /* Reload arg registers from stack. (see sysenter_tracesys) */ - movl RCX(%rsp), %ecx - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl %eax, %eax /* zero extension */ - - RESTORE_EXTRA_REGS - jmp sysenter_do_call ENDPROC(entry_SYSENTER_compat) /*
@@ -280,142 +174,10 @@ ENTRY(entry_SYSCALL_compat) pushq $-ENOSYS /* pt_regs->ax */ sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ - /* - * No need to do an access_ok check here because r8 has been - * 32-bit zero extended: - */ - ASM_STAC -1: movl (%r8), %r9d - _ASM_EXTABLE(1b, ia32_badarg) - ASM_CLAC - orl $TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - testl $_TIF_WORK_SYSCALL_ENTRY, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz cstar_tracesys - -cstar_do_call: - /* 32-bit syscall -> 64-bit C ABI argument conversion */ - movl %edi, %r8d /* arg5 */ - /* r9 already loaded */ /* arg6 */ - xchg %ecx, %esi /* rsi:arg2, rcx:arg4 */ - movl %ebx, %edi /* arg1 */ - movl %edx, %edx /* arg3 (zero extension) */ - -cstar_dispatch: - cmpq $(IA32_NR_syscalls-1), %rax - ja 1f - - call *ia32_sys_call_table(, %rax, 8) - movq %rax, RAX(%rsp) -1: - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - testl $_TIF_ALLWORK_MASK, ASM_THREAD_INFO(TI_flags, %rsp, SIZEOF_PTREGS) - jnz ia32_ret_from_sys_call_irqs_off - -sysretl_from_sys_call: - andl $~TS_COMPAT, ASM_THREAD_INFO(TI_status, %rsp, SIZEOF_PTREGS) - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl RIP(%rsp), %ecx - movl EFLAGS(%rsp), %r11d - movq RAX(%rsp), %rax - xorq %r10, %r10 - xorq %r9, %r9 - xorq %r8, %r8 - TRACE_IRQS_ON - movl RSP(%rsp), %esp - /* - * 64-bit->32-bit SYSRET restores eip from ecx, - * eflags from r11 (but RF and VM bits are forced to 0), - * cs and ss are loaded from MSRs. - * (Note: 32-bit->32-bit SYSRET is different: since r11 - * does not exist, it merely sets eflags.IF=1). - * - * NB: On AMD CPUs with the X86_BUG_SYSRET_SS_ATTRS bug, the ss - * descriptor is not reinitialized. This means that we must - * avoid SYSRET with SS == NULL, which could happen if we schedule, - * exit the kernel, and re-enter using an interrupt vector. (All - * interrupt entries on x86_64 set SS to NULL.) We prevent that - * from happening by reloading SS in __switch_to. - */ - USERGS_SYSRET32 - -cstar_tracesys: - xchgl %r9d, %ebp - SAVE_EXTRA_REGS - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %r9, R9(%rsp) - movq %rax, R8(%rsp) - movq %rsp, %rdi /* &pt_regs -> arg1 */ - call syscall_trace_enter - movl R9(%rsp), %r9d - - /* Reload arg registers from stack. (see sysenter_tracesys) */ - movl RCX(%rsp), %ecx - movl RDX(%rsp), %edx - movl RSI(%rsp), %esi - movl RDI(%rsp), %edi - movl %eax, %eax /* zero extension */ - - RESTORE_EXTRA_REGS - xchgl %ebp, %r9d - jmp cstar_do_call + /* Unreachable. */ + ud2 END(entry_SYSCALL_compat) -ia32_badarg: - /* - * So far, we've entered kernel mode, set AC, turned on IRQs, and - * saved C regs except r8-r11. We haven't done any of the other - * standard entry work, though. We want to bail, but we shouldn't - * treat this as a syscall entry since we don't even know what the - * args are. Instead, treat this as a non-syscall entry, finish - * the entry work, and immediately exit after setting AX = -EFAULT. - * - * We're really just being polite here. Killing the task outright - * would be a reasonable action, too. Given that the only valid - * way to have gotten here is through the vDSO, and we already know - * that the stack pointer is bad, the task isn't going to survive - * for long no matter what we do. - */ - - ASM_CLAC /* undo STAC */ - movq $-EFAULT, RAX(%rsp) /* return -EFAULT if possible */ - - /* Fill in the rest of pt_regs */ - xorl %eax, %eax - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - SAVE_EXTRA_REGS - - /* Turn IRQs back off. */ - DISABLE_INTERRUPTS(CLBR_NONE) - TRACE_IRQS_OFF - - /* Now finish entering normal kernel mode. */ -#ifdef CONFIG_CONTEXT_TRACKING - call enter_from_user_mode -#endif - - /* And exit again. */ - jmp retint_user - -ia32_ret_from_sys_call_irqs_off: - TRACE_IRQS_ON - ENABLE_INTERRUPTS(CLBR_NONE) - -ia32_ret_from_sys_call: - xorl %eax, %eax /* Do not leak kernel information */ - movq %rax, R11(%rsp) - movq %rax, R10(%rsp) - movq %rax, R9(%rsp) - movq %rax, R8(%rsp) - jmp int_ret_from_sys_call - /* * Emulated IA32 system calls via int 0x80. *
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/64/compat: Set up full pt_regs for all compat syscalls
In reply to Andy Lutomirskix86/entry/64/compat: Set up full pt_regs for all compat syscalls
Commit-ID: 8169aff611956ed360e3313e8c718f530f58f6cb Gitweb: http://git.kernel.org/tip/8169aff611956ed360e3313e8c718f530f58f6cb Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:05 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:07 +0200
x86/entry/64/compat: Set up full pt_regs for all compat syscalls
This is conceptually simpler. More importantly, it eliminates the PTREGSCALL and execve stubs, which were not compatible with the C ABI. This means that C code can call through the compat syscall table.
The execve stubs are a bit subtle. They did two things: they cleared some registers and they forced slow-path return. Neither is necessary any more: elf_common_init clears the extra registers and start_thread calls force_iret().
--- arch/x86/entry/entry_64.S | 6 +---- arch/x86/entry/entry_64_compat.S | 42 +++++++++++----------------------- arch/x86/entry/syscalls/syscall_32.tbl | 12 +++++----- 3 files changed, 20 insertions(+), 40 deletions(-)
arch/x86/entry/entry_64.S +1 -5
@@ -391,20 +391,16 @@ GLOBAL(stub_execveat) jmp return_from_execve END(stub_execveat) -#if defined(CONFIG_X86_X32_ABI) || defined(CONFIG_IA32_EMULATION) +#if defined(CONFIG_X86_X32_ABI) .align 8 GLOBAL(stub_x32_execve) -GLOBAL(stub32_execve) call compat_sys_execve jmp return_from_execve -END(stub32_execve) END(stub_x32_execve) .align 8 GLOBAL(stub_x32_execveat) -GLOBAL(stub32_execveat) call compat_sys_execveat jmp return_from_execve -END(stub32_execveat) END(stub_x32_execveat) #endif
arch/x86/entry/entry_64_compat.S +13 -29
72 lines not on this page · raw message
arch/x86/entry/syscalls/syscall_32.tbl +6 -6
@@ -8,7 +8,7 @@ # 0 i386 restart_syscall sys_restart_syscall 1 i386 exit sys_exit -2 i386 fork sys_fork stub32_fork +2 i386 fork sys_fork sys_fork 3 i386 read sys_read 4 i386 write sys_write 5 i386 open sys_open compat_sys_open
@@ -17,7 +17,7 @@ 8 i386 creat sys_creat 9 i386 link sys_link 10 i386 unlink sys_unlink -11 i386 execve sys_execve stub32_execve +11 i386 execve sys_execve compat_sys_execve 12 i386 chdir sys_chdir 13 i386 time sys_time compat_sys_time 14 i386 mknod sys_mknod
@@ -125,7 +125,7 @@ 116 i386 sysinfo sys_sysinfo compat_sys_sysinfo 117 i386 ipc sys_ipc compat_sys_ipc 118 i386 fsync sys_fsync -119 i386 sigreturn sys_sigreturn stub32_sigreturn +119 i386 sigreturn sys_sigreturn sys32_sigreturn 120 i386 clone sys_clone stub32_clone 121 i386 setdomainname sys_setdomainname 122 i386 uname sys_newuname
@@ -179,7 +179,7 @@ 170 i386 setresgid sys_setresgid16 171 i386 getresgid sys_getresgid16 172 i386 prctl sys_prctl -173 i386 rt_sigreturn sys_rt_sigreturn stub32_rt_sigreturn +173 i386 rt_sigreturn sys_rt_sigreturn sys32_rt_sigreturn 174 i386 rt_sigaction sys_rt_sigaction compat_sys_rt_sigaction 175 i386 rt_sigprocmask sys_rt_sigprocmask 176 i386 rt_sigpending sys_rt_sigpending compat_sys_rt_sigpending
@@ -196,7 +196,7 @@ 187 i386 sendfile sys_sendfile compat_sys_sendfile 188 i386 getpmsg 189 i386 putpmsg -190 i386 vfork sys_vfork stub32_vfork +190 i386 vfork sys_vfork sys_vfork 191 i386 ugetrlimit sys_getrlimit compat_sys_getrlimit 192 i386 mmap2 sys_mmap_pgoff 193 i386 truncate64 sys_truncate64 sys32_truncate64
@@ -364,7 +364,7 @@ 355 i386 getrandom sys_getrandom 356 i386 memfd_create sys_memfd_create 357 i386 bpf sys_bpf -358 i386 execveat sys_execveat stub32_execveat +358 i386 execveat sys_execveat compat_sys_execveat 359 i386 socket sys_socket 360 i386 socketpair sys_socketpair 361 i386 bind sys_bind
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/syscalls: Move syscall table declarations into asm/syscalls.h
In reply to Andy Lutomirskix86/entry/syscalls: Move syscall table declarations into asm/syscalls.h
Commit-ID: 034042cc1e2837a584cda0a5e4fc2b0a96b74543 Gitweb: http://git.kernel.org/tip/034042cc1e2837a584cda0a5e4fc2b0a96b74543 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:06 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:08 +0200
x86/entry/syscalls: Move syscall table declarations into asm/syscalls.h
The header was missing some compat declarations.
Also make sys_call_ptr_t have a consistent type.
--- arch/x86/entry/syscall_32.c | 5 +---- arch/x86/include/asm/syscall.h | 12 +++++++++++- arch/x86/um/sys_call_table_32.c | 3 +-- arch/x86/um/sys_call_table_64.c | 3 +-- 4 files changed, 14 insertions(+), 9 deletions(-)
arch/x86/entry/syscall_32.c +1 -4
@@ -4,13 +4,12 @@#include <linux/sys.h> #include <linux/cache.h> #include <asm/asm-offsets.h> +#include <asm/syscall.h> #ifdef CONFIG_IA32_EMULATION #define SYM (sym, compat) compat#else #define SYM (sym, compat) sym -#define ia32_sys_call_table sys_call_table -#define __NR_syscall_compat_max __NR_syscall_max#endif #define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void SYM(sym, compat)(void) ;
@@ -19,8 +18,6 @@#define __SYSCALL_I386 (nr, sym, compat) [nr] = SYM(sym, compat), -typedef asmlinkage void (*sys_call_ptr_t)(void); - extern asmlinkage void sys_ni_syscall(void); __visible const sys_call_ptr_t ia32_sys_call_table[__NR_syscall_compat_max+1] = {
arch/x86/include/asm/syscall.h +11 -1
@@ -20,9 +20,19 @@#include <asm/thread_info.h> /* for TS_COMPAT */ #include <asm/unistd.h> -typedef void (*sys_call_ptr_t )(void ); +typedef asmlinkage void (*sys_call_ptr_t)(void);extern const sys_call_ptr_t sys_call_table[]; +#if defined(CONFIG_X86_32 ) +#define ia32_sys_call_table sys_call_table +#define __NR_syscall_compat_max __NR_syscall_max +#define IA32_NR_syscalls NR_syscalls +#endif + +#if defined(CONFIG_IA32_EMULATION ) +extern const sys_call_ptr_t ia32_sys_call_table[]; +#endif +/* * Only the low 32 bits of orig_ax are meaningful, so we return int. * This importantly ignores the high bits on 64-bit, so comparisons
arch/x86/um/sys_call_table_32.c +1 -2
@@ -7,6 +7,7 @@#include <linux/sys.h> #include <linux/cache.h> #include <generated/user_constants.h> +#include <asm/syscall.h> #define __NO_STUBS
@@ -30,8 +31,6 @@#undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [ nr ] = sym, -typedef asmlinkage void (*sys_call_ptr_t)(void); - extern asmlinkage void sys_ni_syscall(void);const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {
arch/x86/um/sys_call_table_64.c +1 -2
@@ -7,6 +7,7 @@#include <linux/sys.h> #include <linux/cache.h> #include <generated/user_constants.h> +#include <asm/syscall.h> #define __NO_STUBS
@@ -43,8 +44,6 @@#undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [ nr ] = sym, -typedef void (*sys_call_ptr_t )(void ); -extern void sys_ni_syscall (void );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/syscalls: Give sys_call_ptr_t a useful type
In reply to Andy Lutomirskix86/syscalls: Give sys_call_ptr_t a useful type
Commit-ID: eb974c62565072e10c1422eb3205f5b611dd99a1 Gitweb: http://git.kernel.org/tip/eb974c62565072e10c1422eb3205f5b611dd99a1 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:07 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:08 +0200
x86/syscalls: Give sys_call_ptr_t a useful type
Syscalls are asmlinkage functions (on 32-bit kernels), take six args of type unsigned long, and return long. Note that uml could probably be slightly cleaned up on top of this patch.
--- arch/x86/entry/syscall_32.c | 4 ++-- arch/x86/entry/syscall_64.c | 4 ++-- arch/x86/include/asm/syscall.h | 4 +++- arch/x86/um/sys_call_table_32.c | 4 ++-- arch/x86/um/sys_call_table_64.c | 4 ++-- 5 files changed, 11 insertions(+), 9 deletions(-)
arch/x86/entry/syscall_32.c +2 -2
@@ -12,13 +12,13 @@#define SYM (sym, compat) sym#endif -#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void SYM(sym, compat)(void) ; +#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage long SYM(sym, compat)(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_32.h> #undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [nr] = SYM(sym, compat), -extern asmlinkage void sys_ni_syscall(void); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ); __visible const sys_call_ptr_t ia32_sys_call_table[__NR_syscall_compat_max+1] = {/*
arch/x86/entry/syscall_64.c +2 -2
@@ -14,13 +14,13 @@# define __SYSCALL_X32 (nr, sym, compat)/* nothing */ #endif -#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_64.h> #undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [nr] = sym, -extern void sys_ni_syscall (void ); +extern long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ); asmlinkage const sys_call_ptr_t sys_call_table[__NR_syscall_max+1] = {/*
arch/x86/include/asm/syscall.h +3 -1
@@ -20,7 +20,9 @@#include <asm/thread_info.h> /* for TS_COMPAT */ #include <asm/unistd.h> -typedef asmlinkage void (*sys_call_ptr_t)(void); +typedef asmlinkage long (*sys_call_ptr_t )(unsigned long ,unsigned long , +unsigned long ,unsigned long , +unsigned long ,unsigned long );extern const sys_call_ptr_t sys_call_table[];#if defined(CONFIG_X86_32 )
arch/x86/um/sys_call_table_32.c +2 -2
@@ -25,13 +25,13 @@#define old_mmap sys_old_mmap -#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_I386 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_32.h> #undef __SYSCALL_I386#define __SYSCALL_I386 (nr, sym, compat) [ nr ] = sym, -extern asmlinkage void sys_ni_syscall(void); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {/*
arch/x86/um/sys_call_table_64.c +2 -2
@@ -38,13 +38,13 @@#define __SYSCALL_COMMON (nr, sym, compat) __SYSCALL_64(nr, sym, compat)#define __SYSCALL_X32 (nr, sym, compat)/* Not supported */ -#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage void sym(void) ; +#define __SYSCALL_64 (nr, sym, compat) extern asmlinkage long sym(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long) ;#include <asm/syscalls_64.h> #undef __SYSCALL_64#define __SYSCALL_64 (nr, sym, compat) [ nr ] = sym, -extern void sys_ni_syscall (void ); +extern asmlinkage long sys_ni_syscall (unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long ,unsigned long );const sys_call_ptr_t sys_call_table[]____cacheline_aligned = {/*
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Add do_syscall_32(), a C function to do 32-bit syscalls
In reply to Andy Lutomirskix86/entry: Add do_syscall_32(), a C function to do 32-bit syscalls
Commit-ID: bd2d3a3ba67ac580f6e809aac36bf942f5447f91 Gitweb: http://git.kernel.org/tip/bd2d3a3ba67ac580f6e809aac36bf942f5447f91 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:08 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:08 +0200
x86/entry: Add do_syscall_32(), a C function to do 32-bit syscalls
System calls are really quite simple. Add a helper to call a 32-bit system call.
--- arch/x86/entry/common.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/entry/common.c +43 -0
@@ -318,3 +318,46 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)local_irq_disable ();prepare_exit_to_usermode (regs); } + +#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION ) +/* +* Does a 32-bit syscall. Called with IRQs off and does all entry and +* exit work. +*/ +__visible void do_int80_syscall_32(struct pt_regs *regs) +{ +struct thread_info *ti =pt_regs_to_thread_info (regs); +unsigned int nr = (unsigned int )regs->orig_ax; + +#ifdef CONFIG_IA32_EMULATION + ti->status |=TS_COMPAT ; +#endif + +local_irq_enable (); + +if (READ_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY) { +/* +* Subtlety here: if ptrace pokes something larger than +* 2^32-1 into orig_ax, this truncates it. This may or +* may not be necessary, but it matches the old asm +* behavior. +*/ + nr =syscall_trace_enter (regs); + } + +if (nr < IA32_NR_syscalls) { +/* +* It's possible that a 32-bit syscall implementation +* takes a 64-bit parameter but nonetheless assumes that +* the high bits are zero. Make sure we zero-extend all +* of the args. +*/ + regs->ax = ia32_sys_call_table[nr]( + (unsigned int )regs->bx, (unsigned int )regs->cx, + (unsigned int )regs->dx, (unsigned int )regs->si, + (unsigned int )regs->di, (unsigned int )regs->bp); + } + +syscall_return_slowpath (regs); +} +#endif
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/64/compat: Migrate the body of the syscall entry to C
In reply to Andy Lutomirskix86/entry/64/compat: Migrate the body of the syscall entry to C
Commit-ID: ee08c6bd315e70756ad2c47ee6ea708a4a882b55 Gitweb: http://git.kernel.org/tip/ee08c6bd315e70756ad2c47ee6ea708a4a882b55 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:09 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:09 +0200
x86/entry/64/compat: Migrate the body of the syscall entry to C
--- arch/x86/entry/entry_64.S | 2 +- arch/x86/entry/entry_64_compat.S | 56 +++++++++++++--------------------------- 2 files changed, 19 insertions(+), 39 deletions(-)
arch/x86/entry/entry_64.S +1 -1
@@ -582,7 +582,7 @@ retint_kernel: * At this label, code paths which return to kernel and to user, * which come from interrupts/exception and from syscalls, merge. */ -restore_regs_and_iret: +GLOBAL(restore_regs_and_iret) RESTORE_EXTRA_REGS restore_c_regs_and_iret: RESTORE_C_REGS
arch/x86/entry/entry_64_compat.S +18 -38
74 lines not on this page · raw message
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Add C code for fast system call entries
In reply to Andy Lutomirskix86/entry: Add C code for fast system call entries
Commit-ID: 710246df58041106b7de645f4b45770f8a59a269 Gitweb: http://git.kernel.org/tip/710246df58041106b7de645f4b45770f8a59a269 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:10 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:09 +0200
x86/entry: Add C code for fast system call entries
This handles both SYSENTER and SYSCALL. The asm glue will take care of the differences.
--- arch/x86/entry/common.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
arch/x86/entry/common.c +43 -0
@@ -24,6 +24,8 @@#include <asm/desc.h> #include <asm/traps.h> +#include <asm/vdso.h> +#include <asm/uaccess.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -360,4 +362,45 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } + +__visible void do_fast_syscall_32(struct pt_regs *regs) +{ +/* +* Called using the internal vDSO SYSENTER/SYSCALL32 calling +* convention. Adjust regs so it looks like we entered using int80. +*/ + +unsigned long landing_pad = (unsigned long )current->mm->context.vdso + + vdso_image_32.sym_int80_landing_pad; + +/* +* SYSENTER loses EIP, and even SYSCALL32 needs us to skip forward +* so that 'regs->ip -= 2' lands back on an int $0x80 instruction. +* Fix it up. +*/ + regs->ip = landing_pad; + +/* +* Fetch ECX from where the vDSO stashed it. +* +* WARNING: We are in CONTEXT_USER and RCU isn't paying attention! +*/ +local_irq_enable (); +if (get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long )(u32 )regs->sp)) { +/* User code screwed up. */ +local_irq_disable (); + regs->ax = -EFAULT ; +#ifdef CONFIG_CONTEXT_TRACKING +enter_from_user_mode (); +#endif +prepare_exit_to_usermode (regs); +return ; + } +local_irq_disable (); + +/* Now this is just like a normal syscall. */ +do_int80_syscall_32 (regs); +return ; +}#endif
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace
In reply to Andy Lutomirskix86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace
Commit-ID: a474e67c913d3ebaf02ba9d7835d5299d226c3ed Gitweb: http://git.kernel.org/tip/a474e67c913d3ebaf02ba9d7835d5299d226c3ed Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:11 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:09 +0200
x86/vdso/compat: Wire up SYSENTER and SYSCSALL for compat userspace
What, you didn't realize that SYSENTER and SYSCALL were actually the same thing? :)
Unlike the old code, this actually passes the ptrace_syscall_32 test on AMD systems.
--- arch/x86/entry/entry_64_compat.S | 90 +++++++++++++++++++------------- arch/x86/entry/vdso/vdso32/system_call.S | 8 +++ 2 files changed, 62 insertions(+), 36 deletions(-)
arch/x86/entry/entry_64_compat.S +54 -36
@@ -52,15 +52,18 @@ ENTRY(entry_SYSENTER_compat) SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - /* Zero-extending 32-bit regs, do not remove */ - movl %ebp, %ebp + /* + * User tracing code (ptrace or signal handlers) might assume that + * the saved RAX contains a 32-bit number when we're invoking a 32-bit + * syscall. Just in case the high bits are nonzero, zero-extend + * the syscall number. (This could almost certainly be deleted + * with no ill effects.) + */ movl %eax, %eax - movl ASM_THREAD_INFO(TI_sysenter_return, %rsp, 0), %r10d - /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ - pushq %rbp /* pt_regs->sp */ + pushq %rcx /* pt_regs->sp */ /* * Push flags. This is nasty. First, interrupts are currently
@@ -70,17 +73,28 @@ ENTRY(entry_SYSENTER_compat) */ pushfq /* pt_regs->flags (except IF = 0) */ orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */ + ASM_CLAC /* Clear AC after saving FLAGS */ pushq $__USER32_CS /* pt_regs->cs */ - pushq %r10 /* pt_regs->ip = thread_info->sysenter_return */ + xorq %r8,%r8 + pushq %r8 /* pt_regs->ip = 0 (placeholder) */ pushq %rax /* pt_regs->orig_ax */ pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */ pushq %rdx /* pt_regs->dx */ - pushq %rcx /* pt_regs->cx */ + pushq %rcx /* pt_regs->cx (will be overwritten) */ pushq $-ENOSYS /* pt_regs->ax */ + pushq %r8 /* pt_regs->r8 = 0 */ + pushq %r8 /* pt_regs->r9 = 0 */ + pushq %r8 /* pt_regs->r10 = 0 */ + pushq %r8 /* pt_regs->r11 = 0 */ + pushq %rbx /* pt_regs->rbx */ + pushq %rbp /* pt_regs->rbp */ + pushq %r8 /* pt_regs->r12 = 0 */ + pushq %r8 /* pt_regs->r13 = 0 */ + pushq %r8 /* pt_regs->r14 = 0 */ + pushq %r8 /* pt_regs->r15 = 0 */ cld - sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ /* * Sysenter doesn't filter flags, so we need to clear NT
@@ -93,16 +107,15 @@ ENTRY(entry_SYSENTER_compat) jnz sysenter_fix_flags sysenter_flags_fixed: - /* Temporary: SYSENTER is disabled. */ -#ifdef CONFIG_CONTEXT_TRACKING - call enter_from_user_mode -#endif - ENABLE_INTERRUPTS(CLBR_NONE) - movl $11, %edi - call do_exit + /* + * User mode is traced as though IRQs are on, and SYSENTER + * turned them off. + */ + TRACE_IRQS_OFF - /* Unreachable. */ - ud2 + movq %rsp, %rdi + call do_fast_syscall_32 + jmp .Lsyscall_32_done sysenter_fix_flags: pushq $X86_EFLAGS_FIXED
@@ -135,26 +148,14 @@ ENDPROC(entry_SYSENTER_compat) * edi arg5 * esp user stack * 0(%esp) arg6 - * - * This is purely a fast path. For anything complicated we use the int 0x80 - * path below. We set up a complete hardware stack frame to share code - * with the int 0x80 path. */ ENTRY(entry_SYSCALL_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK - /* Temporary: SYSCALL32 is disabled. */ - movl $-ENOSYS, %eax - USERGS_SYSRET32 - + /* Stash user ESP and switch to the kernel stack. */ movl %esp, %r8d movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %eax, %eax
@@ -169,13 +170,29 @@ ENTRY(entry_SYSCALL_compat) pushq %rdi /* pt_regs->di */ pushq %rsi /* pt_regs->si */ pushq %rdx /* pt_regs->dx */ - pushq %rbp /* pt_regs->cx */ - movl %ebp, %ecx + pushq %rcx /* pt_regs->cx (will be overwritten) */ pushq $-ENOSYS /* pt_regs->ax */ - sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */ + xorq %r8,%r8 + pushq %r8 /* pt_regs->r8 = 0 */ + pushq %r8 /* pt_regs->r9 = 0 */ + pushq %r8 /* pt_regs->r10 = 0 */ + pushq %r8 /* pt_regs->r11 = 0 */ + pushq %rbx /* pt_regs->rbx */ + pushq %rbp /* pt_regs->rbp */ + pushq %r8 /* pt_regs->r12 = 0 */ + pushq %r8 /* pt_regs->r13 = 0 */ + pushq %r8 /* pt_regs->r14 = 0 */ + pushq %r8 /* pt_regs->r15 = 0 */ - /* Unreachable. */ - ud2 + /* + * User mode is traced as though IRQs are on, and SYSENTER + * turned them off. + */ + TRACE_IRQS_OFF + + movq %rsp, %rdi + call do_fast_syscall_32 + jmp .Lsyscall_32_done END(entry_SYSCALL_compat) /*
@@ -243,6 +260,7 @@ ENTRY(entry_INT80_compat) movq %rsp, %rdi call do_int80_syscall_32 +.Lsyscall_32_done: /* Go back to user mode. */ TRACE_IRQS_ON
arch/x86/entry/vdso/vdso32/system_call.S +8 -0
@@ -3,6 +3,8 @@ */ #include <asm/dwarf2.h> +#include <asm/cpufeature.h> +#include <asm/alternative-asm.h> /* * First get the common code for the sigreturn entry points.
@@ -28,6 +30,12 @@ __kernel_vsyscall: CFI_REL_OFFSET ecx, 0 movl %esp, %ecx +#ifdef CONFIG_X86_64 + /* If SYSENTER (Intel) or SYSCALL32 (AMD) is available, use it. */ + ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ + "syscall", X86_FEATURE_SYSCALL32 +#endif + /* Enter using int $0x80 */ movl (%esp), %ecx int $0x80
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
In reply to Andy Lutomirskix86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
Commit-ID: 7841b408717d4c3b1b334c8f1fef7f18c98cd2bd Gitweb: http://git.kernel.org/tip/7841b408717d4c3b1b334c8f1fef7f18c98cd2bd Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:12 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200
x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
If CS, SS and IP are as expected and FLAGS is compatible with SYSRETL, then return from fast compat syscalls (both SYSCALL and SYSENTER) using SYSRETL.
Unlike native 64-bit opportunistic SYSRET, this is not invisible to user code: RCX and R8-R15 end up in a different state than shown saved in pt_regs. To compensate, we only do this when returning to the vDSO fast syscall return path. This won't interfere with syscall restart, as we won't use SYSRETL when returning to the INT80 restart instruction.
--- arch/x86/entry/common.c | 23 +++++++++++++++++++--- arch/x86/entry/entry_64_compat.S | 42 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 5 deletions(-)
arch/x86/entry/common.c +20 -3
@@ -363,7 +363,8 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } -__visible void do_fast_syscall_32(struct pt_regs *regs) +/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ +__visible long do_fast_syscall_32 (struct pt_regs *regs) {/* * Called using the internal vDSO SYSENTER/SYSCALL32 calling
@@ -395,12 +396,28 @@ __visible void do_fast_syscall_32(struct pt_regs *regs)enter_from_user_mode ();#endif prepare_exit_to_usermode (regs); - return; + return 0; /* Keep it simple: use IRET. */ }local_irq_disable ();/* Now this is just like a normal syscall. */ do_int80_syscall_32 (regs); -return ; + +#ifdef CONFIG_X86_64 +/* +* Opportunistic SYSRETL: if possible, try to return using SYSRETL. +* SYSRETL is available on all 64-bit CPUs, so we don't need to +* bother with SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +*/ +return regs->cs == __USER32_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ; +#else +return 0 ; +#endif }#endif
arch/x86/entry/entry_64_compat.S +40 -2
54 lines not on this page · raw message
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/32: Open-code return tracking from fork and kthreads
In reply to Andy Lutomirskix86/entry/32: Open-code return tracking from fork and kthreads
Commit-ID: 39e8701f33d65c7f51d749a5d12a1379065e0926 Gitweb: http://git.kernel.org/tip/39e8701f33d65c7f51d749a5d12a1379065e0926 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:13 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200
x86/entry/32: Open-code return tracking from fork and kthreads
syscall_exit is going away, and return tracing is just a function call now, so open-code the two non-syscall 32-bit users.
While we're at it, update the big register layout comment.
--- arch/x86/entry/entry_32.S | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
arch/x86/entry/entry_32.S +15 -3
@@ -3,7 +3,7 @@ * * entry_32.S contains the system-call and low-level fault and trap handling routines. * - * Stack layout in 'syscall_exit': + * Stack layout while running C code: * ptrace needs to have all registers on the stack. * If the order here is changed, it needs to be * updated in fork.c:copy_process(), signal.c:do_signal(),
@@ -211,7 +211,11 @@ ENTRY(ret_from_fork) popl %eax pushl $0x0202 # Reset kernel eflags popfl - jmp syscall_exit + + /* When we fork, we trace the syscall return in the child, too. */ + movl %esp, %eax + call syscall_return_slowpath + jmp restore_all END(ret_from_fork) ENTRY(ret_from_kernel_thread)
@@ -224,7 +228,15 @@ ENTRY(ret_from_kernel_thread) movl PT_EBP(%esp), %eax call *PT_EBX(%esp) movl $0, PT_EAX(%esp) - jmp syscall_exit + + /* + * Kernel threads return to userspace as if returning from a syscall. + * We should check whether anything actually uses this path and, if so, + * consider switching it over to ret_from_fork. + */ + movl %esp, %eax + call syscall_return_slowpath + jmp restore_all ENDPROC(ret_from_kernel_thread) /*
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
In reply to Andy Lutomirskix86/entry/32: Switch INT80 to the new C syscall path
Commit-ID: 150ac78d63afb96360dab448b7b4d33c98c8266c Gitweb: http://git.kernel.org/tip/150ac78d63afb96360dab448b7b4d33c98c8266c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:14 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200
x86/entry/32: Switch INT80 to the new C syscall path
--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)
arch/x86/entry/entry_32.S +13 -21
@@ -153,13 +153,13 @@ #endif /* CONFIG_X86_32_LAZY_GS */ -.macro SAVE_ALL +.macro SAVE_ALL pt_regs_ax=%eax cld PUSH_GS pushl %fs pushl %es pushl %ds - pushl %eax + pushl \pt_regs_ax pushl %ebp pushl %edi pushl %esi
@@ -370,20 +370,17 @@ ENDPROC(entry_SYSENTER_32) # system call handler stub ENTRY(entry_INT80_32) ASM_CLAC - pushl %eax # save orig_eax - SAVE_ALL - GET_THREAD_INFO(%ebp) - # system call tracing in operation / emulation - testl $_TIF_WORK_SYSCALL_ENTRY, TI_flags(%ebp) - jnz syscall_trace_entry - cmpl $(NR_syscalls), %eax - jae syscall_badsys -syscall_call: - call *sys_call_table(, %eax, 4) -syscall_after_call: - movl %eax, PT_EAX(%esp) # store the return value -syscall_exit: - jmp syscall_exit_work + pushl %eax /* pt_regs->orig_ax */ + SAVE_ALL pt_regs_ax=$-ENOSYS /* save rest, load -ENOSYS into ax */ + + /* + * User mode is traced as though IRQs are on, and the interrupt gate + * turned them off. + */ + TRACE_IRQS_OFF + + movl %esp, %eax + call do_int80_syscall_32 restore_all: TRACE_IRQS_IRET
@@ -491,11 +488,6 @@ syscall_fault: jmp resume_userspace END(syscall_fault) -syscall_badsys: - movl $-ENOSYS, %eax - jmp syscall_after_call -END(syscall_badsys) - sysenter_badsys: movl $-ENOSYS, %eax jmp sysenter_after_call
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry/32: Re-implement SYSENTER using the new C path
In reply to Andy Lutomirskix86/entry/32: Re-implement SYSENTER using the new C path
Commit-ID: 5f310f739b4cc343f3f087681e41bbc2f0ce902d Gitweb: http://git.kernel.org/tip/5f310f739b4cc343f3f087681e41bbc2f0ce902d Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:15 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200
x86/entry/32: Re-implement SYSENTER using the new C path
--- arch/x86/entry/common.c | 17 +++- arch/x86/entry/entry_32.S | 132 ++++++++----------------------- arch/x86/entry/vdso/vdso32/system_call.S | 2 + 3 files changed, 51 insertions(+), 100 deletions(-)
arch/x86/entry/common.c +15 -2
@@ -363,7 +363,7 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } -/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ +/* Returns 0 to return using IRET or 1 to return using SYSEXIT/SYSRETL. */ __visible long do_fast_syscall_32 (struct pt_regs *regs) {/*
@@ -417,7 +417,20 @@ __visible long do_fast_syscall_32(struct pt_regs *regs) regs->ip == landing_pad && (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ;#else -return 0 ; +/* +* Opportunistic SYSEXIT: if possible, try to return using SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +* +* We don't allow syscalls at all from VM86 mode, but we still +* need to check VM, because we might be returning from sys_vm86. +*/ +return static_cpu_has (X86_FEATURE_SEP ) && + regs->cs == __USER_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF |X86_EFLAGS_VM )) ==0 ; #endif }#endif
arch/x86/entry/entry_32.S +34 -98
165 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/system_call.S +2 -0
@@ -34,6 +34,8 @@ __kernel_vsyscall: /* If SYSENTER (Intel) or SYSCALL32 (AMD) is available, use it. */ ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ "syscall", X86_FEATURE_SYSCALL32 +#else + ALTERNATIVE "", "sysenter", X86_FEATURE_SEP #endif /* Enter using int $0x80 */
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/asm: Remove thread_info.sysenter_return
In reply to Andy Lutomirskix86/asm: Remove thread_info.sysenter_return
Commit-ID: 487e3bf4f77699160aa81a414200060a78a67c3d Gitweb: http://git.kernel.org/tip/487e3bf4f77699160aa81a414200060a78a67c3d Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:16 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:11 +0200
x86/asm: Remove thread_info.sysenter_return
It's no longer needed.
We could reinstate something like it as an optimization, which would remove two cachelines from the fast syscall entry working set. I benchmarked it, and it makes no difference whatsoever to the performance of cache-hot compat syscalls on Sandy Bridge.
--- arch/x86/include/asm/thread_info.h | 1 - arch/x86/kernel/asm-offsets.c | 3 --- 2 files changed, 4 deletions(-)
arch/x86/include/asm/thread_info.h +0 -1
@@ -59,7 +59,6 @@ struct thread_info {__u32 cpu;/* current CPU */ int saved_preempt_count;mm_segment_t addr_limit; -void __user *sysenter_return; unsigned int sig_on_uaccess_error:1; unsigned int uaccess_err:1; /* uaccess failed */ };
arch/x86/kernel/asm-offsets.c +0 -3
@@ -54,9 +54,6 @@ void common(void) {OFFSET (IA32_SIGCONTEXT_ip, sigcontext_ia32, ip);BLANK (); -OFFSET (TI_sysenter_return, thread_info, sysenter_return); - -BLANK ();OFFSET (IA32_RT_SIGFRAME_sigcontext, rt_sigframe_ia32, uc.uc_mcontext);#endif
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls
In reply to Andy Lutomirskix86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls
Commit-ID: 8b13c2552ffc8e54e57598df36707183933e8e8c Gitweb: http://git.kernel.org/tip/8b13c2552ffc8e54e57598df36707183933e8e8c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:17 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:11 +0200
x86/entry: Remove unnecessary IRQ twiddling in fast 32-bit syscalls
This is slightly messy, but it eliminates an unnecessary cli;sti pair.
--- arch/x86/entry/common.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
arch/x86/entry/common.c +11 -7
@@ -323,10 +323,10 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )/* -* Does a 32-bit syscall. Called with IRQs off and does all entry and -* exit work. +* Does a 32-bit syscall. Called with IRQs on and does all entry and +* exit work and returns with IRQs off. */ -__visible void do_int80_syscall_32(struct pt_regs *regs) +static void do_syscall_32_irqs_on (struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned int nr = (unsigned int )regs->orig_ax;
@@ -335,8 +335,6 @@ __visible void do_int80_syscall_32(struct pt_regs *regs) ti->status |=TS_COMPAT ;#endif -local_irq_enable (); -if (READ_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY) {/* * Subtlety here: if ptrace pokes something larger than
@@ -363,6 +361,13 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } +/* Handles int $0x80 */ +__visible void do_int80_syscall_32(struct pt_regs *regs) +{ +local_irq_enable (); +do_syscall_32_irqs_on (regs); +} +/* Returns 0 to return using IRET or 1 to return using SYSEXIT/SYSRETL. */ __visible long do_fast_syscall_32 (struct pt_regs *regs) {
@@ -398,10 +403,9 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)prepare_exit_to_usermode (regs);return 0 ;/* Keep it simple: use IRET. */ } -local_irq_disable ();/* Now this is just like a normal syscall. */ -do_int80_syscall_32 (regs); +do_syscall_32_irqs_on (regs);#ifdef CONFIG_X86_64 /*
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Make irqs_disabled checks in exit code depend on lockdep
In reply to Andy Lutomirskix86/entry: Make irqs_disabled checks in exit code depend on lockdep
Commit-ID: 460d12453e1afe20416ce9536cfecb31d17a9abd Gitweb: http://git.kernel.org/tip/460d12453e1afe20416ce9536cfecb31d17a9abd Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:18 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:11 +0200
x86/entry: Make irqs_disabled checks in exit code depend on lockdep
These checks are quite slow. Disable them in non-lockdep kernels to reduce the performance hit.
--- arch/x86/entry/common.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +3 -3
@@ -219,7 +219,7 @@ static struct thread_info *pt_regs_to_thread_info(struct pt_regs *regs)/* Called with IRQs disabled. */ __visible void prepare_exit_to_usermode(struct pt_regs *regs) { -if (WARN_ON (!irqs_disabled ())) +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) &&WARN_ON (!irqs_disabled ()))local_irq_disable ();lockdep_sys_exit ();
@@ -281,8 +281,8 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)CT_WARN_ON (ct_state () !=CONTEXT_KERNEL ); -if (WARN (irqs_disabled (),"syscall %ld left IRQs disabled" , - regs->orig_ax)) +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) && +WARN (irqs_disabled (),"syscall %ld left IRQs disabled" , regs->orig_ax))local_irq_enable ();/*
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Force inlining of 32-bit syscall code
In reply to Andy Lutomirskix86/entry: Force inlining of 32-bit syscall code
Commit-ID: 33c52129f45e06d9ce23e1a3d50bf9fd6770748b Gitweb: http://git.kernel.org/tip/33c52129f45e06d9ce23e1a3d50bf9fd6770748b Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:19 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:12 +0200
x86/entry: Force inlining of 32-bit syscall code
On systems that support fast syscalls, we only really care about the performance of the fast syscall path. Forcibly inline it and add a likely annotation.
This saves 4-6 cycles.
--- arch/x86/entry/common.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/x86/entry/common.c +5 -3
@@ -324,9 +324,11 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION )/* * Does a 32-bit syscall. Called with IRQs on and does all entry and -* exit work and returns with IRQs off. +* exit work and returns with IRQs off. This function is extremely hot +* in workloads that use it, and it's usually called from +* do_fast_syscall_32, so forcibly inline it to improve performance. */ -static void do_syscall_32_irqs_on (struct pt_regs *regs) +static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned int nr = (unsigned int )regs->orig_ax;
@@ -345,7 +347,7 @@ static void do_syscall_32_irqs_on(struct pt_regs *regs) nr =syscall_trace_enter (regs); } -if (nr < IA32_NR_syscalls) { +if (likely (nr < IA32_NR_syscalls)) {/* * It's possible that a 32-bit syscall implementation * takes a 64-bit parameter but nonetheless assumes that
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Micro-optimize compat fast syscall arg fetch
In reply to Andy Lutomirskix86/entry: Micro-optimize compat fast syscall arg fetch
Commit-ID: c68ca6787bdd6d2df37cf950135aa11e71af358a Gitweb: http://git.kernel.org/tip/c68ca6787bdd6d2df37cf950135aa11e71af358a Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:20 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:12 +0200
x86/entry: Micro-optimize compat fast syscall arg fetch
We're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the access_ok() check isn't needed.
This saves about three cycles on a cache-hot fast syscall.
--- arch/x86/entry/common.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
arch/x86/entry/common.c +14 -2
@@ -394,8 +394,20 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)* WARNING: We are in CONTEXT_USER and RCU isn't paying attention! */ local_irq_enable (); -if (get_user (*(u32 *)®s->cx, - (u32 __user __force *)(unsigned long )(u32 )regs->sp)) { + if ( +#ifdef CONFIG_X86_64 +/* +* Micro-optimization: the pointer we're following is explicitly +* 32 bits, so it can't be out of range. +*/ +__get_user (*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long)(u32)regs->sp) +#else + get_user(*(u32 *)®s->cx, + (u32 __user __force *)(unsigned long)(u32)regs->sp) +#endif + ) { +/* User code screwed up. */ local_irq_disable (); regs->ax = -EFAULT ;
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Hide two syscall entry assertions behind CONFIG_DEBUG_ENTRY
In reply to Andy Lutomirskix86/entry: Hide two syscall entry assertions behind CONFIG_DEBUG_ENTRY
Commit-ID: 4aabd140f9cbe0361401a1368bac74df1010abf5 Gitweb: http://git.kernel.org/tip/4aabd140f9cbe0361401a1368bac74df1010abf5 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:21 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:12 +0200
x86/entry: Hide two syscall entry assertions behind CONFIG_DEBUG_ENTRY
This shaves a few cycles off the slow paths.
--- arch/x86/entry/common.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/entry/common.c +4 -2
@@ -71,7 +71,8 @@ unsigned long syscall_trace_enter_phase1(struct pt_regs *regs, u32 arch)unsigned long ret =0 ;u32 work; -BUG_ON (regs !=task_pt_regs (current)); +if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) +BUG_ON (regs !=task_pt_regs (current)); work =ACCESS_ONCE (current_thread_info ()->flags) & _TIF_WORK_SYSCALL_ENTRY;
@@ -160,7 +161,8 @@ long syscall_trace_enter_phase2(struct pt_regs *regs, u32 arch,u32 work =ACCESS_ONCE (current_thread_info ()->flags) & _TIF_WORK_SYSCALL_ENTRY; -BUG_ON (regs !=task_pt_regs (current)); +if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) +BUG_ON (regs !=task_pt_regs (current));/* * If we stepped into a sysenter/syscall insn, it trapped in
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Use pt_regs_to_thread_info() in syscall entry tracing
In reply to Andy Lutomirskix86/entry: Use pt_regs_to_thread_info() in syscall entry tracing
Commit-ID: dd636071c3d8044c802b7a365e9934724a929530 Gitweb: http://git.kernel.org/tip/dd636071c3d8044c802b7a365e9934724a929530 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:22 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:12 +0200
x86/entry: Use pt_regs_to_thread_info() in syscall entry tracing
It generates simpler and faster code than current_thread_info().
--- arch/x86/entry/common.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
arch/x86/entry/common.c +11 -11
@@ -30,6 +30,13 @@#define CREATE_TRACE_POINTS #include <trace/events/syscalls.h> +static struct thread_info *pt_regs_to_thread_info (struct pt_regs *regs) +{ +unsigned long top_of_stack = + (unsigned long )(regs +1 ) +TOP_OF_KERNEL_STACK_PADDING ; +return (struct thread_info *)(top_of_stack -THREAD_SIZE ); +} +#ifdef CONFIG_CONTEXT_TRACKING /* Called on entry from user mode with IRQs off. */ __visible void enter_from_user_mode(void)
@@ -68,14 +75,14 @@ static void do_audit_syscall_entry(struct pt_regs *regs, u32 arch)*/ unsigned long syscall_trace_enter_phase1 (struct pt_regs *regs,u32 arch) { +struct thread_info *ti =pt_regs_to_thread_info (regs);unsigned long ret =0 ;u32 work;if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))BUG_ON (regs !=task_pt_regs (current)); - work =ACCESS_ONCE (current_thread_info ()->flags) & - _TIF_WORK_SYSCALL_ENTRY; + work =ACCESS_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY;#ifdef CONFIG_CONTEXT_TRACKING /*
@@ -157,9 +164,9 @@ unsigned long syscall_trace_enter_phase1(struct pt_regs *regs, u32 arch)long syscall_trace_enter_phase2 (struct pt_regs *regs,u32 arch,unsigned long phase1_result) { +struct thread_info *ti =pt_regs_to_thread_info (regs);long ret =0 ; -u32 work =ACCESS_ONCE (current_thread_info ()->flags) & - _TIF_WORK_SYSCALL_ENTRY; +u32 work =ACCESS_ONCE (ti->flags) & _TIF_WORK_SYSCALL_ENTRY;if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))BUG_ON (regs !=task_pt_regs (current));
@@ -211,13 +218,6 @@ long syscall_trace_enter(struct pt_regs *regs)return syscall_trace_enter_phase2 (regs, arch, phase1_result); } -static struct thread_info *pt_regs_to_thread_info (struct pt_regs *regs) -{ -unsigned long top_of_stack = - (unsigned long )(regs +1 ) +TOP_OF_KERNEL_STACK_PADDING ; -return (struct thread_info *)(top_of_stack -THREAD_SIZE ); -} -/* Called with IRQs disabled. */ __visible void prepare_exit_to_usermode(struct pt_regs *regs) {
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Split and inline prepare_exit_to_usermode()
In reply to Andy Lutomirskix86/entry: Split and inline prepare_exit_to_usermode()
Commit-ID: 39b48e575e92e31251b74b4b48cea2129cee90bd Gitweb: http://git.kernel.org/tip/39b48e575e92e31251b74b4b48cea2129cee90bd Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:23 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:13 +0200
x86/entry: Split and inline prepare_exit_to_usermode()
GCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part.
Help it out with prepare_exit_to_usermode() by splitting it into two parts and inline the hot part.
Saves 6-8 cycles for compat syscalls.
--- arch/x86/entry/common.c | 43 ++++++++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 15 deletions(-)
arch/x86/entry/common.c +28 -15
@@ -218,14 +218,12 @@ long syscall_trace_enter(struct pt_regs *regs)return syscall_trace_enter_phase2 (regs, arch, phase1_result); } -/* Called with IRQs disabled. */ -__visible void prepare_exit_to_usermode(struct pt_regs *regs) -{ -if (IS_ENABLED (CONFIG_PROVE_LOCKING ) &&WARN_ON (!irqs_disabled ())) -local_irq_disable (); - -lockdep_sys_exit (); +#define EXIT_TO_USERMODE_LOOP_FLAGS \ + (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ + _TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY) +static void exit_to_usermode_loop (struct pt_regs *regs,u32 cached_flags) +{/* * In order to return to user mode, we need to have IRQs off with * none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY,
@@ -235,14 +233,6 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)* work to clear some of the flags can sleep. */ while (true) { -u32 cached_flags = -READ_ONCE (pt_regs_to_thread_info (regs)->flags); - -if (!(cached_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | - _TIF_UPROBE | _TIF_NEED_RESCHED | - _TIF_USER_RETURN_NOTIFY))) -break ; -/* We have work to do. */ local_irq_enable ();
@@ -266,7 +256,30 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)/* Disable IRQs and retry */ local_irq_disable (); + + cached_flags =READ_ONCE (pt_regs_to_thread_info (regs)->flags); + +if (!(cached_flags &EXIT_TO_USERMODE_LOOP_FLAGS )) +break ; + } +} + +/* Called with IRQs disabled. */ +__visible inline void prepare_exit_to_usermode(struct pt_regs *regs) +{ +u32 cached_flags; + +if (IS_ENABLED (CONFIG_PROVE_LOCKING ) &&WARN_ON (!irqs_disabled ())) +local_irq_disable (); + +lockdep_sys_exit (); + + cached_flags = +READ_ONCE (pt_regs_to_thread_info (regs)->flags); + +if (unlikely (cached_flags &EXIT_TO_USERMODE_LOOP_FLAGS )) +exit_to_usermode_loop (regs, cached_flags);user_enter (); }
tip-bot for Andy Lutomirski
merge bot[tip:x86/asm] x86/entry: Split and inline syscall_return_slowpath ()
In reply to Andy Lutomirskix86/entry: Split and inline syscall_return_slowpath()
Commit-ID: f5e6a9753ac2965564a14e6285a06f44043ed9c8 Gitweb: http://git.kernel.org/tip/f5e6a9753ac2965564a14e6285a06f44043ed9c8 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:24 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:13 +0200
x86/entry: Split and inline syscall_return_slowpath()
GCC is unable to properly optimize functions that have a very short likely case and a longer and register-heavier cold part -- it fails to sink all of the register saving and stack frame setup code into the unlikely part.
Help it out with syscall_return_slowpath() by splitting it into two parts and inline the hot part.
Saves 6 cycles for compat syscalls.
--- arch/x86/entry/common.c | 50 ++++++++++++++++++++++++++++--------------------- 1 file changed, 29 insertions(+), 21 deletions(-)
arch/x86/entry/common.c +29 -21
@@ -284,15 +284,40 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs)user_enter (); } +#define SYSCALL_EXIT_WORK_FLAGS \ + (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ + _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT) + +static void syscall_slow_exit_work (struct pt_regs *regs,u32 cached_flags) +{ +bool step; + +audit_syscall_exit (regs); + +if (cached_flags & _TIF_SYSCALL_TRACEPOINT) +trace_sys_exit (regs, regs->ax); + +/* +* If TIF_SYSCALL_EMU is set, we only get here because of +* TIF_SINGLESTEP (i.e. this is PTRACE_SYSEMU_SINGLESTEP). +* We already reported this syscall instruction in +* syscall_trace_enter(). +*/ + step =unlikely ( + (cached_flags & (_TIF_SINGLESTEP | _TIF_SYSCALL_EMU)) + == _TIF_SINGLESTEP); +if (step || cached_flags & _TIF_SYSCALL_TRACE) +tracehook_report_syscall_exit (regs, step); +} +/* * Called with IRQs on and fully valid regs. Returns with IRQs off in a * state such that we can immediately switch to user mode. */ -__visible void syscall_return_slowpath(struct pt_regs *regs) +__visible inline void syscall_return_slowpath(struct pt_regs *regs) {struct thread_info *ti =pt_regs_to_thread_info (regs);u32 cached_flags =READ_ONCE (ti->flags); -bool step;CT_WARN_ON (ct_state () !=CONTEXT_KERNEL );
@@ -304,25 +329,8 @@ __visible void syscall_return_slowpath(struct pt_regs *regs)* First do one-time work. If these work items are enabled, we * want to run them exactly once per syscall exit with IRQs on. */ -if (cached_flags & (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | - _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT)) { -audit_syscall_exit (regs); - -if (cached_flags & _TIF_SYSCALL_TRACEPOINT) -trace_sys_exit (regs, regs->ax); - -/* -* If TIF_SYSCALL_EMU is set, we only get here because of -* TIF_SINGLESTEP (i.e. this is PTRACE_SYSEMU_SINGLESTEP). -* We already reported this syscall instruction in -* syscall_trace_enter(). -*/ - step =unlikely ( - (cached_flags & (_TIF_SINGLESTEP | _TIF_SYSCALL_EMU)) - == _TIF_SINGLESTEP); -if (step || cached_flags & _TIF_SYSCALL_TRACE) -tracehook_report_syscall_exit (regs, step); - } +if (unlikely (cached_flags &SYSCALL_EXIT_WORK_FLAGS )) +syscall_slow_exit_work (regs, cached_flags);#ifdef CONFIG_COMPAT /*
Borislav Petkov
maintainerPATCH 1/1 x86/entry/64/compat: Document sysenter_fix_flags's reason for existence
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Date: Fri, 9 Oct 2015 19:04:09 +0200
Subject: [PATCH] x86/entry/64/compat: Document sysenter_fix_flags's reason for
existence
The code under the label can normally be inline, without the jumping back and forth but the latter is an optimization. Document that.
--- arch/x86/entry/entry_64_compat.S | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/entry/entry_64_compat.S +6 -0
@@ -102,6 +102,12 @@ ENTRY(entry_SYSENTER_compat) * NT was set instead of doing an unconditional popfq. * This needs to happen before enabling interrupts so that * we don't get preempted with NT set. + * + * NB.: sysenter_fix_flags is a label with the code under it moved + * out-of-line as an optimization: NT is unlikely to be set in the + * majority of the cases and instead of polluting the I$ unnecessarily, + * we're keeping that code behind a branch which will predict as + * not-taken and therefore its instructions won't be fetched. */ testl $X86_EFLAGS_NT, EFLAGS(%rsp) jnz sysenter_fix_flags
Signature · 7 lines
2.3.5
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Re: [PATCH] x86/entry/64/compat: Document sysenter_fix_flags's reason for existence
Quoted 7 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
Date: Fri, 9 Oct 2015 19:04:09 +0200
Subject: [PATCH] x86/entry/64/compat: Document sysenter_fix_flags's reason for
existenceThe code under the label can normally be inline, without the jumping back and forth but the latter is an optimization. Document that.
--Andy
Re: [PATCH v2 14/36] x86/entry/64/compat: Disable SYSENTER and SYSCALL32 entries
In reply to Borislav PetkovQuoted 10 lines · Borislav Petkov
Quoted 7 lines · Andy Lutomirski
We've disabled the vDSO helpers to call them, so turn off the entries entirely (temporarily) in preparation for cleaning them up.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_64_compat.S | 13 +++++++++++++ 1 file changed, 13 insertions(+)
Quoted 5 lines
What do we say for the case when someone bisects, lands at this patch and, as Murphy would generally do it, has a reproducer which is calling SYSENTER/SYSCALL32?
Tough luck?
Probably. I really hope that no one does that. Given how problematic those instructions are outside the vdso, it would have to be a rather strange reproducer.
Quoted 7 lines
--
Regards/Gruss,
Boris.ECO tip #101: Trim your mails when you reply.
--
Re: [PATCH v2 32/36] x86/entry: Micro-optimize compat fast syscall arg fetch
In reply to Ingo MolnarQuoted 12 lines · Ingo Molnar
Quoted 4 lines · Andy Lutomirski
we're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the
access_okcheck isn't needed.This saves about three cycles on a cache-hot fast syscall.
Another request: could you please stick the benchmarking code of the various x86 system call variants into 'perf bench' - under
tools/perf/bench/, so that measurements can be done on more hardware and can be reproduced easily?
Quoted 6 lines
I'd suggest we dedicate an entirely new benchmark family to it: 'perf bench x86' and then have:
perf bench x86 syscall vdso
perf bench x86 syscall int80
perf bench x86 syscall vdso-compat
I'll play with this. I'm not too familiar with the perf bench stuff.
--Andy
Re: [tip:x86/asm] x86/entry, locking/lockdep: Move lockdep_sys_exit() to prepare_exit_to_usermode()
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
Quoted 14 lines · tip-bot for Andy Lutomirski
Commit-ID: 72f924783b8a87e4454516520ffb5f35e4930371 Gitweb: http://git.kernel.org/tip/72f924783b8a87e4454516520ffb5f35e4930371 Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:47:54 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 7 Oct 2015 11:34:07 +0200x86/entry, locking/lockdep: Move
lockdep_sys_exit()toprepare_exit_to_usermode()Rather than worrying about exactly where
LOCKDEP_SYS_EXITshould go in the asm code, add it toprepare_exit_from_usermode()and remove all of the asm calls that are followed byprepare_exit_to_usermode().
Quoted 4 lines
Quoted 1 line
LOCKDEP_SYS_EXITnow appears only in the syscall fast paths.Does that not pose a risk that something that always takes the slow path (signals? tracing?) will leak a lock to userspace?
No, because while LOCKDEP_SYS_EXIT only happens in the fast path, lockdep_sys_exit (lowercase) will get called in the other cases.
--Andy
Re: [PATCH v2 07/36] x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
In reply to Denys VlasenkoQuoted 49 lines · Denys Vlasenko
Quoted 33 lines · Andy Lutomirski
We eventually want to make it all the way into C code before enabling interrupts. We need to rework our flags handling slightly to delay enabling interrupts.
Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/entry_64_compat.S | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)arch/x86/entry/entry_64_compat.S +1
-6@@ -58,14 +58,9 @@ ENDPROC(native_usergs_sysret32) * with the int 0x80 path. */ ENTRY(entry_SYSENTER_compat) - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ + /* Interrupts are off on entry. */ SWAPGS_UNSAFE_STACK movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp - ENABLE_INTERRUPTS(CLBR_NONE) /* Zero-extending 32-bit regs, do not remove */ movl %ebp, %ebp@@ -76,7 +71,16 @@ ENTRY(entry_SYSENTER_compat) /* Construct struct pt_regs on stack */ pushq $__USER32_DS /* pt_regs->ss */ pushq %rbp /* pt_regs->sp */arch/x86/entry/entry_64_compat.S ENTRY(entry_SYSENTER_compat) · line 82
Quoted 10 lines
arch/x86/entry/entry_64_compat.S +9
-1- pushfq /* pt_regs->flags */ + + /* + * Push flags. This is nasty. First, interrupts are currently + * off, but we need pt_regs->flags to have IF set. Second, even + * if TF was set when SYSENTER started, it's clear by now. We fix + * that later using TIF_SINGLESTEP. + */ + pushfq /* pt_regs->flags (except IF = 0) */ + orl $X86_EFLAGS_IF, (%rsp) /* Fix saved flags */The sequence of "push +
insn_using_rsp" is a bit slow on most CPUs because stack engine (the machinery which makes consecutive pushes fast) needs syncronizing with register file.
Quoted 9 lines
It may be better to move the ORL insn here:
push, push, push
cld
sub $(10*8), %rsp /* pt_regs->r8-11, bp, bx, r12-15 not saved */
+ orl $X86_EFLAGS_IF, EFLAGS(%rsp) /* Fix saved flags to have .IF = 1 */where we already eat that penalty.
I played with it a bit. It seems to save one cycle, which is probably worthwhile.
I think that the gain is small because neither case is particular bad. I read Agner Fog's writeup on the stack engine. I think that, with the code as written in my patch, the µops are, roughly:
store $__USER32_DS to [rsp - 0]
store $rcx to [rsp - 8]
store flags to [rsp - 12]
sync rsp (rsp -= 12)
<-- wait for ALU here
or $X86_EFLAGS_IF, $rsp (might be two µops)
store $__USER32_CS to [rsp]
...
sync rsp (rsp -= something)
<-- wait for ALU here
testl ...
with the change, it's:
store $__USER32_DS to [rsp - 0]
store $rcx to [rsp - 8]
store flags to [rsp - 12]
store $__USER32_CS to [rsp-16]
...
sync rsp (rsp -= something)
sync rsp (rsp -= 12)
<-- wait for ALU here
or $X86_EFLAGS_IF, EFLAGS($rsp) (might be two µops) testl ...
So the change removes one rsp sync and one wait. On the other hand, the ALU is probably otherwise idle for most of this sequence, so the latency is probably almost entirely hidden because the dispatcher can do the rsp syncs much earlier than they appear before reordering.
Also, deferring the orl makes the code longer and adds an address calculation, and both of those may have some cost. And moving the orl right before the testl worsens that dependency. It's too bad we don't have orx on all supported CPUs.
Anyway, it's still a win, but I'll keep playing. There may be even better ways to do this.
--Andy
Re: [PATCH v2 32/36] x86/entry: Micro-optimize compat fast syscall arg fetch
In reply to Andy LutomirskiQuoted 13 lines · Andy Lutomirski
Quoted 12 lines · Ingo Molnar
Quoted 4 lines · Andy Lutomirski
we're following a 32-bit pointer, and the uaccess code isn't smart enough to figure out that the
access_okcheck isn't needed.This saves about three cycles on a cache-hot fast syscall.
Another request: could you please stick the benchmarking code of the various x86 system call variants into 'perf bench' - under
tools/perf/bench/, so that measurements can be done on more hardware and can be reproduced easily?
Quoted 8 lines
Quoted 6 lines
I'd suggest we dedicate an entirely new benchmark family to it: 'perf bench x86' and then have:
perf bench x86 syscall vdso
perf bench x86 syscall int80
perf bench x86 syscall vdso-compatI'll play with this. I'm not too familiar with the perf bench stuff.
So the perf bench stuff is meant to be a familiar home to kernel developers we'd like to slap a micro (or macro) benchmark into an easy to modify place.
Over the years it has gathered a number of benchmarks - but more are always welcome.
Just copy one of the existing benchmark modules (the tools/perf/bench/numa.c one is the most advanced one, tools/perf/bench/sched-pipe.c is the simplest one) and off you go.
Here's a commit that adds a new benchmark suite:
a043971141f1 ("perf bench: Add futex-hash microbenchmark")
There are no big restrictions on the benchmarks: just put your existing code in that produces stdout output and it will be likely very close to upstream acceptable.
Can help should you get stuck anywhere.
Thanks,
Ingo
tip-bot for Borislav Petkov
merge bot[tip:x86/asm] x86/entry/64/compat: Document sysenter_fix_flags' s reason for existence
In reply to Borislav Petkovx86/entry/64/compat: Document sysenter_fix_flags's reason for existence
Commit-ID: 374a3a3916a70fc6236bc2b8f8ac02548a128a54 Gitweb: http://git.kernel.org/tip/374a3a3916a70fc6236bc2b8f8ac02548a128a54 Author: Borislav Petkov <[email protected]> AuthorDate: Fri, 9 Oct 2015 19:08:59 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Oct 2015 11:06:40 +0200
x86/entry/64/compat: Document sysenter_fix_flags's reason for existence
The code under the label can normally be inline, without the jumping back and forth but the latter is an optimization.
Document that.
--- arch/x86/entry/entry_64_compat.S | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/entry/entry_64_compat.S +6 -0
@@ -102,6 +102,12 @@ ENTRY(entry_SYSENTER_compat) * NT was set instead of doing an unconditional popfq. * This needs to happen before enabling interrupts so that * we don't get preempted with NT set. + * + * NB.: sysenter_fix_flags is a label with the code under it moved + * out-of-line as an optimization: NT is unlikely to be set in the + * majority of the cases and instead of polluting the I$ unnecessarily, + * we're keeping that code behind a branch which will predict as + * not-taken and therefore its instructions won't be fetched. */ testl $X86_EFLAGS_NT, EFLAGS(%rsp) jnz sysenter_fix_flags
Borislav Petkov
maintainerRe: [PATCH v2 24/36] x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
In reply to Andy LutomirskiQuoted 47 lines · Andy Lutomirski
If CS, SS and IP are as expected and FLAGS is compatible with SYSRETL, then return from fast compat syscalls (both SYSCALL and SYSENTER) using SYSRETL.
Unlike native 64-bit opportunistic SYSRET, this is not invisible to user code: RCX and R8-R15 end up in a different state than shown saved in
pt_regs. To compensate, we only do this when returning to the vDSO fast syscall return path. This won't interfere with syscall restart, as we won't use SYSRETL when returning to the INT80 restart instruction.Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/common.c | 23 +++++++++++++++++++--- arch/x86/entry/entry_64_compat.S | 42 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 5 deletions(-)arch/x86/entry/common.c +7
-3@@ -363,7 +363,8 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } -__visible void do_fast_syscall_32(struct pt_regs *regs) +/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ +__visible long do_fast_syscall_32 (struct pt_regs *regs) {/* * Called using the internal vDSO SYSENTER/SYSCALL32 calling @@ -395,12 +396,28 @@ __visible void do_fast_syscall_32(struct pt_regs *regs)enter_from_user_mode ();#endif prepare_exit_to_usermode (regs); -return ; +return 0 ;/* Keep it simple: use IRET. */ }local_irq_disable ();/* Now this is just like a normal syscall. */ do_int80_syscall_32 (regs); -return ; + +#ifdef CONFIG_X86_64 +/* +* Opportunistic SYSRETL: if possible, try to return using SYSRETL.
arch/x86/entry/common.c __visible void do_fast_syscall_32(struct pt_regs *regs) · line 418
Quoted 10 lines
arch/x86/entry/common.c +10
-0+* SYSRETL is available on all 64-bit CPUs, so we don't need to +* bother with SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +*/ +return regs->cs == __USER32_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ;
This could've used some readability massaging:
return regs->cs == __USER32_CS && regs->ss == __USER_DS && regs->ip == landing_pad && !(regs->flags & (X86_EFLAGS_RF | X86_EFLAGS_TF));
Although I'm not crazy about that one either. This expression is simply not parseable at a glance.
Why not write it like the 64-bit opportunistic SYSRET:
if (regs->cs != __USER32_CS) return 0; if (regs->ss != __USER_DS) return 0;
and so on...
asm is almost the same, except in the "after" case, the CALL to syscall_trace_enter is after the last JMP.
before:
.L270:
xorl %eax, %eax # D.34060
cmpq $35, 136(%rbx) #, regs_12(D)->cs
je .L296 #,
.L263:
movq -32(%rbp), %rbx #,
movq -24(%rbp), %r12 #,
movq -16(%rbp), %r13 #,
movq -8(%rbp), %r14 #,
leave
ret
.L291:
movq %rbx, %rdi # regs,
call syscall_trace_enter #
jmp .L265 #
.L296:
cmpq $43, 160(%rbx) #, regs_12(D)->ss
jne .L263 #,
cmpq 128(%rbx), %r12 # regs_12(D)->ip, landing_pad
jne .L263 #,
xorl %eax, %eax # D.34060
testq $65792, 144(%rbx) #, regs_12(D)->flags
sete %al #, D.34060
jmp .L263 #after:
.L271:
cmpq $35, 136(%rbx) #, regs_11(D)->cs
je .L293 #,
.L286:
xorl %eax, %eax # D.34060
.L264:
movq -32(%rbp), %rbx #,
movq -24(%rbp), %r12 #,
movq -16(%rbp), %r13 #,
movq -8(%rbp), %r14 #,
leave
ret
.L293:
cmpq $43, 160(%rbx) #, regs_11(D)->ss
jne .L286 #,
cmpq 128(%rbx), %r12 # regs_11(D)->ip, landing_pad
jne .L286 #,
xorl %eax, %eax # D.34060
testq $65792, 144(%rbx) #, regs_11(D)->flags
sete %al #, D.34060
jmp .L264 #Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Re: [PATCH v2 07/36] x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
In reply to Andy LutomirskiQuoted 2 lines · Andy Lutomirski
Anyway, it's still a win, but I'll keep playing. There may be even better ways to do this.
Such as this?
Drop OR, check for cleared IF here:
testl $X86_EFLAGS_NT, EFLAGS(%rsp)
jnz sysenter_fix_flags
testl $X86_EFLAGS_IF, EFLAGS(%rsp)
jz sysenter_fix_flagssysenter_flags_fixed:
and make sysenter_fix_flags code set IF in addition to TF clearing. This way, we don't even have an unconditional RMW op.
Re: [PATCH v2 07/36] x86/entry/64/compat: After SYSENTER, move STI after the NT fixup
Quoted 7 lines · Denys Vlasenko
Quoted 2 lines · Andy Lutomirski
Anyway, it's still a win, but I'll keep playing. There may be even better ways to do this.
Such as this?
Drop OR, check for cleared IF here:
Quoted 8 lines
testl $X86_EFLAGS_NT, EFLAGS(%rsp) jnz sysenter_fix_flags testl $X86_EFLAGS_IF, EFLAGS(%rsp) jz sysenter_fix_flags
sysenter_flags_fixed:and make
sysenter_fix_flagscode set IF in addition to TF clearing. This way, we don't even have an unconditional RMW op.
IF should normally be set in userspace, but since SYSENTER doesn't save the old flags and always clears IF when entering the kernel, the flags pushed on the stack will always have IF clear.
--
Brian Gerst
Quoted 4 lines · Ingo Molnar
Ok, so I applied all of them to tip:x86/asm, in two phases, with small (stylistic) edits - it all seems to work fine for me so far, so I pushed it all out to -tip and linux-next.
FYI, this breaks the UML build:
CC arch/x86/um/sys_call_table_64.oarch/x86/um/sys_call_table_64.c:49:1: error: unknown type name ‘sys_call_ptr_t’
const sys_call_ptr_t sys_call_table[] ____cacheline_aligned = {
^arch/x86/um/sys_call_table_64.c:54:2: warning: initialization makes integer from pointer without a cast [enabled by default]
[0 ... __NR_syscall_max] = &sys_ni_syscall,
^arch/x86/um/sys_call_table_64.c:54:2: warning: (near initialization for ‘sys_call_table[0]’) [enabled by default] arch/x86/um/sys_call_table_64.c:54:2: error: initializer element is not computable at load timearch/x86/um/sys_call_table_64.c:54:2: error: (near initialization for ‘sys_call_table[0]’)
Andy, I suspect UML's syscall.h needs the sys_call_ptr_t type too?
Signature · 2 lines
Thanks, //richard
Quoted 23 lines · Richard Weinberger
Quoted 4 lines · Ingo Molnar
Ok, so I applied all of them to tip:x86/asm, in two phases, with small (stylistic) edits - it all seems to work fine for me so far, so I pushed it all out to -tip and linux-next.
FYI, this breaks the UML build:
CC
arch/x86/um/sys_call_table_64.oarch/x86/um/sys_call_table_64.c:49:1: error: unknown type name ‘sys_call_ptr_t’
constsys_call_ptr_tsys_call_table[]____cacheline_aligned= {
^arch/x86/um/sys_call_table_64.c:54:2: warning: initialization makes integer from pointer without a cast [enabled by default]
[0 ...__NR_syscall_max] =&sys_ni_syscall,
^arch/x86/um/sys_call_table_64.c:54:2: warning: (near initialization for ‘sys_call_table[0]’) [enabled by default]arch/x86/um/sys_call_table_64.c:54:2: error: initializer element is not computable at load timearch/x86/um/sys_call_table_64.c:54:2: error: (near initialization for ‘sys_call_table[0]’)
Quoted 2 lines
Andy, I suspect UML's
syscall.hneeds thesys_call_ptr_ttype too?
Whoops, yes. UML has some weird casts in the syscall code that might be removable with that change, too. Want to fix it or should I?
--Andy
Richard Weinberger
maintainerAm 12.10.2015 um 20:41 schrieb Andy Lutomirski:
Quoted 25 lines
Quoted 23 lines · Richard Weinberger
Quoted 4 lines · Ingo Molnar
Ok, so I applied all of them to tip:x86/asm, in two phases, with small (stylistic) edits - it all seems to work fine for me so far, so I pushed it all out to -tip and linux-next.
FYI, this breaks the UML build:
CC
arch/x86/um/sys_call_table_64.oarch/x86/um/sys_call_table_64.c:49:1: error: unknown type name ‘sys_call_ptr_t’
constsys_call_ptr_tsys_call_table[]____cacheline_aligned= {
^arch/x86/um/sys_call_table_64.c:54:2: warning: initialization makes integer from pointer without a cast [enabled by default]
[0 ...__NR_syscall_max] =&sys_ni_syscall,
^arch/x86/um/sys_call_table_64.c:54:2: warning: (near initialization for ‘sys_call_table[0]’) [enabled by default]arch/x86/um/sys_call_table_64.c:54:2: error: initializer element is not computable at load timearch/x86/um/sys_call_table_64.c:54:2: error: (near initialization for ‘sys_call_table[0]’)
Quoted 5 lines
Quoted 2 lines
Andy, I suspect UML's
syscall.hneeds thesys_call_ptr_ttype too?Whoops, yes. UML has some weird casts in the syscall code that might be removable with that change, too. Want to fix it or should I?
Please send a fix. I'll happily review/test it.
Thanks,
//richard
Re: [PATCH v2 24/36] x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
In reply to Borislav PetkovQuoted 71 lines · Borislav Petkov
Quoted 47 lines · Andy Lutomirski
If CS, SS and IP are as expected and FLAGS is compatible with SYSRETL, then return from fast compat syscalls (both SYSCALL and SYSENTER) using SYSRETL.
Unlike native 64-bit opportunistic SYSRET, this is not invisible to user code: RCX and R8-R15 end up in a different state than shown saved in
pt_regs. To compensate, we only do this when returning to the vDSO fast syscall return path. This won't interfere with syscall restart, as we won't use SYSRETL when returning to the INT80 restart instruction.Signed-off-by Andy Lutomirski <[email protected]>--- arch/x86/entry/common.c | 23 +++++++++++++++++++--- arch/x86/entry/entry_64_compat.S | 42 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 60 insertions(+), 5 deletions(-)arch/x86/entry/common.c +7
-3@@ -363,7 +363,8 @@ __visible void do_int80_syscall_32(struct pt_regs *regs)syscall_return_slowpath (regs); } -__visible void do_fast_syscall_32(struct pt_regs *regs) +/* Returns 0 to return using IRET or 1 to return using SYSRETL. */ +__visible long do_fast_syscall_32 (struct pt_regs *regs) {/* * Called using the internal vDSO SYSENTER/SYSCALL32 calling @@ -395,12 +396,28 @@ __visible void do_fast_syscall_32(struct pt_regs *regs)enter_from_user_mode ();#endif prepare_exit_to_usermode (regs); -return ; +return 0 ;/* Keep it simple: use IRET. */ }local_irq_disable ();/* Now this is just like a normal syscall. */ do_int80_syscall_32 (regs); -return ; + +#ifdef CONFIG_X86_64 +/* +* Opportunistic SYSRETL: if possible, try to return using SYSRETL. arch/x86/entry/common.c __visible void do_fast_syscall_32(struct pt_regs *regs) · line 418
Quoted 10 lines
arch/x86/entry/common.c +10
-0+* SYSRETL is available on all 64-bit CPUs, so we don't need to +* bother with SYSEXIT. +* +* Unlike 64-bit opportunistic SYSRET, we can't check that CX == IP, +* because the ECX fixup above will ensure that this is essentially +* never the case. +*/ +return regs->cs == __USER32_CS && regs->ss == __USER_DS && + regs->ip == landing_pad && + (regs->flags & (X86_EFLAGS_RF |X86_EFLAGS_TF )) ==0 ;This could've used some readability massaging:
return regs->cs == __USER32_CS && regs->ss == __USER_DS && regs->ip == landing_pad && !(regs->flags & (X86_EFLAGS_RF | X86_EFLAGS_TF));Although I'm not crazy about that one either. This expression is simply not parseable at a glance.
Why not write it like the 64-bit opportunistic SYSRET:
Quoted 7 lines
if (regs->cs != __USER32_CS) return 0; if (regs->ss != __USER_DS) return 0;and so on...
We could do that. It might also be nice to factor it into its own function:
static bool may_fast_exit_32(struct pt_regs *regs);
I'll try that eventually if you don't beat me.
--Andy
Borislav Petkov
maintainerRe: [PATCH v2 24/36] x86/entry/compat: Implement opportunistic SYSRETL for compat syscalls
Quoted 3 lines · Andy Lutomirski
We could do that. It might also be nice to factor it into its own function:
static bool
may_fast_exit_32(struct pt_regs *regs);
Right.
Quoted 1 line
I'll try that eventually if you don't beat me.
I promise I won't beat you! :-P
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Borislav Petkov
maintainerRe: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
In reply to tip-bot for Andy LutomirskiQuoted 21 lines · tip-bot for Andy Lutomirski
Commit-ID: 150ac78d63afb96360dab448b7b4d33c98c8266c Gitweb: http://git.kernel.org/tip/150ac78d63afb96360dab448b7b4d33c98c8266c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:14 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200x86/entry/32: Switch INT80 to the new C syscall path
Signed-off-by Andy Lutomirski <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Borislav Petkov <[email protected]>Cc Brian Gerst <[email protected]>Cc Denys Vlasenko <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Linus Torvalds <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Link http://lkml.kernel.org/r/a7e8d8df96838eae3208dd0441023f3ce7a81831.1444091585.git.luto@kernel.org open hereSigned-off-by Ingo Molnar <[email protected]>
Quoted 3 lines
--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)
Just triggered this here on rc5+tip/master, 32-bit. Any ideas?
------------[ cut here ]------------ WARNING: CPU: 1 PID: 1 at /mnt/kernel/kernel/linux-2.6/kernel/locking/lockdep.c:2639 trace_hardirqs_off_caller+0xef/0x150() DEBUG_LOCKS_WARN_ON(!irqs_disabled()) Modules linked in: CPU: 1 PID: 1 Comm: init Tainted: G W 4.3.0-rc5+ #1 Hardware name: LENOVO 30515QG/30515QG, BIOS 8RET30WW (1.12 ) 09/15/2011 00000000 00000000 f44fbf34 c1301072 f44fbf74 f44fbf64 c105658d c1819094 f44fbf90 00000001 c181f838 00000a4f c10a284f c10a284f f4520000 c1662048 00000009 f44fbf7c c10565f3 00000009 f44fbf74 c1819094 f44fbf90 f44fbf9c Call Trace: [<c1301072>] dump_stack+0x4b/0x79 [<c105658d>] warn_slowpath_common+0x8d/0xc0 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c1662048>] ? entry_INT80_32+0x28/0x2f [<c10565f3>] warn_slowpath_fmt+0x33/0x40 [<c10a284f>] trace_hardirqs_off_caller+0xef/0x150 [<c1000f1c>] trace_hardirqs_off_thunk+0xc/0x10 [<c1662048>] ? entry_INT80_32+0x28/0x2f ---[ end trace dd0b2c83c5d36016 ]---
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Re: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
Quoted 28 lines · Borislav Petkov
Quoted 21 lines · tip-bot for Andy Lutomirski
Commit-ID: 150ac78d63afb96360dab448b7b4d33c98c8266c Gitweb: http://git.kernel.org/tip/150ac78d63afb96360dab448b7b4d33c98c8266c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:14 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200x86/entry/32: Switch INT80 to the new C syscall path
Signed-off-by Andy Lutomirski <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Borislav Petkov <[email protected]>Cc Brian Gerst <[email protected]>Cc Denys Vlasenko <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Linus Torvalds <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Link http://lkml.kernel.org/r/a7e8d8df96838eae3208dd0441023f3ce7a81831.1444091585.git.luto@kernel.org open hereSigned-off-by Ingo Molnar <[email protected]>Quoted 3 lines
--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)Just triggered this here on rc5+tip/master, 32-bit. Any ideas?
Quoted 16 lines
------------[ cut here ]------------ WARNING: CPU: 1 PID: 1 at /mnt/kernel/kernel/linux-2.6/kernel/locking/lockdep.c:2639 trace_hardirqs_off_caller+0xef/0x150() DEBUG_LOCKS_WARN_ON(!irqs_disabled()) Modules linked in: CPU: 1 PID: 1 Comm: init Tainted: G W 4.3.0-rc5+ #1 Hardware name: LENOVO 30515QG/30515QG, BIOS 8RET30WW (1.12 ) 09/15/2011 00000000 00000000 f44fbf34 c1301072 f44fbf74 f44fbf64 c105658d c1819094 f44fbf90 00000001 c181f838 00000a4f c10a284f c10a284f f4520000 c1662048 00000009 f44fbf7c c10565f3 00000009 f44fbf74 c1819094 f44fbf90 f44fbf9c Call Trace: [<c1301072>] dump_stack+0x4b/0x79 [<c105658d>] warn_slowpath_common+0x8d/0xc0 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c1662048>] ? entry_INT80_32+0x28/0x2f
Can you turn that entry_INT80_32 address into either a line number of some assembly code? I'm not seeing the code path that could do this, and there are two unlikely choices.
Based on the warning that triggered, I assume you have CONFIG_PROVE_LOCKING=y, which means that the obvious failures that would have blown up like this should also have warned earlier.
--Andy
Borislav Petkov
maintainerRe: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
Quoted 46 lines · Andy Lutomirski
Quoted 28 lines · Borislav Petkov
Quoted 21 lines · tip-bot for Andy Lutomirski
Commit-ID: 150ac78d63afb96360dab448b7b4d33c98c8266c Gitweb: http://git.kernel.org/tip/150ac78d63afb96360dab448b7b4d33c98c8266c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:14 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200x86/entry/32: Switch INT80 to the new C syscall path
Signed-off-by Andy Lutomirski <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Borislav Petkov <[email protected]>Cc Brian Gerst <[email protected]>Cc Denys Vlasenko <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Linus Torvalds <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Link http://lkml.kernel.org/r/a7e8d8df96838eae3208dd0441023f3ce7a81831.1444091585.git.luto@kernel.org open hereSigned-off-by Ingo Molnar <[email protected]>Quoted 3 lines
--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)Just triggered this here on rc5+tip/master, 32-bit. Any ideas?
Quoted 16 lines
------------[ cut here ]------------ WARNING: CPU: 1 PID: 1 at /mnt/kernel/kernel/linux-2.6/kernel/locking/lockdep.c:2639 trace_hardirqs_off_caller+0xef/0x150() DEBUG_LOCKS_WARN_ON(!irqs_disabled()) Modules linked in: CPU: 1 PID: 1 Comm: init Tainted: G W 4.3.0-rc5+ #1 Hardware name: LENOVO 30515QG/30515QG, BIOS 8RET30WW (1.12 ) 09/15/2011 00000000 00000000 f44fbf34 c1301072 f44fbf74 f44fbf64 c105658d c1819094 f44fbf90 00000001 c181f838 00000a4f c10a284f c10a284f f4520000 c1662048 00000009 f44fbf7c c10565f3 00000009 f44fbf74 c1819094 f44fbf90 f44fbf9c Call Trace: [<c1301072>] dump_stack+0x4b/0x79 [<c105658d>] warn_slowpath_common+0x8d/0xc0 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c1662048>] ? entry_INT80_32+0x28/0x2f
Quoted 3 lines
Can you turn that
entry_INT80_32address into either a line number of some assembly code? I'm not seeing the code path that could do this, and there are two unlikely choices.
Why, that's the TRACE_IRQS_OFF at the end of entry_INT80_32. It calls trace_hardirqs_off_caller through the thunk. That's pretty obvious. Or am I misunderstanding you?
c1662020 <entry_INT80_32>:
c1662020: 90 nop c1662021: 90 nop c1662022: 90 nop c1662023: 50 push %eax c1662024: fc cld c1662025: 6a 00 push $0x0 c1662027: 0f a0 push %fs c1662029: 06 push %es c166202a: 1e push %ds c166202b: 6a da push $0xffffffda c166202d: 55 push %ebp c166202e: 57 push %edi c166202f: 56 push %esi c1662030: 52 push %edx c1662031: 51 push %ecx c1662032: 53 push %ebx c1662033: ba 7b 00 00 00 mov $0x7b,%edx c1662038: 8e da mov %edx,%ds c166203a: 8e c2 mov %edx,%es c166203c: ba d8 00 00 00 mov $0xd8,%edx c1662041: 8e e2 mov %edx,%fs c1662043: e8 c8 ee 99 ff call c1000f10 <trace_hardirqs_off_thunk> c1662048: 89 e0 mov %esp,%eax c166204a: e8 61 f9 99 ff call c10019b0 <do_int80_syscall_32>
/*
* User mode is traced as though IRQs are on, and the interrupt gate
* turned them off.
*/
TRACE_IRQS_OFF
Sounds like the gate didn't disable IRQs, right? Or did the irqs_disabled() check get tricked into looking at the wrong flags...? But I don't see it. Hmmm..
Quoted 2 lines
Based on the warning that triggered, I assume you have
CONFIG_PROVE_LOCKING=y,
yes.
Quoted 2 lines
which means that the obvious failures that would have blown up like this should also have warned earlier.
If you mean, earlier in the boot, the only splat before that one is (see the "not tainted").
[ 2.703038] NX-protecting the kernel data: 5740k [ 2.703822] ------------[ cut here ]------------ [ 2.703973] WARNING: CPU: 0 PID: 1 at /mnt/kernel/kernel/linux-2.6/arch/x86/mm/dump_pagetables.c:225 note_page+0x63e/0x800() [ 2.704177] x86/mm: Found insecure W+X mapping at address c00a0000/0xc00a0000 [ 2.704324] Modules linked in: [ 2.704461] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc5+ #1
...
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Re: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
In reply to a message not in the archiveQuoted 9 lines · Borislav Petkov
Quoted 6 lines · Borislav Petkov
Quoted 6 lines
Quoted 6 lines
Just triggered this here on rc5+tip/master, 32-bit. Any ideas?
------------[ cut here ]------------ WARNING: CPU: 1 PID: 1 at /mnt/kernel/kernel/linux-2.6/kernel/locking/lockdep.c:2639 trace_hardirqs_off_caller+0xef/0x150() DEBUG_LOCKS_WARN_ON(!irqs_disabled()) Modules linked in:
Quoted 7 lines
Hmm, something more is b0rked with this. I'm attaching a #GP splat from the intel laptop. Yeah, the box has paniced and the only way to catch it is a photo (how convenient that cameras are ubiquitous nowadays...).
I see
prepare_exit_to_usermodeanddo_signal...Fun stuff :-\
Is this AMD or Intel? If it's Intel, can you see if booting with nosep fixes it?
--Andy
Re: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
In reply to Borislav PetkovQuoted 47 lines · Borislav Petkov
Quoted 46 lines · Andy Lutomirski
Quoted 44 lines · Borislav Petkov
Quoted 21 lines · tip-bot for Andy Lutomirski
Commit-ID: 150ac78d63afb96360dab448b7b4d33c98c8266c Gitweb: http://git.kernel.org/tip/150ac78d63afb96360dab448b7b4d33c98c8266c Author: Andy Lutomirski <[email protected]> AuthorDate: Mon, 5 Oct 2015 17:48:14 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 9 Oct 2015 09:41:10 +0200x86/entry/32: Switch INT80 to the new C syscall path
Signed-off-by Andy Lutomirski <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Borislav Petkov <[email protected]>Cc Brian Gerst <[email protected]>Cc Denys Vlasenko <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Linus Torvalds <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Link http://lkml.kernel.org/r/a7e8d8df96838eae3208dd0441023f3ce7a81831.1444091585.git.luto@kernel.org open hereSigned-off-by Ingo Molnar <[email protected]>Quoted 3 lines
--- arch/x86/entry/entry_32.S | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-)Just triggered this here on rc5+tip/master, 32-bit. Any ideas?
------------[ cut here ]------------ WARNING: CPU: 1 PID: 1 at /mnt/kernel/kernel/linux-2.6/kernel/locking/lockdep.c:2639 trace_hardirqs_off_caller+0xef/0x150() DEBUG_LOCKS_WARN_ON(!irqs_disabled()) Modules linked in: CPU: 1 PID: 1 Comm: init Tainted: G W 4.3.0-rc5+ #1 Hardware name: LENOVO 30515QG/30515QG, BIOS 8RET30WW (1.12 ) 09/15/2011 00000000 00000000 f44fbf34 c1301072 f44fbf74 f44fbf64 c105658d c1819094 f44fbf90 00000001 c181f838 00000a4f c10a284f c10a284f f4520000 c1662048 00000009 f44fbf7c c10565f3 00000009 f44fbf74 c1819094 f44fbf90 f44fbf9c Call Trace: [<c1301072>] dump_stack+0x4b/0x79 [<c105658d>] warn_slowpath_common+0x8d/0xc0 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c10a284f>] ? trace_hardirqs_off_caller+0xef/0x150 [<c1662048>] ? entry_INT80_32+0x28/0x2f
Quoted 7 lines
Quoted 3 lines
Can you turn that
entry_INT80_32address into either a line number of some assembly code? I'm not seeing the code path that could do this, and there are two unlikely choices.Why, that's the
TRACE_IRQS_OFFat the end ofentry_INT80_32. It callstrace_hardirqs_off_callerthrough the thunk. That's pretty obvious. Or am I misunderstanding you?
I was thinking it could also be TRACE_IRQS_IRETQ, but I was wrong (that would be trace_hardirqs_on_caller).
Quoted 34 lines
c1662020<entry_INT80_32>:c1662020: 90 nop c1662021: 90 nop c1662022: 90 nop c1662023: 50 push %eax c1662024: fc cld c1662025: 6a 00 push $0x0 c1662027: 0f a0 push %fs c1662029: 06 push %es c166202a: 1e push %ds c166202b: 6a da push $0xffffffda c166202d: 55 push %ebp c166202e: 57 push %edi c166202f: 56 push %esi c1662030: 52 push %edx c1662031: 51 push %ecx c1662032: 53 push %ebx c1662033: ba 7b 00 00 00 mov $0x7b,%edx c1662038: 8e da mov %edx,%ds c166203a: 8e c2 mov %edx,%es c166203c: ba d8 00 00 00 mov $0xd8,%edx c1662041: 8e e2 mov %edx,%fs c1662043: e8 c8 ee 99 ff call c1000f10 <trace_hardirqs_off_thunk> c1662048: 89 e0 mov %esp,%eax c166204a: e8 61 f9 99 ff call c10019b0 <do_int80_syscall_32>/*
* User mode is traced as though IRQs are on, and the interrupt gate
* turned them off.
*/
TRACE_IRQS_OFF
Quoted 3 lines
Sounds like the gate didn't disable IRQs, right? Or did the
irqs_disabled()check get tricked into looking at the wrong flags...? But I don't see it. Hmmm..
Wow I am incompetent.
set_system_trap_gate(IA32_SYSCALL_VECTOR, entry_INT80_32);
How did I not catch that in testing? Can you change that to set_system_intr_gate and see if that helps?
--Andy
Borislav Petkov
maintainerRe: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
In reply to Andy LutomirskiQuoted 1 line · Andy Lutomirski
Is this AMD or Intel?
Intel.
Quoted 1 line
If it's Intel, can you see if booting with nosep fixes it?
It does.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Borislav Petkov
maintainerRe: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
In reply to Andy LutomirskiQuoted 1 line · Andy Lutomirski
Wow I am incompetent.
Bah, it can happen to anyone...
Quoted 4 lines
set_system_trap_gate(IA32_SYSCALL_VECTOR, entry_INT80_32);How did I not catch that in testing? Can you change that to
set_system_intr_gateand see if that helps?
Yeah, that was it. Well spotted, thanks!
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Re: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
Quoted 11 lines · Borislav Petkov
Quoted 1 line · Andy Lutomirski
Wow I am incompetent.
Bah, it can happen to anyone...
Quoted 4 lines
set_system_trap_gate(IA32_SYSCALL_VECTOR, entry_INT80_32);How did I not catch that in testing? Can you change that to
set_system_intr_gateand see if that helps?Yeah, that was it. Well spotted, thanks!
The INT80 handler doesn't do anything that requires interrupts to be off (it is already on the process stack), so the tracing should be fixed to expect interrupts on. do_int80_syscall_32() can be eliminated too.
--
Brian Gerst
Re: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
Quoted 11 lines · Brian Gerst
Quoted 10 lines · Borislav Petkov
Quoted 1 line · Andy Lutomirski
Wow I am incompetent.
Bah, it can happen to anyone...
Quoted 4 lines
set_system_trap_gate(IA32_SYSCALL_VECTOR, entry_INT80_32);How did I not catch that in testing? Can you change that to
set_system_intr_gateand see if that helps?
Quoted 6 lines
Quoted 1 line
Yeah, that was it. Well spotted, thanks!
The INT80 handler doesn't do anything that requires interrupts to be off (it is already on the process stack), so the tracing should be fixed to expect interrupts on.
do_int80_syscall_32()can be eliminated too.
Good point. Then we blow up in potentially interesting ways if an iopl-using process does int80 with interrupts off. Oh well.
--Andy
Re: [tip:x86/asm] x86/entry/32: Switch INT80 to the new C syscall path
Quoted 14 lines · Andy Lutomirski
Quoted 13 lines · Brian Gerst
Quoted 11 lines · Borislav Petkov
Quoted 1 line · Andy Lutomirski
Wow I am incompetent.
Bah, it can happen to anyone...
Quoted 4 lines
set_system_trap_gate(IA32_SYSCALL_VECTOR, entry_INT80_32);How did I not catch that in testing? Can you change that to
set_system_intr_gateand see if that helps?Yeah, that was it. Well spotted, thanks!
Quoted 7 lines
Quoted 4 lines
The INT80 handler doesn't do anything that requires interrupts to be off (it is already on the process stack), so the tracing should be fixed to expect interrupts on.
do_int80_syscall_32()can be eliminated too.Good point. Then we blow up in potentially interesting ways if an iopl-using process does int80 with interrupts off. Oh well.
The code this replaced ran with interrupts enabled too, so I don't see this as a regression. Usermode drivers could already blow up the system in many different ways. I doubt there were any that actually did a system call while interrupts were disabled.
--
Brian Gerst