x86 vdso32 cleanups
Subsystems (5)
cover x86 vdso32 cleanups
This patch set contains several cleanups to the 32-bit VDSO. The main change is to only build one VDSO image, and select the syscall entry point at runtime.
arch/x86/entry/vdso/.gitignore | 4 +---
arch/x86/entry/vdso/Makefile | 53 ++++++++++++++++++++++-------------------------------
arch/x86/entry/vdso/{vdso32 => }/int80.S | 13 +------------
arch/x86/entry/vdso/{vdso32 => }/sigreturn.S | 9 +++++++--
arch/x86/entry/vdso/{vdso32 => }/syscall.S | 23 +++++------------------
arch/x86/entry/vdso/{vdso32 => }/sysenter.S | 19 +++++--------------
arch/x86/entry/vdso/vclock_gettime.c | 31 +++++++++++++++++++++++++++++++
arch/x86/entry/vdso/vdso-note.S | 32 +++++++++++++++++++++++++++++++-
arch/x86/entry/vdso/vdso2c.c | 2 ++
arch/x86/entry/vdso/vdso32-setup.c | 15 ++++++++-------
arch/x86/entry/vdso/{vdso32 => }/vdso32.lds.S | 2 +-
arch/x86/entry/vdso/vdso32/.gitignore | 1 -
arch/x86/entry/vdso/vdso32/note.S | 44 --------------------------------------------
arch/x86/entry/vdso/vdso32/vclock_gettime.c | 30 ------------------------------
arch/x86/entry/vdso/vdso32/vdso-fakesections.c | 1 -
arch/x86/entry/vdso/vma.c | 6 +++---
arch/x86/ia32/ia32_signal.c | 4 ++--
arch/x86/include/asm/elf.h | 3 +--
arch/x86/include/asm/vdso.h | 20 +++++++++++++-------
arch/x86/kernel/signal.c | 4 ++--
arch/x86/xen/setup.c | 13 ++-----------
arch/x86/xen/vdso.h | 4 ----
22 files changed, 137 insertions(+), 196 deletions(-)PATCH 1/7 x86/vdso32: Separate sigreturn code
Compile a separate sigreturn.o instead of including it in the three syscall entry stub files. Use alternatives to patch in a syscall instruction when supported.
--- arch/x86/entry/vdso/Makefile | 3 ++- arch/x86/entry/vdso/vdso32/int80.S | 5 +---- arch/x86/entry/vdso/vdso32/sigreturn.S | 9 +++++++-- arch/x86/entry/vdso/vdso32/syscall.S | 7 +------ arch/x86/entry/vdso/vdso32/sysenter.S | 5 +---- 5 files changed, 12 insertions(+), 17 deletions(-)
arch/x86/entry/vdso/Makefile +2 -1
@@ -140,7 +140,7 @@ override obj-dirs = $(dir $(obj)) $(obj)/vdso32/ targets += vdso32/vdso32.lds targets += vdso32/note.o vdso32/vclock_gettime.o $(vdso32.so-y:%=vdso32/%.o) -targets += vdso32/vclock_gettime.o +targets += vdso32/vclock_gettime.o vdso32/sigreturn.o $(obj)/vdso32.o: $(vdso32-images:%=$(obj)/%)
@@ -163,6 +163,7 @@ $(vdso32-images:%=$(obj)/%.dbg): $(obj)/vdso32-%.so.dbg: FORCE \ $(obj)/vdso32/vdso32.lds \ $(obj)/vdso32/vclock_gettime.o \ $(obj)/vdso32/note.o \ + $(obj)/vdso32/sigreturn.o \ $(obj)/vdso32/%.o $(call if_changed,vdso)
arch/x86/entry/vdso/vdso32/int80.S +1 -4
@@ -1,10 +1,7 @@ /* * Code for the vDSO. This version uses the old int $0x80 method. - * - * First get the common code for the sigreturn entry points. - * This must come first. */ -#include "sigreturn.S" +#include <linux/linkage.h> .text .globl __kernel_vsyscall
arch/x86/entry/vdso/vdso32/sigreturn.S +7 -2
@@ -9,9 +9,14 @@ #include <linux/linkage.h> #include <asm/unistd_32.h> #include <asm/asm-offsets.h> +#include <asm/alternative-asm.h> +#include <asm/cpufeature.h> -#ifndef SYSCALL_ENTER_KERNEL -#define SYSCALL_ENTER_KERNEL int $0x80 +#ifdef CONFIG_COMPAT +#define SYSCALL_ENTER_KERNEL \ + ALTERNATIVE "int $0x80", "syscall", X86_FEATURE_SYSCALL32 +#else +#define SYSCALL_ENTER_KERNEL int $0x80 #endif .text
arch/x86/entry/vdso/vdso32/syscall.S +1 -6
@@ -1,12 +1,7 @@ /* * Code for the vDSO. This version uses the syscall instruction. - * - * First get the common code for the sigreturn entry points. - * This must come first. */ -#define SYSCALL_ENTER_KERNEL syscall -#include "sigreturn.S" - +#include <linux/linkage.h> #include <asm/segment.h> .text
arch/x86/entry/vdso/vdso32/sysenter.S +1 -4
@@ -1,10 +1,7 @@ /* * Code for the vDSO. This version uses the sysenter instruction. - * - * First get the common code for the sigreturn entry points. - * This must come first. */ -#include "sigreturn.S" +#include <linux/linkage.h> /* * The caller puts arg2 in %ecx, which gets pushed. The kernel will use
Signature · 1 line
2.4.3
PATCH 2/7 x86/vdso32: Remove VDSO32_vsyscall_eh_frame_size
In reply to Brian GerstThis symbol and the padding are unnecessary since we no longer rely on the symbols being exactly the same in each variant of the vdso32.
--- arch/x86/entry/vdso/vdso32/int80.S | 8 -------- arch/x86/entry/vdso/vdso32/syscall.S | 8 -------- arch/x86/entry/vdso/vdso32/sysenter.S | 6 ------ 3 files changed, 22 deletions(-)
arch/x86/entry/vdso/vdso32/int80.S +0 -8
@@ -43,11 +43,3 @@ __kernel_vsyscall: .align 4 .LENDFDEDLSI: .previous - - /* - * Pad out the segment to match the size of the sysenter.S version. - */ -VDSO32_vsyscall_eh_frame_size = 0x40 - .section .data,"aw",@progbits - .space VDSO32_vsyscall_eh_frame_size-(.LENDFDEDLSI-.LSTARTFRAMEDLSI), 0 - .previous
arch/x86/entry/vdso/vdso32/syscall.S +0 -8
@@ -60,11 +60,3 @@ __kernel_vsyscall: .align 4 .LENDFDE1: .previous - - /* - * Pad out the segment to match the size of the sysenter.S version. - */ -VDSO32_vsyscall_eh_frame_size = 0x40 - .section .data,"aw",@progbits - .space VDSO32_vsyscall_eh_frame_size-(.LENDFDE1-.LSTARTFRAME), 0 - .previous
arch/x86/entry/vdso/vdso32/sysenter.S +0 -6
@@ -105,9 +105,3 @@ VDSO32_SYSENTER_RETURN: /* Symbol used by sysenter.c via vdso32-syms.h */ .align 4 .LENDFDEDLSI: .previous - - /* - * Emit a symbol with the size of this .eh_frame data, - * to verify it matches the other versions. - */ -VDSO32_vsyscall_eh_frame_size = (.LENDFDEDLSI-.LSTARTFRAMEDLSI)
Signature · 1 line
2.4.3
PATCH 3/7 x86/vdso32: Remove unused vdso-fakesections.c
In reply to Brian Gerst--- arch/x86/entry/vdso/vdso32/vdso-fakesections.c | 1 - 1 file changed, 1 deletion(-) delete mode 100644 arch/x86/entry/vdso/vdso32/vdso-fakesections.c
arch/x86/entry/vdso/vdso32/vdso-fakesections.c deleted +0 -1
@@ -1 +0,0 @@ -#include "../vdso-fakesections.c"
Signature · 1 line
2.4.3
Re: [PATCH 3/7] x86/vdso32: Remove unused vdso-fakesections.c
Quoted 1 line · Brian Gerst
Signed-off-by Brian Gerst <[email protected]>
--Andy
PATCH 4/7 x86/vdso32: Build single vdso32 image
In reply to Brian GerstCurrently, there are three images that are built for vdso32, differing only in the syscall entry code. The syscall entry is a tiny fraction of the total code, so most of the vdso code is duplicated in memory three times. This patch merges all the syscall entry points into one image, and instead of selecting the image, selects the entry point that is placed in the AT_SYSINFO vector and the ELF entry point.
--- arch/x86/entry/vdso/.gitignore | 3 --- arch/x86/entry/vdso/Makefile | 44 ++++++++++++----------------------- arch/x86/entry/vdso/vdso2c.c | 2 ++ arch/x86/entry/vdso/vdso32-setup.c | 15 ++++++------ arch/x86/entry/vdso/vdso32/syscall.S | 8 +++---- arch/x86/entry/vdso/vdso32/sysenter.S | 8 +++---- arch/x86/entry/vdso/vma.c | 6 ++--- arch/x86/ia32/ia32_signal.c | 4 ++-- arch/x86/include/asm/elf.h | 3 +-- arch/x86/include/asm/vdso.h | 11 ++++----- arch/x86/kernel/signal.c | 4 ++-- arch/x86/xen/setup.c | 12 ++-------- 12 files changed, 47 insertions(+), 73 deletions(-)
arch/x86/entry/vdso/.gitignore +0 -3
@@ -1,7 +1,4 @@ vdso.lds vdsox32.lds -vdso32-syscall-syms.lds -vdso32-sysenter-syms.lds -vdso32-int80-syms.lds vdso-image-*.c vdso2c
arch/x86/entry/vdso/Makefile +15 -29
72 lines not on this page · raw message
arch/x86/entry/vdso/vdso2c.c +2 -0
@@ -100,6 +100,8 @@ struct vdso_sym required_syms[] = { {"VDSO32_NOTE_MASK", true}, {"VDSO32_SYSENTER_RETURN", true}, {"__kernel_vsyscall", true}, + {"__kernel_vsyscall_syscall", true}, + {"__kernel_vsyscall_sysenter", true}, {"__kernel_sigreturn", true}, {"__kernel_rt_sigreturn", true}, };
arch/x86/entry/vdso/vdso32-setup.c +8 -7
@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/kernel.h> #include <linux/mm_types.h> +#include <linux/elf.h> #include <asm/cpufeature.h> #include <asm/processor.h>
@@ -60,23 +61,23 @@ __setup_param("vdso=", vdso_setup, vdso32_setup, 0);#endif /* CONFIG_X86_64 */ -#if defined(CONFIG_X86_32 ) || defined(CONFIG_COMPAT ) -const struct vdso_image *selected_vdso32; -#endif +unsigned long selected_vsyscall;int __init sysenter_setup (void ) {#ifdef CONFIG_COMPAT if (vdso32_syscall ()) - selected_vdso32 = &vdso_image_32_syscall; + selected_vsyscall = vdso_image_32.sym___kernel_vsyscall_syscall; else#endif if (vdso32_sysenter ()) - selected_vdso32 = &vdso_image_32_sysenter; + selected_vsyscall = vdso_image_32.sym___kernel_vsyscall_sysenter;else - selected_vdso32 = &vdso_image_32_int80; + selected_vsyscall = vdso_image_32.sym___kernel_vsyscall; + + ((struct elf32_hdr *)vdso_image_32.data)->e_entry = selected_vsyscall; -init_vdso_image (selected_vdso32); +init_vdso_image (&vdso_image_32);return 0 ; }
arch/x86/entry/vdso/vdso32/syscall.S +4 -4
@@ -5,10 +5,10 @@ #include <asm/segment.h> .text - .globl __kernel_vsyscall - .type __kernel_vsyscall,@function + .globl __kernel_vsyscall_syscall + .type __kernel_vsyscall_syscall,@function ALIGN -__kernel_vsyscall: +__kernel_vsyscall_syscall: .LSTART_vsyscall: push %ebp .Lpush_ebp:
@@ -19,7 +19,7 @@ __kernel_vsyscall: .Lpop_ebp: ret .LEND_vsyscall: - .size __kernel_vsyscall,.-.LSTART_vsyscall + .size __kernel_vsyscall_syscall,.-.LSTART_vsyscall .section .eh_frame,"a",@progbits .LSTARTFRAME:
arch/x86/entry/vdso/vdso32/sysenter.S +4 -4
@@ -22,10 +22,10 @@ * three words on the parent stack do not get copied to the child. */ .text - .globl __kernel_vsyscall - .type __kernel_vsyscall,@function + .globl __kernel_vsyscall_sysenter + .type __kernel_vsyscall_sysenter,@function ALIGN -__kernel_vsyscall: +__kernel_vsyscall_sysenter: .LSTART_vsyscall: push %ecx .Lpush_ecx:
@@ -51,7 +51,7 @@ VDSO32_SYSENTER_RETURN: /* Symbol used by sysenter.c via vdso32-syms.h */ .Lpop_ecx: ret .LEND_vsyscall: - .size __kernel_vsyscall,.-.LSTART_vsyscall + .size __kernel_vsyscall_sysenter,.-.LSTART_vsyscall .previous .section .eh_frame,"a",@progbits
arch/x86/entry/vdso/vma.c +3 -3
@@ -185,14 +185,14 @@ static int load_vdso32(void)if (vdso32_enabled !=1 )/* Other values all mean "disabled" */ return 0 ; - ret =map_vdso (selected_vdso32, false); + ret =map_vdso (&vdso_image_32, false);if (ret)return ret; -if (selected_vdso32->sym_VDSO32_SYSENTER_RETURN) +if (vdso_image_32.sym_VDSO32_SYSENTER_RETURN)current_thread_info ()->sysenter_return = current->mm->context.vdso + - selected_vdso32->sym_VDSO32_SYSENTER_RETURN; + vdso_image_32.sym_VDSO32_SYSENTER_RETURN;return 0 ; }
arch/x86/ia32/ia32_signal.c +2 -2
@@ -289,7 +289,7 @@ int ia32_setup_frame(int sig, struct ksignal *ksig,/* Return stub is in 32bit vsyscall page */ if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode; }
@@ -368,7 +368,7 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, restorer = ksig->ka.sa.sa_restorer; else restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;put_user_ex (ptr_to_compat (restorer), &frame->pretcode);/*
arch/x86/include/asm/elf.h +1 -2
@@ -327,8 +327,7 @@ else \#define VDSO_CURRENT_BASE ((unsigned long)current->mm->context.vdso)#define VDSO_ENTRY \ - ((unsigned long)current->mm->context.vdso + \ - selected_vdso32->sym___kernel_vsyscall) + ((unsigned long)current->mm->context.vdso + selected_vsyscall)struct linux_binprm ;
arch/x86/include/asm/vdso.h +4 -7
@@ -26,6 +26,8 @@ struct vdso_image {long sym___kernel_sigreturn;long sym___kernel_rt_sigreturn;long sym___kernel_vsyscall; +long sym___kernel_vsyscall_syscall; +long sym___kernel_vsyscall_sysenter;long sym_VDSO32_SYSENTER_RETURN; };
@@ -38,13 +40,8 @@ extern const struct vdso_image vdso_image_x32;#endif #if definedCONFIG_X86_32 || definedCONFIG_COMPAT -extern const struct vdso_image vdso_image_32_int80; -#ifdef CONFIG_COMPAT -extern const struct vdso_image vdso_image_32_syscall; -#endif -extern const struct vdso_image vdso_image_32_sysenter; - -extern const struct vdso_image *selected_vdso32; +extern const struct vdso_image vdso_image_32; +extern unsigned long selected_vsyscall;#endif extern void __init init_vdso_image (const struct vdso_image *image);
arch/x86/kernel/signal.c +2 -2
@@ -299,7 +299,7 @@ __setup_frame(int sig, struct ksignal *ksig, sigset_t *set,if (current->mm->context.vdso) restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_sigreturn; + vdso_image_32.sym___kernel_sigreturn;else restorer = &frame->retcode;if (ksig->ka.sa.sa_flags &SA_RESTORER )
@@ -363,7 +363,7 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig,/* Set up to return from userspace. */ restorer = current->mm->context.vdso + - selected_vdso32->sym___kernel_rt_sigreturn; + vdso_image_32.sym___kernel_rt_sigreturn;if (ksig->ka.sa.sa_flags &SA_RESTORER ) restorer = ksig->ka.sa.sa_restorer;put_user_ex (restorer, &frame->pretcode);
arch/x86/xen/setup.c +2 -10
@@ -753,17 +753,9 @@ char * __init xen_auto_xlated_memory_setup(void)static void __init fiddle_vdso (void ) {#ifdef CONFIG_X86_32 -/* -* This could be called before selected_vdso32 is initialized, so -* just fiddle with both possible images. vdso_image_32_syscall -* can't be selected, since it only exists on 64-bit systems. -*/ u32 *mask; - mask = vdso_image_32_int80.data + - vdso_image_32_int80.sym_VDSO32_NOTE_MASK; - *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ; - mask = vdso_image_32_sysenter.data + - vdso_image_32_sysenter.sym_VDSO32_NOTE_MASK; + mask = vdso_image_32.data + + vdso_image_32.sym_VDSO32_NOTE_MASK; *mask |=1 <<VDSO_NOTE_NONEGSEG_BIT ;#endif }
Signature · 1 line
2.4.3
PATCH 5/7 x86/vdso: Merge 32-bit and 64-bit source files
In reply to Brian GerstMerge the 32-bit versions of vclock_gettime and note.S into the 64-bit code. Add some make rules to handle the combined code.
--- arch/x86/entry/vdso/Makefile | 10 ++++++- arch/x86/entry/vdso/vclock_gettime.c | 31 ++++++++++++++++++++ arch/x86/entry/vdso/vdso-note.S | 34 +++++++++++++++++++++- arch/x86/entry/vdso/vdso32/note.S | 44 ----------------------------- arch/x86/entry/vdso/vdso32/vclock_gettime.c | 30 -------------------- 5 files changed, 73 insertions(+), 76 deletions(-) delete mode 100644 arch/x86/entry/vdso/vdso32/note.S delete mode 100644 arch/x86/entry/vdso/vdso32/vclock_gettime.c
arch/x86/entry/vdso/Makefile +9 -1
@@ -128,6 +128,7 @@ VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1 override obj-dirs = $(dir $(obj)) $(obj)/vdso32/ KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) +KBUILD_AFLAGS_32 += -DBUILD_VDSO32 $(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32) $(obj)/vdso32.so.dbg: asflags-$(CONFIG_X86_64) += -m32
@@ -140,8 +141,9 @@ KBUILD_CFLAGS_32 += $(call cc-option, -fno-stack-protector) KBUILD_CFLAGS_32 += $(call cc-option, -foptimize-sibling-calls) KBUILD_CFLAGS_32 += -fno-omit-frame-pointer KBUILD_CFLAGS_32 += -DDISABLE_BRANCH_PROFILING +KBUILD_CFLAGS_32 += -DBUILD_VDSO32 -vobjs32-y := vdso32/vclock_gettime.o vdso32/note.o vdso32/sigreturn.o +vobjs32-y := vclock_gettime-32.o vdso-note-32.o vdso32/sigreturn.o vobjs32-y += vdso32/int80.o vdso32/sysenter.o vobjs32-$(CONFIG_COMPAT) += vdso32/syscall.o
@@ -149,6 +151,12 @@ vobjs32 := $(foreach F,$(vobjs32-y),$(obj)/$F) targets += vdso32/vdso32.lds $(vobjs32-y) +$(obj)/%-32.o: $(src)/%.c FORCE + $(call if_changed_dep,cc_o_c) + +$(obj)/%-32.o: $(src)/%.S FORCE + $(call if_changed_dep,as_o_S) + $(obj)/vdso32.so.dbg: KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) $(obj)/vdso32.so.dbg: $(obj)/vdso32/vdso32.lds $(vobjs32) FORCE $(call if_changed,vdso)
arch/x86/entry/vdso/vclock_gettime.c +31 -0
@@ -11,6 +11,37 @@* Check with readelf after changing. */ +#ifdef BUILD_VDSO32 + +#ifndef CONFIG_CC_OPTIMIZE_FOR_SIZE +#undef CONFIG_OPTIMIZE_INLINING +#endif + +#undef CONFIG_X86_PPRO_FENCE + +#ifdef CONFIG_X86_64 + +/* +* in case of a 32 bit VDSO for a 64 bit kernel fake a 32 bit kernel +* configuration +*/ +#undef CONFIG_64BIT +#undef CONFIG_X86_64 +#undef CONFIG_ILLEGAL_POINTER_VALUE +#undef CONFIG_SPARSEMEM_VMEMMAP +#undef CONFIG_NR_CPUS + +#define CONFIG_X86_32 1 +#define CONFIG_PAGE_OFFSET 0 +#define CONFIG_ILLEGAL_POINTER_VALUE 0 +#define CONFIG_NR_CPUS 1 + +#define BUILD_VDSO32_64 + +#endif /* CONFIG_X86_64 */ + +#endif /* BUILD_VDSO32 */ +#include <uapi/linux/time.h> #include <asm/vgtod.h> #include <asm/hpet.h>
arch/x86/entry/vdso/vdso-note.S +33 -1
@@ -3,10 +3,42 @@ * Here we can supply some information useful to userland. */ -#include <linux/uts.h> #include <linux/version.h> #include <linux/elfnote.h> +/* Ideally this would use UTS_NAME, but using a quoted string here + doesn't work. Remember to change this when changing the + kernel's name. */ ELFNOTE_START(Linux, 0, "a") .long LINUX_VERSION_CODE ELFNOTE_END + +#if defined(CONFIG_XEN) && defined(BUILD_VDSO32) +/* + * Add a special note telling glibc's dynamic linker a fake hardware + * flavor that it will use to choose the search path for libraries in the + * same way it uses real hardware capabilities like "mmx". + * We supply "nosegneg" as the fake capability, to indicate that we + * do not like negative offsets in instructions using segment overrides, + * since we implement those inefficiently. This makes it possible to + * install libraries optimized to avoid those access patterns in someplace + * like /lib/i686/tls/nosegneg. Note that an /etc/ld.so.conf.d/file + * corresponding to the bits here is needed to make ldconfig work right. + * It should contain: + * hwcap 1 nosegneg + * to match the mapping of bit to name that we give here. + * + * At runtime, the fake hardware feature will be considered to be present + * if its bit is set in the mask word. So, we start with the mask 0, and + * at boot time we set VDSO_NOTE_NONEGSEG_BIT if running under Xen. + */ + +#include "../../xen/vdso.h" /* Defines VDSO_NOTE_NONEGSEG_BIT. */ + +ELFNOTE_START(GNU, 2, "a") + .long 1 /* ncaps */ +VDSO32_NOTE_MASK: /* Symbol used by arch/x86/xen/setup.c */ + .long 0 /* mask */ + .byte VDSO_NOTE_NONEGSEG_BIT; .asciz "nosegneg" /* bit, name */ +ELFNOTE_END +#endif
arch/x86/entry/vdso/vdso32/note.S deleted +0 -44
44 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/vclock_gettime.c deleted +0 -30
@@ -1,30 +0,0 @@ -#define BUILD_VDSO32 - -#ifndef CONFIG_CC_OPTIMIZE_FOR_SIZE -#undef CONFIG_OPTIMIZE_INLINING -#endif - -#undef CONFIG_X86_PPRO_FENCE - -#ifdef CONFIG_X86_64 - -/* -* in case of a 32 bit VDSO for a 64 bit kernel fake a 32 bit kernel -* configuration -*/ -#undef CONFIG_64BIT -#undef CONFIG_X86_64 -#undef CONFIG_ILLEGAL_POINTER_VALUE -#undef CONFIG_SPARSEMEM_VMEMMAP -#undef CONFIG_NR_CPUS - -#define CONFIG_X86_32 1 -#define CONFIG_PAGE_OFFSET 0 -#define CONFIG_ILLEGAL_POINTER_VALUE 0 -#define CONFIG_NR_CPUS 1 - -#define BUILD_VDSO32_64 - -#endif - -#include "../vclock_gettime.c"
Signature · 1 line
2.4.3
PATCH 6/7 x86/vdso32/xen: Move VDSO_NOTE_NONEGSEG_BIT define
In reply to Brian GerstXen had its own vdso.h just to define VDSO_NOTE_NONEGSEG_BIT. Move it to the main vdso.h.
--- arch/x86/entry/vdso/vdso-note.S | 4 +--- arch/x86/include/asm/vdso.h | 9 +++++++++ arch/x86/xen/setup.c | 1 - arch/x86/xen/vdso.h | 4 ---- 4 files changed, 10 insertions(+), 8 deletions(-) delete mode 100644 arch/x86/xen/vdso.h
arch/x86/entry/vdso/vdso-note.S +1 -3
@@ -5,6 +5,7 @@ #include <linux/version.h> #include <linux/elfnote.h> +#include <asm/vdso.h> /* Ideally this would use UTS_NAME, but using a quoted string here doesn't work. Remember to change this when changing the
@@ -32,9 +33,6 @@ ELFNOTE_END * if its bit is set in the mask word. So, we start with the mask 0, and * at boot time we set VDSO_NOTE_NONEGSEG_BIT if running under Xen. */ - -#include "../../xen/vdso.h" /* Defines VDSO_NOTE_NONEGSEG_BIT. */ - ELFNOTE_START(GNU, 2, "a") .long 1 /* ncaps */ VDSO32_NOTE_MASK: /* Symbol used by arch/x86/xen/setup.c */
arch/x86/include/asm/vdso.h +9 -0
@@ -48,4 +48,13 @@ extern void __init init_vdso_image(const struct vdso_image *image);#endif /* __ASSEMBLER__ */ +#if defined(CONFIG_X86_32 ) || defined(CONFIG_IA32_EMULATION ) +/* +* Bit used for the pseudo-hwcap for non-negative segments. We use +* bit 1 to avoid bugs in some versions of glibc when bit 0 is +* used; the choice is otherwise arbitrary. +*/ +#define VDSO_NOTE_NONEGSEG_BIT 1 +#endif +#endif /* _ASM_X86_VDSO_H */
arch/x86/xen/setup.c +0 -1
@@ -28,7 +28,6 @@#include <xen/interface/physdev.h> #include <xen/features.h> #include "xen-ops.h" -#include "vdso.h" #include "p2m.h" #include "mmu.h"
arch/x86/xen/vdso.h deleted +0 -4
@@ -1,4 +0,0 @@ -/* Bit used for the pseudo-hwcap for non-negative segments. We use -bit 1 to avoid bugs in some versions of glibc when bit 0 is -used; the choice is otherwise arbitrary. */ -#define VDSO_NOTE_NONEGSEG_BIT 1
Signature · 1 line
2.4.3
PATCH 7/7 x86/vdso32: Remove vdso32 subdirectory
In reply to Brian GerstSince the vdso32 subdirectory doesn't have a proper Makefile, it is more difficult to work with. Move the remaining files up one level.
--- arch/x86/entry/vdso/.gitignore | 1 + arch/x86/entry/vdso/Makefile | 14 ++- arch/x86/entry/vdso/int80.S | 45 ++++++++++ arch/x86/entry/vdso/sigreturn.S | 150 ++++++++++++++++++++++++++++++++ arch/x86/entry/vdso/syscall.S | 62 +++++++++++++ arch/x86/entry/vdso/sysenter.S | 107 +++++++++++++++++++++++ arch/x86/entry/vdso/vdso32.lds.S | 37 ++++++++ arch/x86/entry/vdso/vdso32/.gitignore | 1 - arch/x86/entry/vdso/vdso32/int80.S | 45 ---------- arch/x86/entry/vdso/vdso32/sigreturn.S | 150 -------------------------------- arch/x86/entry/vdso/vdso32/syscall.S | 62 ------------- arch/x86/entry/vdso/vdso32/sysenter.S | 107 ----------------------- arch/x86/entry/vdso/vdso32/vdso32.lds.S | 37 -------- 13 files changed, 407 insertions(+), 411 deletions(-) create mode 100644 arch/x86/entry/vdso/int80.S create mode 100644 arch/x86/entry/vdso/sigreturn.S create mode 100644 arch/x86/entry/vdso/syscall.S create mode 100644 arch/x86/entry/vdso/sysenter.S create mode 100644 arch/x86/entry/vdso/vdso32.lds.S delete mode 100644 arch/x86/entry/vdso/vdso32/.gitignore delete mode 100644 arch/x86/entry/vdso/vdso32/int80.S delete mode 100644 arch/x86/entry/vdso/vdso32/sigreturn.S delete mode 100644 arch/x86/entry/vdso/vdso32/syscall.S delete mode 100644 arch/x86/entry/vdso/vdso32/sysenter.S delete mode 100644 arch/x86/entry/vdso/vdso32/vdso32.lds.S
arch/x86/entry/vdso/.gitignore +1 -0
@@ -1,4 +1,5 @@ vdso.lds vdsox32.lds +vdso32.lds vdso-image-*.c vdso2c
arch/x86/entry/vdso/Makefile +5 -9
@@ -123,10 +123,6 @@ $(obj)/vdsox32.so.dbg: $(src)/vdsox32.lds $(vobjx32s) FORCE CPPFLAGS_vdso32.lds = $(CPPFLAGS_vdso.lds) VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1 -# This makes sure the $(obj) subdirectory exists even though vdso32/ -# is not a kbuild sub-make subdirectory. -override obj-dirs = $(dir $(obj)) $(obj)/vdso32/ - KBUILD_AFLAGS_32 := $(filter-out -m64,$(KBUILD_AFLAGS)) KBUILD_AFLAGS_32 += -DBUILD_VDSO32 $(obj)/vdso32.so.dbg: KBUILD_AFLAGS = $(KBUILD_AFLAGS_32)
@@ -143,13 +139,13 @@ KBUILD_CFLAGS_32 += -fno-omit-frame-pointer KBUILD_CFLAGS_32 += -DDISABLE_BRANCH_PROFILING KBUILD_CFLAGS_32 += -DBUILD_VDSO32 -vobjs32-y := vclock_gettime-32.o vdso-note-32.o vdso32/sigreturn.o -vobjs32-y += vdso32/int80.o vdso32/sysenter.o -vobjs32-$(CONFIG_COMPAT) += vdso32/syscall.o +vobjs32-y := vclock_gettime-32.o vdso-note-32.o sigreturn.o +vobjs32-y += int80.o sysenter.o +vobjs32-$(CONFIG_COMPAT) += syscall.o vobjs32 := $(foreach F,$(vobjs32-y),$(obj)/$F) -targets += vdso32/vdso32.lds $(vobjs32-y) +targets += vdso32.lds $(vobjs32-y) $(obj)/%-32.o: $(src)/%.c FORCE $(call if_changed_dep,cc_o_c)
@@ -158,7 +154,7 @@ $(obj)/%-32.o: $(src)/%.S FORCE $(call if_changed_dep,as_o_S) $(obj)/vdso32.so.dbg: KBUILD_CFLAGS = $(KBUILD_CFLAGS_32) -$(obj)/vdso32.so.dbg: $(obj)/vdso32/vdso32.lds $(vobjs32) FORCE +$(obj)/vdso32.so.dbg: $(obj)/vdso32.lds $(vobjs32) FORCE $(call if_changed,vdso) #
arch/x86/entry/vdso/int80.S new file +45 -0
45 lines not on this page · raw message
arch/x86/entry/vdso/sigreturn.S new file +150 -0
150 lines not on this page · raw message
arch/x86/entry/vdso/syscall.S new file +62 -0
62 lines not on this page · raw message
arch/x86/entry/vdso/sysenter.S new file +107 -0
107 lines not on this page · raw message
arch/x86/entry/vdso/vdso32.lds.S new file +37 -0
@@ -0,0 +1,37 @@ +/* + * Linker script for 32-bit vDSO. + * We #include the file to define the layout details. + * + * This file defines the version script giving the user-exported symbols in + * the DSO. + */ + +#include <asm/page.h> + +#define BUILD_VDSO32 + +#include "vdso-layout.lds.S" + +/* The ELF entry point can be used to set the AT_SYSINFO value. */ +ENTRY(__kernel_vsyscall); + +/* + * This controls what userland symbols we export from the vDSO. + */ +VERSION +{ + LINUX_2.6 { + global: + __vdso_clock_gettime; + __vdso_gettimeofday; + __vdso_time; + }; + + LINUX_2.5 { + global: + __kernel_vsyscall; + __kernel_sigreturn; + __kernel_rt_sigreturn; + local: *; + }; +}
arch/x86/entry/vdso/vdso32/.gitignore deleted +0 -1
@@ -1 +0,0 @@ -vdso32.lds
arch/x86/entry/vdso/vdso32/int80.S deleted +0 -45
45 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/sigreturn.S deleted +0 -150
150 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/syscall.S deleted +0 -62
62 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/sysenter.S deleted +0 -107
107 lines not on this page · raw message
arch/x86/entry/vdso/vdso32/vdso32.lds.S deleted +0 -37
@@ -1,37 +0,0 @@ -/* - * Linker script for 32-bit vDSO. - * We #include the file to define the layout details. - * - * This file defines the version script giving the user-exported symbols in - * the DSO. - */ - -#include <asm/page.h> - -#define BUILD_VDSO32 - -#include "../vdso-layout.lds.S" - -/* The ELF entry point can be used to set the AT_SYSINFO value. */ -ENTRY(__kernel_vsyscall); - -/* - * This controls what userland symbols we export from the vDSO. - */ -VERSION -{ - LINUX_2.6 { - global: - __vdso_clock_gettime; - __vdso_gettimeofday; - __vdso_time; - }; - - LINUX_2.5 { - global: - __kernel_vsyscall; - __kernel_sigreturn; - __kernel_rt_sigreturn; - local: *; - }; -}
Signature · 1 line
2.4.3
Quoted 3 lines · Brian Gerst
This patch set contains several cleanups to the 32-bit VDSO. The main change is to only build one VDSO image, and select the syscall entry point at runtime.
Oh no, we have dueling patches!
I have a 2/3 finished series that cleans up the AT_SYSINFO mess differently, as I outlined earlier. I've only done the compat and common bits (no 32-bit native support quite yet), and it enters successfully on Intel using SYSENTER and on (fake) AMD using SYSCALL. The SYSRET bit isn't there yet.
Other than some ifdeffery, the final system_call.S looks like this:
The meat is (sorry for whitespace damage):
.text
.globl __kernel_vsyscall
.type __kernel_vsyscall,@function
ALIGN__kernel_vsyscall:CFI_STARTPROC
/*
* Reshuffle regs so that all of any of the entry instructions
* will preserve enough state.
*/
pushl %edxCFI_ADJUST_CFA_OFFSET 4CFI_REL_OFFSET edx, 0
pushl %ecxCFI_ADJUST_CFA_OFFSET 4CFI_REL_OFFSET ecx, 0
movl %esp, %ecx
#ifdef CONFIG_X86_64
/* If SYSENTER is available, use it. */
ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \
"syscall", X86_FEATURE_SYSCALL32#endif
/* Enter using int $0x80 */
movl (%esp), %ecx
int $0x80GLOBAL(int80_landing_pad)
/* Restore ECX and EDX in case they were clobbered. */ popl %ecxCFI_RESTORE ecxCFI_ADJUST_CFA_OFFSET -4
popl %edxCFI_RESTORE edxCFI_ADJUST_CFA_OFFSET -4
retCFI_ENDPROC
.size __kernel_vsyscall,.-__kernel_vsyscall
.previous
And that's it.
What do you think? This comes with massively cleaned up kernel-side asm as well as a test case that actually validates the CFI directives.
Certainly, a bunch of your patches make sense regardless, and I'll review them and add them to my queue soon.
--Andy
Quoted 63 lines · Andy Lutomirski
Quoted 3 lines · Brian Gerst
This patch set contains several cleanups to the 32-bit VDSO. The main change is to only build one VDSO image, and select the syscall entry point at runtime.
Oh no, we have dueling patches!
I have a 2/3 finished series that cleans up the
AT_SYSINFOmess differently, as I outlined earlier. I've only done the compat and common bits (no 32-bit native support quite yet), and it enters successfully on Intel using SYSENTER and on (fake) AMD using SYSCALL. The SYSRET bit isn't there yet.Other than some ifdeffery, the final
system_call.Slooks like this:The meat is (sorry for whitespace damage):
.text
.globl__kernel_vsyscall
.type __kernel_vsyscall,@function
ALIGN__kernel_vsyscall:CFI_STARTPROC
/*
* Reshuffle regs so that all of any of the entry instructions
* will preserve enough state.
*/
pushl %edxCFI_ADJUST_CFA_OFFSET4CFI_REL_OFFSETedx, 0
pushl %ecxCFI_ADJUST_CFA_OFFSET4CFI_REL_OFFSETecx, 0
movl %esp, %ecx#ifdef
CONFIG_X86_64
/* If SYSENTER is available, use it. */ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ "syscall", X86_FEATURE_SYSCALL32#endif
/* Enter using int $0x80 */
movl (%esp), %ecx
int $0x80GLOBAL(int80_landing_pad)/* Restore ECX and EDX in case they were clobbered. */ popl %ecx
CFI_RESTOREecxCFI_ADJUST_CFA_OFFSET-4
popl %edxCFI_RESTOREedxCFI_ADJUST_CFA_OFFSET-4
retCFI_ENDPROC.size __kernel_vsyscall,.-__kernel_vsyscall
.previousAnd that's it.
Quoted 7 lines
What do you think? This comes with massively cleaned up kernel-side asm as well as a test case that actually validates the CFI directives.
Certainly, a bunch of your patches make sense regardless, and I'll review them and add them to my queue soon.
--Andy
How does the performance compare to the original? Looking at the disassembly, there are two added function calls, and it reloads the args from the stack instead of just shuffling registers.
--
Brian Gerst
Quoted 67 lines · Brian Gerst
Quoted 66 lines · Andy Lutomirski
Quoted 3 lines · Brian Gerst
This patch set contains several cleanups to the 32-bit VDSO. The main change is to only build one VDSO image, and select the syscall entry point at runtime.
Oh no, we have dueling patches!
I have a 2/3 finished series that cleans up the
AT_SYSINFOmess differently, as I outlined earlier. I've only done the compat and common bits (no 32-bit native support quite yet), and it enters successfully on Intel using SYSENTER and on (fake) AMD using SYSCALL. The SYSRET bit isn't there yet.Other than some ifdeffery, the final
system_call.Slooks like this:The meat is (sorry for whitespace damage):
.text
.globl__kernel_vsyscall
.type __kernel_vsyscall,@function
ALIGN__kernel_vsyscall:CFI_STARTPROC
/*
* Reshuffle regs so that all of any of the entry instructions
* will preserve enough state.
*/
pushl %edxCFI_ADJUST_CFA_OFFSET4CFI_REL_OFFSETedx, 0
pushl %ecxCFI_ADJUST_CFA_OFFSET4CFI_REL_OFFSETecx, 0
movl %esp, %ecx#ifdef
CONFIG_X86_64
/* If SYSENTER is available, use it. */ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ "syscall", X86_FEATURE_SYSCALL32#endif
/* Enter using int $0x80 */
movl (%esp), %ecx
int $0x80GLOBAL(int80_landing_pad)/* Restore ECX and EDX in case they were clobbered. */ popl %ecx
CFI_RESTOREecxCFI_ADJUST_CFA_OFFSET-4
popl %edxCFI_RESTOREedxCFI_ADJUST_CFA_OFFSET-4
retCFI_ENDPROC.size __kernel_vsyscall,.-__kernel_vsyscall
.previousAnd that's it.
What do you think? This comes with massively cleaned up kernel-side asm as well as a test case that actually validates the CFI directives.
Quoted 8 lines
Quoted 4 lines
Certainly, a bunch of your patches make sense regardless, and I'll review them and add them to my queue soon.
--Andy
How does the performance compare to the original? Looking at the disassembly, there are two added function calls, and it reloads the args from the stack instead of just shuffling registers.
The replacement is dramatically faster, which means I probably benchmarked it wrong. I'll try again in a day or two.
--Andy
A message that is not in the archive (sent off-list or not yet received): CALCETrVdaDuSuLLByWbFsEei6LYaxZK6qFtJAHZhRu3ZqUc+RQ@mail.gmail.com
Quoted 80 lines · Andy Lutomirski
Quoted 78 lines · Andy Lutomirski
Quoted 75 lines · Brian Gerst
Quoted 63 lines · Andy Lutomirski
Quoted 3 lines · Brian Gerst
This patch set contains several cleanups to the 32-bit VDSO. The main change is to only build one VDSO image, and select the syscall entry point at runtime.
Oh no, we have dueling patches!
I have a 2/3 finished series that cleans up the
AT_SYSINFOmess differently, as I outlined earlier. I've only done the compat and common bits (no 32-bit native support quite yet), and it enters successfully on Intel using SYSENTER and on (fake) AMD using SYSCALL. The SYSRET bit isn't there yet.Other than some ifdeffery, the final
system_call.Slooks like this:The meat is (sorry for whitespace damage):
.text
.globl__kernel_vsyscall
.type __kernel_vsyscall,@function
ALIGN__kernel_vsyscall:CFI_STARTPROC
/*
* Reshuffle regs so that all of any of the entry instructions
* will preserve enough state.
*/
pushl %edxCFI_ADJUST_CFA_OFFSET4CFI_REL_OFFSETedx, 0
pushl %ecxCFI_ADJUST_CFA_OFFSET4CFI_REL_OFFSETecx, 0
movl %esp, %ecx#ifdef
CONFIG_X86_64
/* If SYSENTER is available, use it. */ALTERNATIVE_2 "", "sysenter", X86_FEATURE_SYSENTER32, \ "syscall", X86_FEATURE_SYSCALL32#endif
/* Enter using int $0x80 */
movl (%esp), %ecx
int $0x80GLOBAL(int80_landing_pad)/* Restore ECX and EDX in case they were clobbered. */ popl %ecx
CFI_RESTOREecxCFI_ADJUST_CFA_OFFSET-4
popl %edxCFI_RESTOREedxCFI_ADJUST_CFA_OFFSET-4
retCFI_ENDPROC.size __kernel_vsyscall,.-__kernel_vsyscall
.previousAnd that's it.
Quoted 7 lines
What do you think? This comes with massively cleaned up kernel-side asm as well as a test case that actually validates the CFI directives.
Certainly, a bunch of your patches make sense regardless, and I'll review them and add them to my queue soon.
--Andy
How does the performance compare to the original? Looking at the disassembly, there are two added function calls, and it reloads the args from the stack instead of just shuffling registers.
Quoted 8 lines
Quoted 2 lines
The replacement is dramatically faster, which means I probably benchmarked it wrong. I'll try again in a day or two.
It's enough slower to be problematic. I need to figure out how to trace it properly. (Hmm? Maybe it's time to learn how to get perf on the host to trace a KVM guest.)
Everything is and was hilariously slow with context tracking on. That needs to get fixed, and hopefully once this entry stuff is done someone will do the other end of it.
I got random errors from perf kvm, but I think I found at least part of the issue. The two irqs_disabled() calls in common.c are kind of expensive. I should disable them on non-lockdep kernels.
The context tracking hooks are also too expensive, even when disabled. I should do something to optimize those. Hello, static keys? This doesn't affect syscalls, though.
With context tracking off and the irqs_disabled checks commented out, we're probably doing well enough. We can always tweak the C code and aggressively force inlining if we want a few cycles back.
--Andy
Quoted 11 lines · Andy Lutomirski
I got random errors from perf kvm, but I think I found at least part of the issue. The two
irqs_disabled()calls incommon.care kind of expensive. I should disable them on non-lockdep kernels.The context tracking hooks are also too expensive, even when disabled. I should do something to optimize those. Hello, static keys? This doesn't affect syscalls, though.
With context tracking off and the
irqs_disabledchecks commented out, we're probably doing well enough. We can always tweak the C code and aggressively force inlining if we want a few cycles back.
Currently, a compat AT_SYSINFO syscall (getpid) is 171 cycles for me. With my patches, it's 196 cycles, so it's really not that bad. The impact will probably be slightly worse on native 32-bit because of increased register pressure and because one of the micro-optimizations I threw in are 64-bit specific. We could probably tune the C code a bit more to get a few of the cycles back.
On the flip side, the rewrite is *far* faster in some of the slow path cases because the slow path no longer forces IRET.
On 32-bit, there's the added benefit that we could drop asmlinkage from the syscall bodies on top of the rewrite.
--Andy
Quoted 2 lines
--Andy
Signature · 2 lines
Andy Lutomirski AMA Capital Management, LLC