x86/cpufeature: Cleanups and improvements v1
Subsystems (13)
Borislav Petkov
maintainercover x86/cpufeature: Cleanups and improvements v1
From: Borislav Petkov <[email protected]>
Ok, here's v1. This time it has been boot-tested on boxes here. I have added patch 6 which needs Andy to look at.
Patchset is ontop of tip/master because Linus master has currently a bug which doesn't really let me do testing on 32-bit:
https://lkml.kernel.org/r/[email protected]
Anyway, please take a look.
Thanks.
Changelog:
v0:
So here's the first version of what we talked about. I've ran randbuilds on it for a night because of patch 1 with all the header reorg and we all know how that can be painful. So this version passes all randbuild configs I had generated.
In talking about patch 1, it is a bit big but if you look at the diffstat, it basically is fixing a bunch of includes and the meat of it being the carving out of the X86_FEATURE_* things to a separate file.
Rest of the patches should be pretty self-explanatory, check the respective commit messages.
I haven't tested it functionally yet (booting in kvm doesn't really count) - thus the RFC tag. I thought I should send it out first so that people can doublecheck me on the direction this is going.
Borislav Petkov (5):
x86/cpufeature: Carve out X86_FEATURE_*
x86/cpufeature: Replace the old static_cpu_has() with safe variant x86/cpufeature: Get rid of the non-asm goto variant x86/alternatives: Add an auxilary section
x86/vdso: Use static_cpu_has()
Brian Gerst (1):
x86/alternatives: Discard dynamic check after init
Documentation/kernel-parameters.txt | 2 +- arch/x86/Kconfig.debug | 10 - arch/x86/boot/cpuflags.h | 2 +- arch/x86/boot/mkcpustr.c | 2 +- arch/x86/crypto/crc32-pclmul_glue.c | 2 +- arch/x86/crypto/crc32c-intel_glue.c | 2 +- arch/x86/crypto/crct10dif-pclmul_glue.c | 2 +- arch/x86/entry/common.c | 1 + arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/vdso/vdso32-setup.c | 1 - arch/x86/entry/vdso/vdso32/system_call.S | 2 +- arch/x86/entry/vdso/vma.c | 3 +- arch/x86/include/asm/alternative.h | 6 - arch/x86/include/asm/apic.h | 1 - arch/x86/include/asm/arch_hweight.h | 2 + arch/x86/include/asm/cmpxchg.h | 1 + arch/x86/include/asm/cpufeature.h | 446 ++----------------------------- arch/x86/include/asm/cpufeatures.h | 288 ++++++++++++++++++++ arch/x86/include/asm/fpu/internal.h | 15 +- arch/x86/include/asm/irq_work.h | 2 +- arch/x86/include/asm/mwait.h | 2 + arch/x86/include/asm/processor.h | 3 +- arch/x86/include/asm/smap.h | 2 +- arch/x86/include/asm/smp.h | 1 - arch/x86/include/asm/thread_info.h | 2 +- arch/x86/include/asm/tlbflush.h | 1 + arch/x86/include/asm/uaccess_64.h | 2 +- arch/x86/kernel/apic/apic_numachip.c | 4 +- arch/x86/kernel/cpu/Makefile | 2 +- arch/x86/kernel/cpu/centaur.c | 2 +- arch/x86/kernel/cpu/common.c | 14 - arch/x86/kernel/cpu/cyrix.c | 1 + arch/x86/kernel/cpu/intel.c | 2 +- arch/x86/kernel/cpu/intel_cacheinfo.c | 2 +- arch/x86/kernel/cpu/match.c | 2 +- arch/x86/kernel/cpu/mkcapflags.sh | 6 +- arch/x86/kernel/cpu/mtrr/main.c | 2 +- arch/x86/kernel/cpu/transmeta.c | 2 +- arch/x86/kernel/e820.c | 1 + arch/x86/kernel/head_32.S | 2 +- arch/x86/kernel/hpet.c | 1 + arch/x86/kernel/msr.c | 2 +- arch/x86/kernel/verify_cpu.S | 2 +- arch/x86/kernel/vm86_32.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 11 + arch/x86/lib/clear_page_64.S | 2 +- arch/x86/lib/copy_page_64.S | 2 +- arch/x86/lib/copy_user_64.S | 2 +- arch/x86/lib/memcpy_64.S | 2 +- arch/x86/lib/memmove_64.S | 2 +- arch/x86/lib/memset_64.S | 2 +- arch/x86/mm/setup_nx.c | 1 + arch/x86/oprofile/op_model_amd.c | 1 - arch/x86/um/asm/barrier.h | 2 +- drivers/cpufreq/intel_pstate.c | 2 +- fs/btrfs/disk-io.c | 2 +- lib/atomic64_test.c | 2 +- 57 files changed, 381 insertions(+), 506 deletions(-) create mode 100644 arch/x86/include/asm/cpufeatures.h
Signature · 1 line
2.3.5
Borislav Petkov
maintainerPATCH 1/6 x86/cpufeature: Carve out X86_FEATURE_*
From: Borislav Petkov <[email protected]>
Move them to a separate header and have the following dependency:
x86/cpufeatures.h <- x86/processor.h <- x86/cpufeature.h
This makes it easier to use the header in asm code and not include the whole cpufeature.h and add guards for asm.
--- Documentation/kernel-parameters.txt | 2 +- arch/x86/boot/cpuflags.h | 2 +- arch/x86/boot/mkcpustr.c | 2 +- arch/x86/crypto/crc32-pclmul_glue.c | 2 +- arch/x86/crypto/crc32c-intel_glue.c | 2 +- arch/x86/crypto/crct10dif-pclmul_glue.c | 2 +- arch/x86/entry/common.c | 1 + arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/vdso/vdso32-setup.c | 1 - arch/x86/entry/vdso/vdso32/system_call.S | 2 +- arch/x86/entry/vdso/vma.c | 1 + arch/x86/include/asm/alternative.h | 6 - arch/x86/include/asm/apic.h | 1 - arch/x86/include/asm/arch_hweight.h | 2 + arch/x86/include/asm/cmpxchg.h | 1 + arch/x86/include/asm/cpufeature.h | 284 +----------------------------- arch/x86/include/asm/cpufeatures.h | 288 +++++++++++++++++++++++++++++++ arch/x86/include/asm/fpu/internal.h | 1 + arch/x86/include/asm/irq_work.h | 2 +- arch/x86/include/asm/mwait.h | 2 + arch/x86/include/asm/processor.h | 3 +- arch/x86/include/asm/smap.h | 2 +- arch/x86/include/asm/smp.h | 1 - arch/x86/include/asm/thread_info.h | 2 +- arch/x86/include/asm/tlbflush.h | 1 + arch/x86/include/asm/uaccess_64.h | 2 +- arch/x86/kernel/cpu/Makefile | 2 +- arch/x86/kernel/cpu/centaur.c | 2 +- arch/x86/kernel/cpu/cyrix.c | 1 + arch/x86/kernel/cpu/intel.c | 2 +- arch/x86/kernel/cpu/intel_cacheinfo.c | 2 +- arch/x86/kernel/cpu/match.c | 2 +- arch/x86/kernel/cpu/mkcapflags.sh | 6 +- arch/x86/kernel/cpu/mtrr/main.c | 2 +- arch/x86/kernel/cpu/transmeta.c | 2 +- arch/x86/kernel/e820.c | 1 + arch/x86/kernel/head_32.S | 2 +- arch/x86/kernel/hpet.c | 1 + arch/x86/kernel/msr.c | 2 +- arch/x86/kernel/verify_cpu.S | 2 +- arch/x86/lib/clear_page_64.S | 2 +- arch/x86/lib/copy_page_64.S | 2 +- arch/x86/lib/copy_user_64.S | 2 +- arch/x86/lib/memcpy_64.S | 2 +- arch/x86/lib/memmove_64.S | 2 +- arch/x86/lib/memset_64.S | 2 +- arch/x86/mm/setup_nx.c | 1 + arch/x86/oprofile/op_model_amd.c | 1 - arch/x86/um/asm/barrier.h | 2 +- lib/atomic64_test.c | 2 +- 50 files changed, 336 insertions(+), 328 deletions(-) create mode 100644 arch/x86/include/asm/cpufeatures.h
Documentation/kernel-parameters.txt +1 -1
@@ -661,7 +661,7 @@ bytes respectively. Such letter suffixes can also be entirely omitted. clearcpuid=BITNUM [X86] Disable CPUID feature X for the kernel. See - arch/x86/include/asm/cpufeature.h for the valid bit + arch/x86/include/asm/cpufeatures.h for the valid bit numbers. Note the Linux specific bits are not necessarily stable over kernel options, but the vendor specific ones should be.
arch/x86/boot/cpuflags.h +1 -1
@@ -1,7 +1,7 @@#ifndef BOOT_CPUFLAGS_H #define BOOT_CPUFLAGS_H -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/processor-flags.h> struct cpu_features {
arch/x86/boot/mkcpustr.c +1 -1
@@ -17,7 +17,7 @@#include "../include/asm/required-features.h" #include "../include/asm/disabled-features.h" -#include "../include/asm/cpufeature.h" +#include "../include/asm/cpufeatures.h" #include "../kernel/cpu/capflags.c" int main (void )
arch/x86/crypto/crc32-pclmul_glue.c +1 -1
@@ -33,7 +33,7 @@#include <linux/crc32.h> #include <crypto/internal/hash.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> #include <asm/fpu/api.h>
arch/x86/crypto/crc32c-intel_glue.c +1 -1
@@ -30,7 +30,7 @@#include <linux/kernel.h> #include <crypto/internal/hash.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> #include <asm/fpu/internal.h>
arch/x86/crypto/crct10dif-pclmul_glue.c +1 -1
@@ -30,7 +30,7 @@#include <linux/string.h> #include <linux/kernel.h> #include <asm/fpu/api.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/cpu_device_id.h> asmlinkage __u16 crc_t10dif_pcl(__u16 crc, const unsigned char *buf,
arch/x86/entry/common.c +1 -0
@@ -26,6 +26,7 @@#include <asm/traps.h> #include <asm/vdso.h> #include <asm/uaccess.h> +#include <asm/cpufeature.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
arch/x86/entry/entry_32.S +1 -1
@@ -40,7 +40,7 @@ #include <asm/processor-flags.h> #include <asm/ftrace.h> #include <asm/irq_vectors.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> #include <asm/asm.h> #include <asm/smap.h>
arch/x86/entry/vdso/vdso32-setup.c +0 -1
@@ -11,7 +11,6 @@#include <linux/kernel.h> #include <linux/mm_types.h> -#include <asm/cpufeature.h> #include <asm/processor.h> #include <asm/vdso.h>
arch/x86/entry/vdso/vdso32/system_call.S +1 -1
@@ -3,7 +3,7 @@ */ #include <asm/dwarf2.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> /*
arch/x86/entry/vdso/vma.c +1 -0
@@ -20,6 +20,7 @@#include <asm/page.h> #include <asm/hpet.h> #include <asm/desc.h> +#include <asm/cpufeature.h> #if defined(CONFIG_X86_64 )unsigned int __read_mostly vdso64_enabled =1 ;
arch/x86/include/asm/alternative.h +0 -6
@@ -152,12 +152,6 @@ static inline int alternatives_text_reserved(void *start, void *end)".popsection" /* -* This must be included *after* the definition of ALTERNATIVE due to -* <asm/arch_hweight.h> - */ -#include <asm/cpufeature.h> - -/* * Alternative instructions for different CPU types or capabilities. * * This allows to use optimized instructions even on generic binary
arch/x86/include/asm/apic.h +0 -1
@@ -6,7 +6,6 @@#include <asm/alternative.h> #include <asm/cpufeature.h> -#include <asm/processor.h> #include <asm/apicdef.h> #include <linux/atomic.h> #include <asm/fixmap.h>
arch/x86/include/asm/arch_hweight.h +2 -0
@@ -1,6 +1,8 @@#ifndef _ASM_X86_HWEIGHT_H#define _ASM_X86_HWEIGHT_H +#include <asm/cpufeatures.h> +#ifdef CONFIG_64BIT /* popcnt %edi, %eax -- redundant REX prefix for alignment */ #define POPCNT32 ".byte 0xf3,0x40,0x0f,0xb8,0xc7"
arch/x86/include/asm/cmpxchg.h +1 -0
@@ -2,6 +2,7 @@#define ASM_X86_CMPXCHG_H #include <linux/compiler.h> +#include <asm/cpufeatures.h> #include <asm/alternative.h> /* Provides LOCK_PREFIX */ /*
arch/x86/include/asm/cpufeature.h +1 -283
290 lines not on this page · raw message
arch/x86/include/asm/cpufeatures.h new file +288 -0
288 lines not on this page · raw message
arch/x86/include/asm/fpu/internal.h +1 -0
@@ -17,6 +17,7 @@#include <asm/user.h> #include <asm/fpu/api.h> #include <asm/fpu/xstate.h> +#include <asm/cpufeature.h> /* * High level FPU state handling functions:
arch/x86/include/asm/irq_work.h +1 -1
@@ -1,7 +1,7 @@#ifndef _ASM_IRQ_WORK_H#define _ASM_IRQ_WORK_H -#include <asm/processor.h> +#include <asm/cpufeature.h> static inline bool arch_irq_work_has_interrupt (void ) {
arch/x86/include/asm/mwait.h +2 -0
@@ -3,6 +3,8 @@#include <linux/sched.h> +#include <asm/cpufeature.h> +#define MWAIT_SUBSTATE_MASK 0xf#define MWAIT_CSTATE_MASK 0xf#define MWAIT_SUBSTATE_SIZE 4
arch/x86/include/asm/processor.h +1 -2
@@ -13,7 +13,7 @@ struct vm86;#include <asm/types.h> #include <uapi/asm/sigcontext.h> #include <asm/current.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/page.h> #include <asm/pgtable_types.h> #include <asm/percpu.h>
@@ -24,7 +24,6 @@ struct vm86;#include <asm/fpu/types.h> #include <linux/personality.h> -#include <linux/cpumask.h> #include <linux/cache.h> #include <linux/threads.h> #include <linux/math64.h>
arch/x86/include/asm/smap.h +1 -1
@@ -15,7 +15,7 @@#include <linux/stringify.h> #include <asm/nops.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> /* "Raw" instruction opcodes */ #define __ASM_CLAC .byte 0x0f,0x01,0xca
arch/x86/include/asm/smp.h +0 -1
@@ -16,7 +16,6 @@#endif #include <asm/thread_info.h> #include <asm/cpumask.h> -#include <asm/cpufeature.h> extern int smp_num_siblings;extern unsigned int num_processors;
arch/x86/include/asm/thread_info.h +1 -1
@@ -49,7 +49,7 @@*/ #ifndef __ASSEMBLY__struct task_struct ; -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <linux/atomic.h> struct thread_info {
arch/x86/include/asm/tlbflush.h +1 -0
@@ -5,6 +5,7 @@#include <linux/sched.h> #include <asm/processor.h> +#include <asm/cpufeature.h> #include <asm/special_insns.h> #ifdef CONFIG_PARAVIRT
arch/x86/include/asm/uaccess_64.h +1 -1
@@ -8,7 +8,7 @@#include <linux/errno.h> #include <linux/lockdep.h> #include <asm/alternative.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/page.h> /*
arch/x86/kernel/cpu/Makefile +1 -1
@@ -64,7 +64,7 @@ ifdef CONFIG_X86_FEATURE_NAMES quiet_cmd_mkcapflags = MKCAP $@ cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $@ -cpufeature = $(src)/../../include/asm/cpufeature.h +cpufeature = $(src)/../../include/asm/cpufeatures.h targets += capflags.c $(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.sh FORCE
arch/x86/kernel/cpu/centaur.c +1 -1
@@ -1,7 +1,7 @@#include <linux/bitops.h> #include <linux/kernel.h> -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <asm/e820.h> #include <asm/mtrr.h> #include <asm/msr.h>
arch/x86/kernel/cpu/cyrix.c +1 -0
@@ -8,6 +8,7 @@#include <linux/timer.h> #include <asm/pci-direct.h> #include <asm/tsc.h> +#include <asm/cpufeature.h> #include "cpu.h"
arch/x86/kernel/cpu/intel.c +1 -1
@@ -8,7 +8,7 @@#include <linux/module.h> #include <linux/uaccess.h> -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <asm/pgtable.h> #include <asm/msr.h> #include <asm/bugs.h>
arch/x86/kernel/cpu/intel_cacheinfo.c +1 -1
@@ -14,7 +14,7 @@#include <linux/sysfs.h> #include <linux/pci.h> -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <asm/amd_nb.h> #include <asm/smp.h>
arch/x86/kernel/cpu/match.c +1 -1
@@ -1,5 +1,5 @@#include <asm/cpu_device_id.h> -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <linux/cpu.h> #include <linux/module.h> #include <linux/slab.h>
arch/x86/kernel/cpu/mkcapflags.sh +3 -3
@@ -1,6 +1,6 @@ #!/bin/sh # -# Generate the x86_cap/bug_flags[] arrays from include/asm/cpufeature.h +# Generate the x86_cap/bug_flags[] arrays from include/asm/cpufeatures.h # IN=$1
@@ -49,8 +49,8 @@ dump_array() trap 'rm "$OUT"' EXIT ( - echo "#ifndef _ASM_X86_CPUFEATURE_H" - echo "#include <asm/cpufeature.h>" + echo "#ifndef _ASM_X86_CPUFEATURES_H" + echo "#include <asm/cpufeatures.h>" echo "#endif" echo ""
arch/x86/kernel/cpu/mtrr/main.c +1 -1
@@ -47,7 +47,7 @@#include <linux/smp.h> #include <linux/syscore_ops.h> -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <asm/e820.h> #include <asm/mtrr.h> #include <asm/msr.h>
arch/x86/kernel/cpu/transmeta.c +1 -1
@@ -1,6 +1,6 @@#include <linux/kernel.h> #include <linux/mm.h> -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <asm/msr.h> #include "cpu.h"
arch/x86/kernel/e820.c +1 -0
@@ -24,6 +24,7 @@#include <asm/e820.h> #include <asm/proto.h> #include <asm/setup.h> +#include <asm/cpufeature.h> /* * The e820 map is the map that gets modified e.g. with command line parameters
arch/x86/kernel/head_32.S +1 -1
@@ -19,7 +19,7 @@ #include <asm/setup.h> #include <asm/processor-flags.h> #include <asm/msr-index.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/percpu.h> #include <asm/nops.h> #include <asm/bootparam.h>
arch/x86/kernel/hpet.c +1 -0
@@ -12,6 +12,7 @@#include <linux/pm.h> #include <linux/io.h> +#include <asm/cpufeature.h> #include <asm/irqdomain.h> #include <asm/fixmap.h> #include <asm/hpet.h>
arch/x86/kernel/msr.c +1 -1
@@ -40,7 +40,7 @@#include <linux/uaccess.h> #include <linux/gfp.h> -#include <asm/processor.h> +#include <asm/cpufeature.h> #include <asm/msr.h> static struct class *msr_class;
arch/x86/kernel/verify_cpu.S +1 -1
@@ -30,7 +30,7 @@ * appropriately. Either display a message or halt. */ -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/msr-index.h> verify_cpu:
arch/x86/lib/clear_page_64.S +1 -1
@@ -1,5 +1,5 @@ #include <linux/linkage.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> /*
arch/x86/lib/copy_page_64.S +1 -1
@@ -1,7 +1,7 @@ /* Written 2003 by Andi Kleen, based on a kernel by Evandro Menezes */ #include <linux/linkage.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> /*
arch/x86/lib/copy_user_64.S +1 -1
@@ -10,7 +10,7 @@ #include <asm/current.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> #include <asm/asm.h> #include <asm/smap.h>
arch/x86/lib/memcpy_64.S +1 -1
@@ -1,7 +1,7 @@ /* Copyright 2002 Andi Kleen */ #include <linux/linkage.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> /*
arch/x86/lib/memmove_64.S +1 -1
@@ -6,7 +6,7 @@ * - Copyright 2011 Fenghua Yu <[email protected]> */ #include <linux/linkage.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> #undef memmove
arch/x86/lib/memset_64.S +1 -1
@@ -1,7 +1,7 @@ /* Copyright 2002 Andi Kleen, SuSE Labs */ #include <linux/linkage.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/alternative-asm.h> .weak memset
arch/x86/mm/setup_nx.c +1 -0
@@ -4,6 +4,7 @@#include <asm/pgtable.h> #include <asm/proto.h> +#include <asm/cpufeature.h> static int disable_nx;
arch/x86/oprofile/op_model_amd.c +0 -1
@@ -24,7 +24,6 @@#include <asm/nmi.h> #include <asm/apic.h> #include <asm/processor.h> -#include <asm/cpufeature.h> #include "op_x86_model.h" #include "op_counter.h"
arch/x86/um/asm/barrier.h +1 -1
@@ -3,7 +3,7 @@#include <asm/asm.h> #include <asm/segment.h> -#include <asm/cpufeature.h> +#include <asm/cpufeatures.h> #include <asm/cmpxchg.h> #include <asm/nops.h>
lib/atomic64_test.c +1 -1
@@ -17,7 +17,7 @@#include <linux/atomic.h> #ifdef CONFIG_X86 -#include <asm/processor.h> /* for boot_cpu_has below */ +#include <asm/cpufeature.h> /* for boot_cpu_has below */ #endif #define TEST (bit, op, c_op, val) \
Signature · 1 line
2.3.5
Borislav Petkov
maintainerPATCH 2/6 x86/cpufeature: Replace the old static_cpu_has() with safe variant
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
So the old one didn't work properly before alternatives had run. And it was supposed to provide an optimized JMP because the assumption was that the offset it is jumping to is within a signed byte and thus a two-byte JMP.
So I did an x86_64 allyesconfig build and dumped all possible sites where static_cpu_has() was used. The optimization amounted to all in all 12(!) places where static_cpu_has() had generated a 2-byte JMP. Which has saved us a whopping 36 bytes!
This clearly is not worth the trouble so we can remove it. The only place where the optimization might count - in __switch_to() - we will handle differently. But that's not subject of this patch.
--- arch/x86/Kconfig.debug | 10 ---- arch/x86/include/asm/cpufeature.h | 100 +++-------------------------------- arch/x86/include/asm/fpu/internal.h | 14 ++--- arch/x86/kernel/apic/apic_numachip.c | 4 +- arch/x86/kernel/cpu/common.c | 12 +---- arch/x86/kernel/vm86_32.c | 2 +- drivers/cpufreq/intel_pstate.c | 2 +- fs/btrfs/disk-io.c | 2 +- 8 files changed, 21 insertions(+), 125 deletions(-)
arch/x86/Kconfig.debug +0 -10
@@ -350,16 +350,6 @@ config DEBUG_IMR_SELFTEST If unsure say N here. -config X86_DEBUG_STATIC_CPU_HAS - bool "Debug alternatives" - depends on DEBUG_KERNEL - ---help--- - This option causes additional code to be generated which - fails if static_cpu_has() is used before alternatives have - run. - - If unsure, say N. - config X86_DEBUG_FPU bool "Debug the x86 FPU code" depends on DEBUG_KERNEL
arch/x86/include/asm/cpufeature.h +7 -93
146 lines not on this page · raw message
arch/x86/include/asm/fpu/internal.h +7 -7
@@ -59,22 +59,22 @@ extern u64 fpu__get_supported_xfeatures_mask(void);*/ static __always_inline __pure bool use_eager_fpu(void) { -return static_cpu_has_safe (X86_FEATURE_EAGER_FPU ); +return static_cpu_has (X86_FEATURE_EAGER_FPU ); } static __always_inline __pure bool use_xsaveopt(void) { -return static_cpu_has_safe (X86_FEATURE_XSAVEOPT ); +return static_cpu_has (X86_FEATURE_XSAVEOPT ); } static __always_inline __pure bool use_xsave(void) { -return static_cpu_has_safe (X86_FEATURE_XSAVE ); +return static_cpu_has (X86_FEATURE_XSAVE ); } static __always_inline __pure bool use_fxsr(void) { -return static_cpu_has_safe (X86_FEATURE_FXSR ); +return static_cpu_has (X86_FEATURE_FXSR ); }/*
@@ -301,7 +301,7 @@ static inline void copy_xregs_to_kernel_booting(struct xregs_state *xstate)WARN_ON (system_state !=SYSTEM_BOOTING ); -if (static_cpu_has_safe (X86_FEATURE_XSAVES )) +if (static_cpu_has (X86_FEATURE_XSAVES ))XSTATE_OP (XSAVES , xstate, lmask, hmask, err);else XSTATE_OP (XSAVE , xstate, lmask, hmask, err);
@@ -323,7 +323,7 @@ static inline void copy_kernel_to_xregs_booting(struct xregs_state *xstate)WARN_ON (system_state !=SYSTEM_BOOTING ); -if (static_cpu_has_safe (X86_FEATURE_XSAVES )) +if (static_cpu_has (X86_FEATURE_XSAVES ))XSTATE_OP (XRSTORS , xstate, lmask, hmask, err);else XSTATE_OP (XRSTOR , xstate, lmask, hmask, err);
@@ -461,7 +461,7 @@ static inline void copy_kernel_to_fpregs(union fpregs_state *fpstate)* pending. Clear the x87 state here by setting it to fixed values. * "m" is a random variable that should be in L1. */ -if (unlikely (static_cpu_has_bug_safe (X86_BUG_FXSAVE_LEAK ))) { +if (unlikely (static_cpu_has_bug (X86_BUG_FXSAVE_LEAK ))) { asmvolatile ("fnclex\n\t" "emms\n\t"
arch/x86/kernel/apic/apic_numachip.c +2 -2
@@ -30,7 +30,7 @@ static unsigned int numachip1_get_apic_id(unsigned long x)unsigned long value;unsigned int id = (x >>24 ) &0xff ; -if (static_cpu_has_safe (X86_FEATURE_NODEID_MSR )) { +if (static_cpu_has (X86_FEATURE_NODEID_MSR )) {rdmsrl (MSR_FAM10H_NODE_ID , value); id |= (value <<2 ) &0xff00 ; }
@@ -178,7 +178,7 @@ static void fixup_cpu_id(struct cpuinfo_x86 *c, int node)this_cpu_write (cpu_llc_id, node);/* Account for nodes per socket in multi-core-module processors */ -if (static_cpu_has_safe (X86_FEATURE_NODEID_MSR )) { +if (static_cpu_has (X86_FEATURE_NODEID_MSR )) {rdmsrl (MSR_FAM10H_NODE_ID , val); nodes = ((val >>3 ) &7 ) +1 ; }
arch/x86/kernel/cpu/common.c +2 -10
@@ -1475,19 +1475,11 @@ void cpu_init(void) }#endif -#ifdef CONFIG_X86_DEBUG_STATIC_CPU_HAS -void warn_pre_alternatives (void ) -{ -WARN (1 ,"You're using static_cpu_has before alternatives have run!\n" ); -} -EXPORT_SYMBOL_GPL (warn_pre_alternatives); -#endif - -inline bool __static_cpu_has_safe (u16 bit) +inline bool __static_cpu_has (u16 bit) {return boot_cpu_has (bit); } -EXPORT_SYMBOL_GPL (__static_cpu_has_safe); +EXPORT_SYMBOL_GPL (__static_cpu_has);static void bsp_resume (void ) {
arch/x86/kernel/vm86_32.c +1 -1
@@ -358,7 +358,7 @@ static long do_sys_vm86(struct vm86plus_struct __user *user_vm86, bool plus)/* make room for real-mode segments */ tsk->thread.sp0 +=16 ; -if (static_cpu_has_safe (X86_FEATURE_SEP )) +if (static_cpu_has (X86_FEATURE_SEP )) tsk->thread.sysenter_cs =0 ;load_sp0 (tss, &tsk->thread);
drivers/cpufreq/intel_pstate.c +1 -1
@@ -1431,7 +1431,7 @@ static int __init intel_pstate_init(void)if (!all_cpu_data)return -ENOMEM ; -if (static_cpu_has_safe (X86_FEATURE_HWP ) && !no_hwp) { +if (static_cpu_has (X86_FEATURE_HWP ) && !no_hwp) {pr_info ("intel_pstate: HWP enabled\n" ); hwp_active++; }
fs/btrfs/disk-io.c +1 -1
@@ -923,7 +923,7 @@ static int check_async_write(struct inode *inode, unsigned long bio_flags)if (bio_flags &EXTENT_BIO_TREE_LOG )return 0 ;#ifdef CONFIG_X86 -if (static_cpu_has_safe (X86_FEATURE_XMM4_2 )) +if (static_cpu_has (X86_FEATURE_XMM4_2 ))return 0 ;#endif return 1 ;
Signature · 1 line
2.3.5
Borislav Petkov
maintainerPATCH 3/6 x86/cpufeature: Get rid of the non-asm goto variant
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
I can simply quote hpa from the mail:
"Get rid of the non-asm goto variant and just fall back to dynamic if asm goto is unavailable. It doesn't make any sense, really, if it is supposed to be safe, and by now the asm goto-capable gcc is in more wide use. (Originally the gcc 3.x fallback to pure dynamic didn't exist, either.)"
Booy, am I lazy.
Cleanup the whole CC_HAVE_ASM_GOTO ifdeffery too, while at it.
--- arch/x86/include/asm/cpufeature.h | 49 ++++----------------------------------- 1 file changed, 5 insertions(+), 44 deletions(-)
arch/x86/include/asm/cpufeature.h +5 -44
@@ -129,17 +129,16 @@ extern const char * const x86_bug_flags[NBUGINTS*32];* fast paths and boot_cpu_has() otherwise! */ -#if __GNUC__ >=4 && defined(CONFIG_X86_FAST_FEATURE_TESTS ) +#if CC_HAVE_ASM_GOTO && defined(CONFIG_X86_FAST_FEATURE_TESTS )extern bool __static_cpu_has (u16 bit);/* * Static testing of CPU features. Used the same as boot_cpu_has(). -* These are only valid after alternatives have run, but will statically -* patch the target code for additional performance. +* These will statically patch the target code for additional +* performance. */ static __always_inline __pure bool _static_cpu_has(u16 bit) { -#ifdef CC_HAVE_ASM_GOTO asm_volatile_goto ("1: jmp %l[t_dynamic]\n" "2:\n" ".skip -(((5f-4f) - (2b-1b)) > 0) * "
@@ -172,45 +171,6 @@ static __always_inline __pure bool _static_cpu_has(u16 bit)return false; t_dynamic:return __static_cpu_has (bit); -#else -u8 flag; -/* Open-coded due to __stringify() in ALTERNATIVE() */ - asmvolatile ("1: movb $2,%0\n" -"2:\n" -".section .altinstructions,\"a\"\n" -" .long 1b - .\n" /* src offset */ -" .long 3f - .\n" /* repl offset */ -" .word %P2\n" /* always replace */ -" .byte 2b - 1b\n" /* source len */ -" .byte 4f - 3f\n" /* replacement len */ -" .byte 0\n" /* pad len */ -".previous\n" -".section .discard,\"aw\",@progbits\n" -" .byte 0xff + (4f-3f) - (2b-1b)\n" /* size check */ -".previous\n" -".section .altinstr_replacement,\"ax\"\n" -"3: movb $0,%0\n" -"4:\n" -".previous\n" -".section .altinstructions,\"a\"\n" -" .long 1b - .\n" /* src offset */ -" .long 5f - .\n" /* repl offset */ -" .word %P1\n" /* feature bit */ -" .byte 4b - 3b\n" /* src len */ -" .byte 6f - 5f\n" /* repl len */ -" .byte 0\n" /* pad len */ -".previous\n" -".section .discard,\"aw\",@progbits\n" -" .byte 0xff + (6f-5f) - (4b-3b)\n" /* size check */ -".previous\n" -".section .altinstr_replacement,\"ax\"\n" -"5: movb $1,%0\n" -"6:\n" -".previous\n" - :"=qm" (flag) - :"i" (bit),"i" (X86_FEATURE_ALWAYS )); -return (flag ==2 ?__static_cpu_has (bit) : flag); -#endif /* CC_HAVE_ASM_GOTO */ }#define static_cpu_has (bit) \
@@ -221,7 +181,8 @@ static __always_inline __pure bool _static_cpu_has(u16 bit) )#else /* -* gcc 3.x is too stupid to do the static test; fall back to dynamic. +* Fall back to dynamic for gcc versions which don't support asm goto. Should be +* a minority now anyway. */ #define static_cpu_has (bit) boot_cpu_has(bit)#endif
Signature · 1 line
2.3.5
Borislav Petkov
maintainerPATCH 4/6 x86/alternatives: Add an auxilary section
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
Add .altinstr_aux for additional instructions which will be used before and/or during patching. All stuff which needs more sophisticated patching should go there. See next patch.
--- arch/x86/kernel/vmlinux.lds.S | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/x86/kernel/vmlinux.lds.S +11 -0
@@ -195,6 +195,17 @@ SECTIONS :init #endif + /* + * Section for code used exclusively before alternatives are run. All + * references to such code must be patched out by alternatives, normally + * by using X86_FEATURE_ALWAYS CPU feature bit. + * + * See static_cpu_has() for an example. + */ + .altinstr_aux : AT(ADDR(.altinstr_aux) - LOAD_OFFSET) { + *(.altinstr_aux) + } + INIT_DATA_SECTION(16) .x86_cpu_dev.init : AT(ADDR(.x86_cpu_dev.init) - LOAD_OFFSET) {
Signature · 1 line
2.3.5
Borislav Petkov
maintainerPATCH 5/6 x86/alternatives: Discard dynamic check after init
In reply to Borislav PetkovFrom: Brian Gerst <[email protected]>
Move the code to do the dynamic check to the altinstr_aux section so that it is discarded after alternatives have run and a static branch has been chosen.
This way we're changing the dynamic branch from C code to assembly, which makes it *substantially* smaller while avoiding a completely unnecessary call to an out of line function.
Boris: change it to do TESTB, as hpa suggests.
--- arch/x86/include/asm/cpufeature.h | 19 ++++++++++++------- arch/x86/kernel/cpu/common.c | 6 ------ 2 files changed, 12 insertions(+), 13 deletions(-)
arch/x86/include/asm/cpufeature.h +12 -7
@@ -130,8 +130,6 @@ extern const char * const x86_bug_flags[NBUGINTS*32];*/ #if CC_HAVE_ASM_GOTO && defined(CONFIG_X86_FAST_FEATURE_TESTS ) -extern bool __static_cpu_has (u16 bit); -/* * Static testing of CPU features. Used the same as boot_cpu_has(). * These will statically patch the target code for additional
@@ -139,7 +137,7 @@ extern bool __static_cpu_has(u16 bit);*/ static __always_inline __pure bool _static_cpu_has(u16 bit) { -asm_volatile_goto ("1: jmp %l[t_dynamic]\n" +asm_volatile_goto ("1: jmp 6f\n" "2:\n" ".skip -(((5f-4f) - (2b-1b)) > 0) * " "((5f-4f) - (2b-1b)),0x90\n"
@@ -164,13 +162,20 @@ static __always_inline __pure bool _static_cpu_has(u16 bit)" .byte 0\n" /* repl len */ " .byte 0\n" /* pad len */ ".previous\n" - : : "i" (bit), "i" (X86_FEATURE_ALWAYS) - : : t_dynamic, t_no); +".section .altinstr_aux,\"ax\"\n" +"6:\n" +" testb %[bitnum],%[cap_byte]\n" +" jnz %l[t_yes]\n" +" jmp %l[t_no]\n" +".previous\n" + : : "i" (bit), "i" (X86_FEATURE_ALWAYS), + [bitnum] "i" (1 << (bit & 7)), + [cap_byte] "m" (((const char *)boot_cpu_data.x86_capability)[bit >> 3]) + : : t_yes, t_no); + t_yes:return true; t_no:return false; - t_dynamic: -return __static_cpu_has (bit); }#define static_cpu_has (bit) \
arch/x86/kernel/cpu/common.c +0 -6
@@ -1475,12 +1475,6 @@ void cpu_init(void) }#endif -inline bool __static_cpu_has (u16 bit) -{ -return boot_cpu_has (bit); -} -EXPORT_SYMBOL_GPL (__static_cpu_has); -static void bsp_resume (void ) {if (this_cpu->c_bsp_resume)
Signature · 1 line
2.3.5
Borislav Petkov
maintainerPATCH 6/6 x86/vdso: Use static_cpu_has()
In reply to Borislav PetkovFrom: Borislav Petkov <[email protected]>
... and simplify and speed up a tad.
--- arch/x86/entry/vdso/vma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/entry/vdso/vma.c +1 -1
@@ -285,7 +285,7 @@ static void vgetcpu_cpu_init(void *arg)#ifdef CONFIG_NUMA node =cpu_to_node (cpu);#endif -if (cpu_has (&cpu_data (cpu),X86_FEATURE_RDTSCP )) +if (static_cpu_has (X86_FEATURE_RDTSCP ))write_rdtscp_aux ((node <<12 ) | cpu);/*
Signature · 1 line
2.3.5
Re: [PATCH 6/6] x86/vdso: Use static_cpu_has()
Quoted 3 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
... and simplify and speed up a tad.
This function is only used when initializing CPUs, so the "tad" is very small indeed.
If there are systems for which some cpus support rdtscp and some don't, then this patch is wrong. Of course, if the bsp has rdtscp and the aps don't, then we're screwed anyway. I left it as cpu_has because this is a cpu init function and it seemed reasonable.
That being said, I have no meaningful objection to this patch.
--Andy
Borislav Petkov
maintainerRe: [PATCH 6/6] x86/vdso: Use static_cpu_has()
We discussed this on IRC, here's the gist:
Quoted 7 lines · Andy Lutomirski
Quoted 3 lines · Borislav Petkov
From: Borislav Petkov <[email protected]>
... and simplify and speed up a tad.
This function is only used when initializing CPUs, so the "tad" is very small indeed.
... except it'll pay out when the branch is patched in. Considering that the majority of the modern CPUs out there - BSP and APs :-) - have RDTSCP, this check will turn into a 5-byte NOP which is the most optimal we can get. Yeah, it is still an init path so called once on each CPU but still.
Quoted 3 lines
If there are systems for which some cpus support rdtscp and some don't, then this patch is wrong. Of course, if the bsp has rdtscp and the aps don't, then we're screwed anyway.
That would be a very odd case.
Quoted 2 lines
I left it as
cpu_hasbecause this is a cpu init function and it seemed reasonable.
Yeah, I see what you mean. But it costs us only the patching and after that we win from not needing for fetch boot_cpu_data anymore on the APs coming up.
Not a panties-dropper speedup but I still think it is worth the trouble.
Quoted 1 line
That being said, I have no meaningful objection to this patch.
Thanks.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.