x86-64: Stack protector and percpu improvements
Subsystems (6)
v2 cover x86-64: Stack protector and percpu improvements
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like most other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
v2:
- Include PVH boot in GSBASE changes.
- Split out removal of 64-bit test script to give full context on why it's not needed anymore.
- Formatting and comment cleanups.
Brian Gerst (11):
x86/stackprotector/32: Remove stack protector test script x86/stackprotector/64: Remove stack protector test script x86/boot: Disable stack protector for early boot code x86/pvh: Use fixed_percpu_data for early boot GSBASE x86/stackprotector/64: Convert stack protector to normal percpu
variable
x86/percpu/64: Remove fixed_percpu_data x86/percpu/64: Use relative percpu offsets x86/boot/64: Remove inverse relocations x86/percpu/64: Remove INIT_PER_CPU macros
percpu: Remove PER_CPU_FIRST_SECTION
kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU
arch/x86/Kconfig | 7 +- arch/x86/Makefile | 19 +-- arch/x86/boot/compressed/misc.c | 14 +-- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/percpu.h | 22 ---- arch/x86/include/asm/processor.h | 28 +---- arch/x86/include/asm/stackprotector.h | 37 ++---- arch/x86/kernel/Makefile | 2 + arch/x86/kernel/asm-offsets_64.c | 6 - arch/x86/kernel/cpu/common.c | 8 +- arch/x86/kernel/head_64.S | 10 +- arch/x86/kernel/irq_64.c | 1 - arch/x86/kernel/setup_percpu.c | 12 +- arch/x86/kernel/vmlinux.lds.S | 35 ------ arch/x86/platform/pvh/head.S | 4 +- arch/x86/tools/relocs.c | 136 +--------------------- arch/x86/xen/xen-head.S | 6 +- include/asm-generic/vmlinux.lds.h | 1 - include/linux/percpu-defs.h | 12 -- init/Kconfig | 11 +- kernel/kallsyms.c | 12 +- scripts/gcc-x86_32-has-stack-protector.sh | 8 -- scripts/gcc-x86_64-has-stack-protector.sh | 4 - scripts/kallsyms.c | 80 +++---------- scripts/link-vmlinux.sh | 4 - 25 files changed, 61 insertions(+), 420 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
Signature · 1 line
2.41.0
PATCH v2 1/11 x86/stackprotector/32: Remove stack protector test script
Test for compiler support directly in Kconfig.
--- arch/x86/Kconfig | 2 +- scripts/gcc-x86_32-has-stack-protector.sh | 8 -------- 2 files changed, 1 insertion(+), 9 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh
arch/x86/Kconfig +1 -1
@@ -411,7 +411,7 @@ config PGTABLE_LEVELS config CC_HAS_SANE_STACKPROTECTOR bool default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT - default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) + default $(cc-option,-mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard) help We have to make sure stack protector is unconditionally disabled if the compiler produces broken code or if it does not let us control
scripts/gcc-x86_32-has-stack-protector.sh deleted +0 -8
@@ -1,8 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -# This requires GCC 8.1 or better. Specifically, we require -# -mstack-protector-guard-reg, added by -# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81708 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m32 -O0 -fstack-protector -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - -o - 2> /dev/null | grep -q "%fs"
Signature · 1 line
2.41.0
Re: [PATCH v2 01/11] x86/stackprotector/32: Remove stack protector test script
Quoted 4 lines · Brian Gerst
Test for compiler support directly in
Kconfig.Signed-off-by Brian Gerst <[email protected]>
Quoted 36 lines
--- arch/x86/Kconfig | 2 +- scripts/gcc-x86_32-has-stack-protector.sh | 8 -------- 2 files changed, 1 insertion(+), 9 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sharch/x86/Kconfig +1
-1@@ -411,7 +411,7 @@ config PGTABLE_LEVELS config CC_HAS_SANE_STACKPROTECTOR bool default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT - default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) + default $(cc-option,-mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard) help We have to make sure stack protector is unconditionally disabled if the compiler produces broken code or if it does not let us controlscripts/gcc-x86_32-has-stack-protector.sh deleted +0
-9@@ -1,8 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -# This requires GCC 8.1 or better. Specifically, we require -# -mstack-protector-guard-reg, added by -# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81708 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m32 -O0 -fstack-protector -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - -o - 2> /dev/null | grep -q "%fs" --2.41.0
PATCH v2 2/11 x86/stackprotector/64: Remove stack protector test script
In reply to Brian GerstThis test for the stack protector was added in 2006 to make sure the compiler had the PR28281 patch applied. With GCC 5.1 being the minimum supported compiler now, it is no longer necessary.
--- arch/x86/Kconfig | 2 +- scripts/gcc-x86_64-has-stack-protector.sh | 4 ---- 2 files changed, 1 insertion(+), 5 deletions(-) delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
arch/x86/Kconfig +1 -1
@@ -410,7 +410,7 @@ config PGTABLE_LEVELS config CC_HAS_SANE_STACKPROTECTOR bool - default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT + default y if 64BIT default $(cc-option,-mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard) help We have to make sure stack protector is unconditionally disabled if
scripts/gcc-x86_64-has-stack-protector.sh deleted +0 -4
@@ -1,4 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m64 -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
Signature · 1 line
2.41.0
Re: [PATCH v2 02/11] x86/stackprotector/64: Remove stack protector test script
Quoted 6 lines · Brian Gerst
This test for the stack protector was added in 2006 to make sure the compiler had the PR28281 patch applied. With GCC 5.1 being the minimum supported compiler now, it is no longer necessary.
Signed-off-by Brian Gerst <[email protected]>
Quoted 32 lines
--- arch/x86/Kconfig | 2 +- scripts/gcc-x86_64-has-stack-protector.sh | 4 ---- 2 files changed, 1 insertion(+), 5 deletions(-) delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sharch/x86/Kconfig +1
-1@@ -410,7 +410,7 @@ config PGTABLE_LEVELS config CC_HAS_SANE_STACKPROTECTOR bool - default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT + default y if 64BIT default $(cc-option,-mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard) help We have to make sure stack protector is unconditionally disabled ifscripts/gcc-x86_64-has-stack-protector.sh deleted +0
-5@@ -1,4 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m64 -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs" --2.41.0
PATCH v2 3/11 x86/boot: Disable stack protector for early boot code
In reply to Brian GerstOn 64-bit, this will prevent crashes when the canary access is changed from %gs:40 to %gs:__stack_chk_guard(%rip). RIP-relative addresses from the identity-mapped early boot code will target the wrong address with zero-based percpu. KASLR could then shift that address to an unmapped page causing a crash on boot.
This early boot code runs well before userspace is active and does not need stack protector enabled.
--- arch/x86/kernel/Makefile | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/Makefile +2 -0
@@ -39,6 +39,8 @@ KMSAN_SANITIZE_nmi.o := n KCOV_INSTRUMENT_head$(BITS).o := n KCOV_INSTRUMENT_sev.o := n +CFLAGS_head32.o := -fno-stack-protector +CFLAGS_head64.o := -fno-stack-protector CFLAGS_irq.o := -I $(srctree)/$(src)/../include/asm/trace obj-y += head_$(BITS).o
Signature · 1 line
2.41.0
PATCH v2 4/11 x86/pvh: Use fixed_percpu_data for early boot GSBASE
In reply to Brian GerstInstead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.
--- arch/x86/platform/pvh/head.S | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/x86/platform/pvh/head.S +3 -4
@@ -96,8 +96,9 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx - mov $_pa(canary), %eax - xor %edx, %edx + lea INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + mov %edx, %eax + shr $32, %rdx wrmsr call xen_prepare_pvh
@@ -156,8 +157,6 @@ SYM_DATA_START_LOCAL(gdt_start) SYM_DATA_END_LABEL(gdt_start, SYM_L_LOCAL, gdt_end) .balign 16 -SYM_DATA_LOCAL(canary, .fill 48, 1, 0) - SYM_DATA_START_LOCAL(early_stack) .fill BOOT_STACK_SIZE, 1, 0 SYM_DATA_END_LABEL(early_stack, SYM_L_LOCAL, early_stack_end)
Signature · 1 line
2.41.0
PATCH v2 5/11 x86/stackprotector/64: Convert stack protector to normal percpu variable
In reply to Brian GerstOlder versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at virtual address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.
Since version 8.1, GCC has options to configure the location of the canary value. This allows the canary to be turned into a normal percpu variable and removes the constraint that the percpu section be zero-based.
--- arch/x86/Kconfig | 5 ++-- arch/x86/Makefile | 19 +++++++++----- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/processor.h | 15 +---------- arch/x86/include/asm/stackprotector.h | 37 +++++---------------------- arch/x86/kernel/asm-offsets_64.c | 6 ----- arch/x86/kernel/cpu/common.c | 4 +-- arch/x86/kernel/head_64.S | 3 +-- arch/x86/xen/xen-head.S | 3 +-- 9 files changed, 26 insertions(+), 68 deletions(-)
arch/x86/Kconfig +2 -3
@@ -410,12 +410,11 @@ config PGTABLE_LEVELS config CC_HAS_SANE_STACKPROTECTOR bool - default y if 64BIT + default $(cc-option,-mstack-protector-guard-reg=gs -mstack-protector-guard-symbol=__stack_chk_guard) if 64BIT default $(cc-option,-mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard) help We have to make sure stack protector is unconditionally disabled if - the compiler produces broken code or if it does not let us control - the segment on 32-bit kernels. + the compiler does not allow control of the segment and symbol. menu "Processor type and features"
arch/x86/Makefile +12 -7
@@ -111,13 +111,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64
@@ -167,6 +161,17 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs +endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) + KBUILD_CFLAGS += -mstack-protector-guard-symbol=__stack_chk_guard + else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif endif #
arch/x86/entry/entry_64.S +1 -1
@@ -190,7 +190,7 @@ SYM_FUNC_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(fixed_percpu_data + FIXED_stack_canary) + movq %rbx, PER_CPU_VAR(__stack_chk_guard) #endif /*
arch/x86/include/asm/processor.h +1 -14
@@ -394,16 +394,7 @@ struct irq_stack {#ifdef CONFIG_X86_64 struct fixed_percpu_data { -/* -* GCC hardcodes the stack canary as %gs:40. Since the -* irq_stack is the object at %gs:0, we reserve the bottom -* 48 bytes of the irq stack for the canary. -* -* Once we are willing to require -mstack-protector-guard-symbol= -* support for x86_64 stackprotector, we can get rid of this. -*/ char gs_base[40 ]; -unsigned long stack_canary; }; DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
@@ -418,11 +409,7 @@ extern asmlinkage void entry_SYSCALL32_ignore(void);/* Save actual FS/GS selectors and bases to current->thread */ void current_save_fsgs (void ); -#else /* X86_64 */ -#ifdef CONFIG_STACKPROTECTOR -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); -#endif -#endif /* !X86_64 */ +#endif /* X86_64 */ struct perf_event ;
arch/x86/include/asm/stackprotector.h +7 -30
@@ -2,26 +2,13 @@/* * GCC stack protector support. * -* Stack protector works by putting predefined pattern at the start of +* Stack protector works by putting a predefined pattern at the start of * the stack frame and verifying that it hasn't been overwritten when -* returning from the function. The pattern is called stack canary -* and unfortunately gcc historically required it to be at a fixed offset -* from the percpu segment base. On x86_64, the offset is 40 bytes. +* returning from the function. The pattern is called the stack canary +* and is a unique value for each task. * -* The same segment is shared by percpu area and stack canary. On -* x86_64, percpu symbols are zero based and %gs (64-bit) points to the -* base of percpu area. The first occupant of the percpu area is always -* fixed_percpu_data which contains stack_canary at the appropriate -* offset. On x86_32, the stack canary is just a regular percpu -* variable. -* -* Putting percpu data in %fs on 32-bit is a minor optimization compared to -* using %gs. Since 32-bit userspace normally has %fs == 0, we are likely -* to load 0 into %fs on exit to usermode, whereas with percpu data in -* %gs, we are likely to load a non-null %gs on return to user mode. -* -* Once we are willing to require GCC 8.1 or better for 64-bit stackprotector -* support, we can remove some of this complexity. +* GCC is configured to read the stack canary value from the __stack_chk_guard +* per-cpu variable, which is changed on task switch. */ #ifndef _ASM_STACKPROTECTOR_H
@@ -36,6 +23,8 @@#include <linux/sched.h> +DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +/* * Initialize the stackprotector canary value. *
@@ -51,25 +40,13 @@ static __always_inline void boot_init_stack_canary(void) {unsigned long canary =get_random_canary (); -#ifdef CONFIG_X86_64 -BUILD_BUG_ON (offsetof(struct fixed_percpu_data , stack_canary) !=40 ); -#endif - current->stack_canary = canary; -#ifdef CONFIG_X86_64 -this_cpu_write (fixed_percpu_data.stack_canary, canary); -#else this_cpu_write (__stack_chk_guard, canary); -#endif }static inline void cpu_init_stack_canary (int cpu,struct task_struct *idle) { -#ifdef CONFIG_X86_64 -per_cpu (fixed_percpu_data.stack_canary, cpu) = idle->stack_canary; -#else per_cpu (__stack_chk_guard, cpu) = idle->stack_canary; -#endif }#else /* STACKPROTECTOR */
arch/x86/kernel/asm-offsets_64.c +0 -6
@@ -54,11 +54,5 @@ int main(void)BLANK ();#undef ENTRY -BLANK (); - -#ifdef CONFIG_STACKPROTECTOR -OFFSET (FIXED_stack_canary, fixed_percpu_data, stack_canary); -BLANK (); -#endif return 0 ; }
arch/x86/kernel/cpu/common.c +1 -3
@@ -2104,15 +2104,13 @@ void syscall_init(void) X86_EFLAGS_AC|X86_EFLAGS_ID); } -#else /* CONFIG_X86_64 */ +#endif /* CONFIG_X86_64 */ #ifdef CONFIG_STACKPROTECTOR DEFINE_PER_CPU(unsigned long, __stack_chk_guard);EXPORT_PER_CPU_SYMBOL (__stack_chk_guard);#endif -#endif /* CONFIG_X86_64 */ -/* * Clear all 6 debug registers: */
arch/x86/kernel/head_64.S +1 -2
@@ -345,8 +345,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */
arch/x86/xen/xen-head.S +1 -2
@@ -53,8 +53,7 @@ SYM_CODE_START(startup_xen) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */
Signature · 1 line
2.41.0
Re: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable
Quoted 9 lines · Brian Gerst
Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at virtual address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.
Quoted 6 lines
Since version 8.1, GCC has options to configure the location of the canary value. This allows the canary to be turned into a normal percpu variable and removes the constraint that the percpu section be zero-based.
Signed-off-by Brian Gerst <[email protected]>
Quoted 254 lines
--- arch/x86/Kconfig | 5 ++-- arch/x86/Makefile | 19 +++++++++----- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/processor.h | 15 +---------- arch/x86/include/asm/stackprotector.h | 37 +++++---------------------- arch/x86/kernel/asm-offsets_64.c | 6 ----- arch/x86/kernel/cpu/common.c | 4 +-- arch/x86/kernel/head_64.S | 3 +-- arch/x86/xen/xen-head.S | 3 +-- 9 files changed, 26 insertions(+), 68 deletions(-)arch/x86/Kconfig +2
-3@@ -410,12 +410,11 @@ config PGTABLE_LEVELS config CC_HAS_SANE_STACKPROTECTOR bool - default y if 64BIT + default $(cc-option,-mstack-protector-guard-reg=gs -mstack-protector-guard-symbol=__stack_chk_guard) if 64BIT default $(cc-option,-mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard) help We have to make sure stack protector is unconditionally disabled if - the compiler produces broken code or if it does not let us control - the segment on 32-bit kernels. + the compiler does not allow control of the segment and symbol. menu "Processor type and features"arch/x86/Makefile +12
-7@@ -111,13 +111,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64@@ -167,6 +161,17 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs +endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) + KBUILD_CFLAGS += -mstack-protector-guard-symbol=__stack_chk_guard + else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif endif #arch/x86/entry/entry_64.S +1
-1@@ -190,7 +190,7 @@ SYM_FUNC_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(fixed_percpu_data + FIXED_stack_canary) + movq %rbx, PER_CPU_VAR(__stack_chk_guard) #endif /*arch/x86/include/asm/processor.h +1
-14@@ -394,16 +394,7 @@ struct irq_stack {#ifdef CONFIG_X86_64 struct fixed_percpu_data { -/* -* GCC hardcodes the stack canary as %gs:40. Since the -* irq_stack is the object at %gs:0, we reserve the bottom -* 48 bytes of the irq stack for the canary. -* -* Once we are willing to require -mstack-protector-guard-symbol= -* support for x86_64 stackprotector, we can get rid of this. -*/ char gs_base[40 ]; -unsigned long stack_canary; }; DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;@@ -418,11 +409,7 @@ extern asmlinkage void entry_SYSCALL32_ignore(void);/* Save actual FS/GS selectors and bases to current->thread */ void current_save_fsgs (void ); -#else /* X86_64 */ -#ifdef CONFIG_STACKPROTECTOR -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); -#endif -#endif /* !X86_64 */ +#endif /* X86_64 */ struct perf_event ;arch/x86/include/asm/stackprotector.h +7
-30@@ -2,26 +2,13 @@/* * GCC stack protector support. * -* Stack protector works by putting predefined pattern at the start of +* Stack protector works by putting a predefined pattern at the start of * the stack frame and verifying that it hasn't been overwritten when -* returning from the function. The pattern is called stack canary -* and unfortunately gcc historically required it to be at a fixed offset -* from the percpu segment base. On x86_64, the offset is 40 bytes. +* returning from the function. The pattern is called the stack canary +* and is a unique value for each task. * -* The same segment is shared by percpu area and stack canary. On -* x86_64, percpu symbols are zero based and %gs (64-bit) points to the -* base of percpu area. The first occupant of the percpu area is always -* fixed_percpu_data which contains stack_canary at the appropriate -* offset. On x86_32, the stack canary is just a regular percpu -* variable. -* -* Putting percpu data in %fs on 32-bit is a minor optimization compared to -* using %gs. Since 32-bit userspace normally has %fs == 0, we are likely -* to load 0 into %fs on exit to usermode, whereas with percpu data in -* %gs, we are likely to load a non-null %gs on return to user mode. -* -* Once we are willing to require GCC 8.1 or better for 64-bit stackprotector -* support, we can remove some of this complexity. +* GCC is configured to read the stack canary value from the __stack_chk_guard +* per-cpu variable, which is changed on task switch. */ #ifndef _ASM_STACKPROTECTOR_H@@ -36,6 +23,8 @@#include <linux/sched.h> +DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +/* * Initialize the stackprotector canary value. * @@ -51,25 +40,13 @@ static __always_inline void boot_init_stack_canary(void) {unsigned long canary =get_random_canary (); -#ifdef CONFIG_X86_64 -BUILD_BUG_ON (offsetof(struct fixed_percpu_data , stack_canary) !=40 ); -#endif - current->stack_canary = canary; -#ifdef CONFIG_X86_64 -this_cpu_write (fixed_percpu_data.stack_canary, canary); -#else this_cpu_write (__stack_chk_guard, canary); -#endif }static inline void cpu_init_stack_canary (int cpu,struct task_struct *idle) { -#ifdef CONFIG_X86_64 -per_cpu (fixed_percpu_data.stack_canary, cpu) = idle->stack_canary; -#else per_cpu (__stack_chk_guard, cpu) = idle->stack_canary; -#endif }#else /* STACKPROTECTOR */ arch/x86/kernel/asm-offsets_64.c +0
-6@@ -54,11 +54,5 @@ int main(void)BLANK ();#undef ENTRY -BLANK (); - -#ifdef CONFIG_STACKPROTECTOR -OFFSET (FIXED_stack_canary, fixed_percpu_data, stack_canary); -BLANK (); -#endif return 0 ; }arch/x86/kernel/cpu/common.c +1
-3@@ -2104,15 +2104,13 @@ void syscall_init(void) X86_EFLAGS_AC|X86_EFLAGS_ID); } -#else /* CONFIG_X86_64 */ +#endif /* CONFIG_X86_64 */ #ifdef CONFIG_STACKPROTECTOR DEFINE_PER_CPU(unsigned long, __stack_chk_guard);EXPORT_PER_CPU_SYMBOL (__stack_chk_guard);#endif -#endif /* CONFIG_X86_64 */ -/* * Clear all 6 debug registers: */ arch/x86/kernel/head_64.S +1
-2@@ -345,8 +345,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */arch/x86/xen/xen-head.S +1
-3@@ -53,8 +53,7 @@ SYM_CODE_START(startup_xen) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ --2.41.0
kernel test robot
build botRe: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable
In reply to Brian Gerstkernel test robot noticed the following build errors:
Hi Brian,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/master]
[also build test ERROR on next-20231027]
[cannot apply to tip/x86/core dennis-percpu/for-next linus/master tip/auto-latest v6.6-rc7] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Brian-Gerst/x86-stackprotector-32-Remove-stack-protector-test-script/20231027-000533 base: tip/master patch link: https://lore.kernel.org/r/20231026160100.195099-6-brgerst%40gmail.com patch subject: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20231029/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231029/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> Unsupported relocation type: unknown type rel type name (42)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable
Quoted 20 lines · kernel test robot
Hi Brian,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/master]
[also build test ERROR on next-20231027]
[cannot apply to tip/x86/core dennis-percpu/for-next linus/master tip/auto-latest v6.6-rc7] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Brian-Gerst/x86-stackprotector-32-Remove-stack-protector-test-script/20231027-000533 base: tip/master patch link: https://lore.kernel.org/r/20231026160100.195099-6-brgerst%40gmail.com patch subject: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20231029/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231029/[email protected]/reproduce)
Quoted 8 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
Unsupported relocation type: unknown type rel type name (42)
Clang is generating a new relocation type (R_X86_64_REX_GOTPCRELX) that the relocs tool doesn't know about. This is supposed to allow
movq __stack_chk_guard@GOTPCREL(%rip), %rax
movq %gs:(%rax), %rax
to be relaxed to
leaq __stack_chk_guard(%rip), %rax
movq %gs:(%rax), %rax
But why is clang doing this instead of what GCC does?
movq %gs:__stack_chk_guard(%rip), %rax
Brian Gerst
Re: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable
Quoted 39 lines · Brian Gerst
Quoted 20 lines · kernel test robot
Hi Brian,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/master]
[also build test ERROR on next-20231027]
[cannot apply to tip/x86/core dennis-percpu/for-next linus/master tip/auto-latest v6.6-rc7] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Brian-Gerst/x86-stackprotector-32-Remove-stack-protector-test-script/20231027-000533 base: tip/master patch link: https://lore.kernel.org/r/20231026160100.195099-6-brgerst%40gmail.com patch subject: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20231029/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231029/[email protected]/reproduce)Quoted 8 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
Unsupported relocation type: unknown type rel type name (42)
Clang is generating a new relocation type (
R_X86_64_REX_GOTPCRELX) that the relocs tool doesn't know about. This is supposed to allow
movq __stack_chk_guard@GOTPCREL(%rip), %rax
movq %gs:(%rax), %rax
to be relaxed to
leaq__stack_chk_guard(%rip), %rax
movq %gs:(%rax), %rax
Quoted 2 lines
But why is clang doing this instead of what GCC does?
movq %gs:__stack_chk_guard(%rip), %rax
Digging a bit deeper, there also appears to be differences in how the linkers behave with this new relocation:
make CC=clang LD=ld:ffffffff81002838: 48 c7 c0 c0 5c 42 83 mov $0xffffffff83425cc0,%rax
ffffffff8100283b: R_X86_64_32S __stack_chk_guard
ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%rax
make CC=clang LD=ld.lld:ffffffff81002838: 48 8d 05 81 34 42 02 lea
0x2423481(%rip),%rax # ffffffff83425cc0 <__stack_chk_guard>
ffffffff8100283b: R_X86_64_REX_GOTPCRELX
__stack_chk_guard-0x4ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%rax
The LLVM linker keeps the R_X86_64_REX_GOTPCRELX even after performing the relaxation. It should be R_X86_64_32S based on it changing to an LEA instruction. The GNU linker changes it to R_X86_64_32S and a MOV immediate.
So I think there are two issues here. 1) clang is producing code referencing the GOT for stack protector accesses, despite -fno-PIE on the command line and no other GOT references, and 2) ld.lld is using the wrong relocation type after the relaxation step is performed.
I think the quick fix here is to teach the relocs tool about this new relocation. It should be able to be safely ignored since it's PC-relative. The code clang produces is functionally correct, although not optimal.
Brian Gerst
Re: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable
Quoted 69 lines · Brian Gerst
Quoted 39 lines · Brian Gerst
Quoted 20 lines · kernel test robot
Hi Brian,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/master]
[also build test ERROR on next-20231027]
[cannot apply to tip/x86/core dennis-percpu/for-next linus/master tip/auto-latest v6.6-rc7] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Brian-Gerst/x86-stackprotector-32-Remove-stack-protector-test-script/20231027-000533 base: tip/master patch link: https://lore.kernel.org/r/20231026160100.195099-6-brgerst%40gmail.com patch subject: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20231029/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231029/[email protected]/reproduce)Quoted 8 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
Unsupported relocation type: unknown type rel type name (42)
Clang is generating a new relocation type (
R_X86_64_REX_GOTPCRELX) that the relocs tool doesn't know about. This is supposed to allow
movq __stack_chk_guard@GOTPCREL(%rip), %rax
movq %gs:(%rax), %rax
to be relaxed to
leaq__stack_chk_guard(%rip), %rax
movq %gs:(%rax), %raxQuoted 2 lines
But why is clang doing this instead of what GCC does?
movq %gs:__stack_chk_guard(%rip), %raxDigging a bit deeper, there also appears to be differences in how the linkers behave with this new relocation:
make CC=clang LD=ld:
ffffffff81002838: 48 c7 c0 c0 5c 42 83 mov $0xffffffff83425cc0,%raxffffffff8100283b: R_X86_64_32S __stack_chk_guard
ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%raxmake CC=clang LD=ld.lld:
ffffffff81002838: 48 8d 05 81 34 42 02 lea
0x2423481(%rip),%rax #ffffffff83425cc0<__stack_chk_guard>ffffffff8100283b: R_X86_64_REX_GOTPCRELX__stack_chk_guard-0x4
ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%raxThe LLVM linker keeps the
R_X86_64_REX_GOTPCRELXeven after performing the relaxation. It should beR_X86_64_32Sbased on it changing to an LEA instruction. The GNU linker changes it toR_X86_64_32Sand a MOV immediate.So I think there are two issues here. 1) clang is producing code referencing the GOT for stack protector accesses, despite -fno-PIE on the command line and no other GOT references, and 2) ld.lld is using the wrong relocation type after the relaxation step is performed.
Quoted 4 lines
I think the quick fix here is to teach the relocs tool about this new relocation. It should be able to be safely ignored since it's PC-relative. The code clang produces is functionally correct, although not optimal.
Thanks for the report. + Fangrui for thoughts on relocations against __stack_chk_guard; clang has similar issues for 32b x86 as well.
Quoted 3 lines
Brian Gerst
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable
Quoted 61 lines · Nick Desaulniers
Quoted 59 lines · Brian Gerst
Quoted 39 lines · Brian Gerst
Quoted 20 lines · kernel test robot
Hi Brian,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/master]
[also build test ERROR on next-20231027]
[cannot apply to tip/x86/core dennis-percpu/for-next linus/master tip/auto-latest v6.6-rc7] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Brian-Gerst/x86-stackprotector-32-Remove-stack-protector-test-script/20231027-000533 base: tip/master patch link: https://lore.kernel.org/r/20231026160100.195099-6-brgerst%40gmail.com patch subject: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20231029/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231029/[email protected]/reproduce)Quoted 8 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
Unsupported relocation type: unknown type rel type name (42)
Clang is generating a new relocation type (
R_X86_64_REX_GOTPCRELX) that the relocs tool doesn't know about. This is supposed to allow
movq __stack_chk_guard@GOTPCREL(%rip), %rax
movq %gs:(%rax), %rax
to be relaxed to
leaq__stack_chk_guard(%rip), %rax
movq %gs:(%rax), %raxQuoted 2 lines
But why is clang doing this instead of what GCC does?
movq %gs:__stack_chk_guard(%rip), %raxDigging a bit deeper, there also appears to be differences in how the linkers behave with this new relocation:
make CC=clang LD=ld:
ffffffff81002838: 48 c7 c0 c0 5c 42 83 mov $0xffffffff83425cc0,%raxffffffff8100283b: R_X86_64_32S __stack_chk_guard
ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%raxmake CC=clang LD=ld.lld:
ffffffff81002838: 48 8d 05 81 34 42 02 lea
0x2423481(%rip),%rax #ffffffff83425cc0<__stack_chk_guard>ffffffff8100283b: R_X86_64_REX_GOTPCRELX__stack_chk_guard-0x4
ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%rax
Quoted 4 lines
Quoted 4 lines
The LLVM linker keeps the
R_X86_64_REX_GOTPCRELXeven after performing the relaxation. It should beR_X86_64_32Sbased on it changing to an LEA instruction. The GNU linker changes it toR_X86_64_32Sand a MOV immediate.
Correction: It should be R_X86_64_PC32 for the LEA instruction.
Brian Gerst
Re: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable
Quoted 46 lines · Brian Gerst
Quoted 43 lines · Nick Desaulniers
Quoted 41 lines · Brian Gerst
Quoted 39 lines · Brian Gerst
Quoted 20 lines · kernel test robot
Hi Brian,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/master]
[also build test ERROR on next-20231027]
[cannot apply to tip/x86/core dennis-percpu/for-next linus/master tip/auto-latest v6.6-rc7] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Brian-Gerst/x86-stackprotector-32-Remove-stack-protector-test-script/20231027-000533 base: tip/master patch link: https://lore.kernel.org/r/20231026160100.195099-6-brgerst%40gmail.com patch subject: [PATCH v2 05/11] x86/stackprotector/64: Convert stack protector to normal percpu variable config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20231029/[email protected]/config) compiler: clang version 16.0.4 (https://github.com/llvm/llvm-project.git ae42196bc493ffe877a7e3dff8be32035dea4d07) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20231029/[email protected]/reproduce)Quoted 8 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
Unsupported relocation type: unknown type rel type name (42)
Clang is generating a new relocation type (
R_X86_64_REX_GOTPCRELX) that the relocs tool doesn't know about. This is supposed to allow
movq __stack_chk_guard@GOTPCREL(%rip), %rax
movq %gs:(%rax), %rax
to be relaxed to
leaq__stack_chk_guard(%rip), %rax
movq %gs:(%rax), %rax
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But why is clang doing this instead of what GCC does?
movq %gs:__stack_chk_guard(%rip), %rax
https://github.com/llvm/llvm-project/issues/60116 has some discussions on this topic.
clang-16 -fno-pic -fstack-protector -mstack-protector-guard-reg=gs -mstack-protector-guard-symbol=__stack_chk_guard
uses a GOT-generating relocation for __stack_chk_guard. This avoids a copy relocation for userspace but the kernel seems to really want an absolute relocation,
so https://reviews.llvm.org/D150841 (milestone: clang 17) has implemented it.
Quoted 17 lines
If an
R_X86_64_32relocation is used and__stack_chk_guardis defined by a shared object, copy relocation. We will need an ELF hack called [copy relocation](https://maskray.me/blog/2021-01-09-copy-relocations-canonical-plt-entries-and-protected).Quoted 15 lines
Quoted 15 lines
Digging a bit deeper, there also appears to be differences in how the linkers behave with this new relocation:
make CC=clang LD=ld:
ffffffff81002838: 48 c7 c0 c0 5c 42 83 mov $0xffffffff83425cc0,%raxffffffff8100283b: R_X86_64_32S __stack_chk_guard
ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%raxmake CC=clang LD=ld.lld:
ffffffff81002838: 48 8d 05 81 34 42 02 lea
0x2423481(%rip),%rax #ffffffff83425cc0<__stack_chk_guard>ffffffff8100283b: R_X86_64_REX_GOTPCRELX__stack_chk_guard-0x4
ffffffff8100283f: 65 48 8b 00 mov %gs:(%rax),%rax
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
The LLVM linker keeps the
R_X86_64_REX_GOTPCRELXeven after performing the relaxation. It should beR_X86_64_32Sbased on it changing to an LEA instruction. The GNU linker changes it toR_X86_64_32Sand a MOV immediate.Correction: It should be
R_X86_64_PC32for the LEA instruction.Brian Gerst
Whether --emit-relocs converts the original relocation type is debatable. I have some comments on a similar topic on RISC-V:
https://sourceware.org/bugzilla/show_bug.cgi?id=30844#c6
Quoted 5 lines
So it seems that ppc performed conversion can all be described by existing relocation types, which is nice.
However, I do not know whether the property will hold for all current and future RISC-V relaxation schemes.
When investigating relocation overflow pressure for x86-64 small code model, I have found that preserving the original relocation type gives me more information: I can tell how
many R_X86_64_PC32/R_X86_64_GOTPCRELX/R_X86_64_REX_GOTPCRELX are problematic. If they are converted to R_X86_64_PC32/R_X86_64_32, I'd lose some information.
Quoted 2 lines
Perhaps whether the --emit-relocs uses the original relocation type or the transformed relocation type , does not matter for the majority of use cases. E.g. Linux kernel's objtool, seems to perform a sanity check on relocations. It just needs to know the categories of relocations, e.g. absolute/PC-relative, not the exact type.
Signature · 1 line
宋方睿
PATCH v2 6/11 x86/percpu/64: Remove fixed_percpu_data
In reply to Brian GerstNow that the stack protector canary value is a normal percpu variable, fixed_percpu_data is unused and can be removed.
--- arch/x86/include/asm/processor.h | 13 +++++-------- arch/x86/kernel/cpu/common.c | 4 ---- arch/x86/kernel/head_64.S | 11 ++++++----- arch/x86/kernel/vmlinux.lds.S | 6 ------ arch/x86/platform/pvh/head.S | 7 ++++++- arch/x86/tools/relocs.c | 1 - arch/x86/xen/xen-head.S | 11 ++++++++--- 7 files changed, 25 insertions(+), 28 deletions(-)
arch/x86/include/asm/processor.h +5 -8
@@ -393,16 +393,13 @@ struct irq_stack { }__aligned(IRQ_STACK_SIZE) ;#ifdef CONFIG_X86_64 -struct fixed_percpu_data { -char gs_base[40 ]; -}; - -DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible; -DECLARE_INIT_PER_CPU (fixed_percpu_data); -static inline unsigned long cpu_kernelmode_gs_base (int cpu) { -return (unsigned long )per_cpu (fixed_percpu_data.gs_base, cpu); +#ifdef CONFIG_SMP +return per_cpu_offset (cpu); +#else +return 0 ; +#endif } extern asmlinkage void entry_SYSCALL32_ignore(void);
arch/x86/kernel/cpu/common.c +0 -4
@@ -2052,10 +2052,6 @@ EXPORT_PER_CPU_SYMBOL(pcpu_hot);EXPORT_PER_CPU_SYMBOL (const_pcpu_hot);#ifdef CONFIG_X86_64 -DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, - fixed_percpu_data)__aligned(PAGE_SIZE) __visible; -EXPORT_PER_CPU_SYMBOL_GPL (fixed_percpu_data); -static void wrmsrl_cstar (unsigned long val) {/*
arch/x86/kernel/head_64.S +6 -5
@@ -72,9 +72,14 @@ SYM_CODE_START_NOALIGN(startup_64) /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#ifdef CONFIG_SMP + leaq __per_cpu_load(%rip), %rdx movl %edx, %eax shrq $32, %rdx +#else + xorl %eax, %eax + xorl %edx, %edx +#endif wrmsr call startup_64_setup_env
@@ -345,14 +350,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifndef CONFIG_SMP - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx -#endif movl %edx, %eax shrq $32, %rdx wrmsr
arch/x86/kernel/vmlinux.lds.S +0 -6
@@ -510,14 +510,8 @@ SECTIONS */ #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load INIT_PER_CPU(gdt_page); -INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); -#ifdef CONFIG_SMP -. = ASSERT((fixed_percpu_data == 0), - "fixed_percpu_data is not at start of per-cpu area"); -#endif - #ifdef CONFIG_CPU_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned"); #endif
arch/x86/platform/pvh/head.S +6 -1
@@ -96,9 +96,14 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx - lea INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#ifdef CONFIG_SMP + lea __per_cpu_load(%rip), %rdx mov %edx, %eax shr $32, %rdx +#else + xor %eax, %eax + xor %edx, %edx +#endif wrmsr call xen_prepare_pvh
arch/x86/tools/relocs.c +0 -1
@@ -811,7 +811,6 @@ static void percpu_init(void)* __per_cpu_load * * The "gold" linker incorrectly associates: -* init_per_cpu__fixed_percpu_data * init_per_cpu__gdt_page */ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname)
arch/x86/xen/xen-head.S +8 -3
@@ -53,13 +53,18 @@ SYM_CODE_START(startup_xen) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax - cdq +#ifdef CONFIG_SMP + leaq __per_cpu_load(%rip), %rdx + movl %edx, %eax + shrq $32, %rdx +#else + xorl %eax, %eax + xorl %edx, %edx +#endif wrmsr mov %rsi, %rdi
Signature · 1 line
2.41.0
Re: [PATCH v2 06/11] x86/percpu/64: Remove fixed_percpu_data
Quoted 5 lines · Brian Gerst
Now that the stack protector canary value is a normal percpu variable,
fixed_percpu_datais unused and can be removed.Signed-off-by Brian Gerst <[email protected]>
Quoted 166 lines
--- arch/x86/include/asm/processor.h | 13 +++++-------- arch/x86/kernel/cpu/common.c | 4 ---- arch/x86/kernel/head_64.S | 11 ++++++----- arch/x86/kernel/vmlinux.lds.S | 6 ------ arch/x86/platform/pvh/head.S | 7 ++++++- arch/x86/tools/relocs.c | 1 - arch/x86/xen/xen-head.S | 11 ++++++++--- 7 files changed, 25 insertions(+), 28 deletions(-)arch/x86/include/asm/processor.h +5
-8@@ -393,16 +393,13 @@ struct irq_stack { }__aligned(IRQ_STACK_SIZE) ;#ifdef CONFIG_X86_64 -struct fixed_percpu_data { -char gs_base[40 ]; -}; - -DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible; -DECLARE_INIT_PER_CPU (fixed_percpu_data); -static inline unsigned long cpu_kernelmode_gs_base (int cpu) { -return (unsigned long )per_cpu (fixed_percpu_data.gs_base, cpu); +#ifdef CONFIG_SMP +return per_cpu_offset (cpu); +#else +return 0 ; +#endif } extern asmlinkage void entry_SYSCALL32_ignore(void);arch/x86/kernel/cpu/common.c +0
-4@@ -2052,10 +2052,6 @@ EXPORT_PER_CPU_SYMBOL(pcpu_hot);EXPORT_PER_CPU_SYMBOL (const_pcpu_hot);#ifdef CONFIG_X86_64 -DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, - fixed_percpu_data)__aligned(PAGE_SIZE) __visible; -EXPORT_PER_CPU_SYMBOL_GPL (fixed_percpu_data); -static void wrmsrl_cstar (unsigned long val) {/* arch/x86/kernel/head_64.S +6
-5@@ -72,9 +72,14 @@ SYM_CODE_START_NOALIGN(startup_64) /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#ifdef CONFIG_SMP + leaq __per_cpu_load(%rip), %rdx movl %edx, %eax shrq $32, %rdx +#else + xorl %eax, %eax + xorl %edx, %edx +#endif wrmsr call startup_64_setup_env@@ -345,14 +350,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifndef CONFIG_SMP - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx -#endif movl %edx, %eax shrq $32, %rdx wrmsrarch/x86/kernel/vmlinux.lds.S +0
-6@@ -510,14 +510,8 @@ SECTIONS */ #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load INIT_PER_CPU(gdt_page); -INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); -#ifdef CONFIG_SMP -. = ASSERT((fixed_percpu_data == 0), - "fixed_percpu_data is not at start of per-cpu area"); -#endif - #ifdef CONFIG_CPU_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned"); #endifarch/x86/platform/pvh/head.S +6
-1@@ -96,9 +96,14 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx - lea INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx +#ifdef CONFIG_SMP + lea __per_cpu_load(%rip), %rdx mov %edx, %eax shr $32, %rdx +#else + xor %eax, %eax + xor %edx, %edx +#endif wrmsr call xen_prepare_pvharch/x86/tools/relocs.c +0
-1@@ -811,7 +811,6 @@ static void percpu_init(void)* __per_cpu_load * * The "gold" linker incorrectly associates: -* init_per_cpu__fixed_percpu_data * init_per_cpu__gdt_page */ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname)arch/x86/xen/xen-head.S +8
-4@@ -53,13 +53,18 @@ SYM_CODE_START(startup_xen) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax - cdq +#ifdef CONFIG_SMP + leaq __per_cpu_load(%rip), %rdx + movl %edx, %eax + shrq $32, %rdx +#else + xorl %eax, %eax + xorl %edx, %edx +#endif wrmsr mov %rsi, %rdi --2.41.0
PATCH v2 7/11 x86/percpu/64: Use relative percpu offsets
In reply to Brian GerstThe percpu section is currently linked at virtual address 0, because older compilers hardcoded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section can be linked normally. This means that x86-64 will calculate percpu offsets like most other architectures, as the delta between the initial percpu address and the dynamically allocated memory.
--- arch/x86/kernel/head_64.S | 6 ------ arch/x86/kernel/setup_percpu.c | 12 ++---------- arch/x86/kernel/vmlinux.lds.S | 24 +----------------------- arch/x86/platform/pvh/head.S | 6 ------ arch/x86/tools/relocs.c | 10 +++------- arch/x86/xen/xen-head.S | 6 ------ init/Kconfig | 2 +- 7 files changed, 7 insertions(+), 59 deletions(-)
arch/x86/kernel/head_64.S +0 -6
@@ -72,14 +72,8 @@ SYM_CODE_START_NOALIGN(startup_64) /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx -#ifdef CONFIG_SMP - leaq __per_cpu_load(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx -#else xorl %eax, %eax xorl %edx, %edx -#endif wrmsr call startup_64_setup_env
arch/x86/kernel/setup_percpu.c +2 -10
@@ -23,18 +23,10 @@#include <asm/cpumask.h> #include <asm/cpu.h> -#ifdef CONFIG_X86_64 -#define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load) -#else -#define BOOT_PERCPU_OFFSET 0 -#endif - -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off) = BOOT_PERCPU_OFFSET; +DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off);EXPORT_PER_CPU_SYMBOL (this_cpu_off); -unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init = { - [0 ...NR_CPUS -1 ] =BOOT_PERCPU_OFFSET , -}; +unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init ;EXPORT_SYMBOL (__per_cpu_offset);/*
arch/x86/kernel/vmlinux.lds.S +1 -23
@@ -103,12 +103,6 @@ const_pcpu_hot = pcpu_hot; PHDRS { text PT_LOAD FLAGS(5); /* R_E */ data PT_LOAD FLAGS(6); /* RW_ */ -#ifdef CONFIG_X86_64 -#ifdef CONFIG_SMP - percpu PT_LOAD FLAGS(6); /* RW_ */ -#endif - init PT_LOAD FLAGS(7); /* RWE */ -#endif note PT_NOTE FLAGS(0); /* ___ */ }
@@ -224,21 +218,7 @@ SECTIONS __init_begin = .; /* paired with __init_end */ } -#if defined(CONFIG_X86_64) && defined(CONFIG_SMP) - /* - * percpu offsets are zero-based on SMP. PERCPU_VADDR() changes the - * output PHDR, so the next output section - .init.text - should - * start another segment - init. - */ - PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu) - ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START, - "per-CPU data too large - increase CONFIG_PHYSICAL_START") -#endif - INIT_TEXT_SECTION(PAGE_SIZE) -#ifdef CONFIG_X86_64 - :init -#endif /* * Section for code used exclusively before alternatives are run. All
@@ -368,9 +348,7 @@ SECTIONS EXIT_DATA } -#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP) PERCPU_SECTION(INTERNODE_CACHE_BYTES) -#endif . = ALIGN(PAGE_SIZE);
@@ -508,7 +486,7 @@ SECTIONS * Per-cpu symbols which need to be offset from __per_cpu_load * for the boot processor. */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load +#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) INIT_PER_CPU(gdt_page); INIT_PER_CPU(irq_stack_backing_store);
arch/x86/platform/pvh/head.S +0 -6
@@ -96,14 +96,8 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx -#ifdef CONFIG_SMP - lea __per_cpu_load(%rip), %rdx - mov %edx, %eax - shr $32, %rdx -#else xor %eax, %eax xor %edx, %edx -#endif wrmsr call xen_prepare_pvh
arch/x86/tools/relocs.c +3 -7
@@ -815,12 +815,7 @@ static void percpu_init(void)*/ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname) { -int shndx =sym_index (sym); - -return (shndx == per_cpu_shndx) && -strcmp (symname,"__init_begin" ) && -strcmp (symname,"__per_cpu_load" ) && -strncmp (symname,"init_per_cpu_" ,13 ); +return 0 ; }
@@ -1043,7 +1038,8 @@ static int cmp_relocs(const void *va, const void *vb)static void sort_relocs (struct relocs *r) { -qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); +if (r->count) +qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); }static int write32 (uint32_t v,FILE *f)
arch/x86/xen/xen-head.S +0 -6
@@ -57,14 +57,8 @@ SYM_CODE_START(startup_xen) * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifdef CONFIG_SMP - leaq __per_cpu_load(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx -#else xorl %eax, %eax xorl %edx, %edx -#endif wrmsr mov %rsi, %rdi
init/Kconfig +1 -1
@@ -1718,7 +1718,7 @@ config KALLSYMS_ALL config KALLSYMS_ABSOLUTE_PERCPU bool depends on KALLSYMS - default X86_64 && SMP + default n config KALLSYMS_BASE_RELATIVE bool
Signature · 1 line
2.41.0
Re: [PATCH v2 07/11] x86/percpu/64: Use relative percpu offsets
Quoted 8 lines · Brian Gerst
The percpu section is currently linked at virtual address 0, because older compilers hardcoded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section can be linked normally. This means that x86-64 will calculate percpu offsets like most other architectures, as the delta between the initial percpu address and the dynamically allocated memory.
The comments above MSR_GS_BASE setup should be reviewed or removed. I don't think they need to be set up to access stack canary, they are just clearing MSR now.
Quoted 2 lines
Signed-off-by Brian Gerst <[email protected]>
With fixed comments:
Quoted 194 lines
--- arch/x86/kernel/head_64.S | 6 ------ arch/x86/kernel/setup_percpu.c | 12 ++---------- arch/x86/kernel/vmlinux.lds.S | 24 +----------------------- arch/x86/platform/pvh/head.S | 6 ------ arch/x86/tools/relocs.c | 10 +++------- arch/x86/xen/xen-head.S | 6 ------ init/Kconfig | 2 +- 7 files changed, 7 insertions(+), 59 deletions(-)arch/x86/kernel/head_64.S +0
-6@@ -72,14 +72,8 @@ SYM_CODE_START_NOALIGN(startup_64) /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx -#ifdef CONFIG_SMP - leaq __per_cpu_load(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx -#else xorl %eax, %eax xorl %edx, %edx -#endif wrmsr call startup_64_setup_envarch/x86/kernel/setup_percpu.c +2
-10@@ -23,18 +23,10 @@#include <asm/cpumask.h> #include <asm/cpu.h> -#ifdef CONFIG_X86_64 -#define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load) -#else -#define BOOT_PERCPU_OFFSET 0 -#endif - -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off) = BOOT_PERCPU_OFFSET; +DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off);EXPORT_PER_CPU_SYMBOL (this_cpu_off); -unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init = { - [0 ...NR_CPUS -1 ] =BOOT_PERCPU_OFFSET , -}; +unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init ;EXPORT_SYMBOL (__per_cpu_offset);/* arch/x86/kernel/vmlinux.lds.S +1
-23@@ -103,12 +103,6 @@ const_pcpu_hot = pcpu_hot; PHDRS { text PT_LOAD FLAGS(5); /* R_E */ data PT_LOAD FLAGS(6); /* RW_ */ -#ifdef CONFIG_X86_64 -#ifdef CONFIG_SMP - percpu PT_LOAD FLAGS(6); /* RW_ */ -#endif - init PT_LOAD FLAGS(7); /* RWE */ -#endif note PT_NOTE FLAGS(0); /* ___ */ }@@ -224,21 +218,7 @@ SECTIONS __init_begin = .; /* paired with __init_end */ } -#if defined(CONFIG_X86_64) && defined(CONFIG_SMP) - /* - * percpu offsets are zero-based on SMP. PERCPU_VADDR() changes the - * output PHDR, so the next output section - .init.text - should - * start another segment - init. - */ - PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu) - ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START, - "per-CPU data too large - increase CONFIG_PHYSICAL_START") -#endif - INIT_TEXT_SECTION(PAGE_SIZE) -#ifdef CONFIG_X86_64 - :init -#endif /* * Section for code used exclusively before alternatives are run. All@@ -368,9 +348,7 @@ SECTIONS EXIT_DATA } -#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP) PERCPU_SECTION(INTERNODE_CACHE_BYTES) -#endif . = ALIGN(PAGE_SIZE);@@ -508,7 +486,7 @@ SECTIONS * Per-cpu symbols which need to be offset from __per_cpu_load * for the boot processor. */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load +#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) INIT_PER_CPU(gdt_page); INIT_PER_CPU(irq_stack_backing_store);arch/x86/platform/pvh/head.S +0
-6@@ -96,14 +96,8 @@ SYM_CODE_START_LOCAL(pvh_start_xen) 1: /* Set base address in stack canary descriptor. */ mov $MSR_GS_BASE,%ecx -#ifdef CONFIG_SMP - lea __per_cpu_load(%rip), %rdx - mov %edx, %eax - shr $32, %rdx -#else xor %eax, %eax xor %edx, %edx -#endif wrmsr call xen_prepare_pvharch/x86/tools/relocs.c +3
-7@@ -815,12 +815,7 @@ static void percpu_init(void)*/ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname) { -int shndx =sym_index (sym); - -return (shndx == per_cpu_shndx) && -strcmp (symname,"__init_begin" ) && -strcmp (symname,"__per_cpu_load" ) && -strncmp (symname,"init_per_cpu_" ,13 ); +return 0 ; }@@ -1043,7 +1038,8 @@ static int cmp_relocs(const void *va, const void *vb)static void sort_relocs (struct relocs *r) { -qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); +if (r->count) +qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); }static int write32 (uint32_t v,FILE *f)arch/x86/xen/xen-head.S +0
-6@@ -57,14 +57,8 @@ SYM_CODE_START(startup_xen) * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifdef CONFIG_SMP - leaq __per_cpu_load(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx -#else xorl %eax, %eax xorl %edx, %edx -#endif wrmsr mov %rsi, %rdiinit/Kconfig +1
-2@@ -1718,7 +1718,7 @@ config KALLSYMS_ALL config KALLSYMS_ABSOLUTE_PERCPU bool depends on KALLSYMS - default X86_64 && SMP + default n config KALLSYMS_BASE_RELATIVE bool --2.41.0
Re: [PATCH v2 07/11] x86/percpu/64: Use relative percpu offsets
Quoted 11 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
The percpu section is currently linked at virtual address 0, because older compilers hardcoded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section can be linked normally. This means that x86-64 will calculate percpu offsets like most other architectures, as the delta between the initial percpu address and the dynamically allocated memory.
Quoted 3 lines
The comments above
MSR_GS_BASEsetup should be reviewed or removed. I don't think they need to be set up to access stack canary, they are just clearing MSR now.
GSBASE is deliberately set to zero offset on SMP for boot because we want any percpu accesses (including stack protector) to use the initial percpu area until the full percpu memory is allocated. It's possible that more stack protector checks could sneak back into the early boot code, and after the conversion to relative percpu offsets they would work properly again. I just didn't reenable them because they are unnecessary that early.
Brian Gerst
Re: [PATCH v2 07/11] x86/percpu/64: Use relative percpu offsets
Quoted 17 lines · Brian Gerst
Quoted 11 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
The percpu section is currently linked at virtual address 0, because older compilers hardcoded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section can be linked normally. This means that x86-64 will calculate percpu offsets like most other architectures, as the delta between the initial percpu address and the dynamically allocated memory.
Quoted 3 lines
The comments above
MSR_GS_BASEsetup should be reviewed or removed. I don't think they need to be set up to access stack canary, they are just clearing MSR now.
Quoted 7 lines
GSBASE is deliberately set to zero offset on SMP for boot because we want any percpu accesses (including stack protector) to use the initial percpu area until the full percpu memory is allocated. It's possible that more stack protector checks could sneak back into the early boot code, and after the conversion to relative percpu offsets they would work properly again. I just didn't reenable them because they are unnecessary that early.
Thanks for the explanation, perhaps this non-obvious fact should be mentioned in the comment .
Thanks,
Uros.
Quoted 1 line
Brian Gerst
PATCH v2 8/11 x86/boot/64: Remove inverse relocations
In reply to Brian GerstNow that the percpu section is not at a fixed virtual address, inverse relocations, which were needed to offset the effects of relocation on RIP-relative percpu references, are no longer needed.
--- arch/x86/boot/compressed/misc.c | 14 +--- arch/x86/tools/relocs.c | 126 +------------------------------- 2 files changed, 2 insertions(+), 138 deletions(-)
arch/x86/boot/compressed/misc.c +1 -13
@@ -221,7 +221,7 @@ static void handle_relocations(void *output, unsigned long output_len,/* * Process relocations: 32 bit relocations first then 64 bit after. -* Three sets of binary relocations are added to the end of the kernel +* Two sets of binary relocations are added to the end of the kernel * before compression. Each relocation table entry is the kernel * address of the location which needs to be updated stored as a * 32-bit value which is sign extended to 64 bits.
@@ -231,8 +231,6 @@ static void handle_relocations(void *output, unsigned long output_len,* kernel bits... * 0 - zero terminator for 64 bit relocations * 64 bit relocation repeated -* 0 - zero terminator for inverse 32 bit relocations -* 32 bit inverse relocation repeated * 0 - zero terminator for 32 bit relocations * 32 bit relocation repeated *
@@ -249,16 +247,6 @@ static void handle_relocations(void *output, unsigned long output_len, *(uint32_t *)ptr += delta; }#ifdef CONFIG_X86_64 -while (*--reloc) { -long extended = *reloc; - extended += map; - - ptr = (unsigned long )extended; -if (ptr < min_addr || ptr > max_addr) -error ("inverse 32-bit relocation outside of kernel!\n" ); - - *(int32_t *)ptr -= delta; - }for (reloc--; *reloc; reloc--) {long extended = *reloc; extended += map;
arch/x86/tools/relocs.c +1 -125
196 lines not on this page · raw message
Signature · 1 line
2.41.0
PATCH v2 9/11 x86/percpu/64: Remove INIT_PER_CPU macros
In reply to Brian GerstThe load and link addresses of percpu variables are now the same, so these macros are no longer necessary.
--- arch/x86/include/asm/percpu.h | 22 ---------------------- arch/x86/kernel/irq_64.c | 1 - arch/x86/kernel/vmlinux.lds.S | 7 ------- arch/x86/tools/relocs.c | 1 - 4 files changed, 31 deletions(-)
arch/x86/include/asm/percpu.h +0 -22
@@ -20,12 +20,6 @@#define PER_CPU_VAR (var) __percpu(var)__percpu_rel -#ifdef CONFIG_X86_64_SMP -#define INIT_PER_CPU_VAR (var) init_per_cpu__##var -#else -#define INIT_PER_CPU_VAR (var) var -#endif -#else /* ...!ASSEMBLY */ #include <linux/kernel.h>
@@ -96,22 +90,6 @@#define __percpu_arg (x) __percpu_prefix "%" #x#define __force_percpu_arg (x) __force_percpu_prefix "%" #x -/* -* Initialized pointers to per-cpu variables needed for the boot -* processor need to use these macros to get the proper address -* offset from __per_cpu_load on SMP. -* -* There also must be an entry in vmlinux_64.lds.S -*/ -#define DECLARE_INIT_PER_CPU (var) \ - extern typeof(var) init_per_cpu_var(var) - -#ifdef CONFIG_X86_64_SMP -#define init_per_cpu_var (var) init_per_cpu__##var -#else -#define init_per_cpu_var (var) var -#endif -/* For arch-specific code, we can use direct single-insn ops (they * don't give an lvalue though). */
arch/x86/kernel/irq_64.c +0 -1
@@ -26,7 +26,6 @@#include <asm/apic.h> DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __visible; -DECLARE_INIT_PER_CPU (irq_stack_backing_store);#ifdef CONFIG_VMAP_STACK /*
arch/x86/kernel/vmlinux.lds.S +0 -7
@@ -482,13 +482,6 @@ SECTIONS "kernel image bigger than KERNEL_IMAGE_SIZE"); #ifdef CONFIG_X86_64 -/* - * Per-cpu symbols which need to be offset from __per_cpu_load - * for the boot processor. - */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) -INIT_PER_CPU(gdt_page); -INIT_PER_CPU(irq_stack_backing_store); #ifdef CONFIG_CPU_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
arch/x86/tools/relocs.c +0 -1
@@ -83,7 +83,6 @@ static const char * const sym_regex_kernel[S_NSYMTYPES] = {"__initramfs_start|" "(jiffies|jiffies_64)|"#if ELF_BITS ==64 -"init_per_cpu__.*|" "__end_rodata_hpage_align|"#endif "__vvar_page|"
Signature · 1 line
2.41.0
Re: [PATCH v2 09/11] x86/percpu/64: Remove INIT_PER_CPU macros
Quoted 5 lines · Brian Gerst
The load and link addresses of percpu variables are now the same, so these macros are no longer necessary.
Signed-off-by Brian Gerst <[email protected]>
Quoted 92 lines
--- arch/x86/include/asm/percpu.h | 22 ---------------------- arch/x86/kernel/irq_64.c | 1 - arch/x86/kernel/vmlinux.lds.S | 7 ------- arch/x86/tools/relocs.c | 1 - 4 files changed, 31 deletions(-)arch/x86/include/asm/percpu.h +0
-22@@ -20,12 +20,6 @@#define PER_CPU_VAR (var) __percpu(var)__percpu_rel -#ifdef CONFIG_X86_64_SMP -#define INIT_PER_CPU_VAR (var) init_per_cpu__##var -#else -#define INIT_PER_CPU_VAR (var) var -#endif -#else /* ...!ASSEMBLY */ #include <linux/kernel.h> @@ -96,22 +90,6 @@#define __percpu_arg (x) __percpu_prefix "%" #x#define __force_percpu_arg (x) __force_percpu_prefix "%" #x -/* -* Initialized pointers to per-cpu variables needed for the boot -* processor need to use these macros to get the proper address -* offset from __per_cpu_load on SMP. -* -* There also must be an entry in vmlinux_64.lds.S -*/ -#define DECLARE_INIT_PER_CPU (var) \ - extern typeof(var) init_per_cpu_var(var) - -#ifdef CONFIG_X86_64_SMP -#define init_per_cpu_var (var) init_per_cpu__##var -#else -#define init_per_cpu_var (var) var -#endif -/* For arch-specific code, we can use direct single-insn ops (they * don't give an lvalue though). */ arch/x86/kernel/irq_64.c +0
-1@@ -26,7 +26,6 @@#include <asm/apic.h> DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __visible; -DECLARE_INIT_PER_CPU (irq_stack_backing_store);#ifdef CONFIG_VMAP_STACK /* arch/x86/kernel/vmlinux.lds.S +0
-7@@ -482,13 +482,6 @@ SECTIONS "kernel image bigger than KERNEL_IMAGE_SIZE"); #ifdef CONFIG_X86_64 -/* - * Per-cpu symbols which need to be offset from __per_cpu_load - * for the boot processor. - */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) -INIT_PER_CPU(gdt_page); -INIT_PER_CPU(irq_stack_backing_store); #ifdef CONFIG_CPU_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");arch/x86/tools/relocs.c +0
-2@@ -83,7 +83,6 @@ static const char * const sym_regex_kernel[S_NSYMTYPES] = {"__initramfs_start|" "(jiffies|jiffies_64)|"#if ELF_BITS ==64 -"init_per_cpu__.*|" "__end_rodata_hpage_align|"#endif "__vvar_page|" --2.41.0
PATCH v2 10/11 percpu: Remove PER_CPU_FIRST_SECTION
In reply to Brian Gerstx86-64 was the only user.
--- include/asm-generic/vmlinux.lds.h | 1 - include/linux/percpu-defs.h | 12 ------------ 2 files changed, 13 deletions(-)
include/asm-generic/vmlinux.lds.h +0 -1
@@ -1032,7 +1032,6 @@*/ #define PERCPU_INPUT (cacheline) \ __per_cpu_start = .; \ - *(.data..percpu..first) \ . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ . = ALIGN(cacheline); \
include/linux/percpu-defs.h +0 -12
@@ -26,13 +26,11 @@#define PER_CPU_SHARED_ALIGNED_SECTION "..shared_aligned"#define PER_CPU_ALIGNED_SECTION "..shared_aligned"#endif -#define PER_CPU_FIRST_SECTION "..first"#else #define PER_CPU_SHARED_ALIGNED_SECTION ""#define PER_CPU_ALIGNED_SECTION "..shared_aligned" -#define PER_CPU_FIRST_SECTION ""#endif
@@ -114,16 +112,6 @@#define DEFINE_PER_CPU (type, name) \ DEFINE_PER_CPU_SECTION(type, name, "") -/* -* Declaration/definition used for per-CPU variables that must come first in -* the set of variables. -*/ -#define DECLARE_PER_CPU_FIRST (type, name) \ - DECLARE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) - -#define DEFINE_PER_CPU_FIRST (type, name) \ - DEFINE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) -/* * Declaration/definition used for per-CPU variables that must be cacheline * aligned under SMP conditions so that, whilst a particular instance of the
Signature · 1 line
2.41.0
Re: [PATCH v2 10/11] percpu: Remove PER_CPU_FIRST_SECTION
Quoted 4 lines · Brian Gerst
x86-64 was the only user.
Signed-off-by Brian Gerst <[email protected]>
Quoted 55 lines
--- include/asm-generic/vmlinux.lds.h | 1 - include/linux/percpu-defs.h | 12 ------------ 2 files changed, 13 deletions(-)include/asm-generic/vmlinux.lds.h +0
-1@@ -1032,7 +1032,6 @@*/ #define PERCPU_INPUT (cacheline) \ __per_cpu_start = .; \ - *(.data..percpu..first) \ . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ . = ALIGN(cacheline); \include/linux/percpu-defs.h +0
-13@@ -26,13 +26,11 @@#define PER_CPU_SHARED_ALIGNED_SECTION "..shared_aligned"#define PER_CPU_ALIGNED_SECTION "..shared_aligned"#endif -#define PER_CPU_FIRST_SECTION "..first"#else #define PER_CPU_SHARED_ALIGNED_SECTION ""#define PER_CPU_ALIGNED_SECTION "..shared_aligned" -#define PER_CPU_FIRST_SECTION ""#endif @@ -114,16 +112,6 @@#define DEFINE_PER_CPU (type, name) \ DEFINE_PER_CPU_SECTION(type, name, "") -/* -* Declaration/definition used for per-CPU variables that must come first in -* the set of variables. -*/ -#define DECLARE_PER_CPU_FIRST (type, name) \ - DECLARE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) - -#define DEFINE_PER_CPU_FIRST (type, name) \ - DEFINE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) -/* * Declaration/definition used for per-CPU variables that must be cacheline * aligned under SMP conditions so that, whilst a particular instance of the -- 2.41.0
PATCH v2 11/11 kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU
In reply to Brian Gerstx86-64 was the only user.
--- init/Kconfig | 11 +----- kernel/kallsyms.c | 12 ++----- scripts/kallsyms.c | 80 ++++++++--------------------------------- scripts/link-vmlinux.sh | 4 --- 4 files changed, 18 insertions(+), 89 deletions(-)
init/Kconfig +1 -10
@@ -1715,11 +1715,6 @@ config KALLSYMS_ALL Say N unless you really need all symbols, or kernel live patching. -config KALLSYMS_ABSOLUTE_PERCPU - bool - depends on KALLSYMS - default n - config KALLSYMS_BASE_RELATIVE bool depends on KALLSYMS
@@ -1727,11 +1722,7 @@ config KALLSYMS_BASE_RELATIVE help Instead of emitting them as absolute values in the native word size, emit the symbol references in the kallsyms table as 32-bit entries, - each containing a relative value in the range [base, base + U32_MAX] - or, when KALLSYMS_ABSOLUTE_PERCPU is in effect, each containing either - an absolute value in the range [0, S32_MAX] or a relative value in the - range [base, base + S32_MAX], where base is the lowest relative symbol - address encountered in the image. + each containing a relative value in the range [base, base + U32_MAX]. On 64-bit builds, this reduces the size of the address table by 50%, but more importantly, it results in entries whose values are build
kernel/kallsyms.c +2 -10
@@ -151,16 +151,8 @@ unsigned long kallsyms_sym_address(int idx)if (!IS_ENABLED (CONFIG_KALLSYMS_BASE_RELATIVE ))return kallsyms_addresses[idx]; -/* values are unsigned offsets if --absolute-percpu is not in effect */ -if (!IS_ENABLED (CONFIG_KALLSYMS_ABSOLUTE_PERCPU )) -return kallsyms_relative_base + (u32 )kallsyms_offsets[idx]; - -/* ...otherwise, positive offsets are absolute values */ -if (kallsyms_offsets[idx] >=0 ) -return kallsyms_offsets[idx]; - -/* ...and negative offsets are relative to kallsyms_relative_base - 1 */ -return kallsyms_relative_base -1 - kallsyms_offsets[idx]; +/* values are unsigned offsets */ +return kallsyms_relative_base + (u32 )kallsyms_offsets[idx]; }static void cleanup_symbol_name (char *s)
scripts/kallsyms.c +15 -65
157 lines not on this page · raw message
scripts/link-vmlinux.sh +0 -4
@@ -148,10 +148,6 @@ kallsyms() kallsymopt="${kallsymopt} --all-symbols" fi - if is_enabled CONFIG_KALLSYMS_ABSOLUTE_PERCPU; then - kallsymopt="${kallsymopt} --absolute-percpu" - fi - if is_enabled CONFIG_KALLSYMS_BASE_RELATIVE; then kallsymopt="${kallsymopt} --base-relative" fi
Signature · 1 line
2.41.0
From: Brian Gerst
Quoted 8 lines
Sent: 26 October 2023 17:01
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
Quoted 5 lines
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like most other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
I didn't think the minimum gcc version was anything like 8.1. I'm using 7.5.0 and I don't think that is the oldest version.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 10 lines · David Laight
From: Brian Gerst
Quoted 8 lines
Sent: 26 October 2023 17:01
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
Quoted 8 lines
Quoted 5 lines
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like most other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
I didn't think the minimum gcc version was anything like 8.1. I'm using 7.5.0 and I don't think that is the oldest version.
What distribution are you using that still relies on that old of a compiler?
Brian Gerst
Quoted 10 lines · David Laight
From: Brian Gerst
Quoted 8 lines
Sent: 26 October 2023 17:01
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
Quoted 8 lines
Quoted 5 lines
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like most other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
I didn't think the minimum gcc version was anything like 8.1. I'm using 7.5.0 and I don't think that is the oldest version.
Please see previous discussion regarding modernizing stack protector on x86_64 [1]
[1] https://lore.kernel.org/lkml/[email protected]/
and x86_32 [2]
[2] https://lore.kernel.org/lkml/[email protected]/
The conclusion in [2] is:
"I'm all in favour of simply requiring GCC-8.1 to build a more secure x86_64 kernel. Gives people an incentive to not use ancient compilers.
And if you do want to use your ancient compiler, we'll still build, you just don't get to have stackprotector."
and in [1]:
"Ack. We did this for 32-bit and got few complaints. Let’s finish the job."
Uros.
From: Uros Bizjak
Quoted 31 lines
Sent: 30 October 2023 08:07
Quoted 10 lines · David Laight
From: Brian Gerst
Quoted 8 lines
Sent: 26 October 2023 17:01
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
Quoted 8 lines
Quoted 5 lines
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like most other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
I didn't think the minimum gcc version was anything like 8.1. I'm using 7.5.0 and I don't think that is the oldest version.
Please see previous discussion regarding modernizing stack protector on
x86_64[1][1] https://lore.kernel.org/lkml/[email protected]/
and
x86_32[2]
Quoted 7 lines
The conclusion in [2] is:
"I'm all in favour of simply requiring GCC-8.1 to build a more secure
x86_64kernel. Gives people an incentive to not use ancient compilers.And if you do want to use your ancient compiler, we'll still build, you just don't get to have stackprotector."
I didn't see a patch that limited 'stackprotector' to gcc >= 8.1 Without that anyone who already has it enabled and is using an older compiler will get very broken kernels.
David
Quoted 6 lines
and in [1]:
"Ack. We did this for 32-bit and got few complaints. Let’s finish the job."
Uros.
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 38 lines · David Laight
From: Uros Bizjak
Quoted 36 lines
Sent: 30 October 2023 08:07
Quoted 10 lines · David Laight
From: Brian Gerst
Quoted 8 lines
Sent: 26 October 2023 17:01
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
Quoted 8 lines
Quoted 5 lines
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like most other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
I didn't think the minimum gcc version was anything like 8.1. I'm using 7.5.0 and I don't think that is the oldest version.
Please see previous discussion regarding modernizing stack protector on
x86_64[1][1] https://lore.kernel.org/lkml/[email protected]/
and
x86_32[2][2] https://lore.kernel.org/lkml/[email protected]/
The conclusion in [2] is:
"I'm all in favour of simply requiring GCC-8.1 to build a more secure
x86_64kernel. Gives people an incentive to not use ancient compilers.
Quoted 6 lines
Quoted 2 lines
And if you do want to use your ancient compiler, we'll still build, you just don't get to have stackprotector."
I didn't see a patch that limited 'stackprotector' to gcc >= 8.1 Without that anyone who already has it enabled and is using an older compiler will get very broken kernels.
It's this part:
--cut here--
(quoted diff) +0 -0
b/scripts/gcc-x86_32-has-stack-protector.sh
index f5c119495254..51f864d76bd6 100755
scripts/gcc-x86_32-has-stack-protector.sh +0 -1
@@ -1,4 +1,8 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0 -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m32
-O0 -fstack-protector - -o - 2> /dev/null | grep -q "%gs" +# This requires GCC 8.1 or better. Specifically, we require +# -mstack-protector-guard-reg, added by
+# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81708
+
+echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m32 -O0 -fstack-protector -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=stack_canary - -o - 2> /dev/null | grep -q "%fs"
--cut here--
Uros.