x86-64: Stack protector and percpu improvements
Subsystems (8)
v6 cover x86-64: Stack protector and percpu improvements
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
v6:
- Rebased to current tip tree
- Dropped patches already applied
- Fixed typos in commit messages
- Added Reviewed-by tags
Ard Biesheuvel (1):
x86/module: Deal with GOT based stack cookie load on Clang < 17
Brian Gerst (14):
x86: Raise minimum GCC version to 8.1 x86/stackprotector: Remove stack protector test scripts x86/boot: Disable stack protector for early boot code x86/pvh: Use fixed_percpu_data for early boot GSBASE x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations x86/stackprotector/64: Convert to normal percpu variable x86/percpu/64: Use relative percpu offsets x86/percpu/64: Remove fixed_percpu_data x86/boot/64: Remove inverse relocations x86/percpu/64: Remove INIT_PER_CPU macros
percpu: Remove PER_CPU_FIRST_SECTION
percpu: Remove PERCPU_VADDR()
percpu: Remove __per_cpu_load
kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU
arch/x86/Kconfig | 11 +- arch/x86/Makefile | 20 +-- arch/x86/boot/compressed/misc.c | 14 +-- arch/x86/entry/entry.S | 2 - arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/desc.h | 1 - arch/x86/include/asm/elf.h | 3 +- arch/x86/include/asm/percpu.h | 22 ---- arch/x86/include/asm/processor.h | 28 +---- arch/x86/include/asm/stackprotector.h | 36 +----- arch/x86/kernel/Makefile | 2 + arch/x86/kernel/asm-offsets_64.c | 6 - arch/x86/kernel/cpu/common.c | 9 +- arch/x86/kernel/head64.c | 2 +- arch/x86/kernel/head_64.S | 20 ++- arch/x86/kernel/irq_64.c | 1 - arch/x86/kernel/module.c | 15 +++ arch/x86/kernel/setup_percpu.c | 12 +- arch/x86/kernel/vmlinux.lds.S | 35 ------ arch/x86/platform/pvh/head.S | 14 ++- arch/x86/tools/relocs.c | 147 ++-------------------- arch/x86/xen/xen-head.S | 10 +- include/asm-generic/sections.h | 2 +- include/asm-generic/vmlinux.lds.h | 38 +----- include/linux/percpu-defs.h | 12 -- init/Kconfig | 5 - kernel/kallsyms.c | 12 +- mm/percpu.c | 4 +- scripts/gcc-x86_32-has-stack-protector.sh | 8 -- scripts/gcc-x86_64-has-stack-protector.sh | 4 - scripts/kallsyms.c | 72 ++--------- scripts/link-vmlinux.sh | 4 - scripts/min-tool-version.sh | 2 + 33 files changed, 100 insertions(+), 475 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
base-commit: b79d90e018587507fd42c4c888956668692ff431
Signature · 1 line
2.47.1
PATCH v6 1/15 x86: Raise minimum GCC version to 8.1
Stack protector support on 64-bit currently requires that the percpu section is linked at absolute address 0 because older compilers fixed the location of the canary value relative to the GS segment base. GCC 8.1 introduced options to change where the canary value is located, allowing it to be configured as a standard percpu variable. This has already been done for 32-bit. Doing the same for 64-bit will enable removing the code needed to support zero-based percpu.
--- scripts/min-tool-version.sh | 2 ++ 1 file changed, 2 insertions(+)
scripts/min-tool-version.sh +2 -0
@@ -19,6 +19,8 @@ binutils) gcc) if [ "$ARCH" = parisc64 ]; then echo 12.0.0 + elif [ "$SRCARCH" = x86 ]; then + echo 8.1.0 else echo 5.1.0 fi
Signature · 1 line
2.47.1
[tip: x86/asm] x86/build: Raise the minimum GCC version to 8.1
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: a3e8fe814ad15c16735cdf394454a8bd96eb4d56 Gitweb: https://git.kernel.org/tip/a3e8fe814ad15c16735cdf394454a8bd96eb4d56 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:33 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:14:40 +01:00
x86/build: Raise the minimum GCC version to 8.1
Stack protector support on 64-bit currently requires that the percpu section is linked at absolute address 0, because older compilers fixed the location of the canary value relative to the GS segment base.
GCC 8.1 introduced options to change where the canary value is located, allowing it to be configured as a standard per-CPU variable. This has already been done for 32-bit. Doing the same for 64-bit will enable removing the code needed to support zero-based percpu.
--- scripts/min-tool-version.sh | 2 ++ 1 file changed, 2 insertions(+)
scripts/min-tool-version.sh +2 -0
@@ -19,6 +19,8 @@ binutils) gcc) if [ "$ARCH" = parisc64 ]; then echo 12.0.0 + elif [ "$SRCARCH" = x86 ]; then + echo 8.1.0 else echo 5.1.0 fi
PATCH v6 2/15 x86/stackprotector: Remove stack protector test scripts
In reply to Brian GerstWith GCC 8.1 now the minimum supported compiler for x86, these scripts are no longer needed.
--- arch/x86/Kconfig | 11 +---------- scripts/gcc-x86_32-has-stack-protector.sh | 8 -------- scripts/gcc-x86_64-has-stack-protector.sh | 4 ---- 3 files changed, 1 insertion(+), 22 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
arch/x86/Kconfig +1 -10
@@ -283,7 +283,7 @@ config X86 select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_SETUP_PER_CPU_AREA select HAVE_SOFTIRQ_ON_OWN_STACK - select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR + select HAVE_STACKPROTECTOR select HAVE_STACK_VALIDATION if HAVE_OBJTOOL select HAVE_STATIC_CALL select HAVE_STATIC_CALL_INLINE if HAVE_OBJTOOL
@@ -423,15 +423,6 @@ config PGTABLE_LEVELS default 3 if X86_PAE default 2 -config CC_HAS_SANE_STACKPROTECTOR - bool - default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT - default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) - help - We have to make sure stack protector is unconditionally disabled if - the compiler produces broken code or if it does not let us control - the segment on 32-bit kernels. - menu "Processor type and features" config SMP
scripts/gcc-x86_32-has-stack-protector.sh deleted +0 -8
@@ -1,8 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -# This requires GCC 8.1 or better. Specifically, we require -# -mstack-protector-guard-reg, added by -# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81708 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m32 -O0 -fstack-protector -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - -o - 2> /dev/null | grep -q "%fs"
scripts/gcc-x86_64-has-stack-protector.sh deleted +0 -4
@@ -1,4 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m64 -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
Signature · 1 line
2.47.1
[tip: x86/asm] x86/stackprotector: Remove stack protector test scripts
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 0ee2689b9374d6fd5f43b703713a532278654749 Gitweb: https://git.kernel.org/tip/0ee2689b9374d6fd5f43b703713a532278654749 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:34 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:14:47 +01:00
x86/stackprotector: Remove stack protector test scripts
With GCC 8.1 now the minimum supported compiler for x86, these scripts are no longer needed.
--- arch/x86/Kconfig | 11 +---------- scripts/gcc-x86_32-has-stack-protector.sh | 8 -------- scripts/gcc-x86_64-has-stack-protector.sh | 4 ---- 3 files changed, 1 insertion(+), 22 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
arch/x86/Kconfig +1 -10
@@ -285,7 +285,7 @@ config X86 select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_SETUP_PER_CPU_AREA select HAVE_SOFTIRQ_ON_OWN_STACK - select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR + select HAVE_STACKPROTECTOR select HAVE_STACK_VALIDATION if HAVE_OBJTOOL select HAVE_STATIC_CALL select HAVE_STATIC_CALL_INLINE if HAVE_OBJTOOL
@@ -426,15 +426,6 @@ config PGTABLE_LEVELS default 3 if X86_PAE default 2 -config CC_HAS_SANE_STACKPROTECTOR - bool - default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT - default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) - help - We have to make sure stack protector is unconditionally disabled if - the compiler produces broken code or if it does not let us control - the segment on 32-bit kernels. - menu "Processor type and features" config SMP
scripts/gcc-x86_32-has-stack-protector.sh deleted +0 -8
@@ -1,8 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -# This requires GCC 8.1 or better. Specifically, we require -# -mstack-protector-guard-reg, added by -# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81708 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m32 -O0 -fstack-protector -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - -o - 2> /dev/null | grep -q "%fs"
scripts/gcc-x86_64-has-stack-protector.sh deleted +0 -4
@@ -1,4 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0 - -echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m64 -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
PATCH v6 3/15 x86/boot: Disable stack protector for early boot code
In reply to Brian GerstOn 64-bit, this will prevent crashes when the canary access is changed from %gs:40 to %gs:__stack_chk_guard(%rip). RIP-relative addresses from the identity-mapped early boot code will target the wrong address with zero-based percpu. KASLR could then shift that address to an unmapped page causing a crash on boot.
This early boot code runs well before userspace is active and does not need stack protector enabled.
--- arch/x86/kernel/Makefile | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/Makefile +2 -0
@@ -44,6 +44,8 @@ KCOV_INSTRUMENT_unwind_orc.o := n KCOV_INSTRUMENT_unwind_frame.o := n KCOV_INSTRUMENT_unwind_guess.o := n +CFLAGS_head32.o := -fno-stack-protector +CFLAGS_head64.o := -fno-stack-protector CFLAGS_irq.o := -I $(src)/../include/asm/trace obj-y += head_$(BITS).o
Signature · 1 line
2.47.1
[tip: x86/asm] x86/boot: Disable stack protector for early boot code
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: a9a76b38aaf577887103e3ebb41d70e6aa5a4b19 Gitweb: https://git.kernel.org/tip/a9a76b38aaf577887103e3ebb41d70e6aa5a4b19 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:35 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:14:51 +01:00
x86/boot: Disable stack protector for early boot code
On 64-bit, this will prevent crashes when the canary access is changed from %gs:40 to %gs:__stack_chk_guard(%rip). RIP-relative addresses from the identity-mapped early boot code will target the wrong address with zero-based percpu. KASLR could then shift that address to an unmapped page causing a crash on boot.
This early boot code runs well before user-space is active and does not need stack protector enabled.
--- arch/x86/kernel/Makefile | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/Makefile +2 -0
@@ -44,6 +44,8 @@ KCOV_INSTRUMENT_unwind_orc.o := n KCOV_INSTRUMENT_unwind_frame.o := n KCOV_INSTRUMENT_unwind_guess.o := n +CFLAGS_head32.o := -fno-stack-protector +CFLAGS_head64.o := -fno-stack-protector CFLAGS_irq.o := -I $(src)/../include/asm/trace obj-y += head_$(BITS).o
PATCH v6 4/15 x86/pvh: Use fixed_percpu_data for early boot GSBASE
In reply to Brian GerstInstead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.
--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +9 -6
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1: UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecx - leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eax + shrq $32, %rdx wrmsr /* Call xen_prepare_pvh() via the kernel virtual mapping */
@@ -238,8 +243,6 @@ SYM_DATA_START_LOCAL(gdt_start) SYM_DATA_END_LABEL(gdt_start, SYM_L_LOCAL, gdt_end) .balign 16 -SYM_DATA_LOCAL(canary, .fill 48, 1, 0) - SYM_DATA_START_LOCAL(early_stack) .fill BOOT_STACK_SIZE, 1, 0 SYM_DATA_END_LABEL(early_stack, SYM_L_LOCAL, early_stack_end)
Signature · 1 line
2.47.1
Borislav Petkov
maintainerRe: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
Quoted 8 lines · Brian Gerst
Instead of having a private area for the stack canary, use
fixed_percpu_datafor GSBASE like the native kernel.Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
Use ./scripts/get_maintainer.pl pls. I've added Juergen now.
Quoted 6 lines
arch/x86/platform/pvh/head.S +0
-0@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1:
arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 179
Quoted 10 lines
arch/x86/platform/pvh/head.S +4
-4UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecx - leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up.
s/cpu/CPU/g
check your whole set pls.
Quoted 6 lines
arch/x86/platform/pvh/head.S +5
-0+ */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eax + shrq $32, %rdx wrmsr
Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
Quoted 33 lines · Borislav Petkov
Quoted 8 lines · Brian Gerst
Instead of having a private area for the stack canary, use
fixed_percpu_datafor GSBASE like the native kernel.Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)Use
./scripts/get_maintainer.plpls. I've added Juergen now.Quoted 6 lines
arch/x86/platform/pvh/head.S +0
-0@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1:arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 179
Quoted 10 lines
arch/x86/platform/pvh/head.S +4
-4UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecx - leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up.s/cpu/CPU/g
Quoted 1 line
check your whole set pls.
To be fair, this was a copy of an existing comment. Is there a style guide where all these grammar rules are documented, so I don't have to keep resending these patches for trivial typos?
Brian Gerst
Borislav Petkov
maintainerRe: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
Quoted 3 lines · Brian Gerst
To be fair, this was a copy of an existing comment. Is there a style guide where all these grammar rules are documented, so I don't have to keep resending these patches for trivial typos?
You don't have to keep resending them for trivial typos - you simply wait 1-2 weeks to gather review feedback, you incorporate it and send a new version of the set. Like it is usually done on lkml. I think you know how the process works...
Signature · 1 line
Sent from a small device: formatting sucks and brevity is inevitable.
[tip: x86/asm] x86/pvh: Use fixed_percpu_data for early boot GSBASE
In reply to Brian GerstThe following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: f58b63857ae38b4484185b799a2759274b930c92 Gitweb: https://git.kernel.org/tip/f58b63857ae38b4484185b799a2759274b930c92 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:36 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:14:59 +01:00
x86/pvh: Use fixed_percpu_data for early boot GSBASE
Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.
--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +9 -6
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1: UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecx - leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that on SMP the boot CPU uses the init data section until + * the per-CPU areas are set up. + */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eax + shrq $32, %rdx wrmsr /* Call xen_prepare_pvh() via the kernel virtual mapping */
@@ -238,8 +243,6 @@ SYM_DATA_START_LOCAL(gdt_start) SYM_DATA_END_LABEL(gdt_start, SYM_L_LOCAL, gdt_end) .balign 16 -SYM_DATA_LOCAL(canary, .fill 48, 1, 0) - SYM_DATA_START_LOCAL(early_stack) .fill BOOT_STACK_SIZE, 1, 0 SYM_DATA_END_LABEL(early_stack, SYM_L_LOCAL, early_stack_end)
Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
In reply to Brian GerstQuoted 19 lines · Brian Gerst
Instead of having a private area for the stack canary, use
fixed_percpu_datafor GSBASE like the native kernel.Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)arch/x86/platform/pvh/head.S +0
-2@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1: UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecx
arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183
Quoted 10 lines
arch/x86/platform/pvh/head.S +8
-2- leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eax
movl
I'm bisecting perf breakage and landing on this commit breaks the build.
H. Peter Anvin
reviewerRe: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
Quoted 33 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst
Instead of having a private area for the stack canary, use
fixed_percpu_datafor GSBASE like the native kernel.Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)arch/x86/platform/pvh/head.S +0
-2@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1: UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecxarch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183
Quoted 10 lines
arch/x86/platform/pvh/head.S +8
-2- leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eaxmovl
Quoted 1 line
I'm bisecting perf breakage and landing on this commit breaks the build.
Breaks the build how?
Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
Quoted 35 lines · H. Peter Anvin
Quoted 33 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst
Instead of having a private area for the stack canary, use
fixed_percpu_datafor GSBASE like the native kernel.Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)arch/x86/platform/pvh/head.S +0
-2@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1: UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecxarch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183
Quoted 10 lines
arch/x86/platform/pvh/head.S +8
-2- leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eaxmovl
Quoted 3 lines
Quoted 1 line
I'm bisecting perf breakage and landing on this commit breaks the build.
Breaks the build how?
AS arch/x86/platform/pvh/head.oarch/x86/platform/pvh/head.S: Assembler messages:arch/x86/platform/pvh/head.S:183: Error: incorrect register %eax' used with q' suffix
Per the above, I presume it was meant to be a movl.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
Quoted 39 lines · Mateusz Guzik
Quoted 37 lines · H. Peter Anvin
Quoted 34 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst
Instead of having a private area for the stack canary, use
fixed_percpu_datafor GSBASE like the native kernel.Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)arch/x86/platform/pvh/head.S +0
-2@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1: UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecxarch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183
Quoted 10 lines
arch/x86/platform/pvh/head.S +8
-2- leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eaxmovl
I'm bisecting perf breakage and landing on this commit breaks the build.
Quoted 8 lines
Quoted 1 line
Breaks the build how?
AS
arch/x86/platform/pvh/head.oarch/x86/platform/pvh/head.S: Assembler messages:arch/x86/platform/pvh/head.S:183: Error: incorrect register%eax' used withq' suffixPer the above, I presume it was meant to be a movl.
Please note that this part is further changed in [PATCH 8/15] [1], so the breakage is temporary.
[1] https://lore.kernel.org/lkml/[email protected]/
Uros.
Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE
Quoted 48 lines · Uros Bizjak
Quoted 46 lines · Mateusz Guzik
Quoted 35 lines · H. Peter Anvin
Quoted 33 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst
Instead of having a private area for the stack canary, use
fixed_percpu_datafor GSBASE like the native kernel.Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>--- arch/x86/platform/pvh/head.S | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)arch/x86/platform/pvh/head.S +0
-2@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen) 1: UNWIND_HINT_END_OF_STACK - /* Set base address in stack canary descriptor. */ - mov $MSR_GS_BASE,%ecxarch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183
Quoted 10 lines
arch/x86/platform/pvh/head.S +8
-2- leal canary(%rip), %eax - xor %edx, %edx + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ + movl $MSR_GS_BASE,%ecx + leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx + movq %edx, %eaxmovl
Quoted 3 lines
Quoted 1 line
I'm bisecting perf breakage and landing on this commit breaks the build.
Breaks the build how?
AS
arch/x86/platform/pvh/head.oarch/x86/platform/pvh/head.S: Assembler messages:arch/x86/platform/pvh/head.S:183: Error: incorrect register%eax' used withq' suffix
Quoted 7 lines
Quoted 1 line
Per the above, I presume it was meant to be a movl.
Please note that this part is further changed in [PATCH 8/15] [1], so the breakage is temporary.
Top of the tree does build, so I assumed this did get addressed in later patches.
I am saying I ran into this build breakage while bisecting. Other people may be in the same spot.
It should pose no difficulty to s/movq/movl/ in the branch and be done with it.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
PATCH v6 5/15 x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations
In reply to Brian GerstClang may produce R_X86_64_REX_GOTPCRELX relocations when redefining the stack protector location. Treat them as another type of PC-relative relocation.
--- arch/x86/tools/relocs.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/tools/relocs.c +7 -0
@@ -32,6 +32,11 @@ static struct relocs relocs32;static struct relocs relocs32neg;static struct relocs relocs64;# define FMT PRIu64 + +#ifndef R_X86_64_REX_GOTPCRELX +#define R_X86_64_REX_GOTPCRELX 42 +#endif +#else # define FMT PRIu32#endif
@@ -227,6 +232,7 @@ static const char *rel_type(unsigned type)REL_TYPE (R_X86_64_PC16 ),REL_TYPE (R_X86_64_8 ),REL_TYPE (R_X86_64_PC8 ), +REL_TYPE (R_X86_64_REX_GOTPCRELX ), #elseREL_TYPE (R_386_NONE ), REL_TYPE(R_386_32),
@@ -861,6 +867,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,case R_X86_64_PC32 :case R_X86_64_PLT32 : +case R_X86_64_REX_GOTPCRELX :/* * PC relative relocations don't need to be adjusted unless * referencing a percpu symbol.
Signature · 1 line
2.47.1
[tip: x86/asm] x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: cb7927fda002ca49ae62e2782c1692acc7b80c67 Gitweb: https://git.kernel.org/tip/cb7927fda002ca49ae62e2782c1692acc7b80c67 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:37 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:02 +01:00
x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations
Clang may produce R_X86_64_REX_GOTPCRELX relocations when redefining the stack protector location. Treat them as another type of PC-relative relocation.
--- arch/x86/tools/relocs.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/tools/relocs.c +7 -0
@@ -32,6 +32,11 @@ static struct relocs relocs32;static struct relocs relocs32neg;static struct relocs relocs64;# define FMT PRIu64 + +#ifndef R_X86_64_REX_GOTPCRELX +# define R_X86_64_REX_GOTPCRELX 42 +#endif +#else # define FMT PRIu32#endif
@@ -227,6 +232,7 @@ static const char *rel_type(unsigned type)REL_TYPE (R_X86_64_PC16 ),REL_TYPE (R_X86_64_8 ),REL_TYPE (R_X86_64_PC8 ), +REL_TYPE (R_X86_64_REX_GOTPCRELX ), #elseREL_TYPE (R_386_NONE ), REL_TYPE(R_386_32),
@@ -861,6 +867,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,case R_X86_64_PC32 :case R_X86_64_PLT32 : +case R_X86_64_REX_GOTPCRELX :/* * PC relative relocations don't need to be adjusted unless * referencing a percpu symbol.
PATCH v6 6/15 x86/module: Deal with GOT based stack cookie load on Clang < 17
In reply to Brian GerstFrom: Ard Biesheuvel <[email protected]>
Clang versions before 17 will not honour -fdirect-access-external-data for the load of the stack cookie emitted into each function's prologue and epilogue.
This is not an issue for the core kernel, as the linker will relax these loads into LEA instructions that take the address of __stack_chk_guard directly. For modules, however, we need to work around this, by dealing with R_X86_64_REX_GOTPCRELX relocations that refer to __stack_chk_guard.
In this case, given that this is a GOT load, the reference should not refer to __stack_chk_guard directly, but to a memory location that holds its address. So take the address of __stack_chk_guard into a static variable, and fix up the relocations to refer to that.
--- arch/x86/include/asm/elf.h | 3 ++- arch/x86/kernel/module.c | 15 +++++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-)
arch/x86/include/asm/elf.h +2 -1
@@ -55,7 +55,8 @@ typedef struct user_i387_struct elf_fpregset_t;#define R_X86_64_JUMP_SLOT 7/* Create PLT entry */ #define R_X86_64_RELATIVE 8/* Adjust by program base */ #define R_X86_64_GOTPCREL 9/* 32 bit signed pc relative -offset to GOT */ +#define R_X86_64_GOTPCRELX 41 offset to GOT */ +#define R_X86_64_REX_GOTPCRELX 42#define R_X86_64_32 10/* Direct 32 bit zero extended */ #define R_X86_64_32S 11/* Direct 32 bit sign extended */ #define R_X86_64_16 12/* Direct 16 bit zero extended */
arch/x86/kernel/module.c +15 -0
@@ -19,6 +19,7 @@#include <linux/jump_label.h> #include <linux/random.h> #include <linux/memory.h> +#include <linux/stackprotector.h> #include <asm/text-patching.h> #include <asm/page.h>
@@ -130,6 +131,20 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#if defined(CONFIG_STACKPROTECTOR ) && \ + defined(CONFIG_CC_IS_CLANG ) &&CONFIG_CLANG_VERSION <170000 +case R_X86_64_REX_GOTPCRELX : { +static unsigned long __percpu *const addr = &__stack_chk_guard; + +if (sym->st_value != (u64 )addr) { +pr_err ("%s: Unsupported GOTPCREL relocation\n" , me->name); +return -ENOEXEC ; + } + + val = (u64 )&addr + rel[i].r_addend; + fallthrough; + } +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;
Signature · 1 line
2.47.1
Re: [PATCH v6 06/15] x86/module: Deal with GOT based stack cookie load on Clang < 17
arch/x86/include/asm/elf.h typedef struct user_i387_struct elf_fpregset_t; · line 58
Quoted 3 lines · Brian Gerst
#define
R_X86_64_GOTPCREL9 /* 32 bit signed pc relativearch/x86/include/asm/elf.h +1
-1-offset to GOT */ +#define R_X86_64_GOTPCRELX 41 offset to GOT */
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line
+#define
R_X86_64_REX_GOTPCRELX42
Uhm, I'm pretty sure this won't even build, should any code use R_X86_64_GOTPCRELX - which it doesn't currently...
Also, each entry here has a comment explaining it - except these two new GOTPCREL variants.
Thanks,
Ingo
Re: [PATCH v6 06/15] x86/module: Deal with GOT based stack cookie load on Clang < 17
Quoted 7 lines · Ingo Molnar
arch/x86/include/asm/elf.h typedef struct user_i387_struct elf_fpregset_t; · line 58
Quoted 3 lines · Brian Gerst
#define
R_X86_64_GOTPCREL9 /* 32 bit signed pc relativearch/x86/include/asm/elf.h +1
-1-offset to GOT */ +#define R_X86_64_GOTPCRELX 41 offset to GOT */
Quoted 8 lines
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Quoted 1 line
+#define
R_X86_64_REX_GOTPCRELX42Uhm, I'm pretty sure this won't even build, should any code use
R_X86_64_GOTPCRELX- which it doesn't currently...Also, each entry here has a comment explaining it - except these two new GOTPCREL variants.
Should we remove the non-REX version that isn't used by the kernel? This header is not exported to userspace.
Brian Gerst
H. Peter Anvin
reviewerRe: [PATCH v6 06/15] x86/module: Deal with GOT based stack cookie load on Clang < 17
Quoted 14 lines · Brian Gerst
Quoted 13 lines · Ingo Molnar
arch/x86/include/asm/elf.h typedef struct user_i387_struct elf_fpregset_t; · line 58
Quoted 3 lines · Brian Gerst
#define
R_X86_64_GOTPCREL9 /* 32 bit signed pc relativearch/x86/include/asm/elf.h +1
-1-offset to GOT */ +#define R_X86_64_GOTPCRELX 41 offset to GOT */ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Quoted 1 line
+#define
R_X86_64_REX_GOTPCRELX42Uhm, I'm pretty sure this won't even build, should any code use
R_X86_64_GOTPCRELX- which it doesn't currently...
Quoted 9 lines
Quoted 2 lines
Also, each entry here has a comment explaining it - except these two new GOTPCREL variants.
Should we remove the non-REX version that isn't used by the kernel? This header is not exported to userspace.
Brian Gerst
I would personally like to see them in there for the sake of documentation at least. If something runs into this in the future then at least it will be a little quicker to debug. This is an ABI constant, after all.
[tip: x86/asm] x86/module: Deal with GOT based stack cookie load on Clang < 17
In reply to Brian GerstThe following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 78c4374ef8b842c6abf195d6f963853c7ec464d2 Gitweb: https://git.kernel.org/tip/78c4374ef8b842c6abf195d6f963853c7ec464d2 Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:38 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:05 +01:00
x86/module: Deal with GOT based stack cookie load on Clang < 17
Clang versions before 17 will not honour -fdirect-access-external-data for the load of the stack cookie emitted into each function's prologue and epilogue.
This is not an issue for the core kernel, as the linker will relax these loads into LEA instructions that take the address of __stack_chk_guard directly. For modules, however, we need to work around this, by dealing with R_X86_64_REX_GOTPCRELX relocations that refer to __stack_chk_guard.
In this case, given that this is a GOT load, the reference should not refer to __stack_chk_guard directly, but to a memory location that holds its address. So take the address of __stack_chk_guard into a static variable, and fix up the relocations to refer to that.
[ mingo: Fix broken R_X86_64_GOTPCRELX definition. ]
--- arch/x86/include/asm/elf.h | 5 +++-- arch/x86/kernel/module.c | 15 +++++++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-)
arch/x86/include/asm/elf.h +3 -2
@@ -54,8 +54,9 @@ typedef struct user_i387_struct elf_fpregset_t;#define R_X86_64_GLOB_DAT 6/* Create GOT entry */ #define R_X86_64_JUMP_SLOT 7/* Create PLT entry */ #define R_X86_64_RELATIVE 8/* Adjust by program base */ -#define R_X86_64_GOTPCREL 9/* 32 bit signed pc relative -offset to GOT */ +#define R_X86_64_GOTPCREL 9/* 32 bit signed pc relative offset to GOT */ +#define R_X86_64_GOTPCRELX 41 +#define R_X86_64_REX_GOTPCRELX 42#define R_X86_64_32 10/* Direct 32 bit zero extended */ #define R_X86_64_32S 11/* Direct 32 bit sign extended */ #define R_X86_64_16 12/* Direct 16 bit zero extended */
arch/x86/kernel/module.c +15 -0
@@ -19,6 +19,7 @@#include <linux/jump_label.h> #include <linux/random.h> #include <linux/memory.h> +#include <linux/stackprotector.h> #include <asm/text-patching.h> #include <asm/page.h>
@@ -130,6 +131,20 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs, goto overflow; size =4 ;break ; +#if defined(CONFIG_STACKPROTECTOR ) && \ + defined(CONFIG_CC_IS_CLANG ) &&CONFIG_CLANG_VERSION <170000 +case R_X86_64_REX_GOTPCRELX : { +static unsigned long __percpu *const addr = &__stack_chk_guard; + +if (sym->st_value != (u64 )addr) { +pr_err ("%s: Unsupported GOTPCREL relocation\n" , me->name); +return -ENOEXEC ; + } + + val = (u64 )&addr + rel[i].r_addend; + fallthrough; + } +#endif case R_X86_64_PC32 :case R_X86_64_PLT32 : val -= (u64 )loc;
PATCH v6 7/15 x86/stackprotector/64: Convert to normal percpu variable
In reply to Brian GerstOlder versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.
GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.
--- arch/x86/Makefile | 20 +++++++++------ arch/x86/entry/entry.S | 2 -- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/processor.h | 16 ++---------- arch/x86/include/asm/stackprotector.h | 36 ++++----------------------- arch/x86/kernel/asm-offsets_64.c | 6 ----- arch/x86/kernel/cpu/common.c | 5 +--- arch/x86/kernel/head_64.S | 3 +-- arch/x86/xen/xen-head.S | 3 +-- 9 files changed, 23 insertions(+), 70 deletions(-)
arch/x86/Makefile +12 -8
@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \ - -mstack-protector-guard-symbol=__ref_stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64
@@ -197,6 +190,17 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs +endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) + KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard + else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif endif #
arch/x86/entry/entry.S +0 -2
@@ -52,7 +52,6 @@ EXPORT_SYMBOL_GPL(mds_verw_sel); THUNK warn_thunk_thunk, __warn_thunk -#ifndef CONFIG_X86_64 /* * Clang's implementation of TLS stack cookies requires the variable in * question to be a TLS variable. If the variable happens to be defined as an
@@ -66,4 +65,3 @@ THUNK warn_thunk_thunk, __warn_thunk #ifdef CONFIG_STACKPROTECTOR EXPORT_SYMBOL(__ref_stack_chk_guard); #endif -#endif
arch/x86/entry/entry_64.S +1 -1
@@ -192,7 +192,7 @@ SYM_FUNC_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(fixed_percpu_data + FIXED_stack_canary) + movq %rbx, PER_CPU_VAR(__stack_chk_guard) #endif /*
arch/x86/include/asm/processor.h +2 -14
@@ -422,16 +422,8 @@ struct irq_stack {#ifdef CONFIG_X86_64 struct fixed_percpu_data { -/* -* GCC hardcodes the stack canary as %gs:40. Since the -* irq_stack is the object at %gs:0, we reserve the bottom -* 48 bytes of the irq stack for the canary. -* -* Once we are willing to require -mstack-protector-guard-symbol= -* support for x86_64 stackprotector, we can get rid of this. -*/ char gs_base[40 ]; -unsigned long stack_canary; +unsigned long reserved; }; DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
@@ -446,11 +438,7 @@ extern asmlinkage void entry_SYSCALL32_ignore(void);/* Save actual FS/GS selectors and bases to current->thread */ void current_save_fsgs (void ); -#else /* X86_64 */ -#ifdef CONFIG_STACKPROTECTOR -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); -#endif -#endif /* !X86_64 */ +#endif /* X86_64 */ struct perf_event ;
arch/x86/include/asm/stackprotector.h +5 -31
@@ -2,26 +2,10 @@/* * GCC stack protector support. * -* Stack protector works by putting predefined pattern at the start of +* Stack protector works by putting a predefined pattern at the start of * the stack frame and verifying that it hasn't been overwritten when -* returning from the function. The pattern is called stack canary -* and unfortunately gcc historically required it to be at a fixed offset -* from the percpu segment base. On x86_64, the offset is 40 bytes. -* -* The same segment is shared by percpu area and stack canary. On -* x86_64, percpu symbols are zero based and %gs (64-bit) points to the -* base of percpu area. The first occupant of the percpu area is always -* fixed_percpu_data which contains stack_canary at the appropriate -* offset. On x86_32, the stack canary is just a regular percpu -* variable. -* -* Putting percpu data in %fs on 32-bit is a minor optimization compared to -* using %gs. Since 32-bit userspace normally has %fs == 0, we are likely -* to load 0 into %fs on exit to usermode, whereas with percpu data in -* %gs, we are likely to load a non-null %gs on return to user mode. -* -* Once we are willing to require GCC 8.1 or better for 64-bit stackprotector -* support, we can remove some of this complexity. +* returning from the function. The pattern is called the stack canary +* and is a unique value for each task. */ #ifndef _ASM_STACKPROTECTOR_H
@@ -36,6 +20,8 @@#include <linux/sched.h> +DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +/* * Initialize the stackprotector canary value. *
@@ -51,25 +37,13 @@ static __always_inline void boot_init_stack_canary(void) {unsigned long canary =get_random_canary (); -#ifdef CONFIG_X86_64 -BUILD_BUG_ON (offsetof(struct fixed_percpu_data , stack_canary) !=40 ); -#endif - current->stack_canary = canary; -#ifdef CONFIG_X86_64 -this_cpu_write (fixed_percpu_data.stack_canary, canary); -#else this_cpu_write (__stack_chk_guard, canary); -#endif }static inline void cpu_init_stack_canary (int cpu,struct task_struct *idle) { -#ifdef CONFIG_X86_64 -per_cpu (fixed_percpu_data.stack_canary, cpu) = idle->stack_canary; -#else per_cpu (__stack_chk_guard, cpu) = idle->stack_canary; -#endif }#else /* STACKPROTECTOR */
arch/x86/kernel/asm-offsets_64.c +0 -6
@@ -54,11 +54,5 @@ int main(void)BLANK ();#undef ENTRY -BLANK (); - -#ifdef CONFIG_STACKPROTECTOR -OFFSET (FIXED_stack_canary, fixed_percpu_data, stack_canary); -BLANK (); -#endif return 0 ; }
arch/x86/kernel/cpu/common.c +1 -4
@@ -2089,8 +2089,7 @@ void syscall_init(void)if (!cpu_feature_enabled (X86_FEATURE_FRED ))idt_syscall_init (); } - -#else /* CONFIG_X86_64 */ +#endif /* CONFIG_X86_64 */ #ifdef CONFIG_STACKPROTECTOR DEFINE_PER_CPU(unsigned long, __stack_chk_guard);
@@ -2099,8 +2098,6 @@ EXPORT_PER_CPU_SYMBOL(__stack_chk_guard);#endif #endif -#endif /* CONFIG_X86_64 */ -/* * Clear all 6 debug registers: */
arch/x86/kernel/head_64.S +1 -2
@@ -361,8 +361,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */
arch/x86/xen/xen-head.S +1 -2
@@ -33,8 +33,7 @@ SYM_CODE_START(startup_xen) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */
Signature · 1 line
2.47.1
Re: [PATCH v6 07/15] x86/stackprotector/64: Convert to normal percpu variable
Quoted 3 lines · Brian Gerst
GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.
So there's two typos in this paragraph alone. Might be time to invest into an editor that does spellchecking for you?
s/allowng
/allowing
s/contraint
/constraint
Thanks,
Ingo
[tip: x86/asm] x86/stackprotector/64: Convert to normal per-CPU variable
In reply to Brian GerstThe following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 80d47defddc000271502057ebd7efa4fd6481542 Gitweb: https://git.kernel.org/tip/80d47defddc000271502057ebd7efa4fd6481542 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:39 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:09 +01:00
x86/stackprotector/64: Convert to normal per-CPU variable
Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.
GCC 8.1 supports redefining where the canary is located, allowing it to become a normal percpu variable instead of at a fixed location. This removes the constraint that the percpu section must be zero-based.
--- arch/x86/Makefile | 20 ++++++++------ arch/x86/entry/entry.S | 2 +- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/processor.h | 16 +----------- arch/x86/include/asm/stackprotector.h | 36 +++----------------------- arch/x86/kernel/asm-offsets_64.c | 6 +---- arch/x86/kernel/cpu/common.c | 5 +---- arch/x86/kernel/head_64.S | 3 +-- arch/x86/xen/xen-head.S | 3 +-- 9 files changed, 23 insertions(+), 70 deletions(-)
arch/x86/Makefile +12 -8
@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \ - -mstack-protector-guard-symbol=__ref_stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64
@@ -197,6 +190,17 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs +endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) + KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard + else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif endif #
arch/x86/entry/entry.S +0 -2
@@ -52,7 +52,6 @@ EXPORT_SYMBOL_GPL(mds_verw_sel); THUNK warn_thunk_thunk, __warn_thunk -#ifndef CONFIG_X86_64 /* * Clang's implementation of TLS stack cookies requires the variable in * question to be a TLS variable. If the variable happens to be defined as an
@@ -66,4 +65,3 @@ THUNK warn_thunk_thunk, __warn_thunk #ifdef CONFIG_STACKPROTECTOR EXPORT_SYMBOL(__ref_stack_chk_guard); #endif -#endif
arch/x86/entry/entry_64.S +1 -1
@@ -192,7 +192,7 @@ SYM_FUNC_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(fixed_percpu_data + FIXED_stack_canary) + movq %rbx, PER_CPU_VAR(__stack_chk_guard) #endif /*
arch/x86/include/asm/processor.h +2 -14
@@ -422,16 +422,8 @@ struct irq_stack {#ifdef CONFIG_X86_64 struct fixed_percpu_data { -/* -* GCC hardcodes the stack canary as %gs:40. Since the -* irq_stack is the object at %gs:0, we reserve the bottom -* 48 bytes of the irq stack for the canary. -* -* Once we are willing to require -mstack-protector-guard-symbol= -* support for x86_64 stackprotector, we can get rid of this. -*/ char gs_base[40 ]; -unsigned long stack_canary; +unsigned long reserved; }; DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
@@ -446,11 +438,7 @@ extern asmlinkage void entry_SYSCALL32_ignore(void);/* Save actual FS/GS selectors and bases to current->thread */ void current_save_fsgs (void ); -#else /* X86_64 */ -#ifdef CONFIG_STACKPROTECTOR -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); -#endif -#endif /* !X86_64 */ +#endif /* X86_64 */ struct perf_event ;
arch/x86/include/asm/stackprotector.h +5 -31
@@ -2,26 +2,10 @@/* * GCC stack protector support. * -* Stack protector works by putting predefined pattern at the start of +* Stack protector works by putting a predefined pattern at the start of * the stack frame and verifying that it hasn't been overwritten when -* returning from the function. The pattern is called stack canary -* and unfortunately gcc historically required it to be at a fixed offset -* from the percpu segment base. On x86_64, the offset is 40 bytes. -* -* The same segment is shared by percpu area and stack canary. On -* x86_64, percpu symbols are zero based and %gs (64-bit) points to the -* base of percpu area. The first occupant of the percpu area is always -* fixed_percpu_data which contains stack_canary at the appropriate -* offset. On x86_32, the stack canary is just a regular percpu -* variable. -* -* Putting percpu data in %fs on 32-bit is a minor optimization compared to -* using %gs. Since 32-bit userspace normally has %fs == 0, we are likely -* to load 0 into %fs on exit to usermode, whereas with percpu data in -* %gs, we are likely to load a non-null %gs on return to user mode. -* -* Once we are willing to require GCC 8.1 or better for 64-bit stackprotector -* support, we can remove some of this complexity. +* returning from the function. The pattern is called the stack canary +* and is a unique value for each task. */ #ifndef _ASM_STACKPROTECTOR_H
@@ -36,6 +20,8 @@#include <linux/sched.h> +DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +/* * Initialize the stackprotector canary value. *
@@ -51,25 +37,13 @@ static __always_inline void boot_init_stack_canary(void) {unsigned long canary =get_random_canary (); -#ifdef CONFIG_X86_64 -BUILD_BUG_ON (offsetof(struct fixed_percpu_data , stack_canary) !=40 ); -#endif - current->stack_canary = canary; -#ifdef CONFIG_X86_64 -this_cpu_write (fixed_percpu_data.stack_canary, canary); -#else this_cpu_write (__stack_chk_guard, canary); -#endif }static inline void cpu_init_stack_canary (int cpu,struct task_struct *idle) { -#ifdef CONFIG_X86_64 -per_cpu (fixed_percpu_data.stack_canary, cpu) = idle->stack_canary; -#else per_cpu (__stack_chk_guard, cpu) = idle->stack_canary; -#endif }#else /* STACKPROTECTOR */
arch/x86/kernel/asm-offsets_64.c +0 -6
@@ -54,11 +54,5 @@ int main(void)BLANK ();#undef ENTRY -BLANK (); - -#ifdef CONFIG_STACKPROTECTOR -OFFSET (FIXED_stack_canary, fixed_percpu_data, stack_canary); -BLANK (); -#endif return 0 ; }
arch/x86/kernel/cpu/common.c +1 -4
@@ -2089,8 +2089,7 @@ void syscall_init(void)if (!cpu_feature_enabled (X86_FEATURE_FRED ))idt_syscall_init (); } - -#else /* CONFIG_X86_64 */ +#endif /* CONFIG_X86_64 */ #ifdef CONFIG_STACKPROTECTOR DEFINE_PER_CPU(unsigned long, __stack_chk_guard);
@@ -2099,8 +2098,6 @@ EXPORT_PER_CPU_SYMBOL(__stack_chk_guard);#endif #endif -#endif /* CONFIG_X86_64 */ -/* * Clear all 6 debug registers: */
arch/x86/kernel/head_64.S +1 -2
@@ -361,8 +361,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */
arch/x86/xen/xen-head.S +1 -2
@@ -33,8 +33,7 @@ SYM_CODE_START(startup_xen) /* Set up %gs. * - * The base of %gs always points to fixed_percpu_data. If the - * stack protector canary is enabled, it is located at %gs:40. + * The base of %gs always points to fixed_percpu_data. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */
Nathan Chancellor
maintainerRe: [PATCH v6 07/15] x86/stackprotector/64: Convert to normal percpu variable
In reply to Brian GerstQuoted 15 lines · Brian Gerst
Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.
GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.
Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>Reviewed-by Uros Bizjak <[email protected]>
...
Quoted 26 lines
arch/x86/Makefile +3
-8@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \ - -mstack-protector-guard-symbol=__ref_stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64@@ -197,6 +190,17 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs
arch/x86/Makefile else · line 204
Quoted 10 lines
arch/x86/Makefile +9
-0+endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) + KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard + else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif endif
-mstack-protector-guard-symbol was only added in clang-15, so it looks like min-tool-version.sh will need an adjustment like GCC did, should I send a patch?
https://github.com/llvm/llvm-project/commit/efbaad1c4a526e91b034e56386e98a9268cd87b2
scripts/min-tool-version.sh +1 -1
@@ -26,7 +26,7 @@ gcc) fi ;; llvm) - if [ "$SRCARCH" = s390 ]; then + if [ "$SRCARCH" = s390 -o "$SRCARCH" = x86 ]; then echo 15.0.0 elif [ "$SRCARCH" = loongarch ]; then echo 18.0.0
Re: [PATCH v6 07/15] x86/stackprotector/64: Convert to normal percpu variable
Quoted 50 lines · Nathan Chancellor
Quoted 15 lines · Brian Gerst
Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.
GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.
Signed-off-by Brian Gerst <[email protected]>Reviewed-by Ard Biesheuvel <[email protected]>Reviewed-by Uros Bizjak <[email protected]>...
Quoted 32 lines
arch/x86/Makefile +9
-8@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y) # temporary until string.h is fixed KBUILD_CFLAGS += -ffreestanding - ifeq ($(CONFIG_STACKPROTECTOR),y) - ifeq ($(CONFIG_SMP),y) - KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \ - -mstack-protector-guard-symbol=__ref_stack_chk_guard - else - KBUILD_CFLAGS += -mstack-protector-guard=global - endif - endif + percpu_seg := fs else BITS := 64 UTS_MACHINE := x86_64@@ -197,6 +190,17 @@ else KBUILD_CFLAGS += -mcmodel=kernel KBUILD_RUSTFLAGS += -Cno-redzone=y KBUILD_RUSTFLAGS += -Ccode-model=kernel + + percpu_seg := gs +endif + +ifeq ($(CONFIG_STACKPROTECTOR),y) + ifeq ($(CONFIG_SMP),y) + KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg) + KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard
Quoted 9 lines
arch/x86/Makefile else · line 204
Quoted 4 lines
arch/x86/Makefile +3
-0+ else + KBUILD_CFLAGS += -mstack-protector-guard=global + endif endif-mstack-protector-guard-symbol was only added in clang-15, so it looks like
min-tool-version.shwill need an adjustment like GCC did, should I send a patch?
Yes, please.
[tip: x86/asm] x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP
In reply to Brian GerstThe following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: c0560fbaab0dab0afbdb3e339b3cb61ef8d3115e Gitweb: https://git.kernel.org/tip/c0560fbaab0dab0afbdb3e339b3cb61ef8d3115e Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 12 Mar 2025 12:48:49 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 12 Mar 2025 12:51:35 +01:00
x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP
The __ref_stack_chk_guard symbol doesn't exist on UP:
<stdin>:4:15: error: ‘__ref_stack_chk_guard’ undeclared here (not in a function)
Fix the #ifdef around the entry.S export.
--- arch/x86/entry/entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/entry/entry.S +1 -1
@@ -62,6 +62,6 @@ THUNK warn_thunk_thunk, __warn_thunk * entirely in the C code, and use an alias emitted by the linker script * instead. */ -#ifdef CONFIG_STACKPROTECTOR +#if defined(CONFIG_STACKPROTECTOR) && defined(CONFIG_SMP) EXPORT_SYMBOL(__ref_stack_chk_guard); #endif
[tip: x86/core] x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP
In reply to Brian GerstThe following commit has been merged into the x86/core branch of tip:
The following commit has been merged into the x86/core branch of tip:
Commit-ID: 91d5451d97ce35cbd510277fa3b7abf9caa4e34d Gitweb: https://git.kernel.org/tip/91d5451d97ce35cbd510277fa3b7abf9caa4e34d Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 12 Mar 2025 12:48:49 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 19 Mar 2025 11:26:58 +01:00
x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP
The __ref_stack_chk_guard symbol doesn't exist on UP:
<stdin>:4:15: error: ‘__ref_stack_chk_guard’ undeclared here (not in a function)
Fix the #ifdef around the entry.S export.
--- arch/x86/entry/entry.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/entry/entry.S +1 -1
@@ -64,6 +64,6 @@ THUNK warn_thunk_thunk, __warn_thunk * entirely in the C code, and use an alias emitted by the linker script * instead. */ -#ifdef CONFIG_STACKPROTECTOR +#if defined(CONFIG_STACKPROTECTOR) && defined(CONFIG_SMP) EXPORT_SYMBOL(__ref_stack_chk_guard); #endif
PATCH v6 8/15 x86/percpu/64: Use relative percpu offsets
In reply to Brian GerstThe percpu section is currently linked at absolute address 0, because older compilers hardcoded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section does not need to be linked at a specific address.
x86-64 will now calculate the percpu offsets as the delta between the initial percpu address and the dynamically allocated memory, like other architectures. Note that GSBASE is limited to the canonical address width (48 or 57 bits, sign-extended). As long as the kernel text, modules, and the dynamically allocated percpu memmory are all in the negative address space, the delta will not overflow this limit.
--- arch/x86/include/asm/processor.h | 6 +++++- arch/x86/kernel/head_64.S | 19 +++++++++---------- arch/x86/kernel/setup_percpu.c | 12 ++---------- arch/x86/kernel/vmlinux.lds.S | 29 +---------------------------- arch/x86/platform/pvh/head.S | 5 ++--- arch/x86/tools/relocs.c | 10 +++------- arch/x86/xen/xen-head.S | 9 ++++----- init/Kconfig | 2 +- 8 files changed, 27 insertions(+), 65 deletions(-)
arch/x86/include/asm/processor.h +5 -1
@@ -431,7 +431,11 @@ DECLARE_INIT_PER_CPU(fixed_percpu_data);static inline unsigned long cpu_kernelmode_gs_base (int cpu) { -return (unsigned long )per_cpu (fixed_percpu_data.gs_base, cpu); +#ifdef CONFIG_SMP +return per_cpu_offset (cpu); +#else +return 0 ; +#endif } extern asmlinkage void entry_SYSCALL32_ignore(void);
arch/x86/kernel/head_64.S +9 -10
@@ -61,11 +61,14 @@ SYM_CODE_START_NOALIGN(startup_64) /* Set up the stack for verify_cpu() */ leaq __top_init_kernel_stack(%rip), %rsp - /* Setup GSBASE to allow stack canary access for C code */ + /* + * Set up GSBASE. + * Note that, on SMP, the boot cpu uses init data section until + * the per cpu areas are set up. + */ movl $MSR_GS_BASE, %ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx + xorl %eax, %eax + xorl %edx, %edx wrmsr call startup_64_setup_gdt_idt
@@ -359,16 +362,12 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) movl %eax,%fs movl %eax,%gs - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. + /* + * Set up GSBASE. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifndef CONFIG_SMP - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx -#endif movl %edx, %eax shrq $32, %rdx wrmsr
arch/x86/kernel/setup_percpu.c +2 -10
@@ -23,18 +23,10 @@#include <asm/cpumask.h> #include <asm/cpu.h> -#ifdef CONFIG_X86_64 -#define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load) -#else -#define BOOT_PERCPU_OFFSET 0 -#endif - -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off) = BOOT_PERCPU_OFFSET; +DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off);EXPORT_PER_CPU_SYMBOL (this_cpu_off); -unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init = { - [0 ...NR_CPUS -1 ] =BOOT_PERCPU_OFFSET , -}; +unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init ;EXPORT_SYMBOL (__per_cpu_offset);/*
arch/x86/kernel/vmlinux.lds.S +1 -28
@@ -112,12 +112,6 @@ ASSERT(__relocate_kernel_end - __relocate_kernel_start <= KEXEC_CONTROL_CODE_MAX PHDRS { text PT_LOAD FLAGS(5); /* R_E */ data PT_LOAD FLAGS(6); /* RW_ */ -#ifdef CONFIG_X86_64 -#ifdef CONFIG_SMP - percpu PT_LOAD FLAGS(6); /* RW_ */ -#endif - init PT_LOAD FLAGS(7); /* RWE */ -#endif note PT_NOTE FLAGS(0); /* ___ */ }
@@ -216,21 +210,7 @@ SECTIONS __init_begin = .; /* paired with __init_end */ } -#if defined(CONFIG_X86_64) && defined(CONFIG_SMP) - /* - * percpu offsets are zero-based on SMP. PERCPU_VADDR() changes the - * output PHDR, so the next output section - .init.text - should - * start another segment - init. - */ - PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu) - ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START, - "per-CPU data too large - increase CONFIG_PHYSICAL_START") -#endif - INIT_TEXT_SECTION(PAGE_SIZE) -#ifdef CONFIG_X86_64 - :init -#endif /* * Section for code used exclusively before alternatives are run. All
@@ -347,9 +327,7 @@ SECTIONS EXIT_DATA } -#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP) PERCPU_SECTION(INTERNODE_CACHE_BYTES) -#endif RUNTIME_CONST_VARIABLES RUNTIME_CONST(ptr, USER_PTR_MAX)
@@ -497,16 +475,11 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard); * Per-cpu symbols which need to be offset from __per_cpu_load * for the boot processor. */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load +#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) INIT_PER_CPU(gdt_page); INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); -#ifdef CONFIG_SMP -. = ASSERT((fixed_percpu_data == 0), - "fixed_percpu_data is not at start of per-cpu area"); -#endif - #ifdef CONFIG_MITIGATION_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned"); #endif
arch/x86/platform/pvh/head.S +2 -3
@@ -179,9 +179,8 @@ SYM_CODE_START(pvh_start_xen) * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx - movq %edx, %eax - shrq $32, %rdx + xorl %eax, %eax + xorl %edx, %edx wrmsr /* Call xen_prepare_pvh() via the kernel virtual mapping */
arch/x86/tools/relocs.c +3 -7
@@ -835,12 +835,7 @@ static void percpu_init(void)*/ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname) { -int shndx =sym_index (sym); - -return (shndx == per_cpu_shndx) && -strcmp (symname,"__init_begin" ) && -strcmp (symname,"__per_cpu_load" ) && -strncmp (symname,"init_per_cpu_" ,13 ); +return 0 ; }
@@ -1062,7 +1057,8 @@ static int cmp_relocs(const void *va, const void *vb)static void sort_relocs (struct relocs *r) { -qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); +if (r->count) +qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); }static int write32 (uint32_t v,FILE *f)
arch/x86/xen/xen-head.S +4 -5
@@ -31,15 +31,14 @@ SYM_CODE_START(startup_xen) leaq __top_init_kernel_stack(%rip), %rsp - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. + /* + * Set up GSBASE. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax - cdq + xorl %eax, %eax + xorl %edx, %edx wrmsr mov %rsi, %rdi
init/Kconfig +1 -1
@@ -1873,7 +1873,7 @@ config KALLSYMS_ALL config KALLSYMS_ABSOLUTE_PERCPU bool depends on KALLSYMS - default X86_64 && SMP + default n # end of the "standard kernel features (expert users)" menu
Signature · 1 line
2.47.1
[tip: x86/asm] x86/percpu/64: Use relative percpu offsets
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 9d7de2aa8b41407bc96d89a80dc1fd637d389d42 Gitweb: https://git.kernel.org/tip/9d7de2aa8b41407bc96d89a80dc1fd637d389d42 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:40 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:27 +01:00
x86/percpu/64: Use relative percpu offsets
The percpu section is currently linked at absolute address 0, because older compilers hard-coded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section does not need to be linked at a specific address.
x86-64 will now calculate the percpu offsets as the delta between the initial percpu address and the dynamically allocated memory, like other architectures. Note that GSBASE is limited to the canonical address width (48 or 57 bits, sign-extended). As long as the kernel text, modules, and the dynamically allocated percpu memory are all in the negative address space, the delta will not overflow this limit.
--- arch/x86/include/asm/processor.h | 6 +++++- arch/x86/kernel/head_64.S | 19 +++++++++---------- arch/x86/kernel/setup_percpu.c | 12 ++---------- arch/x86/kernel/vmlinux.lds.S | 29 +---------------------------- arch/x86/platform/pvh/head.S | 5 ++--- arch/x86/tools/relocs.c | 10 +++------- arch/x86/xen/xen-head.S | 9 ++++----- init/Kconfig | 2 +- 8 files changed, 27 insertions(+), 65 deletions(-)
arch/x86/include/asm/processor.h +5 -1
@@ -431,7 +431,11 @@ DECLARE_INIT_PER_CPU(fixed_percpu_data);static inline unsigned long cpu_kernelmode_gs_base (int cpu) { -return (unsigned long )per_cpu (fixed_percpu_data.gs_base, cpu); +#ifdef CONFIG_SMP +return per_cpu_offset (cpu); +#else +return 0 ; +#endif } extern asmlinkage void entry_SYSCALL32_ignore(void);
arch/x86/kernel/head_64.S +9 -10
@@ -61,11 +61,14 @@ SYM_CODE_START_NOALIGN(startup_64) /* Set up the stack for verify_cpu() */ leaq __top_init_kernel_stack(%rip), %rsp - /* Setup GSBASE to allow stack canary access for C code */ + /* + * Set up GSBASE. + * Note that on SMP the boot CPU uses the init data section until + * the per-CPU areas are set up. + */ movl $MSR_GS_BASE, %ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx - movl %edx, %eax - shrq $32, %rdx + xorl %eax, %eax + xorl %edx, %edx wrmsr call startup_64_setup_gdt_idt
@@ -359,16 +362,12 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) movl %eax,%fs movl %eax,%gs - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. + /* + * Set up GSBASE. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx -#ifndef CONFIG_SMP - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx -#endif movl %edx, %eax shrq $32, %rdx wrmsr
arch/x86/kernel/setup_percpu.c +2 -10
@@ -23,18 +23,10 @@#include <asm/cpumask.h> #include <asm/cpu.h> -#ifdef CONFIG_X86_64 -#define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load) -#else -#define BOOT_PERCPU_OFFSET 0 -#endif - -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off) = BOOT_PERCPU_OFFSET; +DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off);EXPORT_PER_CPU_SYMBOL (this_cpu_off); -unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init = { - [0 ...NR_CPUS -1 ] =BOOT_PERCPU_OFFSET , -}; +unsigned long __per_cpu_offset[NR_CPUS ]__ro_after_init ;EXPORT_SYMBOL (__per_cpu_offset);/*
arch/x86/kernel/vmlinux.lds.S +1 -28
@@ -112,12 +112,6 @@ ASSERT(__relocate_kernel_end - __relocate_kernel_start <= KEXEC_CONTROL_CODE_MAX PHDRS { text PT_LOAD FLAGS(5); /* R_E */ data PT_LOAD FLAGS(6); /* RW_ */ -#ifdef CONFIG_X86_64 -#ifdef CONFIG_SMP - percpu PT_LOAD FLAGS(6); /* RW_ */ -#endif - init PT_LOAD FLAGS(7); /* RWE */ -#endif note PT_NOTE FLAGS(0); /* ___ */ }
@@ -216,21 +210,7 @@ SECTIONS __init_begin = .; /* paired with __init_end */ } -#if defined(CONFIG_X86_64) && defined(CONFIG_SMP) - /* - * percpu offsets are zero-based on SMP. PERCPU_VADDR() changes the - * output PHDR, so the next output section - .init.text - should - * start another segment - init. - */ - PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu) - ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START, - "per-CPU data too large - increase CONFIG_PHYSICAL_START") -#endif - INIT_TEXT_SECTION(PAGE_SIZE) -#ifdef CONFIG_X86_64 - :init -#endif /* * Section for code used exclusively before alternatives are run. All
@@ -347,9 +327,7 @@ SECTIONS EXIT_DATA } -#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP) PERCPU_SECTION(INTERNODE_CACHE_BYTES) -#endif RUNTIME_CONST_VARIABLES RUNTIME_CONST(ptr, USER_PTR_MAX)
@@ -497,16 +475,11 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard); * Per-cpu symbols which need to be offset from __per_cpu_load * for the boot processor. */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load +#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) INIT_PER_CPU(gdt_page); INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); -#ifdef CONFIG_SMP -. = ASSERT((fixed_percpu_data == 0), - "fixed_percpu_data is not at start of per-cpu area"); -#endif - #ifdef CONFIG_MITIGATION_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned"); #endif
arch/x86/platform/pvh/head.S +2 -3
@@ -179,9 +179,8 @@ SYM_CODE_START(pvh_start_xen) * the per-CPU areas are set up. */ movl $MSR_GS_BASE,%ecx - leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx - movq %edx, %eax - shrq $32, %rdx + xorl %eax, %eax + xorl %edx, %edx wrmsr /* Call xen_prepare_pvh() via the kernel virtual mapping */
arch/x86/tools/relocs.c +3 -7
@@ -835,12 +835,7 @@ static void percpu_init(void)*/ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname) { -int shndx =sym_index (sym); - -return (shndx == per_cpu_shndx) && -strcmp (symname,"__init_begin" ) && -strcmp (symname,"__per_cpu_load" ) && -strncmp (symname,"init_per_cpu_" ,13 ); +return 0 ; }
@@ -1062,7 +1057,8 @@ static int cmp_relocs(const void *va, const void *vb)static void sort_relocs (struct relocs *r) { -qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); +if (r->count) +qsort (r->offset, r->count,sizeof (r->offset[0 ]), cmp_relocs); }static int write32 (uint32_t v,FILE *f)
arch/x86/xen/xen-head.S +4 -5
@@ -31,15 +31,14 @@ SYM_CODE_START(startup_xen) leaq __top_init_kernel_stack(%rip), %rsp - /* Set up %gs. - * - * The base of %gs always points to fixed_percpu_data. + /* + * Set up GSBASE. * Note that, on SMP, the boot cpu uses init data section until * the per cpu areas are set up. */ movl $MSR_GS_BASE,%ecx - movq $INIT_PER_CPU_VAR(fixed_percpu_data),%rax - cdq + xorl %eax, %eax + xorl %edx, %edx wrmsr mov %rsi, %rdi
init/Kconfig +1 -1
@@ -1872,7 +1872,7 @@ config KALLSYMS_ALL config KALLSYMS_ABSOLUTE_PERCPU bool depends on KALLSYMS - default X86_64 && SMP + default n # end of the "standard kernel features (expert users)" menu
PATCH v6 9/15 x86/percpu/64: Remove fixed_percpu_data
In reply to Brian GerstNow that the stack protector canary value is a normal percpu variable, fixed_percpu_data is unused and can be removed.
--- arch/x86/include/asm/processor.h | 8 -------- arch/x86/kernel/cpu/common.c | 4 ---- arch/x86/kernel/vmlinux.lds.S | 1 - arch/x86/tools/relocs.c | 1 - 4 files changed, 14 deletions(-)
arch/x86/include/asm/processor.h +0 -8
@@ -421,14 +421,6 @@ struct irq_stack { }__aligned(IRQ_STACK_SIZE) ;#ifdef CONFIG_X86_64 -struct fixed_percpu_data { -char gs_base[40 ]; -unsigned long reserved; -}; - -DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible; -DECLARE_INIT_PER_CPU (fixed_percpu_data); -static inline unsigned long cpu_kernelmode_gs_base (int cpu) {#ifdef CONFIG_SMP
arch/x86/kernel/cpu/common.c +0 -4
@@ -2023,10 +2023,6 @@ EXPORT_PER_CPU_SYMBOL(pcpu_hot);EXPORT_PER_CPU_SYMBOL (const_pcpu_hot);#ifdef CONFIG_X86_64 -DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, - fixed_percpu_data)__aligned(PAGE_SIZE) __visible; -EXPORT_PER_CPU_SYMBOL_GPL (fixed_percpu_data); -static void wrmsrl_cstar (unsigned long val) {/*
arch/x86/kernel/vmlinux.lds.S +0 -1
@@ -477,7 +477,6 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard); */ #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) INIT_PER_CPU(gdt_page); -INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); #ifdef CONFIG_MITIGATION_UNRET_ENTRY
arch/x86/tools/relocs.c +0 -1
@@ -830,7 +830,6 @@ static void percpu_init(void)* __per_cpu_load * * The "gold" linker incorrectly associates: -* init_per_cpu__fixed_percpu_data * init_per_cpu__gdt_page */ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname)
Signature · 1 line
2.47.1
[tip: x86/asm] x86/percpu/64: Remove fixed_percpu_data
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: b5c4f95351a097a635c1a7fc8d9efa18308491b5 Gitweb: https://git.kernel.org/tip/b5c4f95351a097a635c1a7fc8d9efa18308491b5 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:41 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:43 +01:00
x86/percpu/64: Remove fixed_percpu_data
Now that the stack protector canary value is a normal percpu variable, fixed_percpu_data is unused and can be removed.
--- arch/x86/include/asm/processor.h | 8 -------- arch/x86/kernel/cpu/common.c | 4 ---- arch/x86/kernel/vmlinux.lds.S | 1 - arch/x86/tools/relocs.c | 1 - 4 files changed, 14 deletions(-)
arch/x86/include/asm/processor.h +0 -8
@@ -421,14 +421,6 @@ struct irq_stack { }__aligned(IRQ_STACK_SIZE) ;#ifdef CONFIG_X86_64 -struct fixed_percpu_data { -char gs_base[40 ]; -unsigned long reserved; -}; - -DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible; -DECLARE_INIT_PER_CPU (fixed_percpu_data); -static inline unsigned long cpu_kernelmode_gs_base (int cpu) {#ifdef CONFIG_SMP
arch/x86/kernel/cpu/common.c +0 -4
@@ -2023,10 +2023,6 @@ EXPORT_PER_CPU_SYMBOL(pcpu_hot);EXPORT_PER_CPU_SYMBOL (const_pcpu_hot);#ifdef CONFIG_X86_64 -DEFINE_PER_CPU_FIRST(struct fixed_percpu_data, - fixed_percpu_data)__aligned(PAGE_SIZE) __visible; -EXPORT_PER_CPU_SYMBOL_GPL (fixed_percpu_data); -static void wrmsrl_cstar (unsigned long val) {/*
arch/x86/kernel/vmlinux.lds.S +0 -1
@@ -477,7 +477,6 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard); */ #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) INIT_PER_CPU(gdt_page); -INIT_PER_CPU(fixed_percpu_data); INIT_PER_CPU(irq_stack_backing_store); #ifdef CONFIG_MITIGATION_UNRET_ENTRY
arch/x86/tools/relocs.c +0 -1
@@ -830,7 +830,6 @@ static void percpu_init(void)* __per_cpu_load * * The "gold" linker incorrectly associates: -* init_per_cpu__fixed_percpu_data * init_per_cpu__gdt_page */ static int is_percpu_sym (ElfW (Sym) *sym,const char *symname)
PATCH v6 10/15 x86/boot/64: Remove inverse relocations
In reply to Brian GerstInverse relocations were needed to offset the effects of relocation for RIP-relative accesses to zero-based percpu data. Now that the percpu section is linked normally as part of the kernel image, they are no longer needed.
--- arch/x86/boot/compressed/misc.c | 14 +--- arch/x86/tools/relocs.c | 130 +------------------------------- 2 files changed, 2 insertions(+), 142 deletions(-)
arch/x86/boot/compressed/misc.c +1 -13
@@ -235,7 +235,7 @@ static void handle_relocations(void *output, unsigned long output_len,/* * Process relocations: 32 bit relocations first then 64 bit after. -* Three sets of binary relocations are added to the end of the kernel +* Two sets of binary relocations are added to the end of the kernel * before compression. Each relocation table entry is the kernel * address of the location which needs to be updated stored as a * 32-bit value which is sign extended to 64 bits.
@@ -245,8 +245,6 @@ static void handle_relocations(void *output, unsigned long output_len,* kernel bits... * 0 - zero terminator for 64 bit relocations * 64 bit relocation repeated -* 0 - zero terminator for inverse 32 bit relocations -* 32 bit inverse relocation repeated * 0 - zero terminator for 32 bit relocations * 32 bit relocation repeated *
@@ -263,16 +261,6 @@ static void handle_relocations(void *output, unsigned long output_len, *(uint32_t *)ptr += delta; }#ifdef CONFIG_X86_64 -while (*--reloc) { -long extended = *reloc; - extended += map; - - ptr = (unsigned long )extended; -if (ptr < min_addr || ptr > max_addr) -error ("inverse 32-bit relocation outside of kernel!\n" ); - - *(int32_t *)ptr -= delta; - }for (reloc--; *reloc; reloc--) {long extended = *reloc; extended += map;
arch/x86/tools/relocs.c +1 -129
200 lines not on this page · raw message
Signature · 1 line
2.47.1
[tip: x86/asm] x86/boot/64: Remove inverse relocations
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: a8327be7b2aa067ff2b11551732d5bd8b49ef7d1 Gitweb: https://git.kernel.org/tip/a8327be7b2aa067ff2b11551732d5bd8b49ef7d1 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:42 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:47 +01:00
x86/boot/64: Remove inverse relocations
Inverse relocations were needed to offset the effects of relocation for RIP-relative accesses to zero-based percpu data. Now that the percpu section is linked normally as part of the kernel image, they are no longer needed.
--- arch/x86/boot/compressed/misc.c | 14 +--- arch/x86/tools/relocs.c | 130 +------------------------------- 2 files changed, 2 insertions(+), 142 deletions(-)
arch/x86/boot/compressed/misc.c +1 -13
@@ -235,7 +235,7 @@ static void handle_relocations(void *output, unsigned long output_len,/* * Process relocations: 32 bit relocations first then 64 bit after. -* Three sets of binary relocations are added to the end of the kernel +* Two sets of binary relocations are added to the end of the kernel * before compression. Each relocation table entry is the kernel * address of the location which needs to be updated stored as a * 32-bit value which is sign extended to 64 bits.
@@ -245,8 +245,6 @@ static void handle_relocations(void *output, unsigned long output_len,* kernel bits... * 0 - zero terminator for 64 bit relocations * 64 bit relocation repeated -* 0 - zero terminator for inverse 32 bit relocations -* 32 bit inverse relocation repeated * 0 - zero terminator for 32 bit relocations * 32 bit relocation repeated *
@@ -263,16 +261,6 @@ static void handle_relocations(void *output, unsigned long output_len, *(uint32_t *)ptr += delta; }#ifdef CONFIG_X86_64 -while (*--reloc) { -long extended = *reloc; - extended += map; - - ptr = (unsigned long )extended; -if (ptr < min_addr || ptr > max_addr) -error ("inverse 32-bit relocation outside of kernel!\n" ); - - *(int32_t *)ptr -= delta; - }for (reloc--; *reloc; reloc--) {long extended = *reloc; extended += map;
arch/x86/tools/relocs.c +1 -129
200 lines not on this page · raw message
PATCH v6 11/15 x86/percpu/64: Remove INIT_PER_CPU macros
In reply to Brian GerstNow that the load and link addresses of percpu variables are the same, these macros are no longer necessary.
--- arch/x86/include/asm/desc.h | 1 - arch/x86/include/asm/percpu.h | 22 ---------------------- arch/x86/kernel/head64.c | 2 +- arch/x86/kernel/irq_64.c | 1 - arch/x86/kernel/vmlinux.lds.S | 7 ------- arch/x86/tools/relocs.c | 1 - 6 files changed, 1 insertion(+), 33 deletions(-)
arch/x86/include/asm/desc.h +0 -1
@@ -46,7 +46,6 @@ struct gdt_page { } __attribute__((aligned(PAGE_SIZE))); DECLARE_PER_CPU_PAGE_ALIGNED(struct gdt_page, gdt_page); -DECLARE_INIT_PER_CPU (gdt_page);/* Provide the original GDT */ static inline struct desc_struct *get_cpu_gdt_rw (unsigned int cpu)
arch/x86/include/asm/percpu.h +0 -22
@@ -20,12 +20,6 @@#define PER_CPU_VAR (var) __percpu(var)__percpu_rel -#ifdef CONFIG_X86_64_SMP -# define INIT_PER_CPU_VAR (var) init_per_cpu__##var -#else -# define INIT_PER_CPU_VAR (var) var -#endif -#else /* !__ASSEMBLY__: */ #include <linux/build_bug.h>
@@ -97,22 +91,6 @@#define __percpu_arg (x) __percpu_prefix "%" #x#define __force_percpu_arg (x) __force_percpu_prefix "%" #x -/* -* Initialized pointers to per-CPU variables needed for the boot -* processor need to use these macros to get the proper address -* offset from __per_cpu_load on SMP. -* -* There also must be an entry in vmlinux_64.lds.S -*/ -#define DECLARE_INIT_PER_CPU (var) \ - extern typeof(var) init_per_cpu_var(var) - -#ifdef CONFIG_X86_64_SMP -# define init_per_cpu_var (var) init_per_cpu__##var -#else -# define init_per_cpu_var (var) var -#endif -/* * For arch-specific code, we can use direct single-insn ops (they * don't give an lvalue though).
arch/x86/kernel/head64.c +1 -1
@@ -567,7 +567,7 @@ void early_setup_idt(void)*/ void __head startup_64_setup_gdt_idt(void) { -struct desc_struct *gdt = (void *)(__force unsigned long )init_per_cpu_var (gdt_page.gdt); +struct desc_struct *gdt = (void *)(__force unsigned long )gdt_page.gdt;void *handler =NULL ;struct desc_ptr startup_gdt_descr = {
arch/x86/kernel/irq_64.c +0 -1
@@ -27,7 +27,6 @@#include <asm/apic.h> DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __visible; -DECLARE_INIT_PER_CPU (irq_stack_backing_store);#ifdef CONFIG_VMAP_STACK /*
arch/x86/kernel/vmlinux.lds.S +0 -7
@@ -471,13 +471,6 @@ SECTIONS PROVIDE(__ref_stack_chk_guard = __stack_chk_guard); #ifdef CONFIG_X86_64 -/* - * Per-cpu symbols which need to be offset from __per_cpu_load - * for the boot processor. - */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) -INIT_PER_CPU(gdt_page); -INIT_PER_CPU(irq_stack_backing_store); #ifdef CONFIG_MITIGATION_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
arch/x86/tools/relocs.c +0 -1
@@ -90,7 +90,6 @@ static const char * const sym_regex_kernel[S_NSYMTYPES] = {"__initramfs_start|" "(jiffies|jiffies_64)|"#if ELF_BITS ==64 -"init_per_cpu__.*|" "__end_rodata_hpage_align|"#endif "_end)$"
Signature · 1 line
2.47.1
[tip: x86/asm] x86/percpu/64: Remove INIT_PER_CPU macros
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 38a4968b3190f873a8a60e953287278eddf037f1 Gitweb: https://git.kernel.org/tip/38a4968b3190f873a8a60e953287278eddf037f1 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:43 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:50 +01:00
x86/percpu/64: Remove INIT_PER_CPU macros
Now that the load and link addresses of percpu variables are the same, these macros are no longer necessary.
--- arch/x86/include/asm/desc.h | 1 - arch/x86/include/asm/percpu.h | 22 ---------------------- arch/x86/kernel/head64.c | 2 +- arch/x86/kernel/irq_64.c | 1 - arch/x86/kernel/vmlinux.lds.S | 7 ------- arch/x86/tools/relocs.c | 1 - 6 files changed, 1 insertion(+), 33 deletions(-)
arch/x86/include/asm/desc.h +0 -1
@@ -46,7 +46,6 @@ struct gdt_page { } __attribute__((aligned(PAGE_SIZE))); DECLARE_PER_CPU_PAGE_ALIGNED(struct gdt_page, gdt_page); -DECLARE_INIT_PER_CPU (gdt_page);/* Provide the original GDT */ static inline struct desc_struct *get_cpu_gdt_rw (unsigned int cpu)
arch/x86/include/asm/percpu.h +0 -22
@@ -20,12 +20,6 @@#define PER_CPU_VAR (var) __percpu(var)__percpu_rel -#ifdef CONFIG_X86_64_SMP -# define INIT_PER_CPU_VAR (var) init_per_cpu__##var -#else -# define INIT_PER_CPU_VAR (var) var -#endif -#else /* !__ASSEMBLY__: */ #include <linux/build_bug.h>
@@ -98,22 +92,6 @@#define __force_percpu_arg (x) __force_percpu_prefix "%" #x/* -* Initialized pointers to per-CPU variables needed for the boot -* processor need to use these macros to get the proper address -* offset from __per_cpu_load on SMP. -* -* There also must be an entry in vmlinux_64.lds.S -*/ -#define DECLARE_INIT_PER_CPU (var) \ - extern typeof(var) init_per_cpu_var(var) - -#ifdef CONFIG_X86_64_SMP -# define init_per_cpu_var (var) init_per_cpu__##var -#else -# define init_per_cpu_var (var) var -#endif - -/* * For arch-specific code, we can use direct single-insn ops (they * don't give an lvalue though). */
arch/x86/kernel/head64.c +1 -1
@@ -567,7 +567,7 @@ void early_setup_idt(void)*/ void __head startup_64_setup_gdt_idt(void) { -struct desc_struct *gdt = (void *)(__force unsigned long )init_per_cpu_var (gdt_page.gdt); +struct desc_struct *gdt = (void *)(__force unsigned long )gdt_page.gdt;void *handler =NULL ;struct desc_ptr startup_gdt_descr = {
arch/x86/kernel/irq_64.c +0 -1
@@ -27,7 +27,6 @@#include <asm/apic.h> DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __visible; -DECLARE_INIT_PER_CPU (irq_stack_backing_store);#ifdef CONFIG_VMAP_STACK /*
arch/x86/kernel/vmlinux.lds.S +0 -7
@@ -471,13 +471,6 @@ SECTIONS PROVIDE(__ref_stack_chk_guard = __stack_chk_guard); #ifdef CONFIG_X86_64 -/* - * Per-cpu symbols which need to be offset from __per_cpu_load - * for the boot processor. - */ -#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) -INIT_PER_CPU(gdt_page); -INIT_PER_CPU(irq_stack_backing_store); #ifdef CONFIG_MITIGATION_UNRET_ENTRY . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
arch/x86/tools/relocs.c +0 -1
@@ -90,7 +90,6 @@ static const char * const sym_regex_kernel[S_NSYMTYPES] = {"__initramfs_start|" "(jiffies|jiffies_64)|"#if ELF_BITS ==64 -"init_per_cpu__.*|" "__end_rodata_hpage_align|"#endif "_end)$"
PATCH v6 12/15 percpu: Remove PER_CPU_FIRST_SECTION
In reply to Brian Gerstx86-64 was the last user.
--- include/asm-generic/vmlinux.lds.h | 1 - include/linux/percpu-defs.h | 12 ------------ 2 files changed, 13 deletions(-)
include/asm-generic/vmlinux.lds.h +0 -1
@@ -1061,7 +1061,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)*/ #define PERCPU_INPUT (cacheline) \ __per_cpu_start = .; \ - *(.data..percpu..first) \ . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ . = ALIGN(cacheline); \
include/linux/percpu-defs.h +0 -12
@@ -26,13 +26,11 @@#define PER_CPU_SHARED_ALIGNED_SECTION "..shared_aligned"#define PER_CPU_ALIGNED_SECTION "..shared_aligned"#endif -#define PER_CPU_FIRST_SECTION "..first"#else #define PER_CPU_SHARED_ALIGNED_SECTION ""#define PER_CPU_ALIGNED_SECTION "..shared_aligned" -#define PER_CPU_FIRST_SECTION ""#endif
@@ -114,16 +112,6 @@#define DEFINE_PER_CPU (type, name) \ DEFINE_PER_CPU_SECTION(type, name, "") -/* -* Declaration/definition used for per-CPU variables that must come first in -* the set of variables. -*/ -#define DECLARE_PER_CPU_FIRST (type, name) \ - DECLARE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) - -#define DEFINE_PER_CPU_FIRST (type, name) \ - DEFINE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) -/* * Declaration/definition used for per-CPU variables that must be cacheline * aligned under SMP conditions so that, whilst a particular instance of the
Signature · 1 line
2.47.1
[tip: x86/asm] percpu: Remove PER_CPU_FIRST_SECTION
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 95b0916118106054e1f3d5d7f8628ef3dc0b3c02 Gitweb: https://git.kernel.org/tip/95b0916118106054e1f3d5d7f8628ef3dc0b3c02 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:44 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:53 +01:00
percpu: Remove PER_CPU_FIRST_SECTION
x86-64 was the last user.
--- include/asm-generic/vmlinux.lds.h | 1 - include/linux/percpu-defs.h | 12 ------------ 2 files changed, 13 deletions(-)
include/asm-generic/vmlinux.lds.h +0 -1
@@ -1062,7 +1062,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)*/ #define PERCPU_INPUT (cacheline) \ __per_cpu_start = .; \ - *(.data..percpu..first) \ . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ . = ALIGN(cacheline); \
include/linux/percpu-defs.h +0 -12
@@ -26,13 +26,11 @@#define PER_CPU_SHARED_ALIGNED_SECTION "..shared_aligned"#define PER_CPU_ALIGNED_SECTION "..shared_aligned"#endif -#define PER_CPU_FIRST_SECTION "..first"#else #define PER_CPU_SHARED_ALIGNED_SECTION ""#define PER_CPU_ALIGNED_SECTION "..shared_aligned" -#define PER_CPU_FIRST_SECTION ""#endif
@@ -115,16 +113,6 @@DEFINE_PER_CPU_SECTION (type, name,"" )/* -* Declaration/definition used for per-CPU variables that must come first in -* the set of variables. - */ -#define DECLARE_PER_CPU_FIRST (type, name) \ - DECLARE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) - -#define DEFINE_PER_CPU_FIRST (type, name) \ - DEFINE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION) - -/* * Declaration/definition used for per-CPU variables that must be cacheline * aligned under SMP conditions so that, whilst a particular instance of the * data corresponds to a particular CPU, inefficiencies due to direct access by
PATCH v6 13/15 percpu: Remove PERCPU_VADDR()
In reply to Brian Gerstx86-64 was the last user.
--- include/asm-generic/vmlinux.lds.h | 36 +------------------------------ 1 file changed, 1 insertion(+), 35 deletions(-)
include/asm-generic/vmlinux.lds.h +1 -35
@@ -1072,47 +1072,13 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG) __per_cpu_end = .;/** -* PERCPU_VADDR - define output section for percpu area +* PERCPU_SECTION - define output section for percpu area * @cacheline: cacheline size -* @vaddr: explicit base address (optional) -* @phdr: destination PHDR (optional) * * Macro which expands to output section for percpu area. * * @cacheline is used to align subsections to avoid false cacheline * sharing between subsections for different purposes. -* -* If @vaddr is not blank, it specifies explicit base address and all -* percpu symbols will be offset from the given address. If blank, -* @vaddr always equals @laddr + LOAD_OFFSET. -* -* @phdr defines the output PHDR to use if not blank. Be warned that -* output PHDR is sticky. If @phdr is specified, the next output -* section in the linker script will go there too. @phdr should have -* a leading colon. -* -* Note that this macros defines __per_cpu_load as an absolute symbol. -* If there is no need to put the percpu section at a predetermined -* address, use PERCPU_SECTION. -*/ -#define PERCPU_VADDR (cacheline, vaddr, phdr) \ - __per_cpu_load = .; \ - .data..percpu vaddr : AT(__per_cpu_load - LOAD_OFFSET) { \ - PERCPU_INPUT(cacheline) \ - } phdr \ - . = __per_cpu_load + SIZEOF(.data..percpu); - -/** -* PERCPU_SECTION - define output section for percpu area, simple version -* @cacheline: cacheline size -* -* Align to PAGE_SIZE and outputs output section for percpu area. This -* macro doesn't manipulate @vaddr or @phdr and __per_cpu_load and -* __per_cpu_start will be identical. -* -* This macro is equivalent to ALIGN(PAGE_SIZE); PERCPU_VADDR(@cacheline,,) -* except that __per_cpu_load is defined as a relative symbol against -* .data..percpu which is required for relocatable x86_32 configuration. */ #define PERCPU_SECTION (cacheline) \ . = ALIGN(PAGE_SIZE); \
Signature · 1 line
2.47.1
[tip: x86/asm] percpu: Remove PERCPU_VADDR()
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: e23cff6861781ac4e15de6c7bf2d2a0b79cb52ef Gitweb: https://git.kernel.org/tip/e23cff6861781ac4e15de6c7bf2d2a0b79cb52ef Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:45 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:15:57 +01:00
percpu: Remove PERCPU_VADDR()
x86-64 was the last user.
--- include/asm-generic/vmlinux.lds.h | 36 +------------------------------ 1 file changed, 1 insertion(+), 35 deletions(-)
include/asm-generic/vmlinux.lds.h +1 -35
@@ -1073,47 +1073,13 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG) __per_cpu_end = .;/** -* PERCPU_VADDR - define output section for percpu area +* PERCPU_SECTION - define output section for percpu area * @cacheline: cacheline size -* @vaddr: explicit base address (optional) -* @phdr: destination PHDR (optional) * * Macro which expands to output section for percpu area. * * @cacheline is used to align subsections to avoid false cacheline * sharing between subsections for different purposes. -* -* If @vaddr is not blank, it specifies explicit base address and all -* percpu symbols will be offset from the given address. If blank, -* @vaddr always equals @laddr + LOAD_OFFSET. -* -* @phdr defines the output PHDR to use if not blank. Be warned that -* output PHDR is sticky. If @phdr is specified, the next output -* section in the linker script will go there too. @phdr should have -* a leading colon. -* -* Note that this macros defines __per_cpu_load as an absolute symbol. -* If there is no need to put the percpu section at a predetermined -* address, use PERCPU_SECTION. -*/ -#define PERCPU_VADDR (cacheline, vaddr, phdr) \ - __per_cpu_load = .; \ - .data..percpu vaddr : AT(__per_cpu_load - LOAD_OFFSET) { \ - PERCPU_INPUT(cacheline) \ - } phdr \ - . = __per_cpu_load + SIZEOF(.data..percpu); - -/** -* PERCPU_SECTION - define output section for percpu area, simple version -* @cacheline: cacheline size -* -* Align to PAGE_SIZE and outputs output section for percpu area. This -* macro doesn't manipulate @vaddr or @phdr and __per_cpu_load and -* __per_cpu_start will be identical. -* -* This macro is equivalent to ALIGN(PAGE_SIZE); PERCPU_VADDR(@cacheline,,) -* except that __per_cpu_load is defined as a relative symbol against -* .data..percpu which is required for relocatable x86_32 configuration. */ #define PERCPU_SECTION (cacheline) \ . = ALIGN(PAGE_SIZE); \
PATCH v6 14/15 percpu: Remove __per_cpu_load
In reply to Brian Gerst__per_cpu_load is now always equal to __per_cpu_start.
--- include/asm-generic/sections.h | 2 +- include/asm-generic/vmlinux.lds.h | 1 - mm/percpu.c | 4 ++-- 3 files changed, 3 insertions(+), 4 deletions(-)
include/asm-generic/sections.h +1 -1
@@ -39,7 +39,7 @@ extern char __init_begin[], __init_end[];extern char _sinittext[], _einittext[];extern char __start_ro_after_init[], __end_ro_after_init[];extern char _end[]; -extern char __per_cpu_load[], __per_cpu_start[], __per_cpu_end[]; +extern char __per_cpu_start[], __per_cpu_end[];extern char __kprobes_text_start[], __kprobes_text_end[];extern char __entry_text_start[], __entry_text_end[];extern char __start_rodata[], __end_rodata[];
include/asm-generic/vmlinux.lds.h +0 -1
@@ -1083,7 +1083,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)#define PERCPU_SECTION (cacheline) \ . = ALIGN(PAGE_SIZE); \ .data..percpu : AT(ADDR(.data..percpu) - LOAD_OFFSET) { \ - __per_cpu_load = .; \ PERCPU_INPUT(cacheline) \ }
mm/percpu.c +2 -2
@@ -3099,7 +3099,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size,continue ; }/* copy and return the unused part */ -memcpy (ptr, __per_cpu_load, ai->static_size); +memcpy (ptr, __per_cpu_start, ai->static_size);pcpu_fc_free (ptr + size_sum, ai->unit_size - size_sum); } }
@@ -3282,7 +3282,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_tflush_cache_vmap_early (unit_addr, unit_addr + ai->unit_size);/* copy static data */ -memcpy ((void *)unit_addr, __per_cpu_load, ai->static_size); +memcpy ((void *)unit_addr, __per_cpu_start, ai->static_size); }/* we're ready, commit */
Signature · 1 line
2.47.1
[tip: x86/asm] percpu: Remove __per_cpu_load
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 4b00c1160a13d8bf7297ebf49ec07a84e1f41132 Gitweb: https://git.kernel.org/tip/4b00c1160a13d8bf7297ebf49ec07a84e1f41132 Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:46 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:16:00 +01:00
percpu: Remove __per_cpu_load
__per_cpu_load is now always equal to __per_cpu_start.
--- include/asm-generic/sections.h | 2 +- include/asm-generic/vmlinux.lds.h | 1 - mm/percpu.c | 4 ++-- 3 files changed, 3 insertions(+), 4 deletions(-)
include/asm-generic/sections.h +1 -1
@@ -39,7 +39,7 @@ extern char __init_begin[], __init_end[];extern char _sinittext[], _einittext[];extern char __start_ro_after_init[], __end_ro_after_init[];extern char _end[]; -extern char __per_cpu_load[], __per_cpu_start[], __per_cpu_end[]; +extern char __per_cpu_start[], __per_cpu_end[];extern char __kprobes_text_start[], __kprobes_text_end[];extern char __entry_text_start[], __entry_text_end[];extern char __start_rodata[], __end_rodata[];
include/asm-generic/vmlinux.lds.h +0 -1
@@ -1084,7 +1084,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)#define PERCPU_SECTION (cacheline) \ . = ALIGN(PAGE_SIZE); \ .data..percpu : AT(ADDR(.data..percpu) - LOAD_OFFSET) { \ - __per_cpu_load = .; \ PERCPU_INPUT(cacheline) \ }
mm/percpu.c +2 -2
@@ -3071,7 +3071,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size,continue ; }/* copy and return the unused part */ -memcpy (ptr, __per_cpu_load, ai->static_size); +memcpy (ptr, __per_cpu_start, ai->static_size);pcpu_fc_free (ptr + size_sum, ai->unit_size - size_sum); } }
@@ -3240,7 +3240,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_tflush_cache_vmap_early (unit_addr, unit_addr + ai->unit_size);/* copy static data */ -memcpy ((void *)unit_addr, __per_cpu_load, ai->static_size); +memcpy ((void *)unit_addr, __per_cpu_start, ai->static_size); }/* we're ready, commit */
PATCH v6 15/15 kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU
In reply to Brian Gerstx86-64 was the only user.
--- init/Kconfig | 5 --- kernel/kallsyms.c | 12 ++----- scripts/kallsyms.c | 72 +++++++---------------------------------- scripts/link-vmlinux.sh | 4 --- 4 files changed, 14 insertions(+), 79 deletions(-)
init/Kconfig +0 -5
@@ -1870,11 +1870,6 @@ config KALLSYMS_ALL Say N unless you really need all symbols, or kernel live patching. -config KALLSYMS_ABSOLUTE_PERCPU - bool - depends on KALLSYMS - default n - # end of the "standard kernel features (expert users)" menu config ARCH_HAS_MEMBARRIER_CALLBACKS
kernel/kallsyms.c +2 -10
@@ -148,16 +148,8 @@ static unsigned int get_symbol_offset(unsigned long pos)unsigned long kallsyms_sym_address (int idx) { -/* values are unsigned offsets if --absolute-percpu is not in effect */ -if (!IS_ENABLED (CONFIG_KALLSYMS_ABSOLUTE_PERCPU )) -return kallsyms_relative_base + (u32 )kallsyms_offsets[idx]; - -/* ...otherwise, positive offsets are absolute values */ -if (kallsyms_offsets[idx] >=0 ) -return kallsyms_offsets[idx]; - -/* ...and negative offsets are relative to kallsyms_relative_base - 1 */ -return kallsyms_relative_base -1 - kallsyms_offsets[idx]; +/* values are unsigned offsets */ +return kallsyms_relative_base + (u32 )kallsyms_offsets[idx]; }static unsigned int get_symbol_seq (int index)
scripts/kallsyms.c +12 -60
145 lines not on this page · raw message
scripts/link-vmlinux.sh +0 -4
@@ -144,10 +144,6 @@ kallsyms() kallsymopt="${kallsymopt} --all-symbols" fi - if is_enabled CONFIG_KALLSYMS_ABSOLUTE_PERCPU; then - kallsymopt="${kallsymopt} --absolute-percpu" - fi - info KSYMS "${2}.S" scripts/kallsyms ${kallsymopt} "${1}" > "${2}.S"
Signature · 1 line
2.47.1
[tip: x86/asm] kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 01157ddc58dc2fe428ec17dd5a18cc13f134639f Gitweb: https://git.kernel.org/tip/01157ddc58dc2fe428ec17dd5a18cc13f134639f Author: Brian Gerst <[email protected]> AuthorDate: Thu, 23 Jan 2025 14:07:47 -05:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 18 Feb 2025 10:16:04 +01:00
kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU
x86-64 was the only user.
--- init/Kconfig | 5 +--- kernel/kallsyms.c | 12 +------ scripts/kallsyms.c | 72 ++++++---------------------------------- scripts/link-vmlinux.sh | 4 +-- 4 files changed, 14 insertions(+), 79 deletions(-)
init/Kconfig +0 -5
@@ -1869,11 +1869,6 @@ config KALLSYMS_ALL Say N unless you really need all symbols, or kernel live patching. -config KALLSYMS_ABSOLUTE_PERCPU - bool - depends on KALLSYMS - default n - # end of the "standard kernel features (expert users)" menu config ARCH_HAS_MEMBARRIER_CALLBACKS
kernel/kallsyms.c +2 -10
@@ -148,16 +148,8 @@ static unsigned int get_symbol_offset(unsigned long pos)unsigned long kallsyms_sym_address (int idx) { -/* values are unsigned offsets if --absolute-percpu is not in effect */ -if (!IS_ENABLED (CONFIG_KALLSYMS_ABSOLUTE_PERCPU )) -return kallsyms_relative_base + (u32 )kallsyms_offsets[idx]; - -/* ...otherwise, positive offsets are absolute values */ -if (kallsyms_offsets[idx] >=0 ) -return kallsyms_offsets[idx]; - -/* ...and negative offsets are relative to kallsyms_relative_base - 1 */ -return kallsyms_relative_base -1 - kallsyms_offsets[idx]; +/* values are unsigned offsets */ +return kallsyms_relative_base + (u32 )kallsyms_offsets[idx]; }static unsigned int get_symbol_seq (int index)
scripts/kallsyms.c +12 -60
145 lines not on this page · raw message
scripts/link-vmlinux.sh +0 -4
@@ -144,10 +144,6 @@ kallsyms() kallsymopt="${kallsymopt} --all-symbols" fi - if is_enabled CONFIG_KALLSYMS_ABSOLUTE_PERCPU; then - kallsymopt="${kallsymopt} --absolute-percpu" - fi - info KSYMS "${2}.S" scripts/kallsyms ${kallsymopt} "${1}" > "${2}.S"
Quoted 37 lines · Brian Gerst
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
v6:
- Rebased to current tip tree
- Dropped patches already applied
- Fixed typos in commit messages
- Added Reviewed-by tagsArd Biesheuvel (1):
x86/module: Deal with GOT based stack cookie load on Clang < 17Brian Gerst (14):
x86: Raise minimum GCC version to 8.1 x86/stackprotector: Remove stack protector test scripts x86/boot: Disable stack protector for early boot code x86/pvh: Usefixed_percpu_datafor early boot GSBASE x86/relocs: HandleR_X86_64_REX_GOTPCRELXrelocations x86/stackprotector/64: Convert to normal percpu variable x86/percpu/64: Use relative percpu offsets x86/percpu/64: Removefixed_percpu_datax86/boot/64: Remove inverse relocations x86/percpu/64: RemoveINIT_PER_CPUmacros
percpu: RemovePER_CPU_FIRST_SECTION
percpu: RemovePERCPU_VADDR()
percpu: Remove__per_cpu_load
kallsyms: RemoveKALLSYMS_ABSOLUTE_PERCPU
Quoted 3 lines
33 files changed, 100 insertions(+), 475 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
Thanks,
Ingo
Quoted 45 lines · Ingo Molnar
Quoted 37 lines · Brian Gerst
Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.
GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.
v6:
- Rebased to current tip tree
- Dropped patches already applied
- Fixed typos in commit messages
- Added Reviewed-by tagsArd Biesheuvel (1):
x86/module: Deal with GOT based stack cookie load on Clang < 17Brian Gerst (14):
x86: Raise minimum GCC version to 8.1 x86/stackprotector: Remove stack protector test scripts x86/boot: Disable stack protector for early boot code x86/pvh: Usefixed_percpu_datafor early boot GSBASE x86/relocs: HandleR_X86_64_REX_GOTPCRELXrelocations x86/stackprotector/64: Convert to normal percpu variable x86/percpu/64: Use relative percpu offsets x86/percpu/64: Removefixed_percpu_datax86/boot/64: Remove inverse relocations x86/percpu/64: RemoveINIT_PER_CPUmacros
percpu: RemovePER_CPU_FIRST_SECTION
percpu: RemovePERCPU_VADDR()
percpu: Remove__per_cpu_load
kallsyms: RemoveKALLSYMS_ABSOLUTE_PERCPUQuoted 3 lines
33 files changed, 100 insertions(+), 475 deletions(-) delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
Quoted 6 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?
Uros.
Quoted 10 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
Thanks,
Ingo
Quoted 12 lines · Ingo Molnar
Quoted 8 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
Quoted 6 lines
Quoted 2 lines
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Brian Gerst
Quoted 17 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Something like the attached patch?
It boots and runs without problems.
However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:
RELOCS arch/x86/boot/compressed/vmlinux.relocs
WARNING: Absolute relocations present
Offset Info Type Sym.Value Sym.Nameffffffff81200826 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guardffffffff81201493 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guardffffffff81201714 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guardffffffff81201d66 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
...ffffffff834e2a13 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guardffffffff834e2a6a 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
RSTRIP vmlinux
which I don't understand. Looking at the first one:
ffffffff8120081d <force_ibs_eilvt_setup.cold>:
ffffffff8120081d: 48 8b 44 24 08 mov 0x8(%rsp),%rax ffffffff81200822: 65 48 2b 05 f6 ed 30 sub
%gs:0x230edf6(%rip),%rax # ffffffff8350f620
<__ref_stack_chk_guard>ffffffff81200829: 02
I don't think this is absolute relocation, see (%rip).
The kernel was compiled with gcc-14.2.1, so clang specific issue was not tested.
Uros.
arch/x86/entry/entry_32.S +1 -1
@@ -691,7 +691,7 @@ SYM_CODE_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movl TASK_stack_canary(%edx), %ebx - movl %ebx, PER_CPU_VAR(__stack_chk_guard) + movl %ebx, PER_CPU_VAR(pcpu_hot + X86_stack_canary) #endif /*
arch/x86/entry/entry_64.S +1 -1
@@ -193,7 +193,7 @@ SYM_FUNC_START(__switch_to_asm) #ifdef CONFIG_STACKPROTECTOR movq TASK_stack_canary(%rsi), %rbx - movq %rbx, PER_CPU_VAR(__stack_chk_guard) + movq %rbx, PER_CPU_VAR(pcpu_hot + X86_stack_canary) #endif /*
arch/x86/include/asm/current.h +3 -0
@@ -22,6 +22,9 @@ struct pcpu_hot {u64 call_depth;#endif unsigned long top_of_stack; +#ifdef CONFIG_STACKPROTECTOR +unsigned long stack_canary; +#endif void *hardirq_stack_ptr;u16 softirq_pending;#ifdef CONFIG_X86_64
arch/x86/include/asm/stackprotector.h +2 -4
@@ -20,8 +20,6 @@#include <linux/sched.h> -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); -/* * Initialize the stackprotector canary value. *
@@ -38,12 +36,12 @@ static __always_inline void boot_init_stack_canary(void)unsigned long canary =get_random_canary (); current->stack_canary = canary; - this_cpu_write(__stack_chk_guard, canary); + this_cpu_write(pcpu_hot.stack_canary, canary); }static inline void cpu_init_stack_canary (int cpu,struct task_struct *idle) { -per_cpu (__stack_chk_guard, cpu) = idle->stack_canary; +per_cpu (pcpu_hot.stack_canary, cpu) = idle->stack_canary; }#else /* STACKPROTECTOR */
arch/x86/kernel/asm-offsets.c +3 -0
@@ -109,6 +109,9 @@ static void __used common(void)OFFSET (TSS_sp2, tss_struct, x86_tss.sp2);OFFSET (X86_top_of_stack, pcpu_hot, top_of_stack);OFFSET (X86_current_task, pcpu_hot, current_task); +#ifdef CONFIG_STACKPROTECTOR +OFFSET (X86_stack_canary, pcpu_hot, stack_canary); +#endif #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING OFFSET (X86_call_depth, pcpu_hot, call_depth);#endif
arch/x86/kernel/cpu/common.c +0 -8
@@ -24,7 +24,6 @@#include <linux/io.h> #include <linux/syscore_ops.h> #include <linux/pgtable.h> -#include <linux/stackprotector.h> #include <linux/utsname.h> #include <asm/alternative.h>
@@ -2087,13 +2086,6 @@ void syscall_init(void) }#endif /* CONFIG_X86_64 */ -#ifdef CONFIG_STACKPROTECTOR -DEFINE_PER_CPU(unsigned long, __stack_chk_guard); -#ifndef CONFIG_SMP -EXPORT_PER_CPU_SYMBOL (__stack_chk_guard); -#endif -#endif -/* * Clear all 6 debug registers: */
arch/x86/kernel/vmlinux.lds.S +1 -1
@@ -468,7 +468,7 @@ SECTIONS "kernel image bigger than KERNEL_IMAGE_SIZE"); /* needed for Clang - see arch/x86/entry/entry.S */ -PROVIDE(__ref_stack_chk_guard = __stack_chk_guard); +PROVIDE(__ref_stack_chk_guard = pcpu_hot + X86_stack_canary); #ifdef CONFIG_X86_64
Hi Uros,
Quoted 19 lines · Uros Bizjak
Quoted 17 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Something like the attached patch?
Interesting take. I'd have tried to put the canary at offset 0x0, and simply use pcpu_hot as the guard symbol.
Quoted 26 lines
It boots and runs without problems.
However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:
RELOCS
arch/x86/boot/compressed/vmlinux.relocs
WARNING: Absolute relocations present
Offset Info Type Sym.Value Sym.Nameffffffff812008260003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff812014930003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff812017140003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff81201d660003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guard
...ffffffff834e2a130003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff834e2a6a0003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardRSTRIP vmlinux
which I don't understand. Looking at the first one:
Quoted 9 lines
ffffffff8120081d<force_ibs_eilvt_setup.cold>:ffffffff8120081d: 48 8b 44 24 08 mov 0x8(%rsp),%rax ffffffff81200822: 65 48 2b 05 f6 ed 30 sub%gs:0x230edf6(%rip),%rax #
ffffffff8350f620
<__ref_stack_chk_guard>ffffffff81200829: 02I don't think this is absolute relocation, see (%rip).
The warning is about the type of __ref_stack_chk_guard, not about the type of the relocation.
$ nm vmlinux |grep \\s__ref_sta ffffffff8350c620 A __ref_stack_chk_guard
Without your patch:
$ nm vmlinux |grep \\s__ref_sta ffffffff834fba10 D __ref_stack_chk_guard
Quoted 70 lines · Ard Biesheuvel
Hi Uros,
Quoted 19 lines · Uros Bizjak
Quoted 17 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Something like the attached patch?
Interesting take. I'd have tried to put the canary at offset
0x0, and simply usepcpu_hotas the guard symbol.Quoted 33 lines
It boots and runs without problems.
However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:
RELOCS
arch/x86/boot/compressed/vmlinux.relocs
WARNING: Absolute relocations present
Offset Info Type Sym.Value Sym.Nameffffffff812008260003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff812014930003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff812017140003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff81201d660003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guard
...ffffffff834e2a130003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardffffffff834e2a6a0003259e00000002R_X86_64_PC32ffffffff8350f620__ref_stack_chk_guardRSTRIP vmlinux
which I don't understand. Looking at the first one:
ffffffff8120081d<force_ibs_eilvt_setup.cold>:ffffffff8120081d: 48 8b 44 24 08 mov 0x8(%rsp),%rax ffffffff81200822: 65 48 2b 05 f6 ed 30 sub%gs:0x230edf6(%rip),%rax #
ffffffff8350f620
<__ref_stack_chk_guard>ffffffff81200829: 02
Quoted 5 lines
Quoted 2 lines
I don't think this is absolute relocation, see (%rip).
The warning is about the type of
__ref_stack_chk_guard, not about the type of the relocation.
Thanks, I got distracted by the text of the warning that mentions relocation.
Quoted 7 lines
$ nm vmlinux |grep \\s__ref_sta ffffffff8350c620 A __ref_stack_chk_guardWithout your patch:
$ nm vmlinux |grep \\s__ref_sta ffffffff834fba10 D __ref_stack_chk_guard
Is this a problem in our specific case? While the symbol is absolute, the relocation is still relative, so IMO it should be OK even with your ongoing rip-relative efforts in mind.
We can list the symbol in arch/x86/tools/relocs.c to quiet the warning, but I would need some help with auditing the symbol itself.
OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));
for good measure.
Uros.
Quoted 33 lines · Uros Bizjak
Quoted 31 lines · Ard Biesheuvel
Hi Uros,
Quoted 10 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · Uros Bizjak
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Interesting take. I'd have tried to put the canary at offset
0x0, and simply usepcpu_hotas the guard symbol.Quoted 5 lines
It boots and runs without problems.
However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:
...
Quoted 9 lines
Quoted 3 lines
The warning is about the type of
__ref_stack_chk_guard, not about the type of the relocation.Thanks, I got distracted by the text of the warning that mentions relocation.
Quoted 3 lines
$ nm vmlinux |grep \\s__ref_sta ffffffff8350c620 A __ref_stack_chk_guard
Quoted 6 lines
Quoted 4 lines
Without your patch:
$ nm vmlinux |grep \\s__ref_sta ffffffff834fba10 D __ref_stack_chk_guardIs this a problem in our specific case?
I don't think so - the whole notion of absolute ELF symbols is rather flaky IME, so I don't think we should be pedantic here.
Quoted 9 lines
We can list the symbol in
arch/x86/tools/relocs.cto quiet the warning, but I would need some help with auditing the symbol itself.OTOH, we could simply do it your way and put stack canary at the beginning of
pcpu_hotstructure, with
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));for good measure.
I think this would be the most straight-forward if there are no other locality concerns this might interfere with.
Quoted 61 lines · Ard Biesheuvel
Quoted 33 lines · Uros Bizjak
Quoted 31 lines · Ard Biesheuvel
Hi Uros,
Quoted 10 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · Uros Bizjak
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Interesting take. I'd have tried to put the canary at offset
0x0, and simply usepcpu_hotas the guard symbol.Quoted 5 lines
It boots and runs without problems.
However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:
...
Quoted 9 lines
Quoted 3 lines
The warning is about the type of
__ref_stack_chk_guard, not about the type of the relocation.Thanks, I got distracted by the text of the warning that mentions relocation.
Quoted 3 lines
$ nm vmlinux |grep \\s__ref_sta ffffffff8350c620 A __ref_stack_chk_guardQuoted 6 lines
Quoted 4 lines
Without your patch:
$ nm vmlinux |grep \\s__ref_sta ffffffff834fba10 D __ref_stack_chk_guardIs this a problem in our specific case?
I don't think so - the whole notion of absolute ELF symbols is rather flaky IME, so I don't think we should be pedantic here.
Quoted 6 lines
We can list the symbol in
arch/x86/tools/relocs.cto quiet the warning, but I would need some help with auditing the symbol itself.OTOH, we could simply do it your way and put stack canary at the beginning of
pcpu_hotstructure, with
Quoted 6 lines
Quoted 3 lines
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));for good measure.
I think this would be the most straight-forward if there are no other locality concerns this might interfere with.
OK, let me prepare a patch then.
Thanks,
Uros.
Quoted 47 lines · Ard Biesheuvel
Quoted 33 lines · Uros Bizjak
Quoted 31 lines · Ard Biesheuvel
Hi Uros,
Quoted 10 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · Uros Bizjak
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Interesting take. I'd have tried to put the canary at offset
0x0, and simply usepcpu_hotas the guard symbol.Quoted 5 lines
It boots and runs without problems.
However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:
...
Quoted 11 lines
Quoted 3 lines
The warning is about the type of
__ref_stack_chk_guard, not about the type of the relocation.Thanks, I got distracted by the text of the warning that mentions relocation.
Quoted 5 lines
$ nm vmlinux |grep \\s__ref_sta ffffffff8350c620 A __ref_stack_chk_guardWithout your patch:
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
$ nm vmlinux |grep \\s__ref_sta ffffffff834fba10 D __ref_stack_chk_guardIs this a problem in our specific case?
I don't think so - the whole notion of absolute ELF symbols is rather flaky IME, so I don't think we should be pedantic here.
From what I understand it stayed relative because there wasn't a constant added. As soon as you add a constant (which the linker treats as absolute), it becomes absolute.
Quoted 6 lines
Quoted 6 lines
We can list the symbol in
arch/x86/tools/relocs.cto quiet the warning, but I would need some help with auditing the symbol itself.OTOH, we could simply do it your way and put stack canary at the beginning of
pcpu_hotstructure, with
Quoted 6 lines
Quoted 3 lines
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));for good measure.
I think this would be the most straight-forward if there are no other locality concerns this might interfere with.
I'd prefer it at the end of pcpu_hot, that way the disassembler doesn't latch on to the __stack_chk_guard symbol when referencing the other fields of pcpu_hot.
Brian Gerst
Quoted 10 lines · Brian Gerst
Quoted 8 lines · Ard Biesheuvel
Quoted 6 lines · Uros Bizjak
OTOH, we could simply do it your way and put stack canary at the beginning of
pcpu_hotstructure, with
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
for good measure.
I think this would be the most straight-forward if there are no other locality concerns this might interfere with.
I'd prefer it at the end of
pcpu_hot, that way the disassembler doesn't latch on to the__stack_chk_guardsymbol when referencing the other fields ofpcpu_hot.
__stack_chk_guard would no longer exist, only __ref_stack_chk_guard, which would be equal to pcpu_hot. We could just call that __ref_pcpu_hot instead if it might cause confusion otherwise. (We can't use pcpu_hot directly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via __ref_stack_chk_guard)
Quoted 22 lines · Ard Biesheuvel
Quoted 10 lines · Brian Gerst
Quoted 8 lines · Ard Biesheuvel
Quoted 6 lines · Uros Bizjak
OTOH, we could simply do it your way and put stack canary at the beginning of
pcpu_hotstructure, with
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));Quoted 9 lines
Quoted 4 lines
Quoted 1 line
for good measure.
I think this would be the most straight-forward if there are no other locality concerns this might interfere with.
I'd prefer it at the end of
pcpu_hot, that way the disassembler doesn't latch on to the__stack_chk_guardsymbol when referencing the other fields ofpcpu_hot.
Quoted 5 lines
__stack_chk_guardwould no longer exist, only__ref_stack_chk_guard, which would be equal topcpu_hot. We could just call that__ref_pcpu_hotinstead if it might cause confusion otherwise. (We can't usepcpu_hotdirectly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via__ref_stack_chk_guard)
That works for me.
Brian Gerst
Quoted 24 lines · Brian Gerst
Quoted 22 lines · Ard Biesheuvel
Quoted 19 lines · Brian Gerst
Quoted 6 lines · Ard Biesheuvel
Quoted 4 lines · Uros Bizjak
OTOH, we could simply do it your way and put stack canary at the beginning of
pcpu_hotstructure, withQuoted 6 lines
Quoted 3 lines
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));for good measure.
I think this would be the most straight-forward if there are no other locality concerns this might interfere with.
I'd prefer it at the end of
pcpu_hot, that way the disassembler doesn't latch on to the__stack_chk_guardsymbol when referencing the other fields ofpcpu_hot.
Quoted 7 lines
Quoted 5 lines
__stack_chk_guardwould no longer exist, only__ref_stack_chk_guard, which would be equal topcpu_hot. We could just call that__ref_pcpu_hotinstead if it might cause confusion otherwise. (We can't usepcpu_hotdirectly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via__ref_stack_chk_guard)That works for me.
Maybe not. One quirk of how GCC implements this is that -mstack-protector-guard=global (used by !SMP builds) ignores the -mstack-protector-guard-symbol option and always uses __stack_chk_guard. That makes things more challenging.
Brian Gerst
Quoted 37 lines · Brian Gerst
Quoted 24 lines · Brian Gerst
Quoted 22 lines · Ard Biesheuvel
Quoted 19 lines · Brian Gerst
Quoted 6 lines · Ard Biesheuvel
Quoted 4 lines · Uros Bizjak
OTOH, we could simply do it your way and put stack canary at the beginning of
pcpu_hotstructure, withQuoted 6 lines
Quoted 3 lines
static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));for good measure.
I think this would be the most straight-forward if there are no other locality concerns this might interfere with.
I'd prefer it at the end of
pcpu_hot, that way the disassembler doesn't latch on to the__stack_chk_guardsymbol when referencing the other fields ofpcpu_hot.Quoted 7 lines
Quoted 5 lines
__stack_chk_guardwould no longer exist, only__ref_stack_chk_guard, which would be equal topcpu_hot. We could just call that__ref_pcpu_hotinstead if it might cause confusion otherwise. (We can't usepcpu_hotdirectly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via__ref_stack_chk_guard)That works for me.
Maybe not. One quirk of how GCC implements this is that
(quoted diff) +0
-2-mstack-protector-guard=global (used by !SMP builds) ignores the -mstack-protector-guard-symbol option and always uses
Quoted 1 line
__stack_chk_guard. That makes things more challenging.
Not really. If we put stack_canary as the first member of struct pcpu_hot, we can just alias __stack_chk_guard to struct pcpu_hot in the linker script, and everything starts to magically work, SMP and !SMP. Please see the proposed patch, effectively a three liner, at [1].
[1] https://lore.kernel.org/lkml/[email protected]/
Uros.
Quoted 16 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Probably not worth it then?
Thanks,
Ingo
Quoted 20 lines · Ingo Molnar
Quoted 16 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Probably not worth it then?
Actually it wasn't so bad since we already had the hack for __ref_stack_chk_guard. Do you want the patches now or when the dust settles on the original series?
Brian Gerst
Quoted 25 lines · Brian Gerst
Quoted 24 lines · Ingo Molnar
Quoted 20 lines · Brian Gerst
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines
Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.
Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.
I wonder if there would be any benefit if stack canary is put into struct
pcpu_hot?It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.
Probably not worth it then?
Actually it wasn't so bad since we already had the hack for
__ref_stack_chk_guard. Do you want the patches now or when the dust settles on the original series?
We can add it now I suppose, while memories are fresh and all that.
Thanks,
Ingo