Skip to contentSkip to conversation

x86-64: Stack protector and percpu improvements

PATCH Applied Brian Gerst 9 participants 64 messages Jan 23, 2025 – Mar 19, 2025

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

v6 cover x86-64: Stack protector and percpu improvements

Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.

GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.

v6:
- Rebased to current tip tree
- Dropped patches already applied
- Fixed typos in commit messages
- Added Reviewed-by tags

Ard Biesheuvel (1):
x86/module: Deal with GOT based stack cookie load on Clang < 17

Brian Gerst (14):
x86: Raise minimum GCC version to 8.1 x86/stackprotector: Remove stack protector test scripts x86/boot: Disable stack protector for early boot code x86/pvh: Use fixed_percpu_data for early boot GSBASE x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations x86/stackprotector/64: Convert to normal percpu variable x86/percpu/64: Use relative percpu offsets x86/percpu/64: Remove fixed_percpu_data x86/boot/64: Remove inverse relocations x86/percpu/64: Remove INIT_PER_CPU macros
percpu: Remove PER_CPU_FIRST_SECTION
percpu: Remove PERCPU_VADDR()
percpu: Remove __per_cpu_load
kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU

 arch/x86/Kconfig                          |  11 +-
 arch/x86/Makefile                         |  20 +--
 arch/x86/boot/compressed/misc.c           |  14 +--
 arch/x86/entry/entry.S                    |   2 -
 arch/x86/entry/entry_64.S                 |   2 +-
 arch/x86/include/asm/desc.h               |   1 -
 arch/x86/include/asm/elf.h                |   3 +-
 arch/x86/include/asm/percpu.h             |  22 ----
 arch/x86/include/asm/processor.h          |  28 +----
 arch/x86/include/asm/stackprotector.h     |  36 +-----
 arch/x86/kernel/Makefile                  |   2 +
 arch/x86/kernel/asm-offsets_64.c          |   6 -
 arch/x86/kernel/cpu/common.c              |   9 +-
 arch/x86/kernel/head64.c                  |   2 +-
 arch/x86/kernel/head_64.S                 |  20 ++-
 arch/x86/kernel/irq_64.c                  |   1 -
 arch/x86/kernel/module.c                  |  15 +++
 arch/x86/kernel/setup_percpu.c            |  12 +-
 arch/x86/kernel/vmlinux.lds.S             |  35 ------
 arch/x86/platform/pvh/head.S              |  14 ++-
 arch/x86/tools/relocs.c                   | 147 ++--------------------
 arch/x86/xen/xen-head.S                   |  10 +-
 include/asm-generic/sections.h            |   2 +-
 include/asm-generic/vmlinux.lds.h         |  38 +-----
 include/linux/percpu-defs.h               |  12 --
 init/Kconfig                              |   5 -
 kernel/kallsyms.c                         |  12 +-
 mm/percpu.c                               |   4 +-
 scripts/gcc-x86_32-has-stack-protector.sh |   8 --
 scripts/gcc-x86_64-has-stack-protector.sh |   4 -
 scripts/kallsyms.c                        |  72 ++---------
 scripts/link-vmlinux.sh                   |   4 -
 scripts/min-tool-version.sh               |   2 +
 33 files changed, 100 insertions(+), 475 deletions(-)
 delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh
 delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh

base-commit: b79d90e018587507fd42c4c888956668692ff431

Signature · 1 line
2.47.1
#1 lore raw

PATCH v6 1/15 x86: Raise minimum GCC version to 8.1

Stack protector support on 64-bit currently requires that the percpu section is linked at absolute address 0 because older compilers fixed the location of the canary value relative to the GS segment base. GCC 8.1 introduced options to change where the canary value is located, allowing it to be configured as a standard percpu variable. This has already been done for 32-bit. Doing the same for 64-bit will enable removing the code needed to support zero-based percpu.

---
 scripts/min-tool-version.sh | 2 ++
 1 file changed, 2 insertions(+)
scripts/min-tool-version.sh +2 -0
index 91c91201212c..06c4e410ecab 100755
@@ -19,6 +19,8 @@ binutils)
 gcc)
 	if [ "$ARCH" = parisc64 ]; then
 		echo 12.0.0
+	elif [ "$SRCARCH" = x86 ]; then
+		echo 8.1.0
 	else
 		echo 5.1.0
 	fi
Signature · 1 line
2.47.1
merge bot Applied
#2 lore raw

[tip: x86/asm] x86/build: Raise the minimum GCC version to 8.1

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     a3e8fe814ad15c16735cdf394454a8bd96eb4d56
Gitweb:        https://git.kernel.org/tip/a3e8fe814ad15c16735cdf394454a8bd96eb4d56
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:33 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:14:40 +01:00

x86/build: Raise the minimum GCC version to 8.1

Stack protector support on 64-bit currently requires that the percpu section is linked at absolute address 0, because older compilers fixed the location of the canary value relative to the GS segment base.

GCC 8.1 introduced options to change where the canary value is located, allowing it to be configured as a standard per-CPU variable. This has already been done for 32-bit. Doing the same for 64-bit will enable removing the code needed to support zero-based percpu.

---
 scripts/min-tool-version.sh | 2 ++
 1 file changed, 2 insertions(+)
scripts/min-tool-version.sh +2 -0
index 91c9120..06c4e41 100755
@@ -19,6 +19,8 @@ binutils)
 gcc)
 	if [ "$ARCH" = parisc64 ]; then
 		echo 12.0.0
+	elif [ "$SRCARCH" = x86 ]; then
+		echo 8.1.0
 	else
 		echo 5.1.0
 	fi
#3 lore raw

PATCH v6 2/15 x86/stackprotector: Remove stack protector test scripts

In reply to Brian Gerst

With GCC 8.1 now the minimum supported compiler for x86, these scripts are no longer needed.

---
 arch/x86/Kconfig                          | 11 +----------
 scripts/gcc-x86_32-has-stack-protector.sh |  8 --------
 scripts/gcc-x86_64-has-stack-protector.sh |  4 ----
 3 files changed, 1 insertion(+), 22 deletions(-)
 delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh
 delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
arch/x86/Kconfig +1 -10
index 27c21c9b6a70..bdfc6b078a1e 100644
@@ -283,7 +283,7 @@ config X86
 	select HAVE_FUNCTION_ARG_ACCESS_API
 	select HAVE_SETUP_PER_CPU_AREA
 	select HAVE_SOFTIRQ_ON_OWN_STACK
-	select HAVE_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
+	select HAVE_STACKPROTECTOR
 	select HAVE_STACK_VALIDATION		if HAVE_OBJTOOL
 	select HAVE_STATIC_CALL
 	select HAVE_STATIC_CALL_INLINE		if HAVE_OBJTOOL
@@ -423,15 +423,6 @@ config PGTABLE_LEVELS
 	default 3 if X86_PAE
 	default 2
 
-config CC_HAS_SANE_STACKPROTECTOR
-	bool
-	default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT
-	default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC) $(CLANG_FLAGS))
-	help
-	  We have to make sure stack protector is unconditionally disabled if
-	  the compiler produces broken code or if it does not let us control
-	  the segment on 32-bit kernels.
-
 menu "Processor type and features"
 
 config SMP
scripts/gcc-x86_32-has-stack-protector.sh deleted +0 -8
deleted file mode 100755
index 9459ca4f0f11..000000000000
@@ -1,8 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0
-
-# This requires GCC 8.1 or better.  Specifically, we require
-# -mstack-protector-guard-reg, added by
-# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81708
-
-echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m32 -O0 -fstack-protector -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - -o - 2> /dev/null | grep -q "%fs"
scripts/gcc-x86_64-has-stack-protector.sh deleted +0 -4
deleted file mode 100755
index f680bb01aeeb..000000000000
@@ -1,4 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0
-
-echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m64 -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
Signature · 1 line
2.47.1
merge bot Applied
#4 lore raw

[tip: x86/asm] x86/stackprotector: Remove stack protector test scripts

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     0ee2689b9374d6fd5f43b703713a532278654749
Gitweb:        https://git.kernel.org/tip/0ee2689b9374d6fd5f43b703713a532278654749
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:34 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:14:47 +01:00

x86/stackprotector: Remove stack protector test scripts

With GCC 8.1 now the minimum supported compiler for x86, these scripts are no longer needed.

---
 arch/x86/Kconfig                          | 11 +----------
 scripts/gcc-x86_32-has-stack-protector.sh |  8 --------
 scripts/gcc-x86_64-has-stack-protector.sh |  4 ----
 3 files changed, 1 insertion(+), 22 deletions(-)
 delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh
 delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
arch/x86/Kconfig +1 -10
index be2c311..6595b35 100644
@@ -285,7 +285,7 @@ config X86
 	select HAVE_FUNCTION_ARG_ACCESS_API
 	select HAVE_SETUP_PER_CPU_AREA
 	select HAVE_SOFTIRQ_ON_OWN_STACK
-	select HAVE_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
+	select HAVE_STACKPROTECTOR
 	select HAVE_STACK_VALIDATION		if HAVE_OBJTOOL
 	select HAVE_STATIC_CALL
 	select HAVE_STATIC_CALL_INLINE		if HAVE_OBJTOOL
@@ -426,15 +426,6 @@ config PGTABLE_LEVELS
 	default 3 if X86_PAE
 	default 2
 
-config CC_HAS_SANE_STACKPROTECTOR
-	bool
-	default $(success,$(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC) $(CLANG_FLAGS)) if 64BIT
-	default $(success,$(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC) $(CLANG_FLAGS))
-	help
-	  We have to make sure stack protector is unconditionally disabled if
-	  the compiler produces broken code or if it does not let us control
-	  the segment on 32-bit kernels.
-
 menu "Processor type and features"
 
 config SMP
scripts/gcc-x86_32-has-stack-protector.sh deleted +0 -8
deleted file mode 100755
index 9459ca4..0000000
@@ -1,8 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0
-
-# This requires GCC 8.1 or better.  Specifically, we require
-# -mstack-protector-guard-reg, added by
-# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81708
-
-echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m32 -O0 -fstack-protector -mstack-protector-guard-reg=fs -mstack-protector-guard-symbol=__stack_chk_guard - -o - 2> /dev/null | grep -q "%fs"
scripts/gcc-x86_64-has-stack-protector.sh deleted +0 -4
deleted file mode 100755
index f680bb0..0000000
@@ -1,4 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0
-
-echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -m64 -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
#5 lore raw

PATCH v6 3/15 x86/boot: Disable stack protector for early boot code

In reply to Brian Gerst

On 64-bit, this will prevent crashes when the canary access is changed from %gs:40 to %gs:__stack_chk_guard(%rip). RIP-relative addresses from the identity-mapped early boot code will target the wrong address with zero-based percpu. KASLR could then shift that address to an unmapped page causing a crash on boot.

This early boot code runs well before userspace is active and does not need stack protector enabled.

---
 arch/x86/kernel/Makefile | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kernel/Makefile +2 -0
index b43eb7e384eb..84cfa179802c 100644
@@ -44,6 +44,8 @@ KCOV_INSTRUMENT_unwind_orc.o				:= n
 KCOV_INSTRUMENT_unwind_frame.o				:= n
 KCOV_INSTRUMENT_unwind_guess.o				:= n
 
+CFLAGS_head32.o := -fno-stack-protector
+CFLAGS_head64.o := -fno-stack-protector
 CFLAGS_irq.o := -I $(src)/../include/asm/trace
 
 obj-y			+= head_$(BITS).o
Signature · 1 line
2.47.1
merge bot Applied
#6 lore raw

[tip: x86/asm] x86/boot: Disable stack protector for early boot code

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     a9a76b38aaf577887103e3ebb41d70e6aa5a4b19
Gitweb:        https://git.kernel.org/tip/a9a76b38aaf577887103e3ebb41d70e6aa5a4b19
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:35 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:14:51 +01:00

x86/boot: Disable stack protector for early boot code

On 64-bit, this will prevent crashes when the canary access is changed from %gs:40 to %gs:__stack_chk_guard(%rip). RIP-relative addresses from the identity-mapped early boot code will target the wrong address with zero-based percpu. KASLR could then shift that address to an unmapped page causing a crash on boot.

This early boot code runs well before user-space is active and does not need stack protector enabled.

---
 arch/x86/kernel/Makefile | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kernel/Makefile +2 -0
index b43eb7e..84cfa17 100644
@@ -44,6 +44,8 @@ KCOV_INSTRUMENT_unwind_orc.o				:= n
 KCOV_INSTRUMENT_unwind_frame.o				:= n
 KCOV_INSTRUMENT_unwind_guess.o				:= n
 
+CFLAGS_head32.o := -fno-stack-protector
+CFLAGS_head64.o := -fno-stack-protector
 CFLAGS_irq.o := -I $(src)/../include/asm/trace
 
 obj-y			+= head_$(BITS).o
#7 lore raw

PATCH v6 4/15 x86/pvh: Use fixed_percpu_data for early boot GSBASE

In reply to Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +9 -6
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:
 	UNWIND_HINT_END_OF_STACK
 
-	/* Set base address in stack canary descriptor. */
-	mov $MSR_GS_BASE,%ecx
-	leal canary(%rip), %eax
-	xor %edx, %edx
+	/*
+	 * Set up GSBASE.
+	 * Note that, on SMP, the boot cpu uses init data section until
+	 * the per cpu areas are set up.
+	 */
+	movl $MSR_GS_BASE,%ecx
+	leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+	movq %edx, %eax
+	shrq $32, %rdx
 	wrmsr
 
 	/* Call xen_prepare_pvh() via the kernel virtual mapping */
@@ -238,8 +243,6 @@ SYM_DATA_START_LOCAL(gdt_start)
 SYM_DATA_END_LABEL(gdt_start, SYM_L_LOCAL, gdt_end)
 
 	.balign 16
-SYM_DATA_LOCAL(canary, .fill 48, 1, 0)
-
 SYM_DATA_START_LOCAL(early_stack)
 	.fill BOOT_STACK_SIZE, 1, 0
 SYM_DATA_END_LABEL(early_stack, SYM_L_LOCAL, early_stack_end)
Signature · 1 line
2.47.1
maintainer
#8 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

Quoted 8 lines · Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

Use ./scripts/get_maintainer.pl pls. I've added Juergen now.

Quoted 6 lines
arch/x86/platform/pvh/head.S +0 -0
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:

arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 179

Quoted 10 lines
arch/x86/platform/pvh/head.S +4 -4
 	UNWIND_HINT_END_OF_STACK
 
-	/* Set base address in stack canary descriptor. */
-	mov $MSR_GS_BASE,%ecx
-	leal canary(%rip), %eax
-	xor %edx, %edx
+	/*
+	 * Set up GSBASE.
+	 * Note that, on SMP, the boot cpu uses init data section until
+	 * the per cpu areas are set up.

s/cpu/CPU/g

check your whole set pls.

Quoted 6 lines
arch/x86/platform/pvh/head.S +5 -0
+	 */
+	movl $MSR_GS_BASE,%ecx
+	leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+	movq %edx, %eax
+	shrq $32, %rdx
 	wrmsr
#9 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

Quoted 33 lines · Borislav Petkov
Quoted 8 lines · Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

Use ./scripts/get_maintainer.pl pls. I've added Juergen now.

Quoted 6 lines
arch/x86/platform/pvh/head.S +0 -0
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:

arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 179

Quoted 10 lines
arch/x86/platform/pvh/head.S +4 -4
      UNWIND_HINT_END_OF_STACK
 
-     /* Set base address in stack canary descriptor. */
-     mov $MSR_GS_BASE,%ecx
-     leal canary(%rip), %eax
-     xor %edx, %edx
+     /*
+      * Set up GSBASE.
+      * Note that, on SMP, the boot cpu uses init data section until
+      * the per cpu areas are set up.

s/cpu/CPU/g

Quoted 1 line

check your whole set pls.

To be fair, this was a copy of an existing comment. Is there a style guide where all these grammar rules are documented, so I don't have to keep resending these patches for trivial typos?

Brian Gerst

maintainer
#10 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

Quoted 3 lines · Brian Gerst

To be fair, this was a copy of an existing comment. Is there a style guide where all these grammar rules are documented, so I don't have to keep resending these patches for trivial typos?

You don't have to keep resending them for trivial typos - you simply wait 1-2 weeks to gather review feedback, you incorporate it and send a new version of the set. Like it is usually done on lkml. I think you know how the process works...

Signature · 1 line
Sent from a small device: formatting sucks and brevity is inevitable.
merge bot Applied
#11 lore raw

[tip: x86/asm] x86/pvh: Use fixed_percpu_data for early boot GSBASE

In reply to Brian Gerst
The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     f58b63857ae38b4484185b799a2759274b930c92
Gitweb:        https://git.kernel.org/tip/f58b63857ae38b4484185b799a2759274b930c92
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:36 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:14:59 +01:00

x86/pvh: Use fixed_percpu_data for early boot GSBASE

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +9 -6
index 4733a5f..723f181 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:
 	UNWIND_HINT_END_OF_STACK
 
-	/* Set base address in stack canary descriptor. */
-	mov $MSR_GS_BASE,%ecx
-	leal canary(%rip), %eax
-	xor %edx, %edx
+	/*
+	 * Set up GSBASE.
+	 * Note that on SMP the boot CPU uses the init data section until
+	 * the per-CPU areas are set up.
+	 */
+	movl $MSR_GS_BASE,%ecx
+	leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+	movq %edx, %eax
+	shrq $32, %rdx
 	wrmsr
 
 	/* Call xen_prepare_pvh() via the kernel virtual mapping */
@@ -238,8 +243,6 @@ SYM_DATA_START_LOCAL(gdt_start)
 SYM_DATA_END_LABEL(gdt_start, SYM_L_LOCAL, gdt_end)
 
 	.balign 16
-SYM_DATA_LOCAL(canary, .fill 48, 1, 0)
-
 SYM_DATA_START_LOCAL(early_stack)
 	.fill BOOT_STACK_SIZE, 1, 0
 SYM_DATA_END_LABEL(early_stack, SYM_L_LOCAL, early_stack_end)
#12 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

In reply to Brian Gerst
Quoted 19 lines · Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +0 -2
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:
 	UNWIND_HINT_END_OF_STACK
 
-	/* Set base address in stack canary descriptor. */
-	mov $MSR_GS_BASE,%ecx

arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183

Quoted 10 lines
arch/x86/platform/pvh/head.S +8 -2
-	leal canary(%rip), %eax
-	xor %edx, %edx
+	/*
+	 * Set up GSBASE.
+	 * Note that, on SMP, the boot cpu uses init data section until
+	 * the per cpu areas are set up.
+	 */
+	movl $MSR_GS_BASE,%ecx
+	leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+	movq %edx, %eax

movl

I'm bisecting perf breakage and landing on this commit breaks the build.

#13 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

Quoted 33 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +0 -2
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:
 	UNWIND_HINT_END_OF_STACK
 
-	/* Set base address in stack canary descriptor. */
-	mov $MSR_GS_BASE,%ecx

arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183

Quoted 10 lines
arch/x86/platform/pvh/head.S +8 -2
-	leal canary(%rip), %eax
-	xor %edx, %edx
+	/*
+	 * Set up GSBASE.
+	 * Note that, on SMP, the boot cpu uses init data section until
+	 * the per cpu areas are set up.
+	 */
+	movl $MSR_GS_BASE,%ecx
+	leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+	movq %edx, %eax

movl

Quoted 1 line

I'm bisecting perf breakage and landing on this commit breaks the build.

Breaks the build how?

#14 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

Quoted 35 lines · H. Peter Anvin
Quoted 33 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +0 -2
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:
     UNWIND_HINT_END_OF_STACK
 
-    /* Set base address in stack canary descriptor. */
-    mov $MSR_GS_BASE,%ecx

arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183

Quoted 10 lines
arch/x86/platform/pvh/head.S +8 -2
-    leal canary(%rip), %eax
-    xor %edx, %edx
+    /*
+     * Set up GSBASE.
+     * Note that, on SMP, the boot cpu uses init data section until
+     * the per cpu areas are set up.
+     */
+    movl $MSR_GS_BASE,%ecx
+    leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+    movq %edx, %eax

movl

Quoted 3 lines
Quoted 1 line

I'm bisecting perf breakage and landing on this commit breaks the build.

Breaks the build how?

AS arch/x86/platform/pvh/head.o
arch/x86/platform/pvh/head.S: Assembler messages:
arch/x86/platform/pvh/head.S:183: Error: incorrect register %eax' used with q' suffix

Per the above, I presume it was meant to be a movl.

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#15 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

Quoted 39 lines · Mateusz Guzik
Quoted 37 lines · H. Peter Anvin
Quoted 34 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +0 -2
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:
     UNWIND_HINT_END_OF_STACK
 
-    /* Set base address in stack canary descriptor. */
-    mov $MSR_GS_BASE,%ecx

arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183

Quoted 10 lines
arch/x86/platform/pvh/head.S +8 -2
-    leal canary(%rip), %eax
-    xor %edx, %edx
+    /*
+     * Set up GSBASE.
+     * Note that, on SMP, the boot cpu uses init data section until
+     * the per cpu areas are set up.
+     */
+    movl $MSR_GS_BASE,%ecx
+    leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+    movq %edx, %eax

movl

I'm bisecting perf breakage and landing on this commit breaks the build.

Quoted 8 lines
Quoted 1 line

Breaks the build how?

AS arch/x86/platform/pvh/head.o
arch/x86/platform/pvh/head.S: Assembler messages:
arch/x86/platform/pvh/head.S:183: Error: incorrect register %eax' used with q' suffix

Per the above, I presume it was meant to be a movl.

Please note that this part is further changed in [PATCH 8/15] [1], so the breakage is temporary.

[1] https://lore.kernel.org/lkml/[email protected]/

Uros.

#16 lore raw

Re: [PATCH v6 04/15] x86/pvh: Use fixed_percpu_data for early boot GSBASE

Quoted 48 lines · Uros Bizjak
Quoted 46 lines · Mateusz Guzik
Quoted 35 lines · H. Peter Anvin
Quoted 33 lines · Mateusz Guzik
Quoted 19 lines · Brian Gerst

Instead of having a private area for the stack canary, use fixed_percpu_data for GSBASE like the native kernel.

---
 arch/x86/platform/pvh/head.S | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/platform/pvh/head.S +0 -2
index 4733a5f467b8..fa0072e0ca43 100644
@@ -173,10 +173,15 @@ SYM_CODE_START(pvh_start_xen)
 1:
     UNWIND_HINT_END_OF_STACK
 
-    /* Set base address in stack canary descriptor. */
-    mov $MSR_GS_BASE,%ecx

arch/x86/platform/pvh/head.S SYM_CODE_START(pvh_start_xen) · line 183

Quoted 10 lines
arch/x86/platform/pvh/head.S +8 -2
-    leal canary(%rip), %eax
-    xor %edx, %edx
+    /*
+     * Set up GSBASE.
+     * Note that, on SMP, the boot cpu uses init data section until
+     * the per cpu areas are set up.
+     */
+    movl $MSR_GS_BASE,%ecx
+    leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
+    movq %edx, %eax

movl

Quoted 3 lines
Quoted 1 line

I'm bisecting perf breakage and landing on this commit breaks the build.

Breaks the build how?

AS arch/x86/platform/pvh/head.o
arch/x86/platform/pvh/head.S: Assembler messages:
arch/x86/platform/pvh/head.S:183: Error: incorrect register %eax' used with q' suffix

Quoted 7 lines
Quoted 1 line

Per the above, I presume it was meant to be a movl.

Please note that this part is further changed in [PATCH 8/15] [1], so the breakage is temporary.

[1] https://lore.kernel.org/lkml/[email protected]/

Top of the tree does build, so I assumed this did get addressed in later patches.

I am saying I ran into this build breakage while bisecting. Other people may be in the same spot.

It should pose no difficulty to s/movq/movl/ in the branch and be done with it.

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#17 lore raw

PATCH v6 5/15 x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations

In reply to Brian Gerst

Clang may produce R_X86_64_REX_GOTPCRELX relocations when redefining the stack protector location. Treat them as another type of PC-relative relocation.

---
 arch/x86/tools/relocs.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/tools/relocs.c +7 -0
index e937be979ec8..33dffc5c30b5 100644
@@ -32,6 +32,11 @@ static struct relocs		relocs32;
 static struct relocs		relocs32neg;
 static struct relocs		relocs64;
 # define FMT PRIu64
+
+#ifndef R_X86_64_REX_GOTPCRELX
+#define R_X86_64_REX_GOTPCRELX 42
+#endif
+
 #else
 # define FMT PRIu32
 #endif
@@ -227,6 +232,7 @@ static const char *rel_type(unsigned type)
 		REL_TYPE(R_X86_64_PC16),
 		REL_TYPE(R_X86_64_8),
 		REL_TYPE(R_X86_64_PC8),
+		REL_TYPE(R_X86_64_REX_GOTPCRELX),
 #else
 		REL_TYPE(R_386_NONE),
 		REL_TYPE(R_386_32),
@@ -861,6 +867,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
 
 	case R_X86_64_PC32:
 	case R_X86_64_PLT32:
+	case R_X86_64_REX_GOTPCRELX:
 		/*
 		 * PC relative relocations don't need to be adjusted unless
 		 * referencing a percpu symbol.
Signature · 1 line
2.47.1
merge bot Applied
#18 lore raw

[tip: x86/asm] x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     cb7927fda002ca49ae62e2782c1692acc7b80c67
Gitweb:        https://git.kernel.org/tip/cb7927fda002ca49ae62e2782c1692acc7b80c67
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:37 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:02 +01:00

x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations

Clang may produce R_X86_64_REX_GOTPCRELX relocations when redefining the stack protector location. Treat them as another type of PC-relative relocation.

---
 arch/x86/tools/relocs.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/tools/relocs.c +7 -0
index e937be9..92a1e50 100644
@@ -32,6 +32,11 @@ static struct relocs		relocs32;
 static struct relocs		relocs32neg;
 static struct relocs		relocs64;
 # define FMT PRIu64
+
+#ifndef R_X86_64_REX_GOTPCRELX
+# define R_X86_64_REX_GOTPCRELX 42
+#endif
+
 #else
 # define FMT PRIu32
 #endif
@@ -227,6 +232,7 @@ static const char *rel_type(unsigned type)
 		REL_TYPE(R_X86_64_PC16),
 		REL_TYPE(R_X86_64_8),
 		REL_TYPE(R_X86_64_PC8),
+		REL_TYPE(R_X86_64_REX_GOTPCRELX),
 #else
 		REL_TYPE(R_386_NONE),
 		REL_TYPE(R_386_32),
@@ -861,6 +867,7 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
 
 	case R_X86_64_PC32:
 	case R_X86_64_PLT32:
+	case R_X86_64_REX_GOTPCRELX:
 		/*
 		 * PC relative relocations don't need to be adjusted unless
 		 * referencing a percpu symbol.
#19 lore raw

PATCH v6 6/15 x86/module: Deal with GOT based stack cookie load on Clang < 17

In reply to Brian Gerst

From: Ard Biesheuvel <[email protected]>

Clang versions before 17 will not honour -fdirect-access-external-data for the load of the stack cookie emitted into each function's prologue and epilogue.

This is not an issue for the core kernel, as the linker will relax these loads into LEA instructions that take the address of __stack_chk_guard directly. For modules, however, we need to work around this, by dealing with R_X86_64_REX_GOTPCRELX relocations that refer to __stack_chk_guard.

In this case, given that this is a GOT load, the reference should not refer to __stack_chk_guard directly, but to a memory location that holds its address. So take the address of __stack_chk_guard into a static variable, and fix up the relocations to refer to that.

---
 arch/x86/include/asm/elf.h |  3 ++-
 arch/x86/kernel/module.c   | 15 +++++++++++++++
 2 files changed, 17 insertions(+), 1 deletion(-)
arch/x86/include/asm/elf.h +2 -1
index 1fb83d47711f..0d6ca771549d 100644
@@ -55,7 +55,8 @@ typedef struct user_i387_struct elf_fpregset_t;
 #define R_X86_64_JUMP_SLOT	7	/* Create PLT entry */
 #define R_X86_64_RELATIVE	8	/* Adjust by program base */
 #define R_X86_64_GOTPCREL	9	/* 32 bit signed pc relative
-					   offset to GOT */
+#define R_X86_64_GOTPCRELX	41	   offset to GOT */
+#define R_X86_64_REX_GOTPCRELX	42
 #define R_X86_64_32		10	/* Direct 32 bit zero extended */
 #define R_X86_64_32S		11	/* Direct 32 bit sign extended */
 #define R_X86_64_16		12	/* Direct 16 bit zero extended */
arch/x86/kernel/module.c +15 -0
index 8984abd91c00..a286f32c5503 100644
@@ -19,6 +19,7 @@
 #include <linux/jump_label.h>
 #include <linux/random.h>
 #include <linux/memory.h>
+#include <linux/stackprotector.h>
 
 #include <asm/text-patching.h>
 #include <asm/page.h>
@@ -130,6 +131,20 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,
 				goto overflow;
 			size = 4;
 			break;
+#if defined(CONFIG_STACKPROTECTOR) && \
+    defined(CONFIG_CC_IS_CLANG) && CONFIG_CLANG_VERSION < 170000
+		case R_X86_64_REX_GOTPCRELX: {
+			static unsigned long __percpu *const addr = &__stack_chk_guard;
+
+			if (sym->st_value != (u64)addr) {
+				pr_err("%s: Unsupported GOTPCREL relocation\n", me->name);
+				return -ENOEXEC;
+			}
+
+			val = (u64)&addr + rel[i].r_addend;
+			fallthrough;
+		}
+#endif
 		case R_X86_64_PC32:
 		case R_X86_64_PLT32:
 			val -= (u64)loc;
Signature · 1 line
2.47.1
#20 lore raw

Re: [PATCH v6 06/15] x86/module: Deal with GOT based stack cookie load on Clang < 17

arch/x86/include/asm/elf.h typedef struct user_i387_struct elf_fpregset_t; · line 58

Quoted 3 lines · Brian Gerst

#define R_X86_64_GOTPCREL 9 /* 32 bit signed pc relative

arch/x86/include/asm/elf.h +1 -1
-					   offset to GOT */
+#define R_X86_64_GOTPCRELX	41	   offset to GOT */
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line

+#define R_X86_64_REX_GOTPCRELX 42

Uhm, I'm pretty sure this won't even build, should any code use R_X86_64_GOTPCRELX - which it doesn't currently...

Also, each entry here has a comment explaining it - except these two new GOTPCREL variants.

Thanks,

Ingo

#21 lore raw

Re: [PATCH v6 06/15] x86/module: Deal with GOT based stack cookie load on Clang < 17

Quoted 7 lines · Ingo Molnar

arch/x86/include/asm/elf.h typedef struct user_i387_struct elf_fpregset_t; · line 58

Quoted 3 lines · Brian Gerst

#define R_X86_64_GOTPCREL 9 /* 32 bit signed pc relative

arch/x86/include/asm/elf.h +1 -1
-                                        offset to GOT */
+#define R_X86_64_GOTPCRELX   41         offset to GOT */
Quoted 8 lines
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line

+#define R_X86_64_REX_GOTPCRELX 42

Uhm, I'm pretty sure this won't even build, should any code use R_X86_64_GOTPCRELX - which it doesn't currently...

Also, each entry here has a comment explaining it - except these two new GOTPCREL variants.

Should we remove the non-REX version that isn't used by the kernel? This header is not exported to userspace.

Brian Gerst

#22 lore raw

Re: [PATCH v6 06/15] x86/module: Deal with GOT based stack cookie load on Clang < 17

Quoted 14 lines · Brian Gerst
Quoted 13 lines · Ingo Molnar

arch/x86/include/asm/elf.h typedef struct user_i387_struct elf_fpregset_t; · line 58

Quoted 3 lines · Brian Gerst

#define R_X86_64_GOTPCREL 9 /* 32 bit signed pc relative

arch/x86/include/asm/elf.h +1 -1
-                                        offset to GOT */
+#define R_X86_64_GOTPCRELX   41         offset to GOT */
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line

+#define R_X86_64_REX_GOTPCRELX 42

Uhm, I'm pretty sure this won't even build, should any code use R_X86_64_GOTPCRELX - which it doesn't currently...

Quoted 9 lines
Quoted 2 lines

Also, each entry here has a comment explaining it - except these two new GOTPCREL variants.

Should we remove the non-REX version that isn't used by the kernel? This header is not exported to userspace.

Brian Gerst

I would personally like to see them in there for the sake of documentation at least. If something runs into this in the future then at least it will be a little quicker to debug. This is an ABI constant, after all.

merge bot Applied
#23 lore raw

[tip: x86/asm] x86/module: Deal with GOT based stack cookie load on Clang < 17

In reply to Brian Gerst
The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     78c4374ef8b842c6abf195d6f963853c7ec464d2
Gitweb:        https://git.kernel.org/tip/78c4374ef8b842c6abf195d6f963853c7ec464d2
Author:        Ard Biesheuvel <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:38 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:05 +01:00

x86/module: Deal with GOT based stack cookie load on Clang < 17

Clang versions before 17 will not honour -fdirect-access-external-data for the load of the stack cookie emitted into each function's prologue and epilogue.

This is not an issue for the core kernel, as the linker will relax these loads into LEA instructions that take the address of __stack_chk_guard directly. For modules, however, we need to work around this, by dealing with R_X86_64_REX_GOTPCRELX relocations that refer to __stack_chk_guard.

In this case, given that this is a GOT load, the reference should not refer to __stack_chk_guard directly, but to a memory location that holds its address. So take the address of __stack_chk_guard into a static variable, and fix up the relocations to refer to that.

[ mingo: Fix broken R_X86_64_GOTPCRELX definition. ]

---
 arch/x86/include/asm/elf.h |  5 +++--
 arch/x86/kernel/module.c   | 15 +++++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)
arch/x86/include/asm/elf.h +3 -2
index 1fb83d4..1286026 100644
@@ -54,8 +54,9 @@ typedef struct user_i387_struct elf_fpregset_t;
 #define R_X86_64_GLOB_DAT	6	/* Create GOT entry */
 #define R_X86_64_JUMP_SLOT	7	/* Create PLT entry */
 #define R_X86_64_RELATIVE	8	/* Adjust by program base */
-#define R_X86_64_GOTPCREL	9	/* 32 bit signed pc relative
-					   offset to GOT */
+#define R_X86_64_GOTPCREL	9	/* 32 bit signed pc relative offset to GOT */
+#define R_X86_64_GOTPCRELX	41
+#define R_X86_64_REX_GOTPCRELX	42
 #define R_X86_64_32		10	/* Direct 32 bit zero extended */
 #define R_X86_64_32S		11	/* Direct 32 bit sign extended */
 #define R_X86_64_16		12	/* Direct 16 bit zero extended */
arch/x86/kernel/module.c +15 -0
index 8984abd..a286f32 100644
@@ -19,6 +19,7 @@
 #include <linux/jump_label.h>
 #include <linux/random.h>
 #include <linux/memory.h>
+#include <linux/stackprotector.h>
 
 #include <asm/text-patching.h>
 #include <asm/page.h>
@@ -130,6 +131,20 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,
 				goto overflow;
 			size = 4;
 			break;
+#if defined(CONFIG_STACKPROTECTOR) && \
+    defined(CONFIG_CC_IS_CLANG) && CONFIG_CLANG_VERSION < 170000
+		case R_X86_64_REX_GOTPCRELX: {
+			static unsigned long __percpu *const addr = &__stack_chk_guard;
+
+			if (sym->st_value != (u64)addr) {
+				pr_err("%s: Unsupported GOTPCREL relocation\n", me->name);
+				return -ENOEXEC;
+			}
+
+			val = (u64)&addr + rel[i].r_addend;
+			fallthrough;
+		}
+#endif
 		case R_X86_64_PC32:
 		case R_X86_64_PLT32:
 			val -= (u64)loc;
#24 lore raw

PATCH v6 7/15 x86/stackprotector/64: Convert to normal percpu variable

In reply to Brian Gerst

Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.

GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.

---
 arch/x86/Makefile                     | 20 +++++++++------
 arch/x86/entry/entry.S                |  2 --
 arch/x86/entry/entry_64.S             |  2 +-
 arch/x86/include/asm/processor.h      | 16 ++----------
 arch/x86/include/asm/stackprotector.h | 36 ++++-----------------------
 arch/x86/kernel/asm-offsets_64.c      |  6 -----
 arch/x86/kernel/cpu/common.c          |  5 +---
 arch/x86/kernel/head_64.S             |  3 +--
 arch/x86/xen/xen-head.S               |  3 +--
 9 files changed, 23 insertions(+), 70 deletions(-)
arch/x86/Makefile +12 -8
index 5b773b34768d..88a1705366f9 100644
@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y)
         # temporary until string.h is fixed
         KBUILD_CFLAGS += -ffreestanding
 
-    ifeq ($(CONFIG_STACKPROTECTOR),y)
-        ifeq ($(CONFIG_SMP),y)
-            KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \
-                             -mstack-protector-guard-symbol=__ref_stack_chk_guard
-        else
-            KBUILD_CFLAGS += -mstack-protector-guard=global
-        endif
-    endif
+        percpu_seg := fs
 else
         BITS := 64
         UTS_MACHINE := x86_64
@@ -197,6 +190,17 @@ else
         KBUILD_CFLAGS += -mcmodel=kernel
         KBUILD_RUSTFLAGS += -Cno-redzone=y
         KBUILD_RUSTFLAGS += -Ccode-model=kernel
+
+        percpu_seg := gs
+endif
+
+ifeq ($(CONFIG_STACKPROTECTOR),y)
+    ifeq ($(CONFIG_SMP),y)
+	KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg)
+	KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard
+    else
+	KBUILD_CFLAGS += -mstack-protector-guard=global
+    endif
 endif
 
 #
arch/x86/entry/entry.S +0 -2
index b7ea3e8e9ecc..fe5344a249a1 100644
@@ -52,7 +52,6 @@ EXPORT_SYMBOL_GPL(mds_verw_sel);
 
 THUNK warn_thunk_thunk, __warn_thunk
 
-#ifndef CONFIG_X86_64
 /*
  * Clang's implementation of TLS stack cookies requires the variable in
  * question to be a TLS variable. If the variable happens to be defined as an
@@ -66,4 +65,3 @@ THUNK warn_thunk_thunk, __warn_thunk
 #ifdef CONFIG_STACKPROTECTOR
 EXPORT_SYMBOL(__ref_stack_chk_guard);
 #endif
-#endif
arch/x86/entry/entry_64.S +1 -1
index f52dbe0ad93c..33a955aa01d8 100644
@@ -192,7 +192,7 @@ SYM_FUNC_START(__switch_to_asm)
 
 #ifdef CONFIG_STACKPROTECTOR
 	movq	TASK_stack_canary(%rsi), %rbx
-	movq	%rbx, PER_CPU_VAR(fixed_percpu_data + FIXED_stack_canary)
+	movq	%rbx, PER_CPU_VAR(__stack_chk_guard)
 #endif
 
 	/*
arch/x86/include/asm/processor.h +2 -14
index c0cd10182e90..a4687122951f 100644
@@ -422,16 +422,8 @@ struct irq_stack {
 
 #ifdef CONFIG_X86_64
 struct fixed_percpu_data {
-	/*
-	 * GCC hardcodes the stack canary as %gs:40.  Since the
-	 * irq_stack is the object at %gs:0, we reserve the bottom
-	 * 48 bytes of the irq stack for the canary.
-	 *
-	 * Once we are willing to require -mstack-protector-guard-symbol=
-	 * support for x86_64 stackprotector, we can get rid of this.
-	 */
 	char		gs_base[40];
-	unsigned long	stack_canary;
+	unsigned long	reserved;
 };
 
 DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
@@ -446,11 +438,7 @@ extern asmlinkage void entry_SYSCALL32_ignore(void);
 
 /* Save actual FS/GS selectors and bases to current->thread */
 void current_save_fsgs(void);
-#else	/* X86_64 */
-#ifdef CONFIG_STACKPROTECTOR
-DECLARE_PER_CPU(unsigned long, __stack_chk_guard);
-#endif
-#endif	/* !X86_64 */
+#endif	/* X86_64 */
 
 struct perf_event;
 
arch/x86/include/asm/stackprotector.h +5 -31
index 00473a650f51..d43fb589fcf6 100644
@@ -2,26 +2,10 @@
 /*
  * GCC stack protector support.
  *
- * Stack protector works by putting predefined pattern at the start of
+ * Stack protector works by putting a predefined pattern at the start of
  * the stack frame and verifying that it hasn't been overwritten when
- * returning from the function.  The pattern is called stack canary
- * and unfortunately gcc historically required it to be at a fixed offset
- * from the percpu segment base.  On x86_64, the offset is 40 bytes.
- *
- * The same segment is shared by percpu area and stack canary.  On
- * x86_64, percpu symbols are zero based and %gs (64-bit) points to the
- * base of percpu area.  The first occupant of the percpu area is always
- * fixed_percpu_data which contains stack_canary at the appropriate
- * offset.  On x86_32, the stack canary is just a regular percpu
- * variable.
- *
- * Putting percpu data in %fs on 32-bit is a minor optimization compared to
- * using %gs.  Since 32-bit userspace normally has %fs == 0, we are likely
- * to load 0 into %fs on exit to usermode, whereas with percpu data in
- * %gs, we are likely to load a non-null %gs on return to user mode.
- *
- * Once we are willing to require GCC 8.1 or better for 64-bit stackprotector
- * support, we can remove some of this complexity.
+ * returning from the function.  The pattern is called the stack canary
+ * and is a unique value for each task.
  */
 
 #ifndef _ASM_STACKPROTECTOR_H
@@ -36,6 +20,8 @@
 
 #include <linux/sched.h>
 
+DECLARE_PER_CPU(unsigned long, __stack_chk_guard);
+
 /*
  * Initialize the stackprotector canary value.
  *
@@ -51,25 +37,13 @@ static __always_inline void boot_init_stack_canary(void)
 {
 	unsigned long canary = get_random_canary();
 
-#ifdef CONFIG_X86_64
-	BUILD_BUG_ON(offsetof(struct fixed_percpu_data, stack_canary) != 40);
-#endif
-
 	current->stack_canary = canary;
-#ifdef CONFIG_X86_64
-	this_cpu_write(fixed_percpu_data.stack_canary, canary);
-#else
 	this_cpu_write(__stack_chk_guard, canary);
-#endif
 }
 
 static inline void cpu_init_stack_canary(int cpu, struct task_struct *idle)
 {
-#ifdef CONFIG_X86_64
-	per_cpu(fixed_percpu_data.stack_canary, cpu) = idle->stack_canary;
-#else
 	per_cpu(__stack_chk_guard, cpu) = idle->stack_canary;
-#endif
 }
 
 #else	/* STACKPROTECTOR */
arch/x86/kernel/asm-offsets_64.c +0 -6
index bb65371ea9df..590b6cd0eac0 100644
@@ -54,11 +54,5 @@ int main(void)
 	BLANK();
 #undef ENTRY
 
-	BLANK();
-
-#ifdef CONFIG_STACKPROTECTOR
-	OFFSET(FIXED_stack_canary, fixed_percpu_data, stack_canary);
-	BLANK();
-#endif
 	return 0;
 }
arch/x86/kernel/cpu/common.c +1 -4
index 7cce91b19fb2..b71178f0ed6c 100644
@@ -2089,8 +2089,7 @@ void syscall_init(void)
 	if (!cpu_feature_enabled(X86_FEATURE_FRED))
 		idt_syscall_init();
 }
-
-#else	/* CONFIG_X86_64 */
+#endif /* CONFIG_X86_64 */
 
 #ifdef CONFIG_STACKPROTECTOR
 DEFINE_PER_CPU(unsigned long, __stack_chk_guard);
@@ -2099,8 +2098,6 @@ EXPORT_PER_CPU_SYMBOL(__stack_chk_guard);
 #endif
 #endif
 
-#endif	/* CONFIG_X86_64 */
-
 /*
  * Clear all 6 debug registers:
  */
arch/x86/kernel/head_64.S +1 -2
index 31345e0ba006..c3d73c04603f 100644
@@ -361,8 +361,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 
 	/* Set up %gs.
 	 *
-	 * The base of %gs always points to fixed_percpu_data. If the
-	 * stack protector canary is enabled, it is located at %gs:40.
+	 * The base of %gs always points to fixed_percpu_data.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
arch/x86/xen/xen-head.S +1 -2
index 9252652afe59..5d3866ec3100 100644
@@ -33,8 +33,7 @@ SYM_CODE_START(startup_xen)
 
 	/* Set up %gs.
 	 *
-	 * The base of %gs always points to fixed_percpu_data.  If the
-	 * stack protector canary is enabled, it is located at %gs:40.
+	 * The base of %gs always points to fixed_percpu_data.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
Signature · 1 line
2.47.1
#25 lore raw

Re: [PATCH v6 07/15] x86/stackprotector/64: Convert to normal percpu variable

Quoted 3 lines · Brian Gerst

GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.

So there's two typos in this paragraph alone. Might be time to invest into an editor that does spellchecking for you?

s/allowng
/allowing

s/contraint
/constraint

Thanks,

Ingo

merge bot Applied
#26 lore raw

[tip: x86/asm] x86/stackprotector/64: Convert to normal per-CPU variable

In reply to Brian Gerst
The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     80d47defddc000271502057ebd7efa4fd6481542
Gitweb:        https://git.kernel.org/tip/80d47defddc000271502057ebd7efa4fd6481542
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:39 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:09 +01:00

x86/stackprotector/64: Convert to normal per-CPU variable

Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.

GCC 8.1 supports redefining where the canary is located, allowing it to become a normal percpu variable instead of at a fixed location. This removes the constraint that the percpu section must be zero-based.

---
 arch/x86/Makefile                     | 20 ++++++++------
 arch/x86/entry/entry.S                |  2 +-
 arch/x86/entry/entry_64.S             |  2 +-
 arch/x86/include/asm/processor.h      | 16 +-----------
 arch/x86/include/asm/stackprotector.h | 36 +++-----------------------
 arch/x86/kernel/asm-offsets_64.c      |  6 +----
 arch/x86/kernel/cpu/common.c          |  5 +----
 arch/x86/kernel/head_64.S             |  3 +--
 arch/x86/xen/xen-head.S               |  3 +--
 9 files changed, 23 insertions(+), 70 deletions(-)
arch/x86/Makefile +12 -8
index 5b773b3..88a1705 100644
@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y)
         # temporary until string.h is fixed
         KBUILD_CFLAGS += -ffreestanding
 
-    ifeq ($(CONFIG_STACKPROTECTOR),y)
-        ifeq ($(CONFIG_SMP),y)
-            KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \
-                             -mstack-protector-guard-symbol=__ref_stack_chk_guard
-        else
-            KBUILD_CFLAGS += -mstack-protector-guard=global
-        endif
-    endif
+        percpu_seg := fs
 else
         BITS := 64
         UTS_MACHINE := x86_64
@@ -197,6 +190,17 @@ else
         KBUILD_CFLAGS += -mcmodel=kernel
         KBUILD_RUSTFLAGS += -Cno-redzone=y
         KBUILD_RUSTFLAGS += -Ccode-model=kernel
+
+        percpu_seg := gs
+endif
+
+ifeq ($(CONFIG_STACKPROTECTOR),y)
+    ifeq ($(CONFIG_SMP),y)
+	KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg)
+	KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard
+    else
+	KBUILD_CFLAGS += -mstack-protector-guard=global
+    endif
 endif
 
 #
arch/x86/entry/entry.S +0 -2
index b7ea3e8..fe5344a 100644
@@ -52,7 +52,6 @@ EXPORT_SYMBOL_GPL(mds_verw_sel);
 
 THUNK warn_thunk_thunk, __warn_thunk
 
-#ifndef CONFIG_X86_64
 /*
  * Clang's implementation of TLS stack cookies requires the variable in
  * question to be a TLS variable. If the variable happens to be defined as an
@@ -66,4 +65,3 @@ THUNK warn_thunk_thunk, __warn_thunk
 #ifdef CONFIG_STACKPROTECTOR
 EXPORT_SYMBOL(__ref_stack_chk_guard);
 #endif
-#endif
arch/x86/entry/entry_64.S +1 -1
index f52dbe0..33a955a 100644
@@ -192,7 +192,7 @@ SYM_FUNC_START(__switch_to_asm)
 
 #ifdef CONFIG_STACKPROTECTOR
 	movq	TASK_stack_canary(%rsi), %rbx
-	movq	%rbx, PER_CPU_VAR(fixed_percpu_data + FIXED_stack_canary)
+	movq	%rbx, PER_CPU_VAR(__stack_chk_guard)
 #endif
 
 	/*
arch/x86/include/asm/processor.h +2 -14
index c0cd101..a468712 100644
@@ -422,16 +422,8 @@ struct irq_stack {
 
 #ifdef CONFIG_X86_64
 struct fixed_percpu_data {
-	/*
-	 * GCC hardcodes the stack canary as %gs:40.  Since the
-	 * irq_stack is the object at %gs:0, we reserve the bottom
-	 * 48 bytes of the irq stack for the canary.
-	 *
-	 * Once we are willing to require -mstack-protector-guard-symbol=
-	 * support for x86_64 stackprotector, we can get rid of this.
-	 */
 	char		gs_base[40];
-	unsigned long	stack_canary;
+	unsigned long	reserved;
 };
 
 DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
@@ -446,11 +438,7 @@ extern asmlinkage void entry_SYSCALL32_ignore(void);
 
 /* Save actual FS/GS selectors and bases to current->thread */
 void current_save_fsgs(void);
-#else	/* X86_64 */
-#ifdef CONFIG_STACKPROTECTOR
-DECLARE_PER_CPU(unsigned long, __stack_chk_guard);
-#endif
-#endif	/* !X86_64 */
+#endif	/* X86_64 */
 
 struct perf_event;
 
arch/x86/include/asm/stackprotector.h +5 -31
index 00473a6..d43fb58 100644
@@ -2,26 +2,10 @@
 /*
  * GCC stack protector support.
  *
- * Stack protector works by putting predefined pattern at the start of
+ * Stack protector works by putting a predefined pattern at the start of
  * the stack frame and verifying that it hasn't been overwritten when
- * returning from the function.  The pattern is called stack canary
- * and unfortunately gcc historically required it to be at a fixed offset
- * from the percpu segment base.  On x86_64, the offset is 40 bytes.
- *
- * The same segment is shared by percpu area and stack canary.  On
- * x86_64, percpu symbols are zero based and %gs (64-bit) points to the
- * base of percpu area.  The first occupant of the percpu area is always
- * fixed_percpu_data which contains stack_canary at the appropriate
- * offset.  On x86_32, the stack canary is just a regular percpu
- * variable.
- *
- * Putting percpu data in %fs on 32-bit is a minor optimization compared to
- * using %gs.  Since 32-bit userspace normally has %fs == 0, we are likely
- * to load 0 into %fs on exit to usermode, whereas with percpu data in
- * %gs, we are likely to load a non-null %gs on return to user mode.
- *
- * Once we are willing to require GCC 8.1 or better for 64-bit stackprotector
- * support, we can remove some of this complexity.
+ * returning from the function.  The pattern is called the stack canary
+ * and is a unique value for each task.
  */
 
 #ifndef _ASM_STACKPROTECTOR_H
@@ -36,6 +20,8 @@
 
 #include <linux/sched.h>
 
+DECLARE_PER_CPU(unsigned long, __stack_chk_guard);
+
 /*
  * Initialize the stackprotector canary value.
  *
@@ -51,25 +37,13 @@ static __always_inline void boot_init_stack_canary(void)
 {
 	unsigned long canary = get_random_canary();
 
-#ifdef CONFIG_X86_64
-	BUILD_BUG_ON(offsetof(struct fixed_percpu_data, stack_canary) != 40);
-#endif
-
 	current->stack_canary = canary;
-#ifdef CONFIG_X86_64
-	this_cpu_write(fixed_percpu_data.stack_canary, canary);
-#else
 	this_cpu_write(__stack_chk_guard, canary);
-#endif
 }
 
 static inline void cpu_init_stack_canary(int cpu, struct task_struct *idle)
 {
-#ifdef CONFIG_X86_64
-	per_cpu(fixed_percpu_data.stack_canary, cpu) = idle->stack_canary;
-#else
 	per_cpu(__stack_chk_guard, cpu) = idle->stack_canary;
-#endif
 }
 
 #else	/* STACKPROTECTOR */
arch/x86/kernel/asm-offsets_64.c +0 -6
index bb65371..590b6cd 100644
@@ -54,11 +54,5 @@ int main(void)
 	BLANK();
 #undef ENTRY
 
-	BLANK();
-
-#ifdef CONFIG_STACKPROTECTOR
-	OFFSET(FIXED_stack_canary, fixed_percpu_data, stack_canary);
-	BLANK();
-#endif
 	return 0;
 }
arch/x86/kernel/cpu/common.c +1 -4
index 7cce91b..b71178f 100644
@@ -2089,8 +2089,7 @@ void syscall_init(void)
 	if (!cpu_feature_enabled(X86_FEATURE_FRED))
 		idt_syscall_init();
 }
-
-#else	/* CONFIG_X86_64 */
+#endif /* CONFIG_X86_64 */
 
 #ifdef CONFIG_STACKPROTECTOR
 DEFINE_PER_CPU(unsigned long, __stack_chk_guard);
@@ -2099,8 +2098,6 @@ EXPORT_PER_CPU_SYMBOL(__stack_chk_guard);
 #endif
 #endif
 
-#endif	/* CONFIG_X86_64 */
-
 /*
  * Clear all 6 debug registers:
  */
arch/x86/kernel/head_64.S +1 -2
index 31345e0..c3d73c0 100644
@@ -361,8 +361,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 
 	/* Set up %gs.
 	 *
-	 * The base of %gs always points to fixed_percpu_data. If the
-	 * stack protector canary is enabled, it is located at %gs:40.
+	 * The base of %gs always points to fixed_percpu_data.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
arch/x86/xen/xen-head.S +1 -2
index 894edf8..a31b057 100644
@@ -33,8 +33,7 @@ SYM_CODE_START(startup_xen)
 
 	/* Set up %gs.
 	 *
-	 * The base of %gs always points to fixed_percpu_data.  If the
-	 * stack protector canary is enabled, it is located at %gs:40.
+	 * The base of %gs always points to fixed_percpu_data.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
#27 lore raw

Re: [PATCH v6 07/15] x86/stackprotector/64: Convert to normal percpu variable

In reply to Brian Gerst
Quoted 15 lines · Brian Gerst

Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.

GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.

...

Quoted 26 lines
arch/x86/Makefile +3 -8
index 5b773b34768d..88a1705366f9 100644
@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y)
         # temporary until string.h is fixed
         KBUILD_CFLAGS += -ffreestanding
 
-    ifeq ($(CONFIG_STACKPROTECTOR),y)
-        ifeq ($(CONFIG_SMP),y)
-            KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \
-                             -mstack-protector-guard-symbol=__ref_stack_chk_guard
-        else
-            KBUILD_CFLAGS += -mstack-protector-guard=global
-        endif
-    endif
+        percpu_seg := fs
 else
         BITS := 64
         UTS_MACHINE := x86_64
@@ -197,6 +190,17 @@ else
         KBUILD_CFLAGS += -mcmodel=kernel
         KBUILD_RUSTFLAGS += -Cno-redzone=y
         KBUILD_RUSTFLAGS += -Ccode-model=kernel
+
+        percpu_seg := gs

arch/x86/Makefile else · line 204

Quoted 10 lines
arch/x86/Makefile +9 -0
+endif
+
+ifeq ($(CONFIG_STACKPROTECTOR),y)
+    ifeq ($(CONFIG_SMP),y)
+	KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg)
+	KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard
+    else
+	KBUILD_CFLAGS += -mstack-protector-guard=global
+    endif
 endif

-mstack-protector-guard-symbol was only added in clang-15, so it looks like min-tool-version.sh will need an adjustment like GCC did, should I send a patch?

https://github.com/llvm/llvm-project/commit/efbaad1c4a526e91b034e56386e98a9268cd87b2

scripts/min-tool-version.sh +1 -1
index 06c4e410ecab..787868183b84 100755
@@ -26,7 +26,7 @@ gcc)
 	fi
 	;;
 llvm)
-	if [ "$SRCARCH" = s390 ]; then
+	if [ "$SRCARCH" = s390 -o "$SRCARCH" = x86 ]; then
 		echo 15.0.0
 	elif [ "$SRCARCH" = loongarch ]; then
 		echo 18.0.0
#28 lore raw

Re: [PATCH v6 07/15] x86/stackprotector/64: Convert to normal percpu variable

Quoted 50 lines · Nathan Chancellor
Quoted 15 lines · Brian Gerst

Older versions of GCC fixed the location of the stack protector canary at %gs:40. This constraint forced the percpu section to be linked at absolute address 0 so that the canary could be the first data object in the percpu section. Supporting the zero-based percpu section requires additional code to handle relocations for RIP-relative references to percpu data, extra complexity to kallsyms, and workarounds for linker bugs due to the use of absolute symbols.

GCC 8.1 supports redefining where the canary is located, allowng it to become a normal percpu variable instead of at a fixed location. This removes the contraint that the percpu section must be zero-based.

...

Quoted 32 lines
arch/x86/Makefile +9 -8
index 5b773b34768d..88a1705366f9 100644
@@ -140,14 +140,7 @@ ifeq ($(CONFIG_X86_32),y)
         # temporary until string.h is fixed
         KBUILD_CFLAGS += -ffreestanding
 
-    ifeq ($(CONFIG_STACKPROTECTOR),y)
-        ifeq ($(CONFIG_SMP),y)
-            KBUILD_CFLAGS += -mstack-protector-guard-reg=fs \
-                             -mstack-protector-guard-symbol=__ref_stack_chk_guard
-        else
-            KBUILD_CFLAGS += -mstack-protector-guard=global
-        endif
-    endif
+        percpu_seg := fs
 else
         BITS := 64
         UTS_MACHINE := x86_64
@@ -197,6 +190,17 @@ else
         KBUILD_CFLAGS += -mcmodel=kernel
         KBUILD_RUSTFLAGS += -Cno-redzone=y
         KBUILD_RUSTFLAGS += -Ccode-model=kernel
+
+        percpu_seg := gs
+endif
+
+ifeq ($(CONFIG_STACKPROTECTOR),y)
+    ifeq ($(CONFIG_SMP),y)
+     KBUILD_CFLAGS += -mstack-protector-guard-reg=$(percpu_seg)
+     KBUILD_CFLAGS += -mstack-protector-guard-symbol=__ref_stack_chk_guard
Quoted 9 lines

arch/x86/Makefile else · line 204

Quoted 4 lines
arch/x86/Makefile +3 -0
+    else
+     KBUILD_CFLAGS += -mstack-protector-guard=global
+    endif
 endif

-mstack-protector-guard-symbol was only added in clang-15, so it looks like min-tool-version.sh will need an adjustment like GCC did, should I send a patch?

Yes, please.

merge bot Applied
#29 lore raw

[tip: x86/asm] x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP

In reply to Brian Gerst
The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     c0560fbaab0dab0afbdb3e339b3cb61ef8d3115e
Gitweb:        https://git.kernel.org/tip/c0560fbaab0dab0afbdb3e339b3cb61ef8d3115e
Author:        Ingo Molnar <[email protected]>
AuthorDate:    Wed, 12 Mar 2025 12:48:49 +01:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Wed, 12 Mar 2025 12:51:35 +01:00

x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP

The __ref_stack_chk_guard symbol doesn't exist on UP:

<stdin>:4:15: error: ‘__ref_stack_chk_guard’ undeclared here (not in a function)

Fix the #ifdef around the entry.S export.

---
 arch/x86/entry/entry.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/entry/entry.S +1 -1
index fe5344a..35a91cf 100644
@@ -62,6 +62,6 @@ THUNK warn_thunk_thunk, __warn_thunk
  * entirely in the C code, and use an alias emitted by the linker script
  * instead.
  */
-#ifdef CONFIG_STACKPROTECTOR
+#if defined(CONFIG_STACKPROTECTOR) && defined(CONFIG_SMP)
 EXPORT_SYMBOL(__ref_stack_chk_guard);
 #endif
merge bot Applied
#30 lore raw

[tip: x86/core] x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP

In reply to Brian Gerst
The following commit has been merged into the x86/core branch of tip:

The following commit has been merged into the x86/core branch of tip:

Commit-ID:     91d5451d97ce35cbd510277fa3b7abf9caa4e34d
Gitweb:        https://git.kernel.org/tip/91d5451d97ce35cbd510277fa3b7abf9caa4e34d
Author:        Ingo Molnar <[email protected]>
AuthorDate:    Wed, 12 Mar 2025 12:48:49 +01:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Wed, 19 Mar 2025 11:26:58 +01:00

x86/stackprotector/64: Only export __ref_stack_chk_guard on CONFIG_SMP

The __ref_stack_chk_guard symbol doesn't exist on UP:

<stdin>:4:15: error: ‘__ref_stack_chk_guard’ undeclared here (not in a function)

Fix the #ifdef around the entry.S export.

---
 arch/x86/entry/entry.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/entry/entry.S +1 -1
index 088f91f..d3caa31 100644
@@ -64,6 +64,6 @@ THUNK warn_thunk_thunk, __warn_thunk
  * entirely in the C code, and use an alias emitted by the linker script
  * instead.
  */
-#ifdef CONFIG_STACKPROTECTOR
+#if defined(CONFIG_STACKPROTECTOR) && defined(CONFIG_SMP)
 EXPORT_SYMBOL(__ref_stack_chk_guard);
 #endif
#31 lore raw

PATCH v6 8/15 x86/percpu/64: Use relative percpu offsets

In reply to Brian Gerst

The percpu section is currently linked at absolute address 0, because older compilers hardcoded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section does not need to be linked at a specific address.

x86-64 will now calculate the percpu offsets as the delta between the initial percpu address and the dynamically allocated memory, like other architectures. Note that GSBASE is limited to the canonical address width (48 or 57 bits, sign-extended). As long as the kernel text, modules, and the dynamically allocated percpu memmory are all in the negative address space, the delta will not overflow this limit.

---
 arch/x86/include/asm/processor.h |  6 +++++-
 arch/x86/kernel/head_64.S        | 19 +++++++++----------
 arch/x86/kernel/setup_percpu.c   | 12 ++----------
 arch/x86/kernel/vmlinux.lds.S    | 29 +----------------------------
 arch/x86/platform/pvh/head.S     |  5 ++---
 arch/x86/tools/relocs.c          | 10 +++-------
 arch/x86/xen/xen-head.S          |  9 ++++-----
 init/Kconfig                     |  2 +-
 8 files changed, 27 insertions(+), 65 deletions(-)
arch/x86/include/asm/processor.h +5 -1
index a4687122951f..b8fee88dac3d 100644
@@ -431,7 +431,11 @@ DECLARE_INIT_PER_CPU(fixed_percpu_data);
 
 static inline unsigned long cpu_kernelmode_gs_base(int cpu)
 {
-	return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
+#ifdef CONFIG_SMP
+	return per_cpu_offset(cpu);
+#else
+	return 0;
+#endif
 }
 
 extern asmlinkage void entry_SYSCALL32_ignore(void);
arch/x86/kernel/head_64.S +9 -10
index c3d73c04603f..905d8be93220 100644
@@ -61,11 +61,14 @@ SYM_CODE_START_NOALIGN(startup_64)
 	/* Set up the stack for verify_cpu() */
 	leaq	__top_init_kernel_stack(%rip), %rsp
 
-	/* Setup GSBASE to allow stack canary access for C code */
+	/*
+	 * Set up GSBASE.
+	 * Note that, on SMP, the boot cpu uses init data section until
+	 * the per cpu areas are set up.
+	 */
 	movl	$MSR_GS_BASE, %ecx
-	leaq	INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
-	movl	%edx, %eax
-	shrq	$32,  %rdx
+	xorl	%eax, %eax
+	xorl	%edx, %edx
 	wrmsr
 
 	call	startup_64_setup_gdt_idt
@@ -359,16 +362,12 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 	movl %eax,%fs
 	movl %eax,%gs
 
-	/* Set up %gs.
-	 *
-	 * The base of %gs always points to fixed_percpu_data.
+	/*
+	 * Set up GSBASE.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
 	movl	$MSR_GS_BASE,%ecx
-#ifndef CONFIG_SMP
-	leaq	INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
-#endif
 	movl	%edx, %eax
 	shrq	$32, %rdx
 	wrmsr
arch/x86/kernel/setup_percpu.c +2 -10
index b30d6e180df7..1e7be9409aa2 100644
@@ -23,18 +23,10 @@
 #include <asm/cpumask.h>
 #include <asm/cpu.h>
 
-#ifdef CONFIG_X86_64
-#define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load)
-#else
-#define BOOT_PERCPU_OFFSET 0
-#endif
-
-DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off) = BOOT_PERCPU_OFFSET;
+DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off);
 EXPORT_PER_CPU_SYMBOL(this_cpu_off);
 
-unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init = {
-	[0 ... NR_CPUS-1] = BOOT_PERCPU_OFFSET,
-};
+unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init;
 EXPORT_SYMBOL(__per_cpu_offset);
 
 /*
arch/x86/kernel/vmlinux.lds.S +1 -28
index 0deb4887d6e9..8a598515239a 100644
@@ -112,12 +112,6 @@ ASSERT(__relocate_kernel_end - __relocate_kernel_start <= KEXEC_CONTROL_CODE_MAX
 PHDRS {
 	text PT_LOAD FLAGS(5);          /* R_E */
 	data PT_LOAD FLAGS(6);          /* RW_ */
-#ifdef CONFIG_X86_64
-#ifdef CONFIG_SMP
-	percpu PT_LOAD FLAGS(6);        /* RW_ */
-#endif
-	init PT_LOAD FLAGS(7);          /* RWE */
-#endif
 	note PT_NOTE FLAGS(0);          /* ___ */
 }
 
@@ -216,21 +210,7 @@ SECTIONS
 		__init_begin = .; /* paired with __init_end */
 	}
 
-#if defined(CONFIG_X86_64) && defined(CONFIG_SMP)
-	/*
-	 * percpu offsets are zero-based on SMP.  PERCPU_VADDR() changes the
-	 * output PHDR, so the next output section - .init.text - should
-	 * start another segment - init.
-	 */
-	PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu)
-	ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START,
-	       "per-CPU data too large - increase CONFIG_PHYSICAL_START")
-#endif
-
 	INIT_TEXT_SECTION(PAGE_SIZE)
-#ifdef CONFIG_X86_64
-	:init
-#endif
 
 	/*
 	 * Section for code used exclusively before alternatives are run. All
@@ -347,9 +327,7 @@ SECTIONS
 		EXIT_DATA
 	}
 
-#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP)
 	PERCPU_SECTION(INTERNODE_CACHE_BYTES)
-#endif
 
 	RUNTIME_CONST_VARIABLES
 	RUNTIME_CONST(ptr, USER_PTR_MAX)
@@ -497,16 +475,11 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
  * Per-cpu symbols which need to be offset from __per_cpu_load
  * for the boot processor.
  */
-#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load
+#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x)
 INIT_PER_CPU(gdt_page);
 INIT_PER_CPU(fixed_percpu_data);
 INIT_PER_CPU(irq_stack_backing_store);
 
-#ifdef CONFIG_SMP
-. = ASSERT((fixed_percpu_data == 0),
-           "fixed_percpu_data is not at start of per-cpu area");
-#endif
-
 #ifdef CONFIG_MITIGATION_UNRET_ENTRY
 . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
 #endif
arch/x86/platform/pvh/head.S +2 -3
index fa0072e0ca43..84bb46f86421 100644
@@ -179,9 +179,8 @@ SYM_CODE_START(pvh_start_xen)
 	 * the per cpu areas are set up.
 	 */
 	movl $MSR_GS_BASE,%ecx
-	leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
-	movq %edx, %eax
-	shrq $32, %rdx
+	xorl %eax, %eax
+	xorl %edx, %edx
 	wrmsr
 
 	/* Call xen_prepare_pvh() via the kernel virtual mapping */
arch/x86/tools/relocs.c +3 -7
index 33dffc5c30b5..9aebc3b18d73 100644
@@ -835,12 +835,7 @@ static void percpu_init(void)
  */
 static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)
 {
-	int shndx = sym_index(sym);
-
-	return (shndx == per_cpu_shndx) &&
-		strcmp(symname, "__init_begin") &&
-		strcmp(symname, "__per_cpu_load") &&
-		strncmp(symname, "init_per_cpu_", 13);
+	return 0;
 }
 
 
@@ -1062,7 +1057,8 @@ static int cmp_relocs(const void *va, const void *vb)
 
 static void sort_relocs(struct relocs *r)
 {
-	qsort(r->offset, r->count, sizeof(r->offset[0]), cmp_relocs);
+	if (r->count)
+		qsort(r->offset, r->count, sizeof(r->offset[0]), cmp_relocs);
 }
 
 static int write32(uint32_t v, FILE *f)
arch/x86/xen/xen-head.S +4 -5
index 5d3866ec3100..0aed24540212 100644
@@ -31,15 +31,14 @@ SYM_CODE_START(startup_xen)
 
 	leaq	__top_init_kernel_stack(%rip), %rsp
 
-	/* Set up %gs.
-	 *
-	 * The base of %gs always points to fixed_percpu_data.
+	/*
+	 * Set up GSBASE.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
 	movl	$MSR_GS_BASE,%ecx
-	movq	$INIT_PER_CPU_VAR(fixed_percpu_data),%rax
-	cdq
+	xorl	%eax, %eax
+	xorl	%edx, %edx
 	wrmsr
 
 	mov	%rsi, %rdi
init/Kconfig +1 -1
index 7fe82a46e88c..01d36a84cf66 100644
@@ -1873,7 +1873,7 @@ config KALLSYMS_ALL
 config KALLSYMS_ABSOLUTE_PERCPU
 	bool
 	depends on KALLSYMS
-	default X86_64 && SMP
+	default n
 
 # end of the "standard kernel features (expert users)" menu
 
Signature · 1 line
2.47.1
merge bot Applied
#32 lore raw

[tip: x86/asm] x86/percpu/64: Use relative percpu offsets

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     9d7de2aa8b41407bc96d89a80dc1fd637d389d42
Gitweb:        https://git.kernel.org/tip/9d7de2aa8b41407bc96d89a80dc1fd637d389d42
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:40 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:27 +01:00

x86/percpu/64: Use relative percpu offsets

The percpu section is currently linked at absolute address 0, because older compilers hard-coded the stack protector canary value at a fixed offset from the start of the GS segment. Now that the canary is a normal percpu variable, the percpu section does not need to be linked at a specific address.

x86-64 will now calculate the percpu offsets as the delta between the initial percpu address and the dynamically allocated memory, like other architectures. Note that GSBASE is limited to the canonical address width (48 or 57 bits, sign-extended). As long as the kernel text, modules, and the dynamically allocated percpu memory are all in the negative address space, the delta will not overflow this limit.

---
 arch/x86/include/asm/processor.h |  6 +++++-
 arch/x86/kernel/head_64.S        | 19 +++++++++----------
 arch/x86/kernel/setup_percpu.c   | 12 ++----------
 arch/x86/kernel/vmlinux.lds.S    | 29 +----------------------------
 arch/x86/platform/pvh/head.S     |  5 ++---
 arch/x86/tools/relocs.c          | 10 +++-------
 arch/x86/xen/xen-head.S          |  9 ++++-----
 init/Kconfig                     |  2 +-
 8 files changed, 27 insertions(+), 65 deletions(-)
arch/x86/include/asm/processor.h +5 -1
index a468712..b8fee88 100644
@@ -431,7 +431,11 @@ DECLARE_INIT_PER_CPU(fixed_percpu_data);
 
 static inline unsigned long cpu_kernelmode_gs_base(int cpu)
 {
-	return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
+#ifdef CONFIG_SMP
+	return per_cpu_offset(cpu);
+#else
+	return 0;
+#endif
 }
 
 extern asmlinkage void entry_SYSCALL32_ignore(void);
arch/x86/kernel/head_64.S +9 -10
index c3d73c0..2843b0a 100644
@@ -61,11 +61,14 @@ SYM_CODE_START_NOALIGN(startup_64)
 	/* Set up the stack for verify_cpu() */
 	leaq	__top_init_kernel_stack(%rip), %rsp
 
-	/* Setup GSBASE to allow stack canary access for C code */
+	/*
+	 * Set up GSBASE.
+	 * Note that on SMP the boot CPU uses the init data section until
+	 * the per-CPU areas are set up.
+	 */
 	movl	$MSR_GS_BASE, %ecx
-	leaq	INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
-	movl	%edx, %eax
-	shrq	$32,  %rdx
+	xorl	%eax, %eax
+	xorl	%edx, %edx
 	wrmsr
 
 	call	startup_64_setup_gdt_idt
@@ -359,16 +362,12 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL)
 	movl %eax,%fs
 	movl %eax,%gs
 
-	/* Set up %gs.
-	 *
-	 * The base of %gs always points to fixed_percpu_data.
+	/*
+	 * Set up GSBASE.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
 	movl	$MSR_GS_BASE,%ecx
-#ifndef CONFIG_SMP
-	leaq	INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
-#endif
 	movl	%edx, %eax
 	shrq	$32, %rdx
 	wrmsr
arch/x86/kernel/setup_percpu.c +2 -10
index b30d6e1..1e7be94 100644
@@ -23,18 +23,10 @@
 #include <asm/cpumask.h>
 #include <asm/cpu.h>
 
-#ifdef CONFIG_X86_64
-#define BOOT_PERCPU_OFFSET ((unsigned long)__per_cpu_load)
-#else
-#define BOOT_PERCPU_OFFSET 0
-#endif
-
-DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off) = BOOT_PERCPU_OFFSET;
+DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off);
 EXPORT_PER_CPU_SYMBOL(this_cpu_off);
 
-unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init = {
-	[0 ... NR_CPUS-1] = BOOT_PERCPU_OFFSET,
-};
+unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init;
 EXPORT_SYMBOL(__per_cpu_offset);
 
 /*
arch/x86/kernel/vmlinux.lds.S +1 -28
index 0deb488..8a59851 100644
@@ -112,12 +112,6 @@ ASSERT(__relocate_kernel_end - __relocate_kernel_start <= KEXEC_CONTROL_CODE_MAX
 PHDRS {
 	text PT_LOAD FLAGS(5);          /* R_E */
 	data PT_LOAD FLAGS(6);          /* RW_ */
-#ifdef CONFIG_X86_64
-#ifdef CONFIG_SMP
-	percpu PT_LOAD FLAGS(6);        /* RW_ */
-#endif
-	init PT_LOAD FLAGS(7);          /* RWE */
-#endif
 	note PT_NOTE FLAGS(0);          /* ___ */
 }
 
@@ -216,21 +210,7 @@ SECTIONS
 		__init_begin = .; /* paired with __init_end */
 	}
 
-#if defined(CONFIG_X86_64) && defined(CONFIG_SMP)
-	/*
-	 * percpu offsets are zero-based on SMP.  PERCPU_VADDR() changes the
-	 * output PHDR, so the next output section - .init.text - should
-	 * start another segment - init.
-	 */
-	PERCPU_VADDR(INTERNODE_CACHE_BYTES, 0, :percpu)
-	ASSERT(SIZEOF(.data..percpu) < CONFIG_PHYSICAL_START,
-	       "per-CPU data too large - increase CONFIG_PHYSICAL_START")
-#endif
-
 	INIT_TEXT_SECTION(PAGE_SIZE)
-#ifdef CONFIG_X86_64
-	:init
-#endif
 
 	/*
 	 * Section for code used exclusively before alternatives are run. All
@@ -347,9 +327,7 @@ SECTIONS
 		EXIT_DATA
 	}
 
-#if !defined(CONFIG_X86_64) || !defined(CONFIG_SMP)
 	PERCPU_SECTION(INTERNODE_CACHE_BYTES)
-#endif
 
 	RUNTIME_CONST_VARIABLES
 	RUNTIME_CONST(ptr, USER_PTR_MAX)
@@ -497,16 +475,11 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
  * Per-cpu symbols which need to be offset from __per_cpu_load
  * for the boot processor.
  */
-#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x) + __per_cpu_load
+#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x)
 INIT_PER_CPU(gdt_page);
 INIT_PER_CPU(fixed_percpu_data);
 INIT_PER_CPU(irq_stack_backing_store);
 
-#ifdef CONFIG_SMP
-. = ASSERT((fixed_percpu_data == 0),
-           "fixed_percpu_data is not at start of per-cpu area");
-#endif
-
 #ifdef CONFIG_MITIGATION_UNRET_ENTRY
 . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
 #endif
arch/x86/platform/pvh/head.S +2 -3
index 723f181..cfa18ec 100644
@@ -179,9 +179,8 @@ SYM_CODE_START(pvh_start_xen)
 	 * the per-CPU areas are set up.
 	 */
 	movl $MSR_GS_BASE,%ecx
-	leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
-	movq %edx, %eax
-	shrq $32, %rdx
+	xorl %eax, %eax
+	xorl %edx, %edx
 	wrmsr
 
 	/* Call xen_prepare_pvh() via the kernel virtual mapping */
arch/x86/tools/relocs.c +3 -7
index 92a1e50..3cb3b30 100644
@@ -835,12 +835,7 @@ static void percpu_init(void)
  */
 static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)
 {
-	int shndx = sym_index(sym);
-
-	return (shndx == per_cpu_shndx) &&
-		strcmp(symname, "__init_begin") &&
-		strcmp(symname, "__per_cpu_load") &&
-		strncmp(symname, "init_per_cpu_", 13);
+	return 0;
 }
 
 
@@ -1062,7 +1057,8 @@ static int cmp_relocs(const void *va, const void *vb)
 
 static void sort_relocs(struct relocs *r)
 {
-	qsort(r->offset, r->count, sizeof(r->offset[0]), cmp_relocs);
+	if (r->count)
+		qsort(r->offset, r->count, sizeof(r->offset[0]), cmp_relocs);
 }
 
 static int write32(uint32_t v, FILE *f)
arch/x86/xen/xen-head.S +4 -5
index a31b057..5ccb4c5 100644
@@ -31,15 +31,14 @@ SYM_CODE_START(startup_xen)
 
 	leaq	__top_init_kernel_stack(%rip), %rsp
 
-	/* Set up %gs.
-	 *
-	 * The base of %gs always points to fixed_percpu_data.
+	/*
+	 * Set up GSBASE.
 	 * Note that, on SMP, the boot cpu uses init data section until
 	 * the per cpu areas are set up.
 	 */
 	movl	$MSR_GS_BASE,%ecx
-	movq	$INIT_PER_CPU_VAR(fixed_percpu_data),%rax
-	cdq
+	xorl	%eax, %eax
+	xorl	%edx, %edx
 	wrmsr
 
 	mov	%rsi, %rdi
init/Kconfig +1 -1
index d0d021b..b5d9c0f 100644
@@ -1872,7 +1872,7 @@ config KALLSYMS_ALL
 config KALLSYMS_ABSOLUTE_PERCPU
 	bool
 	depends on KALLSYMS
-	default X86_64 && SMP
+	default n
 
 # end of the "standard kernel features (expert users)" menu
 
#33 lore raw

PATCH v6 9/15 x86/percpu/64: Remove fixed_percpu_data

In reply to Brian Gerst

Now that the stack protector canary value is a normal percpu variable, fixed_percpu_data is unused and can be removed.

---
 arch/x86/include/asm/processor.h | 8 --------
 arch/x86/kernel/cpu/common.c     | 4 ----
 arch/x86/kernel/vmlinux.lds.S    | 1 -
 arch/x86/tools/relocs.c          | 1 -
 4 files changed, 14 deletions(-)
arch/x86/include/asm/processor.h +0 -8
index b8fee88dac3d..b3d153730f63 100644
@@ -421,14 +421,6 @@ struct irq_stack {
 } __aligned(IRQ_STACK_SIZE);
 
 #ifdef CONFIG_X86_64
-struct fixed_percpu_data {
-	char		gs_base[40];
-	unsigned long	reserved;
-};
-
-DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
-DECLARE_INIT_PER_CPU(fixed_percpu_data);
-
 static inline unsigned long cpu_kernelmode_gs_base(int cpu)
 {
 #ifdef CONFIG_SMP
arch/x86/kernel/cpu/common.c +0 -4
index b71178f0ed6c..8b49b1338f76 100644
@@ -2023,10 +2023,6 @@ EXPORT_PER_CPU_SYMBOL(pcpu_hot);
 EXPORT_PER_CPU_SYMBOL(const_pcpu_hot);
 
 #ifdef CONFIG_X86_64
-DEFINE_PER_CPU_FIRST(struct fixed_percpu_data,
-		     fixed_percpu_data) __aligned(PAGE_SIZE) __visible;
-EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_data);
-
 static void wrmsrl_cstar(unsigned long val)
 {
 	/*
arch/x86/kernel/vmlinux.lds.S +0 -1
index 8a598515239a..93c2fa8a7522 100644
@@ -477,7 +477,6 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
  */
 #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x)
 INIT_PER_CPU(gdt_page);
-INIT_PER_CPU(fixed_percpu_data);
 INIT_PER_CPU(irq_stack_backing_store);
 
 #ifdef CONFIG_MITIGATION_UNRET_ENTRY
arch/x86/tools/relocs.c +0 -1
index 9aebc3b18d73..a9f8bc6aaafc 100644
@@ -830,7 +830,6 @@ static void percpu_init(void)
  *	__per_cpu_load
  *
  * The "gold" linker incorrectly associates:
- *	init_per_cpu__fixed_percpu_data
  *	init_per_cpu__gdt_page
  */
 static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)
Signature · 1 line
2.47.1
merge bot Applied
#34 lore raw

[tip: x86/asm] x86/percpu/64: Remove fixed_percpu_data

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     b5c4f95351a097a635c1a7fc8d9efa18308491b5
Gitweb:        https://git.kernel.org/tip/b5c4f95351a097a635c1a7fc8d9efa18308491b5
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:41 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:43 +01:00

x86/percpu/64: Remove fixed_percpu_data

Now that the stack protector canary value is a normal percpu variable, fixed_percpu_data is unused and can be removed.

---
 arch/x86/include/asm/processor.h | 8 --------
 arch/x86/kernel/cpu/common.c     | 4 ----
 arch/x86/kernel/vmlinux.lds.S    | 1 -
 arch/x86/tools/relocs.c          | 1 -
 4 files changed, 14 deletions(-)
arch/x86/include/asm/processor.h +0 -8
index b8fee88..b3d1537 100644
@@ -421,14 +421,6 @@ struct irq_stack {
 } __aligned(IRQ_STACK_SIZE);
 
 #ifdef CONFIG_X86_64
-struct fixed_percpu_data {
-	char		gs_base[40];
-	unsigned long	reserved;
-};
-
-DECLARE_PER_CPU_FIRST(struct fixed_percpu_data, fixed_percpu_data) __visible;
-DECLARE_INIT_PER_CPU(fixed_percpu_data);
-
 static inline unsigned long cpu_kernelmode_gs_base(int cpu)
 {
 #ifdef CONFIG_SMP
arch/x86/kernel/cpu/common.c +0 -4
index b71178f..8b49b13 100644
@@ -2023,10 +2023,6 @@ EXPORT_PER_CPU_SYMBOL(pcpu_hot);
 EXPORT_PER_CPU_SYMBOL(const_pcpu_hot);
 
 #ifdef CONFIG_X86_64
-DEFINE_PER_CPU_FIRST(struct fixed_percpu_data,
-		     fixed_percpu_data) __aligned(PAGE_SIZE) __visible;
-EXPORT_PER_CPU_SYMBOL_GPL(fixed_percpu_data);
-
 static void wrmsrl_cstar(unsigned long val)
 {
 	/*
arch/x86/kernel/vmlinux.lds.S +0 -1
index 8a59851..93c2fa8 100644
@@ -477,7 +477,6 @@ PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
  */
 #define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x)
 INIT_PER_CPU(gdt_page);
-INIT_PER_CPU(fixed_percpu_data);
 INIT_PER_CPU(irq_stack_backing_store);
 
 #ifdef CONFIG_MITIGATION_UNRET_ENTRY
arch/x86/tools/relocs.c +0 -1
index 3cb3b30..b5e3695 100644
@@ -830,7 +830,6 @@ static void percpu_init(void)
  *	__per_cpu_load
  *
  * The "gold" linker incorrectly associates:
- *	init_per_cpu__fixed_percpu_data
  *	init_per_cpu__gdt_page
  */
 static int is_percpu_sym(ElfW(Sym) *sym, const char *symname)
#35 lore raw

PATCH v6 10/15 x86/boot/64: Remove inverse relocations

In reply to Brian Gerst

Inverse relocations were needed to offset the effects of relocation for RIP-relative accesses to zero-based percpu data. Now that the percpu section is linked normally as part of the kernel image, they are no longer needed.

---
 arch/x86/boot/compressed/misc.c |  14 +---
 arch/x86/tools/relocs.c         | 130 +-------------------------------
 2 files changed, 2 insertions(+), 142 deletions(-)
arch/x86/boot/compressed/misc.c +1 -13
index 0d37420cad02..1cdcd4aaf395 100644
@@ -235,7 +235,7 @@ static void handle_relocations(void *output, unsigned long output_len,
 
 	/*
 	 * Process relocations: 32 bit relocations first then 64 bit after.
-	 * Three sets of binary relocations are added to the end of the kernel
+	 * Two sets of binary relocations are added to the end of the kernel
 	 * before compression. Each relocation table entry is the kernel
 	 * address of the location which needs to be updated stored as a
 	 * 32-bit value which is sign extended to 64 bits.
@@ -245,8 +245,6 @@ static void handle_relocations(void *output, unsigned long output_len,
 	 * kernel bits...
 	 * 0 - zero terminator for 64 bit relocations
 	 * 64 bit relocation repeated
-	 * 0 - zero terminator for inverse 32 bit relocations
-	 * 32 bit inverse relocation repeated
 	 * 0 - zero terminator for 32 bit relocations
 	 * 32 bit relocation repeated
 	 *
@@ -263,16 +261,6 @@ static void handle_relocations(void *output, unsigned long output_len,
 		*(uint32_t *)ptr += delta;
 	}
 #ifdef CONFIG_X86_64
-	while (*--reloc) {
-		long extended = *reloc;
-		extended += map;
-
-		ptr = (unsigned long)extended;
-		if (ptr < min_addr || ptr > max_addr)
-			error("inverse 32-bit relocation outside of kernel!\n");
-
-		*(int32_t *)ptr -= delta;
-	}
 	for (reloc--; *reloc; reloc--) {
 		long extended = *reloc;
 		extended += map;
arch/x86/tools/relocs.c +1 -129

200 lines not on this page · raw message

Signature · 1 line
2.47.1
merge bot Applied
#36 lore raw

[tip: x86/asm] x86/boot/64: Remove inverse relocations

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     a8327be7b2aa067ff2b11551732d5bd8b49ef7d1
Gitweb:        https://git.kernel.org/tip/a8327be7b2aa067ff2b11551732d5bd8b49ef7d1
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:42 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:47 +01:00

x86/boot/64: Remove inverse relocations

Inverse relocations were needed to offset the effects of relocation for RIP-relative accesses to zero-based percpu data. Now that the percpu section is linked normally as part of the kernel image, they are no longer needed.

---
 arch/x86/boot/compressed/misc.c |  14 +---
 arch/x86/tools/relocs.c         | 130 +-------------------------------
 2 files changed, 2 insertions(+), 142 deletions(-)
arch/x86/boot/compressed/misc.c +1 -13
index 0d37420..1cdcd4a 100644
@@ -235,7 +235,7 @@ static void handle_relocations(void *output, unsigned long output_len,
 
 	/*
 	 * Process relocations: 32 bit relocations first then 64 bit after.
-	 * Three sets of binary relocations are added to the end of the kernel
+	 * Two sets of binary relocations are added to the end of the kernel
 	 * before compression. Each relocation table entry is the kernel
 	 * address of the location which needs to be updated stored as a
 	 * 32-bit value which is sign extended to 64 bits.
@@ -245,8 +245,6 @@ static void handle_relocations(void *output, unsigned long output_len,
 	 * kernel bits...
 	 * 0 - zero terminator for 64 bit relocations
 	 * 64 bit relocation repeated
-	 * 0 - zero terminator for inverse 32 bit relocations
-	 * 32 bit inverse relocation repeated
 	 * 0 - zero terminator for 32 bit relocations
 	 * 32 bit relocation repeated
 	 *
@@ -263,16 +261,6 @@ static void handle_relocations(void *output, unsigned long output_len,
 		*(uint32_t *)ptr += delta;
 	}
 #ifdef CONFIG_X86_64
-	while (*--reloc) {
-		long extended = *reloc;
-		extended += map;
-
-		ptr = (unsigned long)extended;
-		if (ptr < min_addr || ptr > max_addr)
-			error("inverse 32-bit relocation outside of kernel!\n");
-
-		*(int32_t *)ptr -= delta;
-	}
 	for (reloc--; *reloc; reloc--) {
 		long extended = *reloc;
 		extended += map;
arch/x86/tools/relocs.c +1 -129

200 lines not on this page · raw message

#37 lore raw

PATCH v6 11/15 x86/percpu/64: Remove INIT_PER_CPU macros

In reply to Brian Gerst

Now that the load and link addresses of percpu variables are the same, these macros are no longer necessary.

---
 arch/x86/include/asm/desc.h   |  1 -
 arch/x86/include/asm/percpu.h | 22 ----------------------
 arch/x86/kernel/head64.c      |  2 +-
 arch/x86/kernel/irq_64.c      |  1 -
 arch/x86/kernel/vmlinux.lds.S |  7 -------
 arch/x86/tools/relocs.c       |  1 -
 6 files changed, 1 insertion(+), 33 deletions(-)
arch/x86/include/asm/desc.h +0 -1
index 62dc9f59ea76..ec95fe44fa3a 100644
@@ -46,7 +46,6 @@ struct gdt_page {
 } __attribute__((aligned(PAGE_SIZE)));
 
 DECLARE_PER_CPU_PAGE_ALIGNED(struct gdt_page, gdt_page);
-DECLARE_INIT_PER_CPU(gdt_page);
 
 /* Provide the original GDT */
 static inline struct desc_struct *get_cpu_gdt_rw(unsigned int cpu)
arch/x86/include/asm/percpu.h +0 -22
index e525cd85f999..1a76eb87c5d8 100644
@@ -20,12 +20,6 @@
 
 #define PER_CPU_VAR(var)	__percpu(var)__percpu_rel
 
-#ifdef CONFIG_X86_64_SMP
-# define INIT_PER_CPU_VAR(var)  init_per_cpu__##var
-#else
-# define INIT_PER_CPU_VAR(var)  var
-#endif
-
 #else /* !__ASSEMBLY__: */
 
 #include <linux/build_bug.h>
@@ -97,22 +91,6 @@
 #define __percpu_arg(x)		__percpu_prefix "%" #x
 #define __force_percpu_arg(x)	__force_percpu_prefix "%" #x
 
-/*
- * Initialized pointers to per-CPU variables needed for the boot
- * processor need to use these macros to get the proper address
- * offset from __per_cpu_load on SMP.
- *
- * There also must be an entry in vmlinux_64.lds.S
- */
-#define DECLARE_INIT_PER_CPU(var) \
-       extern typeof(var) init_per_cpu_var(var)
-
-#ifdef CONFIG_X86_64_SMP
-# define init_per_cpu_var(var)  init_per_cpu__##var
-#else
-# define init_per_cpu_var(var)  var
-#endif
-
 /*
  * For arch-specific code, we can use direct single-insn ops (they
  * don't give an lvalue though).
arch/x86/kernel/head64.c +1 -1
index 22c9ba305ac1..05f8b8acf784 100644
@@ -567,7 +567,7 @@ void early_setup_idt(void)
  */
 void __head startup_64_setup_gdt_idt(void)
 {
-	struct desc_struct *gdt = (void *)(__force unsigned long)init_per_cpu_var(gdt_page.gdt);
+	struct desc_struct *gdt = (void *)(__force unsigned long)gdt_page.gdt;
 	void *handler = NULL;
 
 	struct desc_ptr startup_gdt_descr = {
arch/x86/kernel/irq_64.c +0 -1
index ade0043ce56e..56bdeecd8ee0 100644
@@ -27,7 +27,6 @@
 #include <asm/apic.h>
 
 DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __visible;
-DECLARE_INIT_PER_CPU(irq_stack_backing_store);
 
 #ifdef CONFIG_VMAP_STACK
 /*
arch/x86/kernel/vmlinux.lds.S +0 -7
index 93c2fa8a7522..1769a7126224 100644
@@ -471,13 +471,6 @@ SECTIONS
 PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
 
 #ifdef CONFIG_X86_64
-/*
- * Per-cpu symbols which need to be offset from __per_cpu_load
- * for the boot processor.
- */
-#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x)
-INIT_PER_CPU(gdt_page);
-INIT_PER_CPU(irq_stack_backing_store);
 
 #ifdef CONFIG_MITIGATION_UNRET_ENTRY
 . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
arch/x86/tools/relocs.c +0 -1
index 9e8d5a62ff50..199ab867da7f 100644
@@ -90,7 +90,6 @@ static const char * const	sym_regex_kernel[S_NSYMTYPES] = {
 	"__initramfs_start|"
 	"(jiffies|jiffies_64)|"
 #if ELF_BITS == 64
-	"init_per_cpu__.*|"
 	"__end_rodata_hpage_align|"
 #endif
 	"_end)$"
Signature · 1 line
2.47.1
merge bot Applied
#38 lore raw

[tip: x86/asm] x86/percpu/64: Remove INIT_PER_CPU macros

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     38a4968b3190f873a8a60e953287278eddf037f1
Gitweb:        https://git.kernel.org/tip/38a4968b3190f873a8a60e953287278eddf037f1
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:43 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:50 +01:00

x86/percpu/64: Remove INIT_PER_CPU macros

Now that the load and link addresses of percpu variables are the same, these macros are no longer necessary.

---
 arch/x86/include/asm/desc.h   |  1 -
 arch/x86/include/asm/percpu.h | 22 ----------------------
 arch/x86/kernel/head64.c      |  2 +-
 arch/x86/kernel/irq_64.c      |  1 -
 arch/x86/kernel/vmlinux.lds.S |  7 -------
 arch/x86/tools/relocs.c       |  1 -
 6 files changed, 1 insertion(+), 33 deletions(-)
arch/x86/include/asm/desc.h +0 -1
index 62dc9f5..ec95fe4 100644
@@ -46,7 +46,6 @@ struct gdt_page {
 } __attribute__((aligned(PAGE_SIZE)));
 
 DECLARE_PER_CPU_PAGE_ALIGNED(struct gdt_page, gdt_page);
-DECLARE_INIT_PER_CPU(gdt_page);
 
 /* Provide the original GDT */
 static inline struct desc_struct *get_cpu_gdt_rw(unsigned int cpu)
arch/x86/include/asm/percpu.h +0 -22
index e525cd8..1a76eb8 100644
@@ -20,12 +20,6 @@
 
 #define PER_CPU_VAR(var)	__percpu(var)__percpu_rel
 
-#ifdef CONFIG_X86_64_SMP
-# define INIT_PER_CPU_VAR(var)  init_per_cpu__##var
-#else
-# define INIT_PER_CPU_VAR(var)  var
-#endif
-
 #else /* !__ASSEMBLY__: */
 
 #include <linux/build_bug.h>
@@ -98,22 +92,6 @@
 #define __force_percpu_arg(x)	__force_percpu_prefix "%" #x
 
 /*
- * Initialized pointers to per-CPU variables needed for the boot
- * processor need to use these macros to get the proper address
- * offset from __per_cpu_load on SMP.
- *
- * There also must be an entry in vmlinux_64.lds.S
- */
-#define DECLARE_INIT_PER_CPU(var) \
-       extern typeof(var) init_per_cpu_var(var)
-
-#ifdef CONFIG_X86_64_SMP
-# define init_per_cpu_var(var)  init_per_cpu__##var
-#else
-# define init_per_cpu_var(var)  var
-#endif
-
-/*
  * For arch-specific code, we can use direct single-insn ops (they
  * don't give an lvalue though).
  */
arch/x86/kernel/head64.c +1 -1
index 22c9ba3..05f8b8a 100644
@@ -567,7 +567,7 @@ void early_setup_idt(void)
  */
 void __head startup_64_setup_gdt_idt(void)
 {
-	struct desc_struct *gdt = (void *)(__force unsigned long)init_per_cpu_var(gdt_page.gdt);
+	struct desc_struct *gdt = (void *)(__force unsigned long)gdt_page.gdt;
 	void *handler = NULL;
 
 	struct desc_ptr startup_gdt_descr = {
arch/x86/kernel/irq_64.c +0 -1
index ade0043..56bdeec 100644
@@ -27,7 +27,6 @@
 #include <asm/apic.h>
 
 DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __visible;
-DECLARE_INIT_PER_CPU(irq_stack_backing_store);
 
 #ifdef CONFIG_VMAP_STACK
 /*
arch/x86/kernel/vmlinux.lds.S +0 -7
index 93c2fa8..1769a71 100644
@@ -471,13 +471,6 @@ SECTIONS
 PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
 
 #ifdef CONFIG_X86_64
-/*
- * Per-cpu symbols which need to be offset from __per_cpu_load
- * for the boot processor.
- */
-#define INIT_PER_CPU(x) init_per_cpu__##x = ABSOLUTE(x)
-INIT_PER_CPU(gdt_page);
-INIT_PER_CPU(irq_stack_backing_store);
 
 #ifdef CONFIG_MITIGATION_UNRET_ENTRY
 . = ASSERT((retbleed_return_thunk & 0x3f) == 0, "retbleed_return_thunk not cacheline-aligned");
arch/x86/tools/relocs.c +0 -1
index ae69626..5778bc4 100644
@@ -90,7 +90,6 @@ static const char * const	sym_regex_kernel[S_NSYMTYPES] = {
 	"__initramfs_start|"
 	"(jiffies|jiffies_64)|"
 #if ELF_BITS == 64
-	"init_per_cpu__.*|"
 	"__end_rodata_hpage_align|"
 #endif
 	"_end)$"
#39 lore raw

PATCH v6 12/15 percpu: Remove PER_CPU_FIRST_SECTION

In reply to Brian Gerst

x86-64 was the last user.

---
 include/asm-generic/vmlinux.lds.h |  1 -
 include/linux/percpu-defs.h       | 12 ------------
 2 files changed, 13 deletions(-)
include/asm-generic/vmlinux.lds.h +0 -1
index 54504013c749..4a69658739de 100644
@@ -1061,7 +1061,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
  */
 #define PERCPU_INPUT(cacheline)						\
 	__per_cpu_start = .;						\
-	*(.data..percpu..first)						\
 	. = ALIGN(PAGE_SIZE);						\
 	*(.data..percpu..page_aligned)					\
 	. = ALIGN(cacheline);						\
include/linux/percpu-defs.h +0 -12
index 5b520fe86b60..40d34e032d5b 100644
@@ -26,13 +26,11 @@
 #define PER_CPU_SHARED_ALIGNED_SECTION "..shared_aligned"
 #define PER_CPU_ALIGNED_SECTION "..shared_aligned"
 #endif
-#define PER_CPU_FIRST_SECTION "..first"
 
 #else
 
 #define PER_CPU_SHARED_ALIGNED_SECTION ""
 #define PER_CPU_ALIGNED_SECTION "..shared_aligned"
-#define PER_CPU_FIRST_SECTION ""
 
 #endif
 
@@ -114,16 +112,6 @@
 #define DEFINE_PER_CPU(type, name)					\
 	DEFINE_PER_CPU_SECTION(type, name, "")
 
-/*
- * Declaration/definition used for per-CPU variables that must come first in
- * the set of variables.
- */
-#define DECLARE_PER_CPU_FIRST(type, name)				\
-	DECLARE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION)
-
-#define DEFINE_PER_CPU_FIRST(type, name)				\
-	DEFINE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION)
-
 /*
  * Declaration/definition used for per-CPU variables that must be cacheline
  * aligned under SMP conditions so that, whilst a particular instance of the
Signature · 1 line
2.47.1
merge bot Applied
#40 lore raw

[tip: x86/asm] percpu: Remove PER_CPU_FIRST_SECTION

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     95b0916118106054e1f3d5d7f8628ef3dc0b3c02
Gitweb:        https://git.kernel.org/tip/95b0916118106054e1f3d5d7f8628ef3dc0b3c02
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:44 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:53 +01:00

percpu: Remove PER_CPU_FIRST_SECTION

x86-64 was the last user.

---
 include/asm-generic/vmlinux.lds.h |  1 -
 include/linux/percpu-defs.h       | 12 ------------
 2 files changed, 13 deletions(-)
include/asm-generic/vmlinux.lds.h +0 -1
index 02a4adb..a3c77a1 100644
@@ -1062,7 +1062,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
  */
 #define PERCPU_INPUT(cacheline)						\
 	__per_cpu_start = .;						\
-	*(.data..percpu..first)						\
 	. = ALIGN(PAGE_SIZE);						\
 	*(.data..percpu..page_aligned)					\
 	. = ALIGN(cacheline);						\
include/linux/percpu-defs.h +0 -12
index 5b520fe..40d34e0 100644
@@ -26,13 +26,11 @@
 #define PER_CPU_SHARED_ALIGNED_SECTION "..shared_aligned"
 #define PER_CPU_ALIGNED_SECTION "..shared_aligned"
 #endif
-#define PER_CPU_FIRST_SECTION "..first"
 
 #else
 
 #define PER_CPU_SHARED_ALIGNED_SECTION ""
 #define PER_CPU_ALIGNED_SECTION "..shared_aligned"
-#define PER_CPU_FIRST_SECTION ""
 
 #endif
 
@@ -115,16 +113,6 @@
 	DEFINE_PER_CPU_SECTION(type, name, "")
 
 /*
- * Declaration/definition used for per-CPU variables that must come first in
- * the set of variables.
- */
-#define DECLARE_PER_CPU_FIRST(type, name)				\
-	DECLARE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION)
-
-#define DEFINE_PER_CPU_FIRST(type, name)				\
-	DEFINE_PER_CPU_SECTION(type, name, PER_CPU_FIRST_SECTION)
-
-/*
  * Declaration/definition used for per-CPU variables that must be cacheline
  * aligned under SMP conditions so that, whilst a particular instance of the
  * data corresponds to a particular CPU, inefficiencies due to direct access by
#41 lore raw

PATCH v6 13/15 percpu: Remove PERCPU_VADDR()

In reply to Brian Gerst

x86-64 was the last user.

---
 include/asm-generic/vmlinux.lds.h | 36 +------------------------------
 1 file changed, 1 insertion(+), 35 deletions(-)
include/asm-generic/vmlinux.lds.h +1 -35
index 4a69658739de..b2cbc1628118 100644
@@ -1072,47 +1072,13 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
 	__per_cpu_end = .;
 
 /**
- * PERCPU_VADDR - define output section for percpu area
+ * PERCPU_SECTION - define output section for percpu area
  * @cacheline: cacheline size
- * @vaddr: explicit base address (optional)
- * @phdr: destination PHDR (optional)
  *
  * Macro which expands to output section for percpu area.
  *
  * @cacheline is used to align subsections to avoid false cacheline
  * sharing between subsections for different purposes.
- *
- * If @vaddr is not blank, it specifies explicit base address and all
- * percpu symbols will be offset from the given address.  If blank,
- * @vaddr always equals @laddr + LOAD_OFFSET.
- *
- * @phdr defines the output PHDR to use if not blank.  Be warned that
- * output PHDR is sticky.  If @phdr is specified, the next output
- * section in the linker script will go there too.  @phdr should have
- * a leading colon.
- *
- * Note that this macros defines __per_cpu_load as an absolute symbol.
- * If there is no need to put the percpu section at a predetermined
- * address, use PERCPU_SECTION.
- */
-#define PERCPU_VADDR(cacheline, vaddr, phdr)				\
-	__per_cpu_load = .;						\
-	.data..percpu vaddr : AT(__per_cpu_load - LOAD_OFFSET) {	\
-		PERCPU_INPUT(cacheline)					\
-	} phdr								\
-	. = __per_cpu_load + SIZEOF(.data..percpu);
-
-/**
- * PERCPU_SECTION - define output section for percpu area, simple version
- * @cacheline: cacheline size
- *
- * Align to PAGE_SIZE and outputs output section for percpu area.  This
- * macro doesn't manipulate @vaddr or @phdr and __per_cpu_load and
- * __per_cpu_start will be identical.
- *
- * This macro is equivalent to ALIGN(PAGE_SIZE); PERCPU_VADDR(@cacheline,,)
- * except that __per_cpu_load is defined as a relative symbol against
- * .data..percpu which is required for relocatable x86_32 configuration.
  */
 #define PERCPU_SECTION(cacheline)					\
 	. = ALIGN(PAGE_SIZE);						\
Signature · 1 line
2.47.1
merge bot Applied
#42 lore raw

[tip: x86/asm] percpu: Remove PERCPU_VADDR()

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     e23cff6861781ac4e15de6c7bf2d2a0b79cb52ef
Gitweb:        https://git.kernel.org/tip/e23cff6861781ac4e15de6c7bf2d2a0b79cb52ef
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:45 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:15:57 +01:00

percpu: Remove PERCPU_VADDR()

x86-64 was the last user.

---
 include/asm-generic/vmlinux.lds.h | 36 +------------------------------
 1 file changed, 1 insertion(+), 35 deletions(-)
include/asm-generic/vmlinux.lds.h +1 -35
index a3c77a1..e25a8ae 100644
@@ -1073,47 +1073,13 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
 	__per_cpu_end = .;
 
 /**
- * PERCPU_VADDR - define output section for percpu area
+ * PERCPU_SECTION - define output section for percpu area
  * @cacheline: cacheline size
- * @vaddr: explicit base address (optional)
- * @phdr: destination PHDR (optional)
  *
  * Macro which expands to output section for percpu area.
  *
  * @cacheline is used to align subsections to avoid false cacheline
  * sharing between subsections for different purposes.
- *
- * If @vaddr is not blank, it specifies explicit base address and all
- * percpu symbols will be offset from the given address.  If blank,
- * @vaddr always equals @laddr + LOAD_OFFSET.
- *
- * @phdr defines the output PHDR to use if not blank.  Be warned that
- * output PHDR is sticky.  If @phdr is specified, the next output
- * section in the linker script will go there too.  @phdr should have
- * a leading colon.
- *
- * Note that this macros defines __per_cpu_load as an absolute symbol.
- * If there is no need to put the percpu section at a predetermined
- * address, use PERCPU_SECTION.
- */
-#define PERCPU_VADDR(cacheline, vaddr, phdr)				\
-	__per_cpu_load = .;						\
-	.data..percpu vaddr : AT(__per_cpu_load - LOAD_OFFSET) {	\
-		PERCPU_INPUT(cacheline)					\
-	} phdr								\
-	. = __per_cpu_load + SIZEOF(.data..percpu);
-
-/**
- * PERCPU_SECTION - define output section for percpu area, simple version
- * @cacheline: cacheline size
- *
- * Align to PAGE_SIZE and outputs output section for percpu area.  This
- * macro doesn't manipulate @vaddr or @phdr and __per_cpu_load and
- * __per_cpu_start will be identical.
- *
- * This macro is equivalent to ALIGN(PAGE_SIZE); PERCPU_VADDR(@cacheline,,)
- * except that __per_cpu_load is defined as a relative symbol against
- * .data..percpu which is required for relocatable x86_32 configuration.
  */
 #define PERCPU_SECTION(cacheline)					\
 	. = ALIGN(PAGE_SIZE);						\
#43 lore raw

PATCH v6 14/15 percpu: Remove __per_cpu_load

In reply to Brian Gerst

__per_cpu_load is now always equal to __per_cpu_start.

---
 include/asm-generic/sections.h    | 2 +-
 include/asm-generic/vmlinux.lds.h | 1 -
 mm/percpu.c                       | 4 ++--
 3 files changed, 3 insertions(+), 4 deletions(-)
include/asm-generic/sections.h +1 -1
index c768de6f19a9..0755bc39b0d8 100644
@@ -39,7 +39,7 @@ extern char __init_begin[], __init_end[];
 extern char _sinittext[], _einittext[];
 extern char __start_ro_after_init[], __end_ro_after_init[];
 extern char _end[];
-extern char __per_cpu_load[], __per_cpu_start[], __per_cpu_end[];
+extern char __per_cpu_start[], __per_cpu_end[];
 extern char __kprobes_text_start[], __kprobes_text_end[];
 extern char __entry_text_start[], __entry_text_end[];
 extern char __start_rodata[], __end_rodata[];
include/asm-generic/vmlinux.lds.h +0 -1
index b2cbc1628118..c616749ebcf7 100644
@@ -1083,7 +1083,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
 #define PERCPU_SECTION(cacheline)					\
 	. = ALIGN(PAGE_SIZE);						\
 	.data..percpu	: AT(ADDR(.data..percpu) - LOAD_OFFSET) {	\
-		__per_cpu_load = .;					\
 		PERCPU_INPUT(cacheline)					\
 	}
 
mm/percpu.c +2 -2
index d8dd31a2e407..8c9f2164cea9 100644
@@ -3099,7 +3099,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size,
 				continue;
 			}
 			/* copy and return the unused part */
-			memcpy(ptr, __per_cpu_load, ai->static_size);
+			memcpy(ptr, __per_cpu_start, ai->static_size);
 			pcpu_fc_free(ptr + size_sum, ai->unit_size - size_sum);
 		}
 	}
@@ -3282,7 +3282,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t
 		flush_cache_vmap_early(unit_addr, unit_addr + ai->unit_size);
 
 		/* copy static data */
-		memcpy((void *)unit_addr, __per_cpu_load, ai->static_size);
+		memcpy((void *)unit_addr, __per_cpu_start, ai->static_size);
 	}
 
 	/* we're ready, commit */
Signature · 1 line
2.47.1
merge bot Applied
#44 lore raw

[tip: x86/asm] percpu: Remove __per_cpu_load

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     4b00c1160a13d8bf7297ebf49ec07a84e1f41132
Gitweb:        https://git.kernel.org/tip/4b00c1160a13d8bf7297ebf49ec07a84e1f41132
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:46 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:16:00 +01:00

percpu: Remove __per_cpu_load

__per_cpu_load is now always equal to __per_cpu_start.

---
 include/asm-generic/sections.h    | 2 +-
 include/asm-generic/vmlinux.lds.h | 1 -
 mm/percpu.c                       | 4 ++--
 3 files changed, 3 insertions(+), 4 deletions(-)
include/asm-generic/sections.h +1 -1
index c768de6..0755bc3 100644
@@ -39,7 +39,7 @@ extern char __init_begin[], __init_end[];
 extern char _sinittext[], _einittext[];
 extern char __start_ro_after_init[], __end_ro_after_init[];
 extern char _end[];
-extern char __per_cpu_load[], __per_cpu_start[], __per_cpu_end[];
+extern char __per_cpu_start[], __per_cpu_end[];
 extern char __kprobes_text_start[], __kprobes_text_end[];
 extern char __entry_text_start[], __entry_text_end[];
 extern char __start_rodata[], __end_rodata[];
include/asm-generic/vmlinux.lds.h +0 -1
index e25a8ae..92fc06f 100644
@@ -1084,7 +1084,6 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
 #define PERCPU_SECTION(cacheline)					\
 	. = ALIGN(PAGE_SIZE);						\
 	.data..percpu	: AT(ADDR(.data..percpu) - LOAD_OFFSET) {	\
-		__per_cpu_load = .;					\
 		PERCPU_INPUT(cacheline)					\
 	}
 
mm/percpu.c +2 -2
index ac61e3f..7b58353 100644
@@ -3071,7 +3071,7 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size,
 				continue;
 			}
 			/* copy and return the unused part */
-			memcpy(ptr, __per_cpu_load, ai->static_size);
+			memcpy(ptr, __per_cpu_start, ai->static_size);
 			pcpu_fc_free(ptr + size_sum, ai->unit_size - size_sum);
 		}
 	}
@@ -3240,7 +3240,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t 
 		flush_cache_vmap_early(unit_addr, unit_addr + ai->unit_size);
 
 		/* copy static data */
-		memcpy((void *)unit_addr, __per_cpu_load, ai->static_size);
+		memcpy((void *)unit_addr, __per_cpu_start, ai->static_size);
 	}
 
 	/* we're ready, commit */
#45 lore raw

PATCH v6 15/15 kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU

In reply to Brian Gerst

x86-64 was the only user.

---
 init/Kconfig            |  5 ---
 kernel/kallsyms.c       | 12 ++-----
 scripts/kallsyms.c      | 72 +++++++----------------------------------
 scripts/link-vmlinux.sh |  4 ---
 4 files changed, 14 insertions(+), 79 deletions(-)
init/Kconfig +0 -5
index 01d36a84cf66..c2a92fb00a98 100644
@@ -1870,11 +1870,6 @@ config KALLSYMS_ALL
 
 	  Say N unless you really need all symbols, or kernel live patching.
 
-config KALLSYMS_ABSOLUTE_PERCPU
-	bool
-	depends on KALLSYMS
-	default n
-
 # end of the "standard kernel features (expert users)" menu
 
 config ARCH_HAS_MEMBARRIER_CALLBACKS
kernel/kallsyms.c +2 -10
index a9a0ca605d4a..4198f30aac3c 100644
@@ -148,16 +148,8 @@ static unsigned int get_symbol_offset(unsigned long pos)
 
 unsigned long kallsyms_sym_address(int idx)
 {
-	/* values are unsigned offsets if --absolute-percpu is not in effect */
-	if (!IS_ENABLED(CONFIG_KALLSYMS_ABSOLUTE_PERCPU))
-		return kallsyms_relative_base + (u32)kallsyms_offsets[idx];
-
-	/* ...otherwise, positive offsets are absolute values */
-	if (kallsyms_offsets[idx] >= 0)
-		return kallsyms_offsets[idx];
-
-	/* ...and negative offsets are relative to kallsyms_relative_base - 1 */
-	return kallsyms_relative_base - 1 - kallsyms_offsets[idx];
+	/* values are unsigned offsets */
+	return kallsyms_relative_base + (u32)kallsyms_offsets[idx];
 }
 
 static unsigned int get_symbol_seq(int index)
scripts/kallsyms.c +12 -60

145 lines not on this page · raw message

scripts/link-vmlinux.sh +0 -4
index d853ddb3b28c..8e23066323eb 100755
@@ -144,10 +144,6 @@ kallsyms()
 		kallsymopt="${kallsymopt} --all-symbols"
 	fi
 
-	if is_enabled CONFIG_KALLSYMS_ABSOLUTE_PERCPU; then
-		kallsymopt="${kallsymopt} --absolute-percpu"
-	fi
-
 	info KSYMS "${2}.S"
 	scripts/kallsyms ${kallsymopt} "${1}" > "${2}.S"
 
Signature · 1 line
2.47.1
merge bot Applied
#46 lore raw

[tip: x86/asm] kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU

The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     01157ddc58dc2fe428ec17dd5a18cc13f134639f
Gitweb:        https://git.kernel.org/tip/01157ddc58dc2fe428ec17dd5a18cc13f134639f
Author:        Brian Gerst <[email protected]>
AuthorDate:    Thu, 23 Jan 2025 14:07:47 -05:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 18 Feb 2025 10:16:04 +01:00

kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU

x86-64 was the only user.

---
 init/Kconfig            |  5 +---
 kernel/kallsyms.c       | 12 +------
 scripts/kallsyms.c      | 72 ++++++----------------------------------
 scripts/link-vmlinux.sh |  4 +--
 4 files changed, 14 insertions(+), 79 deletions(-)
init/Kconfig +0 -5
index b5d9c0f..a0ea04c 100644
@@ -1869,11 +1869,6 @@ config KALLSYMS_ALL
 
 	  Say N unless you really need all symbols, or kernel live patching.
 
-config KALLSYMS_ABSOLUTE_PERCPU
-	bool
-	depends on KALLSYMS
-	default n
-
 # end of the "standard kernel features (expert users)" menu
 
 config ARCH_HAS_MEMBARRIER_CALLBACKS
kernel/kallsyms.c +2 -10
index a9a0ca6..4198f30 100644
@@ -148,16 +148,8 @@ static unsigned int get_symbol_offset(unsigned long pos)
 
 unsigned long kallsyms_sym_address(int idx)
 {
-	/* values are unsigned offsets if --absolute-percpu is not in effect */
-	if (!IS_ENABLED(CONFIG_KALLSYMS_ABSOLUTE_PERCPU))
-		return kallsyms_relative_base + (u32)kallsyms_offsets[idx];
-
-	/* ...otherwise, positive offsets are absolute values */
-	if (kallsyms_offsets[idx] >= 0)
-		return kallsyms_offsets[idx];
-
-	/* ...and negative offsets are relative to kallsyms_relative_base - 1 */
-	return kallsyms_relative_base - 1 - kallsyms_offsets[idx];
+	/* values are unsigned offsets */
+	return kallsyms_relative_base + (u32)kallsyms_offsets[idx];
 }
 
 static unsigned int get_symbol_seq(int index)
scripts/kallsyms.c +12 -60

145 lines not on this page · raw message

scripts/link-vmlinux.sh +0 -4
index 56a077d..67e6633 100755
@@ -144,10 +144,6 @@ kallsyms()
 		kallsymopt="${kallsymopt} --all-symbols"
 	fi
 
-	if is_enabled CONFIG_KALLSYMS_ABSOLUTE_PERCPU; then
-		kallsymopt="${kallsymopt} --absolute-percpu"
-	fi
-
 	info KSYMS "${2}.S"
 	scripts/kallsyms ${kallsymopt} "${1}" > "${2}.S"
 
#47 lore raw
In reply to Brian Gerst
Quoted 37 lines · Brian Gerst

Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.

GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.

v6:
- Rebased to current tip tree
- Dropped patches already applied
- Fixed typos in commit messages
- Added Reviewed-by tags

Ard Biesheuvel (1):
x86/module: Deal with GOT based stack cookie load on Clang < 17

Brian Gerst (14):
x86: Raise minimum GCC version to 8.1 x86/stackprotector: Remove stack protector test scripts x86/boot: Disable stack protector for early boot code x86/pvh: Use fixed_percpu_data for early boot GSBASE x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations x86/stackprotector/64: Convert to normal percpu variable x86/percpu/64: Use relative percpu offsets x86/percpu/64: Remove fixed_percpu_data x86/boot/64: Remove inverse relocations x86/percpu/64: Remove INIT_PER_CPU macros
percpu: Remove PER_CPU_FIRST_SECTION
percpu: Remove PERCPU_VADDR()
percpu: Remove __per_cpu_load
kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU

Quoted 3 lines
 33 files changed, 100 insertions(+), 475 deletions(-)
 delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh
 delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

Thanks,

Ingo

#48 lore raw
Quoted 45 lines · Ingo Molnar
Quoted 37 lines · Brian Gerst

Currently, x86-64 uses an unusual percpu layout, where the percpu section is linked at absolute address 0. The reason behind this is that older GCC versions placed the stack protector (if enabled) at a fixed offset from the GS segment base. Since the GS segement is also used for percpu variables, this forced the current layout.

GCC since version 8.1 supports a configurable location for the stack protector value, which allows removal of the restriction on how the percpu section is linked. This allows the percpu section to be linked normally, like other architectures. In turn, this allows removal of code that was needed to support the zero-based percpu section.

v6:
- Rebased to current tip tree
- Dropped patches already applied
- Fixed typos in commit messages
- Added Reviewed-by tags

Ard Biesheuvel (1):
x86/module: Deal with GOT based stack cookie load on Clang < 17

Brian Gerst (14):
x86: Raise minimum GCC version to 8.1 x86/stackprotector: Remove stack protector test scripts x86/boot: Disable stack protector for early boot code x86/pvh: Use fixed_percpu_data for early boot GSBASE x86/relocs: Handle R_X86_64_REX_GOTPCRELX relocations x86/stackprotector/64: Convert to normal percpu variable x86/percpu/64: Use relative percpu offsets x86/percpu/64: Remove fixed_percpu_data x86/boot/64: Remove inverse relocations x86/percpu/64: Remove INIT_PER_CPU macros
percpu: Remove PER_CPU_FIRST_SECTION
percpu: Remove PERCPU_VADDR()
percpu: Remove __per_cpu_load
kallsyms: Remove KALLSYMS_ABSOLUTE_PERCPU

Quoted 3 lines
 33 files changed, 100 insertions(+), 475 deletions(-)
 delete mode 100755 scripts/gcc-x86_32-has-stack-protector.sh
 delete mode 100755 scripts/gcc-x86_64-has-stack-protector.sh
Quoted 6 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Uros.

#49 lore raw
Quoted 10 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

Thanks,

Ingo

#50 lore raw
Quoted 12 lines · Ingo Molnar
Quoted 8 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

Quoted 6 lines
Quoted 2 lines

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Brian Gerst

#51 lore raw
Quoted 17 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Something like the attached patch?

It boots and runs without problems.

However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:

RELOCS arch/x86/boot/compressed/vmlinux.relocs
WARNING: Absolute relocations present
Offset Info Type Sym.Value Sym.Name
ffffffff81200826 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201493 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201714 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201d66 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
...
ffffffff834e2a13 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff834e2a6a 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard

RSTRIP vmlinux

which I don't understand. Looking at the first one:

ffffffff8120081d <force_ibs_eilvt_setup.cold>:

ffffffff8120081d:    48 8b 44 24 08           mov    0x8(%rsp),%rax
ffffffff81200822:    65 48 2b 05 f6 ed 30     sub

%gs:0x230edf6(%rip),%rax # ffffffff8350f620
<__ref_stack_chk_guard>
ffffffff81200829: 02

I don't think this is absolute relocation, see (%rip).

The kernel was compiled with gcc-14.2.1, so clang specific issue was not tested.

Uros.

arch/x86/entry/entry_32.S +1 -1
index 20be5758c2d2..940efc07f2c1 100644
@@ -691,7 +691,7 @@ SYM_CODE_START(__switch_to_asm)
 
 #ifdef CONFIG_STACKPROTECTOR
 	movl	TASK_stack_canary(%edx), %ebx
-	movl	%ebx, PER_CPU_VAR(__stack_chk_guard)
+	movl	%ebx, PER_CPU_VAR(pcpu_hot + X86_stack_canary)
 #endif
 
 	/*
arch/x86/entry/entry_64.S +1 -1
index 49d3b222fe99..4f4c0cf4963f 100644
@@ -193,7 +193,7 @@ SYM_FUNC_START(__switch_to_asm)
 
 #ifdef CONFIG_STACKPROTECTOR
 	movq	TASK_stack_canary(%rsi), %rbx
-	movq	%rbx, PER_CPU_VAR(__stack_chk_guard)
+	movq	%rbx, PER_CPU_VAR(pcpu_hot + X86_stack_canary)
 #endif
 
 	/*
arch/x86/include/asm/current.h +3 -0
index bf5953883ec3..a4d515cd6a31 100644
@@ -22,6 +22,9 @@ struct pcpu_hot {
 			u64			call_depth;
 #endif
 			unsigned long		top_of_stack;
+#ifdef CONFIG_STACKPROTECTOR
+			unsigned long		stack_canary;
+#endif
 			void			*hardirq_stack_ptr;
 			u16			softirq_pending;
 #ifdef CONFIG_X86_64
arch/x86/include/asm/stackprotector.h +2 -4
index d43fb589fcf6..5e5229ac1c46 100644
@@ -20,8 +20,6 @@
 
 #include <linux/sched.h>
 
-DECLARE_PER_CPU(unsigned long, __stack_chk_guard);
-
 /*
  * Initialize the stackprotector canary value.
  *
@@ -38,12 +36,12 @@ static __always_inline void boot_init_stack_canary(void)
 	unsigned long canary = get_random_canary();
 
 	current->stack_canary = canary;
-	this_cpu_write(__stack_chk_guard, canary);
+	this_cpu_write(pcpu_hot.stack_canary, canary);
 }
 
 static inline void cpu_init_stack_canary(int cpu, struct task_struct *idle)
 {
-	per_cpu(__stack_chk_guard, cpu) = idle->stack_canary;
+	per_cpu(pcpu_hot.stack_canary, cpu) = idle->stack_canary;
 }
 
 #else	/* STACKPROTECTOR */
arch/x86/kernel/asm-offsets.c +3 -0
index a98020bf31bb..59e8b294cbdc 100644
@@ -109,6 +109,9 @@ static void __used common(void)
 	OFFSET(TSS_sp2, tss_struct, x86_tss.sp2);
 	OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack);
 	OFFSET(X86_current_task, pcpu_hot, current_task);
+#ifdef CONFIG_STACKPROTECTOR
+	OFFSET(X86_stack_canary, pcpu_hot, stack_canary);
+#endif
 #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING
 	OFFSET(X86_call_depth, pcpu_hot, call_depth);
 #endif
arch/x86/kernel/cpu/common.c +0 -8
index 21078907af57..f30d6a9c4abd 100644
@@ -24,7 +24,6 @@
 #include <linux/io.h>
 #include <linux/syscore_ops.h>
 #include <linux/pgtable.h>
-#include <linux/stackprotector.h>
 #include <linux/utsname.h>
 
 #include <asm/alternative.h>
@@ -2087,13 +2086,6 @@ void syscall_init(void)
 }
 #endif /* CONFIG_X86_64 */
 
-#ifdef CONFIG_STACKPROTECTOR
-DEFINE_PER_CPU(unsigned long, __stack_chk_guard);
-#ifndef CONFIG_SMP
-EXPORT_PER_CPU_SYMBOL(__stack_chk_guard);
-#endif
-#endif
-
 /*
  * Clear all 6 debug registers:
  */
arch/x86/kernel/vmlinux.lds.S +1 -1
index 1769a7126224..a35e4ebe032e 100644
@@ -468,7 +468,7 @@ SECTIONS
 	   "kernel image bigger than KERNEL_IMAGE_SIZE");
 
 /* needed for Clang - see arch/x86/entry/entry.S */
-PROVIDE(__ref_stack_chk_guard = __stack_chk_guard);
+PROVIDE(__ref_stack_chk_guard = pcpu_hot + X86_stack_canary);
 
 #ifdef CONFIG_X86_64
 
#52 lore raw

Hi Uros,

Quoted 19 lines · Uros Bizjak
Quoted 17 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Something like the attached patch?

Interesting take. I'd have tried to put the canary at offset 0x0, and simply use pcpu_hot as the guard symbol.

Quoted 26 lines

It boots and runs without problems.

However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:

RELOCS arch/x86/boot/compressed/vmlinux.relocs
WARNING: Absolute relocations present
Offset Info Type Sym.Value Sym.Name
ffffffff81200826 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201493 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201714 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201d66 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
...
ffffffff834e2a13 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff834e2a6a 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard

RSTRIP vmlinux

which I don't understand. Looking at the first one:

Quoted 9 lines

ffffffff8120081d <force_ibs_eilvt_setup.cold>:

ffffffff8120081d:    48 8b 44 24 08           mov    0x8(%rsp),%rax
ffffffff81200822:    65 48 2b 05 f6 ed 30     sub

%gs:0x230edf6(%rip),%rax # ffffffff8350f620
<__ref_stack_chk_guard>
ffffffff81200829: 02

I don't think this is absolute relocation, see (%rip).

The warning is about the type of __ref_stack_chk_guard, not about the type of the relocation.

$ nm vmlinux |grep \\s__ref_sta
ffffffff8350c620 A __ref_stack_chk_guard

Without your patch:

$ nm vmlinux |grep \\s__ref_sta
ffffffff834fba10 D __ref_stack_chk_guard
#53 lore raw
Quoted 70 lines · Ard Biesheuvel

Hi Uros,

Quoted 19 lines · Uros Bizjak
Quoted 17 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Something like the attached patch?

Interesting take. I'd have tried to put the canary at offset 0x0, and simply use pcpu_hot as the guard symbol.

Quoted 33 lines

It boots and runs without problems.

However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:

RELOCS arch/x86/boot/compressed/vmlinux.relocs
WARNING: Absolute relocations present
Offset Info Type Sym.Value Sym.Name
ffffffff81200826 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201493 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201714 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff81201d66 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
...
ffffffff834e2a13 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard
ffffffff834e2a6a 0003259e00000002 R_X86_64_PC32 ffffffff8350f620 __ref_stack_chk_guard

RSTRIP vmlinux

which I don't understand. Looking at the first one:

ffffffff8120081d <force_ibs_eilvt_setup.cold>:

ffffffff8120081d:    48 8b 44 24 08           mov    0x8(%rsp),%rax
ffffffff81200822:    65 48 2b 05 f6 ed 30     sub

%gs:0x230edf6(%rip),%rax # ffffffff8350f620
<__ref_stack_chk_guard>
ffffffff81200829: 02

Quoted 5 lines
Quoted 2 lines

I don't think this is absolute relocation, see (%rip).

The warning is about the type of __ref_stack_chk_guard, not about the type of the relocation.

Thanks, I got distracted by the text of the warning that mentions relocation.

Quoted 7 lines
$ nm vmlinux |grep \\s__ref_sta
ffffffff8350c620 A __ref_stack_chk_guard

Without your patch:

$ nm vmlinux |grep \\s__ref_sta
ffffffff834fba10 D __ref_stack_chk_guard

Is this a problem in our specific case? While the symbol is absolute, the relocation is still relative, so IMO it should be OK even with your ongoing rip-relative efforts in mind.

We can list the symbol in arch/x86/tools/relocs.c to quiet the warning, but I would need some help with auditing the symbol itself.

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

for good measure.

Uros.

#54 lore raw
Quoted 33 lines · Uros Bizjak
Quoted 31 lines · Ard Biesheuvel

Hi Uros,

Quoted 10 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · Uros Bizjak

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Interesting take. I'd have tried to put the canary at offset 0x0, and simply use pcpu_hot as the guard symbol.

Quoted 5 lines

It boots and runs without problems.

However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:

...

Quoted 9 lines
Quoted 3 lines

The warning is about the type of __ref_stack_chk_guard, not about the type of the relocation.

Thanks, I got distracted by the text of the warning that mentions relocation.

Quoted 3 lines
$ nm vmlinux |grep \\s__ref_sta
ffffffff8350c620 A __ref_stack_chk_guard
Quoted 6 lines
Quoted 4 lines

Without your patch:

$ nm vmlinux |grep \\s__ref_sta
ffffffff834fba10 D __ref_stack_chk_guard

Is this a problem in our specific case?

I don't think so - the whole notion of absolute ELF symbols is rather flaky IME, so I don't think we should be pedantic here.

Quoted 9 lines

We can list the symbol in arch/x86/tools/relocs.c to quiet the warning, but I would need some help with auditing the symbol itself.

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

for good measure.

I think this would be the most straight-forward if there are no other locality concerns this might interfere with.

#55 lore raw
Quoted 61 lines · Ard Biesheuvel
Quoted 33 lines · Uros Bizjak
Quoted 31 lines · Ard Biesheuvel

Hi Uros,

Quoted 10 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · Uros Bizjak

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Interesting take. I'd have tried to put the canary at offset 0x0, and simply use pcpu_hot as the guard symbol.

Quoted 5 lines

It boots and runs without problems.

However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:

...

Quoted 9 lines
Quoted 3 lines

The warning is about the type of __ref_stack_chk_guard, not about the type of the relocation.

Thanks, I got distracted by the text of the warning that mentions relocation.

Quoted 3 lines
$ nm vmlinux |grep \\s__ref_sta
ffffffff8350c620 A __ref_stack_chk_guard
Quoted 6 lines
Quoted 4 lines

Without your patch:

$ nm vmlinux |grep \\s__ref_sta
ffffffff834fba10 D __ref_stack_chk_guard

Is this a problem in our specific case?

I don't think so - the whole notion of absolute ELF symbols is rather flaky IME, so I don't think we should be pedantic here.

Quoted 6 lines

We can list the symbol in arch/x86/tools/relocs.c to quiet the warning, but I would need some help with auditing the symbol itself.

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

Quoted 6 lines
Quoted 3 lines

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

for good measure.

I think this would be the most straight-forward if there are no other locality concerns this might interfere with.

OK, let me prepare a patch then.

Thanks,
Uros.

#56 lore raw
In reply to Ard Biesheuvel
Quoted 47 lines · Ard Biesheuvel
Quoted 33 lines · Uros Bizjak
Quoted 31 lines · Ard Biesheuvel

Hi Uros,

Quoted 10 lines · Uros Bizjak
Quoted 8 lines · Brian Gerst
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · Uros Bizjak

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Interesting take. I'd have tried to put the canary at offset 0x0, and simply use pcpu_hot as the guard symbol.

Quoted 5 lines

It boots and runs without problems.

However, when building the kernel, I get "Absolute relocations present" warning with thousands of locations:

...

Quoted 11 lines
Quoted 3 lines

The warning is about the type of __ref_stack_chk_guard, not about the type of the relocation.

Thanks, I got distracted by the text of the warning that mentions relocation.

Quoted 5 lines
$ nm vmlinux |grep \\s__ref_sta
ffffffff8350c620 A __ref_stack_chk_guard

Without your patch:

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
$ nm vmlinux |grep \\s__ref_sta
ffffffff834fba10 D __ref_stack_chk_guard

Is this a problem in our specific case?

I don't think so - the whole notion of absolute ELF symbols is rather flaky IME, so I don't think we should be pedantic here.

From what I understand it stayed relative because there wasn't a constant added. As soon as you add a constant (which the linker treats as absolute), it becomes absolute.

Quoted 6 lines
Quoted 6 lines

We can list the symbol in arch/x86/tools/relocs.c to quiet the warning, but I would need some help with auditing the symbol itself.

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

Quoted 6 lines
Quoted 3 lines

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

for good measure.

I think this would be the most straight-forward if there are no other locality concerns this might interfere with.

I'd prefer it at the end of pcpu_hot, that way the disassembler doesn't latch on to the __stack_chk_guard symbol when referencing the other fields of pcpu_hot.

Brian Gerst

#57 lore raw
Quoted 10 lines · Brian Gerst
Quoted 8 lines · Ard Biesheuvel
Quoted 6 lines · Uros Bizjak

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

for good measure.

I think this would be the most straight-forward if there are no other locality concerns this might interfere with.

I'd prefer it at the end of pcpu_hot, that way the disassembler doesn't latch on to the __stack_chk_guard symbol when referencing the other fields of pcpu_hot.

__stack_chk_guard would no longer exist, only __ref_stack_chk_guard, which would be equal to pcpu_hot. We could just call that __ref_pcpu_hot instead if it might cause confusion otherwise. (We can't use pcpu_hot directly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via __ref_stack_chk_guard)

#58 lore raw
Quoted 22 lines · Ard Biesheuvel
Quoted 10 lines · Brian Gerst
Quoted 8 lines · Ard Biesheuvel
Quoted 6 lines · Uros Bizjak

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

for good measure.

I think this would be the most straight-forward if there are no other locality concerns this might interfere with.

I'd prefer it at the end of pcpu_hot, that way the disassembler doesn't latch on to the __stack_chk_guard symbol when referencing the other fields of pcpu_hot.

Quoted 5 lines

__stack_chk_guard would no longer exist, only __ref_stack_chk_guard, which would be equal to pcpu_hot. We could just call that __ref_pcpu_hot instead if it might cause confusion otherwise. (We can't use pcpu_hot directly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via __ref_stack_chk_guard)

That works for me.

Brian Gerst

#59 lore raw
Quoted 24 lines · Brian Gerst
Quoted 22 lines · Ard Biesheuvel
Quoted 19 lines · Brian Gerst
Quoted 6 lines · Ard Biesheuvel
Quoted 4 lines · Uros Bizjak

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

Quoted 6 lines
Quoted 3 lines

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

for good measure.

I think this would be the most straight-forward if there are no other locality concerns this might interfere with.

I'd prefer it at the end of pcpu_hot, that way the disassembler doesn't latch on to the __stack_chk_guard symbol when referencing the other fields of pcpu_hot.

Quoted 7 lines
Quoted 5 lines

__stack_chk_guard would no longer exist, only __ref_stack_chk_guard, which would be equal to pcpu_hot. We could just call that __ref_pcpu_hot instead if it might cause confusion otherwise. (We can't use pcpu_hot directly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via __ref_stack_chk_guard)

That works for me.

Maybe not. One quirk of how GCC implements this is that -mstack-protector-guard=global (used by !SMP builds) ignores the -mstack-protector-guard-symbol option and always uses __stack_chk_guard. That makes things more challenging.

Brian Gerst

#60 lore raw
Quoted 37 lines · Brian Gerst
Quoted 24 lines · Brian Gerst
Quoted 22 lines · Ard Biesheuvel
Quoted 19 lines · Brian Gerst
Quoted 6 lines · Ard Biesheuvel
Quoted 4 lines · Uros Bizjak

OTOH, we could simply do it your way and put stack canary at the beginning of pcpu_hot structure, with

Quoted 6 lines
Quoted 3 lines

static_assert(offsetof(struct pcpu_hot, stack_canary) == 0));

for good measure.

I think this would be the most straight-forward if there are no other locality concerns this might interfere with.

I'd prefer it at the end of pcpu_hot, that way the disassembler doesn't latch on to the __stack_chk_guard symbol when referencing the other fields of pcpu_hot.

Quoted 7 lines
Quoted 5 lines

__stack_chk_guard would no longer exist, only __ref_stack_chk_guard, which would be equal to pcpu_hot. We could just call that __ref_pcpu_hot instead if it might cause confusion otherwise. (We can't use pcpu_hot directly in -mstack-check-guard-symbol= for the same reasons I had to add the indirection via __ref_stack_chk_guard)

That works for me.

Maybe not. One quirk of how GCC implements this is that

(quoted diff) +0 -2
-mstack-protector-guard=global (used by !SMP builds) ignores the
-mstack-protector-guard-symbol option and always uses
Quoted 1 line

__stack_chk_guard. That makes things more challenging.

Not really. If we put stack_canary as the first member of struct pcpu_hot, we can just alias __stack_chk_guard to struct pcpu_hot in the linker script, and everything starts to magically work, SMP and !SMP. Please see the proposed patch, effectively a three liner, at [1].

[1] https://lore.kernel.org/lkml/[email protected]/

Uros.

#61 lore raw
In reply to Brian Gerst
Quoted 16 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Probably not worth it then?

Thanks,

Ingo

#62 lore raw
Quoted 20 lines · Ingo Molnar
Quoted 16 lines · Brian Gerst
Quoted 15 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Probably not worth it then?

Actually it wasn't so bad since we already had the hack for __ref_stack_chk_guard. Do you want the patches now or when the dust settles on the original series?

Brian Gerst

#63 lore raw
Quoted 25 lines · Brian Gerst
Quoted 24 lines · Ingo Molnar
Quoted 20 lines · Brian Gerst
Quoted 18 lines · Ingo Molnar
Quoted 10 lines · Uros Bizjak
Quoted 7 lines

Thank you for doing this series - it all looks pretty good from my side and I've applied it experimentally to tip:x86/asm. I fixed up the trivial details other reviewers and me noticed.

Note that the merge is tentative, it might still need a rebase if some fundamental problem comes up - but let's see how testing goes in -next.

I wonder if there would be any benefit if stack canary is put into struct pcpu_hot?

It should definitely be one of the hottest data structures on x86, so moving it there makes sense even if it cannot be measured explicitly.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

It would have to be done with linker tricks, since you can't make the compiler use a struct member directly.

Probably not worth it then?

Actually it wasn't so bad since we already had the hack for __ref_stack_chk_guard. Do you want the patches now or when the dust settles on the original series?

We can add it now I suppose, while memories are fresh and all that.

Thanks,

Ingo