arm64: Implement support for BTI veneers
v2 cover arm64: Implement support for BTI veneers
From: Ard Biesheuvel <[email protected]>
[ Resending this with minimal changes even though some other approaches
were proposed as well. [1][2] Neither of those are appropriate for backporting, while this series can reasonably be applied to older kernels. ]
Recent toolchains will omit BTI landing pads from functions with static linkage that never have their address taken. If a landing pad is needed nonetheless, it is up to the static linker to emit a BTI veneer withing direct branching range of the target, and direct the call to the veneer instead.
Modules are partially linked objects, and so there is no static linker that can do this for us. Instead, the module loader must see to this.
So implement this for the arm64 module loader. Patch #2 contains a test module that was used to validate the approach.
Changes since v1:
- make HAVE_LIVE_PATCH depend on !ARM64_BTI_KERNEL, as live patch has other cases where landing pads may go missing
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Ard Biesheuvel (2):
arm64: module: Emit BTI veneers for cross-section calls
DONOTMERGE: arm64: module: Test module for BTI veneers
arch/arm64/Kconfig | 7 +- arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/bti_veneer_test.c | 26 ++++ arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 6 files changed, 167 insertions(+), 10 deletions(-) create mode 100644 arch/arm64/kernel/bti_veneer_test.c
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
PATCH v2 1/2 arm64: module: Emit BTI veneers for cross-section calls
From: Ard Biesheuvel <[email protected]>
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
--- arch/arm64/Kconfig | 7 +- arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 140 insertions(+), 10 deletions(-)
arch/arm64/Kconfig +2 -5
@@ -210,7 +210,8 @@ config ARM64 select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IOREMAP_PROT select HAVE_IRQ_TIME_ACCOUNTING - select HAVE_LIVEPATCH + # https://lore.kernel.org/all/[email protected]/ + select HAVE_LIVEPATCH if !ARM64_BTI_KERNEL select HAVE_MOD_ARCH_SPECIFIC select HAVE_NMI select HAVE_PERF_EVENTS
@@ -2198,10 +2199,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCC - # https://github.com/llvm/llvm-project/issues/215547 - depends on !CC_IS_CLANG || CLANG_VERSION < 210000 depends on (!FUNCTION_GRAPH_TRACER || DYNAMIC_FTRACE_WITH_ARGS) help Build the kernel with Branch Target Identification annotations
arch/arm64/include/asm/module.h +12 -0
@@ -16,6 +16,7 @@ struct mod_plt_sec {struct mod_arch_specific {struct mod_plt_sec core;struct mod_plt_sec init; +struct mod_plt_sec bti;/* for CONFIG_DYNAMIC_FTRACE */ struct plt_entry *ftrace_trampolines;
@@ -43,6 +44,17 @@ struct plt_entry {__le32 br;/* br x16 */ }; +struct bti_veneer { +/* +* Functions with static linkage may lack BTI landing pads if their +* address is never taken. E.g., a direct call from .init.text to a +* static function in .text may need an additional veneer at the target +* end if it is routed via a PLT entry. +*/ +__le32 bti_c; +__le32 b; +}; +static inline bool is_forbidden_offset_for_adrp (void *place) { return cpus_have_final_cap(ARM64_WORKAROUND_843419) &&
arch/arm64/include/asm/module.lds.h +3 -0
@@ -1,6 +1,9 @@ SECTIONS { .plt 0 : { BYTE(0) } .init.plt 0 : { BYTE(0) } +#ifdef CONFIG_ARM64_BTI_KERNEL + .text.bti_veneer 0 : { BYTE(0) } +#endif .text.ftrace_trampoline 0 : { BYTE(0) } .init.text.ftrace_trampoline 0 : { BYTE(0) }
arch/arm64/kernel/module-plts.c +123 -5
183 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Re: [PATCH v2 1/2] arm64: module: Emit BTI veneers for cross-section calls
Quoted 8 lines · Ard Biesheuvel
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
Thanks for doing this - it'd be great to finally get something like this merged, it's been sitting for far too long.
Catalin Marinas
maintainerRe: [PATCH v2 1/2] arm64: module: Emit BTI veneers for cross-section calls
In reply to Ard BiesheuvelQuoted 50 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 7 +- arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 140 insertions(+), 10 deletions(-)arch/arm64/Kconfig +2
-1@@ -210,7 +210,8 @@ config ARM64 select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IOREMAP_PROT select HAVE_IRQ_TIME_ACCOUNTING - select HAVE_LIVEPATCH + # https://lore.kernel.org/all/[email protected]/ + select HAVE_LIVEPATCH if !ARM64_BTI_KERNEL select HAVE_MOD_ARCH_SPECIFIC select HAVE_NMI select HAVE_PERF_EVENTS
arch/arm64/Kconfig config ARM64_BTI_KERNEL · line 2204
Quoted 10 lines
arch/arm64/Kconfig +0
-4@@ -2198,10 +2199,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCC - # https://github.com/llvm/llvm-project/issues/215547 - depends on !CC_IS_CLANG || CLANG_VERSION < 210000 depends on (!FUNCTION_GRAPH_TRACER || DYNAMIC_FTRACE_WITH_ARGS) help Build the kernel with Branch Target Identification annotations
I think this should be the other way around (and keep HAVE_LIVEPATCH unconditional):
config ARM64_BTI_KERNEL
... depends on !LIVEPATCH || (CC_IS_CLANG && CLANG_VERSION < 210000)
I also wonder whether dropping the compiler dependency should come with some sanity check/assert for (the unlikely case) of a large vmlinux, similar to one of Josh's patches or a boot-time check that skips enabling BTI. At least until we get some generic compiler/linker workaround for large images:
There are some Sashiko comments. I'd ignore the big-endian ones since we removed it and it had been marked broken for some time anyway. But the other points look valid.
https://sashiko.dev/#/patchset/[email protected]
Apart from those, I don't have other technical objections to the patch but I wonder whether we'll find alternative options going forward that make this fix redundant. E.g. allocating .init.text and .text together (you had an RFC) or a new execmem_alloc_near() to try to place them nearby.
In the meantime, would something like the following work for us instead of the veneers? We won't free the .init.text module section but this should be small. There's the ppc64 precedent.
bool module_init_section(const char *name)
{
if (system_supports_bti_kernel() && strstarts(name, ".init.text"))
return false;
return strstarts(name, ".init");
}Signature · 1 line
Catalin
Re: [PATCH v2 1/2] arm64: module: Emit BTI veneers for cross-section calls
Quoted 65 lines · Catalin Marinas
Quoted 50 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 7 +- arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 140 insertions(+), 10 deletions(-)arch/arm64/Kconfig +2
-1@@ -210,7 +210,8 @@ config ARM64 select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IOREMAP_PROT select HAVE_IRQ_TIME_ACCOUNTING - select HAVE_LIVEPATCH + # https://lore.kernel.org/all/[email protected]/ + select HAVE_LIVEPATCH if !ARM64_BTI_KERNEL select HAVE_MOD_ARCH_SPECIFIC select HAVE_NMI select HAVE_PERF_EVENTSarch/arm64/Kconfig config ARM64_BTI_KERNEL · line 2204
Quoted 10 lines
arch/arm64/Kconfig +0
-4@@ -2198,10 +2199,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCC - # https://github.com/llvm/llvm-project/issues/215547 - depends on !CC_IS_CLANG || CLANG_VERSION < 210000 depends on (!FUNCTION_GRAPH_TRACER || DYNAMIC_FTRACE_WITH_ARGS) help Build the kernel with Branch Target Identification annotationsI think this should be the other way around (and keep
HAVE_LIVEPATCHunconditional):
Quoted 4 lines
config
ARM64_BTI_KERNEL... depends on !LIVEPATCH || (CC_IS_CLANG && CLANG_VERSION < 210000)
OK
Quoted 8 lines
I also wonder whether dropping the compiler dependency should come with some sanity check/assert for (the unlikely case) of a large vmlinux, similar to one of Josh's patches or a boot-time check that skips enabling BTI. At least until we get some generic compiler/linker workaround for large images:
According to that patch, binutils 2.42 and LLVM 20 are ok, so we'd need some version checks in Kconfig. We'd simply keep it disabled for older toolchains, as it is today.
Quoted 6 lines
There are some Sashiko comments. I'd ignore the big-endian ones since we removed it and it had been marked broken for some time anyway. But the other points look valid.
I'll take those into account if I end up respinning this (see below)
Quoted 10 lines
Apart from those, I don't have other technical objections to the patch but I wonder whether we'll find alternative options going forward that make this fix redundant. E.g. allocating .init.text and .text together (you had an RFC) or a new
execmem_alloc_near()to try to place them nearby.In the meantime, would something like the following work for us instead of the veneers? We won't free the .init.text module section but this should be small. There's the ppc64 precedent.
Quoted 8 lines
bool
module_init_section(const char *name){ if (system_supports_bti_kernel() && strstarts(name, ".init.text")) return false; return strstarts(name, ".init"); }
I've done an experiment to simply combine these unconditionally, and the impact on module .text footprint on a typical config is negligible (~10k per 100 loaded modules). That would make this change unnecessary, and would allow the init PLTs and init ftrace trampoline to be dropped as well.
We can always follow up with an execmem_alloc() fix but maybe this is not even worth the trouble.
PATCH v2 2/2 DONOTMERGE: arm64: module: Test module for BTI veneers
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
This module has a tiny .text section with a static function, which is called by the module init hook, which is placed in a ~128M init code section, which is therefore guaranteed to be place out of direct branching range when CONFIG_RANDOMIZE_MODULE_REGION_FULL is disabled.
When CONFIG_ARM64_BTI_KERNEL is enabled, this forces the use of a BTI veneer, as recent GCCs and Clangs will elide BTI landing pads for static functions that never have their address taken, even in the presence of cross-section calls to those functions.
--- arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/bti_veneer_test.c | 26 ++++++++++++++++++++ 2 files changed, 27 insertions(+)
arch/arm64/kernel/Makefile +1 -0
@@ -72,6 +72,7 @@ obj-$(CONFIG_ARM64_MPAM) += mpam.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o +obj-m += bti_veneer_test.o # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so
arch/arm64/kernel/bti_veneer_test.c new file +26 -0
@@ -0,0 +1,26 @@ + +#include <linux/init.h> +#include <linux/module.h> +#include <linux/types.h> + +static int noinline static_noninit_function(void) +{ +static volatile int ret; +return ret; +} + +static int __init bti_veneer_test_init (void ) +{ +return static_noninit_function (); +} +module_init (bti_veneer_test_init); + +static void bti_veneer_test_exit (void ) +{ +} +module_exit (bti_veneer_test_exit); + +/* Make the init code region too big to fit in 128M 'near' module region */ +asm(".section .init.padding, \"ax\", %progbits; .space (128 << 20) - 64; .previous" ); + +MODULE_LICENSE ("GPL" );
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Quoted 26 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
[ Resending this with minimal changes even though some other approaches
were proposed as well. [1][2] Neither of those are appropriate for backporting, while this series can reasonably be applied to older kernels. ]Recent toolchains will omit BTI landing pads from functions with static linkage that never have their address taken. If a landing pad is needed nonetheless, it is up to the static linker to emit a BTI veneer withing direct branching range of the target, and direct the call to the veneer instead.
Modules are partially linked objects, and so there is no static linker that can do this for us. Instead, the module loader must see to this.
So implement this for the arm64 module loader. Patch #2 contains a test module that was used to validate the approach.
Changes since v1:
- makeHAVE_LIVE_PATCHdepend on !ARM64_BTI_KERNEL, as live patch has other cases where landing pads may go missing[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 8 lines
Cc Mark Brown <[email protected]>Cc Josh Poimboeuf <[email protected]>Cc Nick Desaulniers <[email protected]>Ard Biesheuvel (2):
arm64: module: Emit BTI veneers for cross-section calls
DONOTMERGE: arm64: module: Test module for BTI veneers
Ping?