arm64: Implement support for BTI veneers
cover arm64: Implement support for BTI veneers
Recent toolchains will omit BTI landing pads from functions with static linkage that never have their address taken. If a landing pad is needed nonetheless, it is up to the static linker to emit a BTI veneer withing direct branching range of the target, and direct the call to the veneer instead.
Modules are partially linked objects, and so there is no static linker that can do this for us. Instead, the module loader must see to this.
So implement this for the arm64 module loader. Patch #2 contains a test module that was used to validate the approach.
Ard Biesheuvel (2):
arm64: module: Emit BTI veneers for cross-section calls
DONOTMERGE: arm64: module: Test module for BTI veneers
arch/arm64/Kconfig | 2 - arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/bti_veneer_test.c | 26 ++++ arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 6 files changed, 165 insertions(+), 7 deletions(-) create mode 100644 arch/arm64/kernel/bti_veneer_test.c
Signature · 1 line
2.47.3
PATCH 1/2 arm64: module: Emit BTI veneers for cross-section calls
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
--- arch/arm64/Kconfig | 2 - arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 138 insertions(+), 7 deletions(-)
arch/arm64/Kconfig +0 -2
@@ -2114,8 +2114,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCC depends on (!FUNCTION_GRAPH_TRACER || DYNAMIC_FTRACE_WITH_ARGS) help Build the kernel with Branch Target Identification annotations
arch/arm64/include/asm/module.h +12 -0
@@ -16,6 +16,7 @@ struct mod_plt_sec {struct mod_arch_specific {struct mod_plt_sec core;struct mod_plt_sec init; +struct mod_plt_sec bti;/* for CONFIG_DYNAMIC_FTRACE */ struct plt_entry *ftrace_trampolines;
@@ -43,6 +44,17 @@ struct plt_entry {__le32 br;/* br x16 */ }; +struct bti_veneer { +/* +* Functions with static linkage may lack BTI landing pads if their +* address is never taken. E.g., a direct call from .init.text to a +* static function in .text may need an additional veneer at the target +* end if it is routed via a PLT entry. +*/ +__le32 bti_c; +__le32 b; +}; +static inline bool is_forbidden_offset_for_adrp (void *place) { return cpus_have_final_cap(ARM64_WORKAROUND_843419) &&
arch/arm64/include/asm/module.lds.h +3 -0
@@ -1,6 +1,9 @@ SECTIONS { .plt 0 : { BYTE(0) } .init.plt 0 : { BYTE(0) } +#ifdef CONFIG_ARM64_BTI_KERNEL + .text.bti_veneer 0 : { BYTE(0) } +#endif .text.ftrace_trampoline 0 : { BYTE(0) } .init.text.ftrace_trampoline 0 : { BYTE(0) }
arch/arm64/kernel/module-plts.c +123 -5
183 lines not on this page · raw message
Signature · 1 line
2.47.3
Re: [PATCH 1/2] arm64: module: Emit BTI veneers for cross-section calls
Quoted 38 lines · Ard Biesheuvel
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 2 - arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 138 insertions(+), 7 deletions(-)arch/arm64/Kconfig +0
-0
arch/arm64/Kconfig config ARM64_BTI_KERNEL · line 2118
Quoted 5 lines
arch/arm64/Kconfig +0
-2@@ -2114,8 +2114,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCC
This doesn't work for livepatch though, and removing the "depends on !CC_IS_GCC" is a livepatch regression as it broadly increases the likelihood of ARM64_BTI_KERNEL (default y) getting enabled.
So "livepatch broken on arm64 clang 21+" now becomes "livepatch broken on arm64".
So either ARM64_BTI_KERNEL needs to depend on !LIVEPATCH (which will keep BTI disabled for most distros), or we need a different approach which uses __vmalloc_node_range() to allocate a veneer within 128MB of the target. I can try to write that up.
Signature · 1 line
Josh
Re: [PATCH 1/2] arm64: module: Emit BTI veneers for cross-section calls
Quoted 14 lines · Josh Poimboeuf
arch/arm64/Kconfig config ARM64_BTI_KERNEL · line 2118
Quoted 8 lines · Ard Biesheuvel
arch/arm64/Kconfig +0
-2@@ -2114,8 +2114,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCCThis doesn't work for livepatch though, and removing the "depends on !CC_IS_GCC" is a livepatch regression as it broadly increases the likelihood of
ARM64_BTI_KERNEL(default y) getting enabled.
Quoted 6 lines
So "livepatch broken on arm64 clang 21+" now becomes "livepatch broken on arm64".
So either
ARM64_BTI_KERNELneeds to depend on !LIVEPATCH (which will keep BTI disabled for most distros), or we need a different approach which uses__vmalloc_node_range()to allocate a veneer within 128MB of the target. I can try to write that up.
Hm, does the omitted BTI also break long intra-vmlinux branches for vmlinux text size > 128MB?
The linker adds veneers for that case, but doesn't make them BTI-friendly unless GNU_PROPERTY_AARCH64_FEATURE_1_BTI is set on *all* .o files, which is not currently happening for .S files.
And when I force that bit with "-z force-bti" on a 700MB text allyesconfig vmlinux with 22k linker-created veneers, it's crashes the GNU linker :-/
Seems kernel BTI is borked all around...
Signature · 1 line
Josh
Re: [PATCH 1/2] arm64: module: Emit BTI veneers for cross-section calls
In reply to Josh PoimboeufQuoted 49 lines · Josh Poimboeuf
Quoted 38 lines · Ard Biesheuvel
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 2 - arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 138 insertions(+), 7 deletions(-)arch/arm64/Kconfig +0
-0arch/arm64/Kconfig config ARM64_BTI_KERNEL · line 2118
Quoted 5 lines
arch/arm64/Kconfig +0
-2@@ -2114,8 +2114,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCCThis doesn't work for livepatch though, and removing the "depends on !CC_IS_GCC" is a livepatch regression as it broadly increases the likelihood of
ARM64_BTI_KERNEL(default y) getting enabled.
Quoted 3 lines
So "livepatch broken on arm64 clang 21+" now becomes "livepatch broken on arm64".
... when kernel mode BTI is enabled.
I have no insight into which pieces of livepatch for arm64 are actually upstream. Is it just the tooling that is missing? In this case, though, I think HAVE_LIVEPATCH should depend on !ARM64_BTI_KERNEL, rather than the other way around. I can add that in v2.
Quoted 5 lines
So either
ARM64_BTI_KERNELneeds to depend on !LIVEPATCH (which will keep BTI disabled for most distros), or we need a different approach which uses__vmalloc_node_range()to allocate a veneer within 128MB of the target. I can try to write that up.
Yes. Or pre-allocate some space in vmlinux for this purpose. I suppose we might just use -fpatchable-function-entry for this, and add one additional NOP to vmlinux for livepatch kernels, so that a landing pad can be patched in as needed. AFAIR the ftrace code already deals with the BTI landing pads gracefully.
Quoted 7 lines
Hm, does the omitted BTI also break long intra-vmlinux branches for vmlinux text size > 128MB?
The linker adds veneers for that case, but doesn't make them BTI-friendly unless
GNU_PROPERTY_AARCH64_FEATURE_1_BTIis set on *all* .o files, which is not currently happening for .S files.
I think we should fix that.
Quoted 5 lines
And when I force that bit with "-z force-bti" on a 700MB text allyesconfig vmlinux with 22k linker-created veneers, it's crashes the GNU linker :-/
Seems kernel BTI is borked all around...
allyesconfig is useful, but it is not a configuration that is expected to boot successfully. But it certainly shouldn't crash the linker, so this is something we should report to the binutils folks.
Re: [PATCH 1/2] arm64: module: Emit BTI veneers for cross-section calls
Quoted 54 lines · Ard Biesheuvel
Quoted 49 lines · Josh Poimboeuf
Quoted 38 lines · Ard Biesheuvel
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 2 - arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 138 insertions(+), 7 deletions(-)arch/arm64/Kconfig +0
-0arch/arm64/Kconfig config ARM64_BTI_KERNEL · line 2118
Quoted 5 lines
arch/arm64/Kconfig +0
-2@@ -2114,8 +2114,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCCThis doesn't work for livepatch though, and removing the "depends on !CC_IS_GCC" is a livepatch regression as it broadly increases the likelihood of
ARM64_BTI_KERNEL(default y) getting enabled.Quoted 3 lines
So "livepatch broken on arm64 clang 21+" now becomes "livepatch broken on arm64".
Quoted 7 lines
... when kernel mode BTI is enabled.
I have no insight into which pieces of livepatch for arm64 are actually upstream. Is it just the tooling that is missing? In this case, though, I think
HAVE_LIVEPATCHshould depend on !ARM64_BTI_KERNEL, rather than the other way around. I can add that in v2.
BTW we may still try and get the compiler folks to add a command line option to force BTI landing pads to be emitted for static functions, and enable it for vmlinux only when livepatch is enabled. That way, we can re-enable kernel mode BTI for all toolchains for the common case, and have livepatch on arm64 (with BTI) depend on recent versions that implement this option.
Re: [PATCH 1/2] arm64: module: Emit BTI veneers for cross-section calls
In reply to Ard BiesheuvelQuoted 55 lines · Ard Biesheuvel
Quoted 49 lines · Josh Poimboeuf
Quoted 38 lines · Ard Biesheuvel
The compiler is permitted to omit BTI landing pads from static functions that never have their address taken, but are only called directly, even if those calls originate from other code sections.
This means that calls into a module's .text section from .init.text, which may need to be routed via a PLT if .text is out of direct branching range, may result in BTI exceptions due to the indirect calls performed by the PLT veneers. (Note that calls to .init.text from .text are not allowed.)
The 'solution' is to emit yet another veneer - this is what the ELF psABI for AArch64 mandates in this case.
So derive an upper bound for the number of veneers that may be needed in the core module region to ensure that any call from init code that ends up needing a PLT can be directed at a veneer with a BTI landing pad, and allocate the additional space.
Then, emit these veneers as needed, i.e., only when emitting a PLT entry for a call from an init code section to a normal code section in the same module. In practice, this only occurs when a module's .init.text happens to be allocated far away from its .text section, which might happen when the initial 128M 'near' module region runs out of space between allocating the core module and allocating its init region.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 2 - arch/arm64/include/asm/module.h | 12 ++ arch/arm64/include/asm/module.lds.h | 3 + arch/arm64/kernel/module-plts.c | 128 +++++++++++++++++++- 4 files changed, 138 insertions(+), 7 deletions(-)arch/arm64/Kconfig +0
-0arch/arm64/Kconfig config ARM64_BTI_KERNEL · line 2118
Quoted 5 lines
arch/arm64/Kconfig +0
-2@@ -2114,8 +2114,6 @@ config ARM64_BTI_KERNEL depends on CC_HAS_BRANCH_PROT_PAC_RET_BTI # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94697 depends on !CC_IS_GCC || GCC_VERSION >= 100100 - # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 - depends on !CC_IS_GCCThis doesn't work for livepatch though, and removing the "depends on !CC_IS_GCC" is a livepatch regression as it broadly increases the likelihood of
ARM64_BTI_KERNEL(default y) getting enabled.Quoted 3 lines
So "livepatch broken on arm64 clang 21+" now becomes "livepatch broken on arm64".
Quoted 6 lines
... when kernel mode BTI is enabled.
I have no insight into which pieces of livepatch for arm64 are actually upstream. Is it just the tooling that is missing? In this case, though, I think
HAVE_LIVEPATCHshould depend on !ARM64_BTI_KERNEL, rather than the other way around. I can add that in v2.
So arm64 has supported livepatch since HAVE_LIVEPATCH was introduced with fd1e0fd71f65 ("arm64: Implement HAVE_LIVEPATCH").
The only production tooling for arm64 livepatch is out-of-tree, though it's possible to create some rudimentary livepatches in tree (see samples/livepatch).
Though I think klp-build for arm64 is solid and ready to be merged soon.
Quoted 11 lines
Quoted 5 lines
So either
ARM64_BTI_KERNELneeds to depend on !LIVEPATCH (which will keep BTI disabled for most distros), or we need a different approach which uses__vmalloc_node_range()to allocate a veneer within 128MB of the target. I can try to write that up.Yes. Or pre-allocate some space in vmlinux for this purpose. I suppose we might just use -fpatchable-function-entry for this, and add one additional NOP to vmlinux for livepatch kernels, so that a landing pad can be patched in as needed. AFAIR the ftrace code already deals with the BTI landing pads gracefully.
It looked to me like ftrace is using RET instead of BR, which I proposed to do in the veneer, but Will objected:
https://lore.kernel.org/ed4fe1f95071897859ec7fbe9176246cbd4962bf.1786138806.git.jpoimboe@kernel.org
I have a patch using __vmalloc_node_range() to allocate the 2nd veneer near the target, which seems to work fine, I'll post it later.
Quoted 13 lines
Quoted 6 lines
vmlinux text size > 128MB?
The linker adds veneers for that case, but doesn't make them BTI-friendly unless
GNU_PROPERTY_AARCH64_FEATURE_1_BTIis set on *all* .o files, which is not currently happening for .S files.I think we should fix that.
Quoted 4 lines
And when I force that bit with "-z force-bti" on a 700MB text allyesconfig vmlinux with 22k linker-created veneers, it's crashes the GNU linker :-/
Quoted 5 lines
Quoted 1 line
Seems kernel BTI is borked all around...
allyesconfig is useful, but it is not a configuration that is expected to boot successfully. But it certainly shouldn't crash the linker, so this is something we should report to the binutils folks.
Yeah, working on reporting that now, I have a couple of small reproducers. The crash isn't specific to allyesconfig though. It doesn't like the use of PROVIDE() in linker scripts, apparently...
Signature · 1 line
Josh
PATCH 2/2 DONOTMERGE: arm64: module: Test module for BTI veneers
In reply to Ard BiesheuvelThis module has a tiny .text section with a static function, which is called by the module init hook, which is placed in a ~128M init code section, which is therefore guaranteed to be place out of direct branching range when CONFIG_RANDOMIZE_MODULE_REGION_FULL is disabled.
When CONFIG_ARM64_BTI_KERNEL is enabled, this forces the use of a BTI veneer, as recent GCCs and Clangs will elide BTI landing pads for static functions that never have their address taken, even in the presence of cross-section calls to those functions.
--- arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/bti_veneer_test.c | 26 ++++++++++++++++++++ 2 files changed, 27 insertions(+)
arch/arm64/kernel/Makefile +1 -0
@@ -72,6 +72,7 @@ obj-$(CONFIG_ARM64_MPAM) += mpam.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o +obj-m += bti_veneer_test.o # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so
arch/arm64/kernel/bti_veneer_test.c new file +26 -0
@@ -0,0 +1,26 @@ + +#include <linux/init.h> +#include <linux/module.h> +#include <linux/types.h> + +static int noinline static_noninit_function(void) +{ +static volatile int ret; +return ret; +} + +static int __init bti_veneer_test_init (void ) +{ +return static_noninit_function (); +} +module_init (bti_veneer_test_init); + +static void bti_veneer_test_exit (void ) +{ +} +module_exit (bti_veneer_test_exit); + +/* Make the init code region too big to fit in 128M 'near' module region */ +asm(".section .init.padding, \"ax\", %progbits; .space (128 << 20) - 64; .previous" ); + +MODULE_LICENSE ("GPL" );
Signature · 1 line
2.47.3