arm64: dynamic shadow call stack support
Ard Biesheuvel
maintainerv3 cover arm64: dynamic shadow call stack support
Generic kernel images such as Android's GKI usually enable all available security features, which are typically implemented in such a way that they only take effect if the underlying hardware can support it, but don't interfere with correct and efficient operation otherwise.
For shadow call stack support, which is always supported by the hardware, it means it will be enabled even if pointer authentication is also supported, and enabled for signing return addresses stored on the stack. The additional security provided by shadow call stack is only marginal in this case, whereas the performance overhead is not.
Given that return address signing is based on PACIASP/AUTIASP instructions that implicitly operate on the return address register (X30) and are not idempotent (i.e., each needs to be emitted exactly once before the return address is stored on the ordinary stack and after it has been retrieved from it), we can convert these instruction 1:1 into shadow call stack pushes and pops involving the register X30. As this is something that can be done at runtime rather than build time, we can do this conditionally based on whether or not return address signing is supported on the underlying hardware.
In order to be able to unwind call stacks that involve return address signing, whether or not the return address is currently signed is tracked by DWARF CFI directives in the unwinding metadata. This means we can use this information to locate all PACIASP/AUTIASP instructions in the binary, instead of having to use brute force and go over all instructions in the entire program.
This series implements this approach for Clang, which has recently been fixed to emit all these CFI directives correctly. This series is based on an older PoC sent out last year [0] that targeted GCC only (due to this issue). This v3 targets Clang only, as GCC has its own issues with CFI accuracy.
Changes since v2 [1]:
- don't enable unwind table generation for nVHE code - it cannot be patched anyway so it has no use for it;
- drop checks for ID reg overrides
- fix some remaining TODOs regarding augmentation data and the code alignment factor
- disable PAC for leaf functions when dynamic SCS is configured, so that we don't end up with SCS pushes and pops in all leaf functions too;
- add I-cache maintenance after code patching
- add Rb's from Nick and Kees.
Changes since RFC v1:
- implement boot time check for PAC/BTI support, and only enable dynamic SCS if neither are supported;
- implement module patching as well;
- switch to Clang, and drop workaround for GCC bug;
[0] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
Ard Biesheuvel (3):
arm64: unwind: add asynchronous unwind tables to kernel and modules
scs: add support for dynamic shadow call stacks
arm64: implement dynamic shadow call stack for Clang
Makefile | 2 + arch/Kconfig | 7 + arch/arm64/Kconfig | 12 + arch/arm64/Makefile | 15 +- arch/arm64/include/asm/module.lds.h | 8 + arch/arm64/include/asm/scs.h | 45 ++++ arch/arm64/kernel/Makefile | 2 + arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/irq.c | 2 +- arch/arm64/kernel/module.c | 8 + arch/arm64/kernel/patch-scs.c | 257 ++++++++++++++++++++ arch/arm64/kernel/sdei.c | 2 +- arch/arm64/kernel/setup.c | 4 + arch/arm64/kernel/vmlinux.lds.S | 13 + arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + include/linux/scs.h | 16 ++ kernel/scs.c | 14 +- 18 files changed, 406 insertions(+), 6 deletions(-) create mode 100644 arch/arm64/kernel/patch-scs.c
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerPATCH v3 1/3 arm64: unwind: add asynchronous unwind tables to kernel and modules
Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.
This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.
--- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + 6 files changed, 31 insertions(+)
arch/arm64/Kconfig +3 -0
@@ -366,6 +366,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"
arch/arm64/Makefile +5 -0
@@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_prepare
arch/arm64/include/asm/module.lds.h +8 -0
@@ -17,4 +17,12 @@ SECTIONS {*/ .text.hot : { *(.text.hot) }#endif + +#ifdef CONFIG_UNWIND_TABLES +/* +* Currently, we only use unwind info at module load time, so we can +* put it into the .init allocation. +*/ + .init.eh_frame : { *(.eh_frame) } +#endif }
arch/arm64/kernel/vmlinux.lds.S +13 -0
@@ -120,6 +120,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif +#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endif + /* * The size of the PE/COFF section that covers the kernel image, which * runs from _stext to _edata, must be a round multiple of the PE/COFF
@@ -231,6 +242,8 @@ SECTIONS __alt_instructions_end = .; } + UNWIND_DATA_SECTIONS + . = ALIGN(SEGMENT_ALIGN); __inittext_end = .; __initdata_begin = .;
arch/arm64/kvm/hyp/nvhe/Makefile +1 -0
@@ -84,6 +84,7 @@ quiet_cmd_hypcopy = HYPCOPY $@ # Remove ftrace, Shadow Call Stack, and CFI CFLAGS. # This is equivalent to the 'notrace', '__noscs', and '__nocfi' annotations. KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI), $(KBUILD_CFLAGS)) +KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables # KVM nVHE code is run at a different exception code with a different map, so # compiler instrumentation that inserts callbacks or checks into the code may
drivers/firmware/efi/libstub/Makefile +1 -0
@@ -20,6 +20,7 @@ cflags-$(CONFIG_X86) += -m$(BITS) -D__KERNEL__ \ # disable the stackleak plugin cflags-$(CONFIG_ARM64) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fpie $(DISABLE_STACKLEAK_PLUGIN) \ + -fno-unwind-tables -fno-asynchronous-unwind-tables \ $(call cc-option,-mbranch-protection=none) cflags-$(CONFIG_ARM) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fno-builtin -fpic \
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v3 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules
Quoted 77 lines · Ard Biesheuvel
Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.
This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>--- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + 6 files changed, 31 insertions(+)arch/arm64/Kconfig +3
-0@@ -366,6 +366,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"arch/arm64/Makefile +5
-0@@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_preparearch/arm64/include/asm/module.lds.h +8
-0@@ -17,4 +17,12 @@ SECTIONS {*/ .text.hot : { *(.text.hot) }#endif + +#ifdef CONFIG_UNWIND_TABLES +/* +* Currently, we only use unwind info at module load time, so we can +* put it into the .init allocation. +*/ + .init.eh_frame : { *(.eh_frame) } +#endif }arch/arm64/kernel/vmlinux.lds.S +0
-0@@ -120,6 +120,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif
arch/arm64/kernel/vmlinux.lds.S jiffies = jiffies_64; · line 132
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S +10
-0+#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endif
How does this work with SANITIZER_DISCARDS dropping .eh_frame in include/asm-generic/vmlinux.lds.h and scripts/module.lds.S? We would definitely want to enable this together with CONFIG_CFI_CLANG, so it seems like we'd have to drop the discard rules as well.
Sami
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerRe: [PATCH v3 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules
Quoted 79 lines · Sami Tolvanen
Quoted 77 lines · Ard Biesheuvel
Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.
This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>--- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + 6 files changed, 31 insertions(+)arch/arm64/Kconfig +3
-0@@ -366,6 +366,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"arch/arm64/Makefile +5
-0@@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_preparearch/arm64/include/asm/module.lds.h +8
-0@@ -17,4 +17,12 @@ SECTIONS {*/ .text.hot : { *(.text.hot) }#endif + +#ifdef CONFIG_UNWIND_TABLES +/* +* Currently, we only use unwind info at module load time, so we can +* put it into the .init allocation. +*/ + .init.eh_frame : { *(.eh_frame) } +#endif }arch/arm64/kernel/vmlinux.lds.S +0
-0@@ -120,6 +120,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif
Quoted 16 lines
arch/arm64/kernel/vmlinux.lds.S jiffies = jiffies_64; · line 132
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S +10
-0+#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endifHow does this work with
SANITIZER_DISCARDSdropping .eh_frame ininclude/asm-generic/vmlinux.lds.handscripts/module.lds.S? We would definitely want to enable this together withCONFIG_CFI_CLANG, so it seems like we'd have to drop the discard rules as well.
Good point, I had no idea that that existed.
Clang 13 should have the fix for the original issue, so we could make this workaround specific to 12 and earlier.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v3 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules
Quoted 98 lines · Ard Biesheuvel
Quoted 79 lines · Sami Tolvanen
Quoted 77 lines · Ard Biesheuvel
Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.
This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>--- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + 6 files changed, 31 insertions(+)arch/arm64/Kconfig +3
-0@@ -366,6 +366,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"arch/arm64/Makefile +5
-0@@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_preparearch/arm64/include/asm/module.lds.h +8
-0@@ -17,4 +17,12 @@ SECTIONS {*/ .text.hot : { *(.text.hot) }#endif + +#ifdef CONFIG_UNWIND_TABLES +/* +* Currently, we only use unwind info at module load time, so we can +* put it into the .init allocation. +*/ + .init.eh_frame : { *(.eh_frame) } +#endif }arch/arm64/kernel/vmlinux.lds.S +0
-0@@ -120,6 +120,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endifQuoted 16 lines
arch/arm64/kernel/vmlinux.lds.S jiffies = jiffies_64; · line 132
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S +10
-0+#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endifHow does this work with
SANITIZER_DISCARDSdropping .eh_frame ininclude/asm-generic/vmlinux.lds.handscripts/module.lds.S? We would definitely want to enable this together withCONFIG_CFI_CLANG, so it seems like we'd have to drop the discard rules as well.
Quoted 4 lines
Good point, I had no idea that that existed.
Clang 13 should have the fix for the original issue, so we could make this workaround specific to 12 and earlier.
Yeah, I like this -- I'd prefer to know when we get "surprise" sections again.
Signature · 6 lines
Kees Cook _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v3 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules
Quoted 82 lines · Kees Cook
Quoted 81 lines · Ard Biesheuvel
Quoted 79 lines · Sami Tolvanen
Quoted 77 lines · Ard Biesheuvel
Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.
This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>--- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + 6 files changed, 31 insertions(+)arch/arm64/Kconfig +3
-0@@ -366,6 +366,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"arch/arm64/Makefile +5
-0@@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_preparearch/arm64/include/asm/module.lds.h +8
-0@@ -17,4 +17,12 @@ SECTIONS {*/ .text.hot : { *(.text.hot) }#endif + +#ifdef CONFIG_UNWIND_TABLES +/* +* Currently, we only use unwind info at module load time, so we can +* put it into the .init allocation. +*/ + .init.eh_frame : { *(.eh_frame) } +#endif }arch/arm64/kernel/vmlinux.lds.S +0
-0@@ -120,6 +120,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S jiffies = jiffies_64; · line 132
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S +10
-0+#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endif
How do you intend to use the encapsulation symbols __eh_frame_start and __eh_frame_end ?
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
How does this work with
SANITIZER_DISCARDSdropping .eh_frame ininclude/asm-generic/vmlinux.lds.handscripts/module.lds.S? We would definitely want to enable this together withCONFIG_CFI_CLANG, so it seems like we'd have to drop the discard rules as well.
Quoted 6 lines
Quoted 4 lines
Good point, I had no idea that that existed.
Clang 13 should have the fix for the original issue, so we could make this workaround specific to 12 and earlier.
Yeah, I like this -- I'd prefer to know when we get "surprise" sections again.
Does this need asynchronous unwind tables or just synchronous unwind tables?
Note: with Clang < 15, the AArch64 codegen was in a quite bad state. It has significantly improved since the https://reviews.llvm.org/D114545 patch series but I am not confident to state that production use may not into an issue :-)
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerRe: [PATCH v3 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules
Quoted 84 lines · Fangrui Song
Quoted 82 lines · Kees Cook
Quoted 81 lines · Ard Biesheuvel
Quoted 79 lines · Sami Tolvanen
Quoted 77 lines · Ard Biesheuvel
Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.
This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>--- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + 6 files changed, 31 insertions(+)arch/arm64/Kconfig +3
-0@@ -366,6 +366,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"arch/arm64/Makefile +5
-0@@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_preparearch/arm64/include/asm/module.lds.h +8
-0@@ -17,4 +17,12 @@ SECTIONS {*/ .text.hot : { *(.text.hot) }#endif + +#ifdef CONFIG_UNWIND_TABLES +/* +* Currently, we only use unwind info at module load time, so we can +* put it into the .init allocation. +*/ + .init.eh_frame : { *(.eh_frame) } +#endif }arch/arm64/kernel/vmlinux.lds.S +0
-0@@ -120,6 +120,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endif
Quoted 14 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S jiffies = jiffies_64; · line 132
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S +10
-0+#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endifHow do you intend to use the encapsulation symbols
__eh_frame_startand__eh_frame_end?
They are used in patch #3 to access the contents of the section from the program itself.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
How does this work with
SANITIZER_DISCARDSdropping .eh_frame ininclude/asm-generic/vmlinux.lds.handscripts/module.lds.S? We would definitely want to enable this together withCONFIG_CFI_CLANG, so it seems like we'd have to drop the discard rules as well.Good point, I had no idea that that existed.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Clang 13 should have the fix for the original issue, so we could make this workaround specific to 12 and earlier.
Yeah, I like this -- I'd prefer to know when we get "surprise" sections again.
Does this need asynchronous unwind tables or just synchronous unwind tables?
I would assume that unwind tables that are only intended for synchronous unwinding may not be instruction accurate when it comes to the placement of the DW_CFA_negate_ra_state opcodes.
Quoted 3 lines
Note: with Clang < 15, the AArch64 codegen was in a quite bad state. It has significantly improved since the https://reviews.llvm.org/D114545 patch series but I am not confident to state that production use may not into an issue :-)
This is why the feature is only enabled for Clang 15 or later.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v3 1/3] arm64: unwind: add asynchronous unwind tables to kernel and modules
Quoted 125 lines · Ard Biesheuvel
Quoted 84 lines · Fangrui Song
Quoted 82 lines · Kees Cook
Quoted 81 lines · Ard Biesheuvel
Quoted 79 lines · Sami Tolvanen
Quoted 77 lines · Ard Biesheuvel
Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.
This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>--- arch/arm64/Kconfig | 3 +++ arch/arm64/Makefile | 5 +++++ arch/arm64/include/asm/module.lds.h | 8 ++++++++ arch/arm64/kernel/vmlinux.lds.S | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + drivers/firmware/efi/libstub/Makefile | 1 + 6 files changed, 31 insertions(+)arch/arm64/Kconfig +3
-0@@ -366,6 +366,9 @@ config KASAN_SHADOW_OFFSET default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff +config UNWIND_TABLES + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"arch/arm64/Makefile +5
-0@@ -45,8 +45,13 @@ KBUILD_CFLAGS += $(call cc-option,-mabi=lp64) KBUILD_AFLAGS += $(call cc-option,-mabi=lp64) # Avoid generating .eh_frame* sections. +ifneq ($(CONFIG_UNWIND_TABLES),y) KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables KBUILD_AFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables +else +KBUILD_CFLAGS += -fasynchronous-unwind-tables +KBUILD_AFLAGS += -fasynchronous-unwind-tables +endif ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y) prepare: stack_protector_preparearch/arm64/include/asm/module.lds.h +8
-0@@ -17,4 +17,12 @@ SECTIONS {*/ .text.hot : { *(.text.hot) }#endif + +#ifdef CONFIG_UNWIND_TABLES +/* +* Currently, we only use unwind info at module load time, so we can +* put it into the .init allocation. +*/ + .init.eh_frame : { *(.eh_frame) } +#endif }arch/arm64/kernel/vmlinux.lds.S +0
-0@@ -120,6 +120,17 @@ jiffies = jiffies_64; #define TRAMP_TEXT #endifQuoted 14 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S jiffies = jiffies_64; · line 132
Quoted 10 lines
arch/arm64/kernel/vmlinux.lds.S +10
-0+#ifdef CONFIG_UNWIND_TABLES +#define UNWIND_DATA_SECTIONS \ + .eh_frame : { \ + __eh_frame_start = .; \ + *(.eh_frame) \ + __eh_frame_end = .; \ + } +#else +#define UNWIND_DATA_SECTIONS +#endifHow do you intend to use the encapsulation symbols
__eh_frame_startand__eh_frame_end?They are used in patch #3 to access the contents of the section from the program itself.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
How does this work with
SANITIZER_DISCARDSdropping .eh_frame ininclude/asm-generic/vmlinux.lds.handscripts/module.lds.S? We would definitely want to enable this together withCONFIG_CFI_CLANG, so it seems like we'd have to drop the discard rules as well.Good point, I had no idea that that existed.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Clang 13 should have the fix for the original issue, so we could make this workaround specific to 12 and earlier.
Yeah, I like this -- I'd prefer to know when we get "surprise" sections again.
Does this need asynchronous unwind tables or just synchronous unwind tables?
I would assume that unwind tables that are only intended for synchronous unwinding may not be instruction accurate when it comes to the placement of the
DW_CFA_negate_ra_stateopcodes.
Quoted 5 lines
Quoted 3 lines
Note: with Clang < 15, the AArch64 codegen was in a quite bad state. It has significantly improved since the https://reviews.llvm.org/D114545 patch series but I am not confident to state that production use may not into an issue :-)
This is why the feature is only enabled for Clang 15 or later.
Ok, thanks for the clarification.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerPATCH v3 2/3 scs: add support for dynamic shadow call stacks
In reply to Ard BiesheuvelIn order to allow arches to use code patching to conditionally emit the shadow stack pushes and pops, rather than always taking the performance hit even on CPUs that implement alternatives such as stack pointer authentication on arm64, add a Kconfig symbol that can be set by the arch to omit the SCS codegen itself, without otherwise affecting how support code for SCS and compiler options (for register reservation, for instance) are emitted.
Also, add a static key and some plumbing to omit the allocation of shadow call stack for dynamic SCS configurations if SCS is disabled at runtime.
--- Makefile | 2 ++ arch/Kconfig | 7 +++++++ include/linux/scs.h | 16 ++++++++++++++++ kernel/scs.c | 14 ++++++++++++-- 4 files changed, 37 insertions(+), 2 deletions(-)
Makefile +2 -0
@@ -883,8 +883,10 @@ LDFLAGS_vmlinux += --gc-sections endif ifdef CONFIG_SHADOW_CALL_STACK +ifndef CONFIG_DYNAMIC_SCS CC_FLAGS_SCS := -fsanitize=shadow-call-stack KBUILD_CFLAGS += $(CC_FLAGS_SCS) +endif export CC_FLAGS_SCS endif
arch/Kconfig +7 -0
@@ -636,6 +636,13 @@ config SHADOW_CALL_STACK reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying the stacks. +config DYNAMIC_SCS + bool + help + Set by the arch code if it relies on code patching to insert the + shadow call stack push and pop instructions rather than on the + compiler. + config LTO bool help
include/linux/scs.h +16 -0
@@ -53,6 +53,20 @@ static inline bool task_scs_end_corrupted(struct task_struct *tsk) return sz >= SCS_SIZE - 1 || READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC; } +DECLARE_STATIC_KEY_TRUE (dynamic_scs_enabled); + +static inline bool scs_is_dynamic (void ) +{ +return static_branch_likely (&dynamic_scs_enabled); +} + +static inline bool scs_is_enabled (void ) +{ +if (!IS_ENABLED (CONFIG_DYNAMIC_SCS )) +return true; +return scs_is_dynamic (); +} +#else /* CONFIG_SHADOW_CALL_STACK */ static inline void *scs_alloc (int node) {return NULL ; }
@@ -62,6 +76,8 @@ static inline void scs_task_reset(struct task_struct *tsk) {}static inline int scs_prepare (struct task_struct *tsk,int node) {return 0 ; }static inline void scs_release (struct task_struct *tsk) {}static inline bool task_scs_end_corrupted (struct task_struct *tsk) {return false; } +static inline bool scs_is_enabled (void ) {return false; } +static inline bool scs_is_dynamic (void ) {return false; }#endif /* CONFIG_SHADOW_CALL_STACK */
kernel/scs.c +12 -2
@@ -12,6 +12,10 @@#include <linux/vmalloc.h> #include <linux/vmstat.h> +#ifdef CONFIG_DYNAMIC_SCS +DEFINE_STATIC_KEY_TRUE (dynamic_scs_enabled); +#endif +static void __scs_account (void *s,int account) {struct page *scs_page =vmalloc_to_page (s);
@@ -101,14 +105,20 @@ static int scs_cleanup(unsigned int cpu)void __init scs_init (void ) { +if (!scs_is_enabled ()) +return ;cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"scs:scs_cache" ,NULL , scs_cleanup); }int scs_prepare (struct task_struct *tsk,int node) { -void *s =scs_alloc (node); +void *s; +if (!scs_is_enabled ()) +return 0 ; + + s =scs_alloc (node);if (!s)return -ENOMEM ;
@@ -148,7 +158,7 @@ void scs_release(struct task_struct *tsk) {void *s =task_scs (tsk); -if (!s) +if (!scs_is_enabled () || !s)return ; WARN(task_scs_end_corrupted(tsk),
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerRe: [PATCH v3 2/3] scs: add support for dynamic shadow call stacks
Quoted 9 lines · Ard Biesheuvel
In order to allow arches to use code patching to conditionally emit the shadow stack pushes and pops, rather than always taking the performance hit even on CPUs that implement alternatives such as stack pointer authentication on arm64, add a
Kconfigsymbol that can be set by the arch to omit the SCS codegen itself, without otherwise affecting how support code for SCS and compiler options (for register reservation, for instance) are emitted.
Quoted 7 lines
Also, add a static key and some plumbing to omit the allocation of shadow call stack for dynamic SCS configurations if SCS is disabled at runtime.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>Reviewed-by Kees Cook <[email protected]>
This patch needs the following hunk applied on top to fix a build error reported by the bots:
include/linux/scs.h +2 -0
@@ -57,6 +57,8 @@ DECLARE_STATIC_KEY_TRUE(dynamic_scs_enabled);static inline bool scs_is_dynamic (void ) { +if (!IS_ENABLED (CONFIG_DYNAMIC_SCS )) +return false;return static_branch_likely (&dynamic_scs_enabled); }
Quoted 121 lines
--- Makefile | 2 ++ arch/Kconfig | 7 +++++++ include/linux/scs.h | 16 ++++++++++++++++ kernel/scs.c | 14 ++++++++++++-- 4 files changed, 37 insertions(+), 2 deletions(-)Makefile +2
-0@@ -883,8 +883,10 @@ LDFLAGS_vmlinux += --gc-sections endif ifdef CONFIG_SHADOW_CALL_STACK +ifndef CONFIG_DYNAMIC_SCS CC_FLAGS_SCS := -fsanitize=shadow-call-stack KBUILD_CFLAGS += $(CC_FLAGS_SCS) +endif export CC_FLAGS_SCS endifarch/Kconfig +7
-0@@ -636,6 +636,13 @@ config SHADOW_CALL_STACK reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying the stacks. +config DYNAMIC_SCS + bool + help + Set by the arch code if it relies on code patching to insert the + shadow call stack push and pop instructions rather than on the + compiler. + config LTO bool helpinclude/linux/scs.h +16
-0@@ -53,6 +53,20 @@ static inline bool task_scs_end_corrupted(struct task_struct *tsk) return sz >= SCS_SIZE - 1 || READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC; } +DECLARE_STATIC_KEY_TRUE (dynamic_scs_enabled); + +static inline bool scs_is_dynamic (void ) +{ +return static_branch_likely (&dynamic_scs_enabled); +} + +static inline bool scs_is_enabled (void ) +{ +if (!IS_ENABLED (CONFIG_DYNAMIC_SCS )) +return true; +return scs_is_dynamic (); +} +#else /* CONFIG_SHADOW_CALL_STACK */ static inline void *scs_alloc (int node) {return NULL ; }@@ -62,6 +76,8 @@ static inline void scs_task_reset(struct task_struct *tsk) {}static inline int scs_prepare (struct task_struct *tsk,int node) {return 0 ; }static inline void scs_release (struct task_struct *tsk) {}static inline bool task_scs_end_corrupted (struct task_struct *tsk) {return false; } +static inline bool scs_is_enabled (void ) {return false; } +static inline bool scs_is_dynamic (void ) {return false; }#endif /* CONFIG_SHADOW_CALL_STACK */ kernel/scs.c +12
-3@@ -12,6 +12,10 @@#include <linux/vmalloc.h> #include <linux/vmstat.h> +#ifdef CONFIG_DYNAMIC_SCS +DEFINE_STATIC_KEY_TRUE (dynamic_scs_enabled); +#endif +static void __scs_account (void *s,int account) {struct page *scs_page =vmalloc_to_page (s);@@ -101,14 +105,20 @@ static int scs_cleanup(unsigned int cpu)void __init scs_init (void ) { +if (!scs_is_enabled ()) +return ;cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"scs:scs_cache" ,NULL , scs_cleanup); }int scs_prepare (struct task_struct *tsk,int node) { -void *s =scs_alloc (node); +void *s; +if (!scs_is_enabled ()) +return 0 ; + + s =scs_alloc (node);if (!s)return -ENOMEM ;@@ -148,7 +158,7 @@ void scs_release(struct task_struct *tsk) {void *s =task_scs (tsk); -if (!s) +if (!scs_is_enabled () || !s)return ; WARN(task_scs_end_corrupted(tsk), --
Quoted 2 lines
2.30.2
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Sami Tolvanen
Reviewed-byRe: [PATCH v3 2/3] scs: add support for dynamic shadow call stacks
Quoted 24 lines · Ard Biesheuvel
Quoted 9 lines · Ard Biesheuvel
In order to allow arches to use code patching to conditionally emit the shadow stack pushes and pops, rather than always taking the performance hit even on CPUs that implement alternatives such as stack pointer authentication on arm64, add a
Kconfigsymbol that can be set by the arch to omit the SCS codegen itself, without otherwise affecting how support code for SCS and compiler options (for register reservation, for instance) are emitted.Quoted 7 lines
Also, add a static key and some plumbing to omit the allocation of shadow call stack for dynamic SCS configurations if SCS is disabled at runtime.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>Reviewed-by Kees Cook <[email protected]>This patch needs the following hunk applied on top to fix a build error reported by the bots:
include/linux/scs.h DECLARE_STATIC_KEY_TRUE(dynamic_scs_enabled); · line 63
Quoted 7 lines
include/linux/scs.h +2
-0@@ -57,6 +57,8 @@ DECLARE_STATIC_KEY_TRUE(dynamic_scs_enabled);static inline bool scs_is_dynamic (void ) { +if (!IS_ENABLED (CONFIG_DYNAMIC_SCS )) +return false;return static_branch_likely (&dynamic_scs_enabled); }
With this:
Sami
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerRe: [PATCH v3 2/3] scs: add support for dynamic shadow call stacks
Quoted 36 lines · Sami Tolvanen
Quoted 24 lines · Ard Biesheuvel
Quoted 9 lines · Ard Biesheuvel
In order to allow arches to use code patching to conditionally emit the shadow stack pushes and pops, rather than always taking the performance hit even on CPUs that implement alternatives such as stack pointer authentication on arm64, add a
Kconfigsymbol that can be set by the arch to omit the SCS codegen itself, without otherwise affecting how support code for SCS and compiler options (for register reservation, for instance) are emitted.Quoted 7 lines
Also, add a static key and some plumbing to omit the allocation of shadow call stack for dynamic SCS configurations if SCS is disabled at runtime.
Signed-off-by Ard Biesheuvel <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>Reviewed-by Kees Cook <[email protected]>This patch needs the following hunk applied on top to fix a build error reported by the bots:
include/linux/scs.h DECLARE_STATIC_KEY_TRUE(dynamic_scs_enabled); · line 63
Quoted 7 lines
include/linux/scs.h +2
-0@@ -57,6 +57,8 @@ DECLARE_STATIC_KEY_TRUE(dynamic_scs_enabled);static inline bool scs_is_dynamic (void ) { +if (!IS_ENABLED (CONFIG_DYNAMIC_SCS )) +return false;return static_branch_likely (&dynamic_scs_enabled); }With this:
Quoted 2 lines
Reviewed-by Sami Tolvanen <[email protected]>
Thanks. I had spotted that too, and came up with the exact same fix.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerPATCH v3 3/3 arm64: implement dynamic shadow call stack for Clang
In reply to Ard BiesheuvelImplement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively.
This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the desired behavior.
--- arch/arm64/Kconfig | 9 + arch/arm64/Makefile | 10 +- arch/arm64/include/asm/scs.h | 45 ++++ arch/arm64/kernel/Makefile | 2 + arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/irq.c | 2 +- arch/arm64/kernel/module.c | 8 + arch/arm64/kernel/patch-scs.c | 257 ++++++++++++++++++++ arch/arm64/kernel/sdei.c | 2 +- arch/arm64/kernel/setup.c | 4 + 10 files changed, 338 insertions(+), 4 deletions(-)
arch/arm64/Kconfig +9 -0
@@ -369,6 +369,15 @@ config KASAN_SHADOW_OFFSET config UNWIND_TABLES bool +config UNWIND_PATCH_PAC_INTO_SCS + bool "Enable shadow call stack dynamically using code patching" + # needs Clang with https://reviews.llvm.org/D111780 incorporated + depends on CC_IS_CLANG && CLANG_VERSION >= 150000 + depends on ARM64_PTR_AUTH_KERNEL && CC_HAS_BRANCH_PROT_PAC_RET + depends on SHADOW_CALL_STACK + select UNWIND_TABLES + select DYNAMIC_SCS + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"
arch/arm64/Makefile +8 -2
@@ -77,10 +77,16 @@ branch-prot-flags-$(CONFIG_CC_HAS_SIGN_RETURN_ADDRESS) := -msign-return-address= # We enable additional protection for leaf functions as there is some # narrow potential for ROP protection benefits and no substantial # performance impact has been observed. +PACRET-y := pac-ret+leaf + +# Using a shadow call stack in leaf functions is too costly, so avoid PAC there +# as well when we may be patching PAC into SCS +PACRET-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) := pac-ret + ifeq ($(CONFIG_ARM64_BTI_KERNEL),y) -branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=pac-ret+leaf+bti +branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=$(PACRET-y)+bti else -branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=pac-ret+leaf +branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=$(PACRET-y) endif # -march=armv8.3-a enables the non-nops instructions for PAC, to avoid the # compiler to generate them and consequently to break the single image contract
arch/arm64/include/asm/scs.h +45 -0
51 lines not on this page · raw message
arch/arm64/kernel/Makefile +2 -0
@@ -74,6 +74,8 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o +CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so
arch/arm64/kernel/head.S +3 -0
@@ -453,6 +453,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x21 // pass FDT address in x0 bl early_fdt_map // Try mapping the FDT early bl init_feature_override // Parse cpu feature overrides +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + bl scs_patch_vmlinux +#endif #ifdef CONFIG_RANDOMIZE_BASE tst x23, ~(MIN_KIMG_ALIGN - 1) // already running randomized? b.ne 0f
arch/arm64/kernel/irq.c +1 -1
@@ -39,7 +39,7 @@ static void init_irq_scs(void) {int cpu; -if (!IS_ENABLED (CONFIG_SHADOW_CALL_STACK )) +if (!scs_is_enabled ())return ;for_each_possible_cpu (cpu)
arch/arm64/kernel/module.c +8 -0
@@ -15,9 +15,11 @@#include <linux/kernel.h> #include <linux/mm.h> #include <linux/moduleloader.h> +#include <linux/scs.h> #include <linux/vmalloc.h> #include <asm/alternative.h> #include <asm/insn.h> +#include <asm/scs.h> #include <asm/sections.h> void *module_alloc (unsigned long size)
@@ -529,5 +531,11 @@ int module_finalize(const Elf_Ehdr *hdr,if (s)apply_alternatives_module ((void *)s->sh_addr, s->sh_size); +if (scs_is_dynamic ()) { + s =find_section (hdr, sechdrs,".init.eh_frame" ); +if (s) +scs_patch ((void *)s->sh_addr, s->sh_size); + } +return module_init_ftrace_plt (hdr, sechdrs, me); }
arch/arm64/kernel/patch-scs.c new file +257 -0
257 lines not on this page · raw message
arch/arm64/kernel/sdei.c +1 -1
@@ -144,7 +144,7 @@ static int init_sdei_scs(void)int cpu;int err =0 ; -if (!IS_ENABLED (CONFIG_SHADOW_CALL_STACK )) +if (!scs_is_enabled ())return 0 ;for_each_possible_cpu (cpu) {
arch/arm64/kernel/setup.c +4 -0
@@ -30,6 +30,7 @@#include <linux/efi.h> #include <linux/psci.h> #include <linux/sched/task.h> +#include <linux/scs.h> #include <linux/mm.h> #include <asm/acpi.h>
@@ -42,6 +43,7 @@#include <asm/cpu_ops.h> #include <asm/kasan.h> #include <asm/numa.h> +#include <asm/scs.h> #include <asm/sections.h> #include <asm/setup.h> #include <asm/smp_plat.h>
@@ -313,6 +315,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)parse_early_param (); +dynamic_scs_init (); +/* * Unmask asynchronous aborts and fiq after bringing up possible * earlycon. (Report possible System Errors once we can report this
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v3 3/3] arm64: implement dynamic shadow call stack for Clang
Quoted 5 lines · Ard Biesheuvel
Implement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively.
Quoted 7 lines
This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the desired behavior.
Specifically, the "PAC available" benefit is the per-thread memory savings (no shadow stack needs to be allocated). Thanks for getting this working!
Quoted 1 line
Signed-off-by Ard Biesheuvel <[email protected]>
Sami, can you test this for the cases you've needed this for?
In the meantime, Will, can you land this for -next so we can get maximal test time?
Signature · 6 lines
Kees Cook _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerRe: [PATCH v3 3/3] arm64: implement dynamic shadow call stack for Clang
Quoted 19 lines · Kees Cook
Quoted 5 lines · Ard Biesheuvel
Implement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively.
Quoted 7 lines
This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the desired behavior.
Specifically, the "PAC available" benefit is the per-thread memory savings (no shadow stack needs to be allocated). Thanks for getting this working!
Quoted 7 lines
Quoted 1 line
Signed-off-by Ard Biesheuvel <[email protected]>Sami, can you test this for the cases you've needed this for?
In the meantime, Will, can you land this for -next so we can get maximal test time?
I should note that this relies on Clang 15 which has not been released yet.
I have been using the clang-15 and lld-15 packages from
deb http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye main
and setting LLVM=-15 on the make command line.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v3 3/3] arm64: implement dynamic shadow call stack for Clang
In reply to Ard BiesheuvelQuoted 107 lines · Ard Biesheuvel
Implement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively.
This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the desired behavior.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 9 + arch/arm64/Makefile | 10 +- arch/arm64/include/asm/scs.h | 45 ++++ arch/arm64/kernel/Makefile | 2 + arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/irq.c | 2 +- arch/arm64/kernel/module.c | 8 + arch/arm64/kernel/patch-scs.c | 257 ++++++++++++++++++++ arch/arm64/kernel/sdei.c | 2 +- arch/arm64/kernel/setup.c | 4 + 10 files changed, 338 insertions(+), 4 deletions(-)arch/arm64/Kconfig +9
-0@@ -369,6 +369,15 @@ config KASAN_SHADOW_OFFSET config UNWIND_TABLES bool +config UNWIND_PATCH_PAC_INTO_SCS + bool "Enable shadow call stack dynamically using code patching" + # needs Clang with https://reviews.llvm.org/D111780 incorporated + depends on CC_IS_CLANG && CLANG_VERSION >= 150000 + depends on ARM64_PTR_AUTH_KERNEL && CC_HAS_BRANCH_PROT_PAC_RET + depends on SHADOW_CALL_STACK + select UNWIND_TABLES + select DYNAMIC_SCS + source "arch/arm64/Kconfig.platforms" menu "Kernel Features"arch/arm64/Makefile +8
-2@@ -77,10 +77,16 @@ branch-prot-flags-$(CONFIG_CC_HAS_SIGN_RETURN_ADDRESS) := -msign-return-address= # We enable additional protection for leaf functions as there is some # narrow potential for ROP protection benefits and no substantial # performance impact has been observed. +PACRET-y := pac-ret+leaf + +# Using a shadow call stack in leaf functions is too costly, so avoid PAC there +# as well when we may be patching PAC into SCS +PACRET-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) := pac-ret + ifeq ($(CONFIG_ARM64_BTI_KERNEL),y) -branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=pac-ret+leaf+bti +branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=$(PACRET-y)+bti else -branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=pac-ret+leaf +branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=$(PACRET-y) endif # -march=armv8.3-a enables the non-nops instructions for PAC, to avoid the # compiler to generate them and consequently to break the single image contractarch/arm64/include/asm/scs.h +29
-0@@ -24,6 +24,51 @@ .endm#endif /* CONFIG_SHADOW_CALL_STACK */ + +#else + +#include <linux/scs.h> +#include <asm/cpufeature.h> + +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS +static inline bool should_patch_pac_into_scs (void ) +{ +/* +* We only enable the shadow call stack dynamically if we are running +* on a system that does not implement PAC or BTI. PAC and SCS provide +* roughly the same level of protection, and BTI relies on the PACIASP +* instructions serving as landing pads, preventing us from patching +* those instructions into something else. +*/ +u64 reg =read_sysreg_s (SYS_ID_AA64ISAR1_EL1 ); + +if (reg & ((0xf <<ID_AA64ISAR1_APA_SHIFT ) | + (0xf <<ID_AA64ISAR1_API_SHIFT ))) +return false; + + reg =read_sysreg_s (SYS_ID_AA64ISAR2_EL1 ); +if (reg & (0xf <<ID_AA64ISAR2_APA3_SHIFT )) +return false; + +if (IS_ENABLED (CONFIG_ARM64_BTI_KERNEL )) { + reg =read_sysreg_s (SYS_ID_AA64PFR1_EL1 ); +if (reg & (0xf <<ID_AA64PFR1_BT_SHIFT ))
arch/arm64/include/asm/scs.h line 65
Quoted 10 lines
arch/arm64/include/asm/scs.h +10
-0+return false; + } +return true; +} + +static inline void dynamic_scs_init (void ) +{ +if (should_patch_pac_into_scs ()) +static_branch_enable (&dynamic_scs_enabled); +}
Should we print out a message to indicate we are actually enabling SCS at runtime? Otherwise I think the only way to know would be to look at /proc/meminfo, for example.
Quoted 41 lines
arch/arm64/include/asm/scs.h +6
-0+#else +static inline void dynamic_scs_init (void ) {} +#endif + +int scs_patch (const u8 eh_frame[],int size); +#endif /* __ASSEMBLY __ */ #endif /* _ASM_SCS_H */ arch/arm64/kernel/Makefile +2
-0@@ -74,6 +74,8 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o +CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.soarch/arm64/kernel/head.S +3
-0@@ -453,6 +453,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x21 // pass FDT address in x0 bl early_fdt_map // Try mapping the FDT early bl init_feature_override // Parse cpu feature overrides +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + bl scs_patch_vmlinux +#endif #ifdef CONFIG_RANDOMIZE_BASE tst x23, ~(MIN_KIMG_ALIGN - 1) // already running randomized? b.ne 0farch/arm64/kernel/irq.c +0
-0
arch/arm64/kernel/irq.c static void init_irq_scs(void) · line 45
Quoted 8 lines
arch/arm64/kernel/irq.c +1
-1@@ -39,7 +39,7 @@ static void init_irq_scs(void) {int cpu; -if (!IS_ENABLED (CONFIG_SHADOW_CALL_STACK )) +if (!scs_is_enabled ())return ;for_each_possible_cpu (cpu)
I applied the series on top of -rc1 and tested it in qemu. With -cpu cortex-a57 everything seems to work and PAC instructions are correctly patched into SCS push/pop:
Thread 1 hit Breakpoint 1, 0xffff8000098b7920 in scs_patch_vmlinux () (gdb) x/1i scs_alloc
0xffff8000081daeb8 <scs_alloc>: paciasp
(gdb) finish
Run till exit from #0 0xffff8000098b7920 in scs_patch_vmlinux () 0xffff8000098b03cc in __primary_switched ()
(gdb) x/1i scs_alloc
0xffff8000081daeb8 <scs_alloc>: str x30, [x18], #8
However, with -cpu max I'm still seeing calls to scs_alloc despite the following:
# dmesg | grep "Address authentication" [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) # zcat /proc/config.gz | grep -E '(SHADOW_|UNWIND|DYNAMIC_SCS)' CONFIG_UNWIND_TABLES=y CONFIG_UNWIND_PATCH_PAC_INTO_SCS=y CONFIG_CC_HAVE_SHADOW_CALL_STACK=y CONFIG_ARCH_SUPPORTS_SHADOW_CALL_STACK=y CONFIG_SHADOW_CALL_STACK=y CONFIG_DYNAMIC_SCS=y
It looks like we're correctly leaving the PAC instructions unpatched, but scs_is_enabled must be returning true:
Thread 1 hit Breakpoint 1, 0xffff8000098b7920 in scs_patch_vmlinux () (gdb) x/1i scs_alloc
0xffff8000081daeb8 <scs_alloc>: paciasp
(gdb) finish
Run till exit from #0 0xffff8000098b7920 in scs_patch_vmlinux () 0xffff8000098b03cc in __primary_switched ()
(gdb) x/1i scs_alloc
0xffff8000081daeb8 <scs_alloc>: paciasp
(gdb) c
Continuing.
Thread 1 hit Breakpoint 2, 0xffff8000081daeb8 in scs_alloc () (gdb) bt
#0 0xffff8000081daeb8 in scs_alloc () #1 0xffff800008015b60 in init_irq_scs () #2 0xffff8000098b3c5c in init_IRQ () #3 0xffff8000098b0954 in start_kernel () #4 0xffff8000098b03f4 in __primary_switched ()
I'm guessing this is also why I'm getting the following panic after attempting to load a module:
[ 25.549517] Unhandled 64-bit el1h sync exception on CPU0, ESR 0x0000000034000003 -- BTI [ 25.549893] CPU: 0 PID: 156 Comm: modprobe Not tainted 5.19.0-rc1-00246-gef5e30ca27c0-dirty #1 [ 25.550139] Hardware name: linux,dummy-virt (DT) [ 25.550426] pstate: 80400c09 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=j-) [ 25.550591] pc : scs_handle_fde_frame+0x80/0x17c [ 25.551341] lr : scs_handle_fde_frame+0x54/0x17c [ 25.551424] sp : ffff80000a67bb90 [ 25.551476] x29: ffff80000a67bb90 x28: 0000000000000000 x27: ffff800000ff6025 [ 25.552440] x26: 0000000000000003 x25: 0000000000000002 x24: ffff800000ff6026 [ 25.552584] x23: ffff800008ffd000 x22: 00000000d50323bf x21: 00000000d503233f [ 25.552725] x20: ffff800000ff0048 x19: ffff800000ff6014 x18: ffff80000a5d1000 [ 25.552867] x17: 0000000000000000 x16: ffff00001fe9cd88 x15: 24003032336f7470 [ 25.553020] x14: 180f0a0700000000 x13: 0000000000000040 x12: 0000000000000005 [ 25.553161] x11: 6c6300656c75646f x10: 0000000000000010 x9 : ffffffffffffffe4 [ 25.553354] x8 : ffff800008fd1870 x7 : 7f7f7f7f7f7f7f7f x6 : 16fefeff0eff1e0b [ 25.553511] x5 : 0080808000800000 x4 : 0000000000000010 x3 : 1800000010001f0c [ 25.553645] x2 : 1b011e7c0100527a x1 : 0000000000000000 x0 : ffff800000ff0048 [ 25.553980] Kernel panic - not syncing: Unhandled exception [ 25.554134] CPU: 0 PID: 156 Comm: modprobe Not tainted 5.19.0-rc1-00246-gef5e30ca27c0-dirty #1 [ 25.554254] Hardware name: linux,dummy-virt (DT) [ 25.554394] Call trace: [ 25.554512] dump_backtrace+0xe4/0x104 [ 25.554773] show_stack+0x18/0x24 [ 25.554893] dump_stack_lvl+0x64/0x7c [ 25.554983] dump_stack+0x18/0x38 [ 25.555067] panic+0x14c/0x370 [ 25.555149] el1t_64_irq_handler+0x0/0x1c [ 25.555231] el1_abort+0x0/0x5c [ 25.555314] el1h_64_sync+0x64/0x68 [ 25.555395] scs_handle_fde_frame+0x80/0x17c [ 25.555482] scs_patch+0x7c/0xa0 [ 25.555565] module_finalize+0xe0/0x100 [ 25.555647] post_relocation+0xd4/0xf0 [ 25.555730] load_module+0x924/0x11fc [ 25.555811] __arm64_sys_finit_module+0xbc/0x108 [ 25.555895] invoke_syscall+0x40/0x118 [ 25.555984] el0_svc_common+0xb4/0xf0 [ 25.556067] do_el0_svc+0x2c/0xb4 [ 25.556150] el0_svc+0x2c/0x7c [ 25.556233] el0t_64_sync_handler+0x84/0xf0 [ 25.556315] el0t_64_sync+0x190/0x194
This is with defconfig + SHADOW_CALL_STACK + UNWIND_PATCH_PAC_INTO_SCS. Any thoughts if I'm doing something wrong here?
Sami
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Ard Biesheuvel
maintainerRe: [PATCH v3 3/3] arm64: implement dynamic shadow call stack for Clang
Quoted 29 lines · Sami Tolvanen
Quoted 27 lines · Ard Biesheuvel
Implement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively.
This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the desired behavior.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/Kconfig | 9 + arch/arm64/Makefile | 10 +- arch/arm64/include/asm/scs.h | 45 ++++ arch/arm64/kernel/Makefile | 2 + arch/arm64/kernel/head.S | 3 + arch/arm64/kernel/irq.c | 2 +- arch/arm64/kernel/module.c | 8 + arch/arm64/kernel/patch-scs.c | 257 ++++++++++++++++++++ arch/arm64/kernel/sdei.c | 2 +- arch/arm64/kernel/setup.c | 4 + 10 files changed, 338 insertions(+), 4 deletions(-)
...
Quoted 2 lines
Quoted 2 lines
arch/arm64/include/asm/scs.h +0
-0
...
Quoted 11 lines
arch/arm64/include/asm/scs.h line 65
Quoted 6 lines
arch/arm64/include/asm/scs.h +6
-0+ +static inline void dynamic_scs_init (void ) +{ +if (should_patch_pac_into_scs ()) +static_branch_enable (&dynamic_scs_enabled); +}Should we print out a message to indicate we are actually enabling SCS at runtime? Otherwise I think the only way to know would be to look at
/proc/meminfo, for example.
Yes, good point. I will add something here.
...
Quoted 5 lines
I applied the series on top of -rc1 and tested it in qemu. With -cpu cortex-a57 everything seems to work and PAC instructions are correctly patched into SCS push/pop:
...
Quoted 6 lines
However, with -cpu max I'm still seeing calls to
scs_allocdespite the following:# dmesg | grep "Address authentication" [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)
...
Quoted 4 lines
I'm guessing this is also why I'm getting the following panic after attempting to load a module:
[ 25.549517] Unhandled 64-bit el1h sync exception on CPU0, ESR 0x0000000034000003 -- BTI
...
Quoted 4 lines
This is with defconfig +
SHADOW_CALL_STACK+UNWIND_PATCH_PAC_INTO_SCS. Any thoughts if I'm doing something wrong here?
No, I made a mistake with the initial value of the dynamic_scs_enabled static key, so it's currently always true.
I'll fix this in the next revision.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel