arm64: dynamic shadow call stack support

Generic kernel images such as Android's GKI usually enable all available security features, which are typically implemented in such a way that they only take effect if the underlying hardware can support it, but don't interfere with correct and efficient operation otherwise.

For shadow call stack support, which is always supported by the hardware, it means it will be enabled even if pointer authentication is also supported, and enabled for signing return addresses stored on the stack. The additional security provided by shadow call stack is only marginal in this case, whereas the performance overhead is not.

Given that return address signing is based on PACIASP/AUTIASP instructions that implicitly operate on the return address register (X30) and are not idempotent (i.e., each needs to be emitted exactly once before the return address is stored on the ordinary stack and after it has been retrieved from it), we can convert these instruction 1:1 into shadow call stack pushes and pops involving the register X30. As this is something that can be done at runtime rather than build time, we can do this conditionally based on whether or not return address signing is supported on the underlying hardware.

In order to allow runtimes to unwind call stacks that involve return address signing, we track whether or not the return address is currently signed by means of DWARF CFI directives in the unwinding metadata. This means we can use this information to locate all PACIASP/AUTIASP instructions in the binary, instead of having to use brute force and go over all instructions in the entire program.

This series implements this approach for Clang, which has been vetted (and fixed in release 15) to ensure that the unwind metadata is 100% accurate when it comes to PACIASP/AUTIASP occurrences. Sadly, GCC does not always get that quite right, so this series is Clang-only for the moment.

Changes since v4 [1]:
- rebase onto v6.0-rc2
- use SYS_FIELD_GET for AA64ISAR1/2 sysreg field accesses
- add Sami's Rb/Tb

Changes since v3 [2]:
- rebase onto arm64/for-next/core
- fix init value of dynamic_scs_enabled static key
- don't discard .eh_frame sections (to work around a bug in an older Clang version if we are keeping them for dynamic SCS patching,
- print a diagnostic if dynamic SCS patching is enabled,
- apply build fix suggested by Sami and add his ack to patch #2

Changes since v2 [3]:
- don't enable unwind table generation for nVHE code - it cannot be patched anyway so it has no use for it;
- drop checks for ID reg overrides
- fix some remaining TODOs regarding augmentation data and the code alignment factor
- disable PAC for leaf functions when dynamic SCS is configured, so that we don't end up with SCS pushes and pops in all leaf functions too;
- add I-cache maintenance after code patching
- add Rb's from Nick and Kees.

Changes since RFC v1:
- implement boot time check for PAC/BTI support, and only enable dynamic SCS if neither are supported;
- implement module patching as well;
- switch to Clang, and drop workaround for GCC bug;

[0] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[3] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Ard Biesheuvel (3):
arm64: unwind: add asynchronous unwind tables to kernel and modules
scs: add support for dynamic shadow call stacks
arm64: implement dynamic shadow call stack for Clang

 Makefile                              |   2 +
 arch/Kconfig                          |   7 +
 arch/arm64/Kconfig                    |  12 +
 arch/arm64/Makefile                   |  15 +-
 arch/arm64/include/asm/module.lds.h   |   8 +
 arch/arm64/include/asm/scs.h          |  49 ++++
 arch/arm64/kernel/Makefile            |   2 +
 arch/arm64/kernel/head.S              |   3 +
 arch/arm64/kernel/irq.c               |   2 +-
 arch/arm64/kernel/module.c            |   8 +
 arch/arm64/kernel/patch-scs.c         | 257 ++++++++++++++++++++
 arch/arm64/kernel/pi/Makefile         |   1 +
 arch/arm64/kernel/sdei.c              |   2 +-
 arch/arm64/kernel/setup.c             |   4 +
 arch/arm64/kernel/vmlinux.lds.S       |  13 +
 arch/arm64/kvm/hyp/nvhe/Makefile      |   1 +
 drivers/firmware/efi/libstub/Makefile |   1 +
 include/asm-generic/vmlinux.lds.h     |   9 +-
 include/linux/scs.h                   |  18 ++
 kernel/scs.c                          |  14 +-
 scripts/module.lds.S                  |   8 +-
 21 files changed, 427 insertions(+), 9 deletions(-)
 create mode 100644 arch/arm64/kernel/patch-scs.c

2.35.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Enable asynchronous unwind table generation for both the core kernel as well as modules, and emit the resulting .eh_frame sections as init code so we can use the unwind directives for code patching at boot or module load time.

This will be used by dynamic shadow call stack support, which will rely on code patching rather than compiler codegen to emit the shadow call stack push and pop instructions.

---
 arch/arm64/Kconfig                    |  3 +++
 arch/arm64/Makefile                   |  5 +++++
 arch/arm64/include/asm/module.lds.h   |  8 ++++++++
 arch/arm64/kernel/pi/Makefile         |  1 +
 arch/arm64/kernel/vmlinux.lds.S       | 13 +++++++++++++
 arch/arm64/kvm/hyp/nvhe/Makefile      |  1 +
 drivers/firmware/efi/libstub/Makefile |  1 +
 include/asm-generic/vmlinux.lds.h     |  9 +++++++--
 scripts/module.lds.S                  |  8 +++++++-
 9 files changed, 46 insertions(+), 3 deletions(-)

index 571cc234d0b3..d293ac1c5b7f 100644
@@ -370,6 +370,9 @@ config KASAN_SHADOW_OFFSET
 	default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS
 	default 0xffffffffffffffff
 
+config UNWIND_TABLES
+	bool
+
 source "arch/arm64/Kconfig.platforms"
 
 menu "Kernel Features"

index 6d9d4a58b898..4fbca56fa602 100644
@@ -45,8 +45,13 @@ KBUILD_CFLAGS	+= $(call cc-option,-mabi=lp64)
 KBUILD_AFLAGS	+= $(call cc-option,-mabi=lp64)
 
 # Avoid generating .eh_frame* sections.
+ifneq ($(CONFIG_UNWIND_TABLES),y)
 KBUILD_CFLAGS	+= -fno-asynchronous-unwind-tables -fno-unwind-tables
 KBUILD_AFLAGS	+= -fno-asynchronous-unwind-tables -fno-unwind-tables
+else
+KBUILD_CFLAGS	+= -fasynchronous-unwind-tables
+KBUILD_AFLAGS	+= -fasynchronous-unwind-tables
+endif
 
 ifeq ($(CONFIG_STACKPROTECTOR_PER_TASK),y)
 prepare: stack_protector_prepare

index 094701ec5500..dbba4b7559aa 100644
@@ -17,4 +17,12 @@ SECTIONS {
 	 */
 	.text.hot : { *(.text.hot) }
 #endif
+
+#ifdef CONFIG_UNWIND_TABLES
+	/*
+	 * Currently, we only use unwind info at module load time, so we can
+	 * put it into the .init allocation.
+	 */
+	.init.eh_frame : { *(.eh_frame) }
+#endif
 }

index 839291430cb3..4c0ea3cd4ea4 100644
@@ -7,6 +7,7 @@ KBUILD_CFLAGS	:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \
 		   -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \
 		   -include $(srctree)/include/linux/hidden.h \
 		   -D__DISABLE_EXPORTS -ffreestanding -D__NO_FORTIFY \
+		   -fno-asynchronous-unwind-tables -fno-unwind-tables \
 		   $(call cc-option,-fno-addrsig)
 
 # remove SCS flags from all objects in this directory

index 45131e354e27..4c13dafc98b8 100644
@@ -121,6 +121,17 @@ jiffies = jiffies_64;
 #define TRAMP_TEXT
 #endif
 
+#ifdef CONFIG_UNWIND_TABLES
+#define UNWIND_DATA_SECTIONS				\
+	.eh_frame : {					\
+		__eh_frame_start = .;			\
+		*(.eh_frame)				\
+		__eh_frame_end = .;			\
+	}
+#else
+#define UNWIND_DATA_SECTIONS
+#endif
+
 /*
  * The size of the PE/COFF section that covers the kernel image, which
  * runs from _stext to _edata, must be a round multiple of the PE/COFF

@@ -231,6 +242,8 @@ SECTIONS
 		__alt_instructions_end = .;
 	}
 
+	UNWIND_DATA_SECTIONS
+
 	. = ALIGN(SEGMENT_ALIGN);
 	__inittext_end = .;
 	__initdata_begin = .;

index b5c5119c7396..f566aad90875 100644
@@ -89,6 +89,7 @@ quiet_cmd_hypcopy = HYPCOPY $@
 # Remove ftrace, Shadow Call Stack, and CFI CFLAGS.
 # This is equivalent to the 'notrace', '__noscs', and '__nocfi' annotations.
 KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI), $(KBUILD_CFLAGS))
+KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables
 
 # KVM nVHE code is run at a different exception code with a different map, so
 # compiler instrumentation that inserts callbacks or checks into the code may

index d0537573501e..78c46638707a 100644
@@ -20,6 +20,7 @@ cflags-$(CONFIG_X86)		+= -m$(BITS) -D__KERNEL__ \
 # disable the stackleak plugin
 cflags-$(CONFIG_ARM64)		:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \
 				   -fpie $(DISABLE_STACKLEAK_PLUGIN) \
+				   -fno-unwind-tables -fno-asynchronous-unwind-tables \
 				   $(call cc-option,-mbranch-protection=none)
 cflags-$(CONFIG_ARM)		:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \
 				   -fno-builtin -fpic \

index 7515a465ec03..6d870275372b 100644
@@ -1008,15 +1008,20 @@
  * keep any .init_array.* sections.
  * https://bugs.llvm.org/show_bug.cgi?id=46478
  */
+#ifdef CONFIG_UNWIND_TABLES
+#define DISCARD_EH_FRAME
+#else
+#define DISCARD_EH_FRAME	*(.eh_frame)
+#endif
 #if defined(CONFIG_GCOV_KERNEL) || defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KCSAN) || \
 	defined(CONFIG_CFI_CLANG)
 # ifdef CONFIG_CONSTRUCTORS
 #  define SANITIZER_DISCARDS						\
-	*(.eh_frame)
+	DISCARD_EH_FRAME
 # else
 #  define SANITIZER_DISCARDS						\
 	*(.init_array) *(.init_array.*)					\
-	*(.eh_frame)
+	DISCARD_EH_FRAME
 # endif
 #else
 # define SANITIZER_DISCARDS

index 3a3aa2354ed8..9b7193d95dad 100644
@@ -3,10 +3,16 @@
  * Archs are free to supply their own linker scripts.  ld will
  * combine them automatically.
  */
+#ifdef CONFIG_UNWIND_TABLES
+#define DISCARD_EH_FRAME
+#else
+#define DISCARD_EH_FRAME	*(.eh_frame)
+#endif
+
 #ifdef CONFIG_CFI_CLANG
 # include <asm/page.h>
 # define ALIGN_CFI 		ALIGN(PAGE_SIZE)
-# define SANITIZER_DISCARDS	*(.eh_frame)
+# define SANITIZER_DISCARDS	DISCARD_EH_FRAME
 #else
 # define ALIGN_CFI
 # define SANITIZER_DISCARDS

2.35.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

In order to allow arches to use code patching to conditionally emit the shadow stack pushes and pops, rather than always taking the performance hit even on CPUs that implement alternatives such as stack pointer authentication on arm64, add a Kconfig symbol that can be set by the arch to omit the SCS codegen itself, without otherwise affecting how support code for SCS and compiler options (for register reservation, for instance) are emitted.

Also, add a static key and some plumbing to omit the allocation of shadow call stack for dynamic SCS configurations if SCS is disabled at runtime.

---
 Makefile            |  2 ++
 arch/Kconfig        |  7 +++++++
 include/linux/scs.h | 18 ++++++++++++++++++
 kernel/scs.c        | 14 ++++++++++++--
 4 files changed, 39 insertions(+), 2 deletions(-)

index c7705f749601..089d372a6f6d 100644
@@ -888,8 +888,10 @@ LDFLAGS_vmlinux += --gc-sections
 endif
 
 ifdef CONFIG_SHADOW_CALL_STACK
+ifndef CONFIG_DYNAMIC_SCS
 CC_FLAGS_SCS	:= -fsanitize=shadow-call-stack
 KBUILD_CFLAGS	+= $(CC_FLAGS_SCS)
+endif
 export CC_FLAGS_SCS
 endif
 

index 5dbf11a5ba4e..f090cdca7508 100644
@@ -645,6 +645,13 @@ config SHADOW_CALL_STACK
 	  reading and writing arbitrary memory may be able to locate them
 	  and hijack control flow by modifying the stacks.
 
+config DYNAMIC_SCS
+	bool
+	help
+	  Set by the arch code if it relies on code patching to insert the
+	  shadow call stack push and pop instructions rather than on the
+	  compiler.
+
 config LTO
 	bool
 	help

index 18122d9e17ff..4ab5bdc898cf 100644
@@ -53,6 +53,22 @@ static inline bool task_scs_end_corrupted(struct task_struct *tsk)
 	return sz >= SCS_SIZE - 1 || READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC;
 }
 
+DECLARE_STATIC_KEY_FALSE(dynamic_scs_enabled);
+
+static inline bool scs_is_dynamic(void)
+{
+	if (!IS_ENABLED(CONFIG_DYNAMIC_SCS))
+		return false;
+	return static_branch_likely(&dynamic_scs_enabled);
+}
+
+static inline bool scs_is_enabled(void)
+{
+	if (!IS_ENABLED(CONFIG_DYNAMIC_SCS))
+		return true;
+	return scs_is_dynamic();
+}
+
 #else /* CONFIG_SHADOW_CALL_STACK */
 
 static inline void *scs_alloc(int node) { return NULL; }

@@ -62,6 +78,8 @@ static inline void scs_task_reset(struct task_struct *tsk) {}
 static inline int scs_prepare(struct task_struct *tsk, int node) { return 0; }
 static inline void scs_release(struct task_struct *tsk) {}
 static inline bool task_scs_end_corrupted(struct task_struct *tsk) { return false; }
+static inline bool scs_is_enabled(void) { return false; }
+static inline bool scs_is_dynamic(void) { return false; }
 
 #endif /* CONFIG_SHADOW_CALL_STACK */
 

index b7e1b096d906..d7809affe740 100644
@@ -12,6 +12,10 @@
 #include <linux/vmalloc.h>
 #include <linux/vmstat.h>
 
+#ifdef CONFIG_DYNAMIC_SCS
+DEFINE_STATIC_KEY_FALSE(dynamic_scs_enabled);
+#endif
+
 static void __scs_account(void *s, int account)
 {
 	struct page *scs_page = vmalloc_to_page(s);

@@ -101,14 +105,20 @@ static int scs_cleanup(unsigned int cpu)
 
 void __init scs_init(void)
 {
+	if (!scs_is_enabled())
+		return;
 	cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "scs:scs_cache", NULL,
 			  scs_cleanup);
 }
 
 int scs_prepare(struct task_struct *tsk, int node)
 {
-	void *s = scs_alloc(node);
+	void *s;
 
+	if (!scs_is_enabled())
+		return 0;
+
+	s = scs_alloc(node);
 	if (!s)
 		return -ENOMEM;
 

@@ -148,7 +158,7 @@ void scs_release(struct task_struct *tsk)
 {
 	void *s = task_scs(tsk);
 
-	if (!s)
+	if (!scs_is_enabled() || !s)
 		return;
 
 	WARN(task_scs_end_corrupted(tsk),

2.35.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Implement dynamic shadow call stack support on Clang, by parsing the unwind tables at init time to locate all occurrences of PACIASP/AUTIASP instructions, and replacing them with the shadow call stack push and pop instructions, respectively.

This is useful because the overhead of the shadow call stack is difficult to justify on hardware that implements pointer authentication (PAC), and given that the PAC instructions are executed as NOPs on hardware that doesn't, we can just replace them without breaking anything. As PACIASP/AUTIASP are guaranteed to be paired with respect to manipulations of the return address, replacing them 1:1 with shadow call stack pushes and pops is guaranteed to result in the desired behavior.

---
 arch/arm64/Kconfig            |   9 +
 arch/arm64/Makefile           |  10 +-
 arch/arm64/include/asm/scs.h  |  49 ++++
 arch/arm64/kernel/Makefile    |   2 +
 arch/arm64/kernel/head.S      |   3 +
 arch/arm64/kernel/irq.c       |   2 +-
 arch/arm64/kernel/module.c    |   8 +
 arch/arm64/kernel/patch-scs.c | 257 ++++++++++++++++++++
 arch/arm64/kernel/sdei.c      |   2 +-
 arch/arm64/kernel/setup.c     |   4 +
 10 files changed, 342 insertions(+), 4 deletions(-)

index d293ac1c5b7f..396edf45a1ba 100644
@@ -2108,6 +2108,15 @@ config ARCH_NR_GPIO
 
           If unsure, leave the default value.
 
+config UNWIND_PATCH_PAC_INTO_SCS
+	bool "Enable shadow call stack dynamically using code patching"
+	# needs Clang with https://reviews.llvm.org/D111780 incorporated
+	depends on CC_IS_CLANG && CLANG_VERSION >= 150000
+	depends on ARM64_PTR_AUTH_KERNEL && CC_HAS_BRANCH_PROT_PAC_RET
+	depends on SHADOW_CALL_STACK
+	select UNWIND_TABLES
+	select DYNAMIC_SCS
+
 endmenu # "Kernel Features"
 
 menu "Boot options"

index 4fbca56fa602..e439ebbd167d 100644
@@ -77,10 +77,16 @@ branch-prot-flags-$(CONFIG_CC_HAS_SIGN_RETURN_ADDRESS) := -msign-return-address=
 # We enable additional protection for leaf functions as there is some
 # narrow potential for ROP protection benefits and no substantial
 # performance impact has been observed.
+PACRET-y := pac-ret+leaf
+
+# Using a shadow call stack in leaf functions is too costly, so avoid PAC there
+# as well when we may be patching PAC into SCS
+PACRET-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) := pac-ret
+
 ifeq ($(CONFIG_ARM64_BTI_KERNEL),y)
-branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=pac-ret+leaf+bti
+branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET_BTI) := -mbranch-protection=$(PACRET-y)+bti
 else
-branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=pac-ret+leaf
+branch-prot-flags-$(CONFIG_CC_HAS_BRANCH_PROT_PAC_RET) := -mbranch-protection=$(PACRET-y)
 endif
 # -march=armv8.3-a enables the non-nops instructions for PAC, to avoid the
 # compiler to generate them and consequently to break the single image contract

index 1add7b01efa7..420b53b24f07 100644
@@ -79,6 +79,8 @@ obj-$(CONFIG_ARM64_PTR_AUTH)		+= pointer_auth.o
 obj-$(CONFIG_ARM64_MTE)			+= mte.o
 obj-y					+= vdso-wrap.o
 obj-$(CONFIG_COMPAT_VDSO)		+= vdso32-wrap.o
+obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS)	+= patch-scs.o
+CFLAGS_patch-scs.o			+= -mbranch-protection=none
 
 # Force dependency (vdso*-wrap.S includes vdso.so through incbin)
 $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so

index cefe6a73ee54..b2f5ee5a700f 100644
@@ -460,6 +460,9 @@ SYM_FUNC_START_LOCAL(__primary_switched)
 	bl	early_fdt_map			// Try mapping the FDT early
 	mov	x0, x20				// pass the full boot status
 	bl	init_feature_override		// Parse cpu feature overrides
+#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS
+	bl	scs_patch_vmlinux
+#endif
 	mov	x0, x20
 	bl	finalise_el2			// Prefer VHE if possible
 	ldp	x29, x30, [sp], #16

index bda49430c9ea..c284ec35c27c 100644
@@ -39,7 +39,7 @@ static void init_irq_scs(void)
 {
 	int cpu;
 
-	if (!IS_ENABLED(CONFIG_SHADOW_CALL_STACK))
+	if (!scs_is_enabled())
 		return;
 
 	for_each_possible_cpu(cpu)

index f2d4bb14bfab..111dc6414e6d 100644
@@ -15,9 +15,11 @@
 #include <linux/kernel.h>
 #include <linux/mm.h>
 #include <linux/moduleloader.h>
+#include <linux/scs.h>
 #include <linux/vmalloc.h>
 #include <asm/alternative.h>
 #include <asm/insn.h>
+#include <asm/scs.h>
 #include <asm/sections.h>
 
 void *module_alloc(unsigned long size)

@@ -529,5 +531,11 @@ int module_finalize(const Elf_Ehdr *hdr,
 	if (s)
 		apply_alternatives_module((void *)s->sh_addr, s->sh_size);
 
+	if (scs_is_dynamic()) {
+		s = find_section(hdr, sechdrs, ".init.eh_frame");
+		if (s)
+			scs_patch((void *)s->sh_addr, s->sh_size);
+	}
+
 	return module_init_ftrace_plt(hdr, sechdrs, me);
 }

index d20620a1c51a..30f3c7563694 100644
@@ -144,7 +144,7 @@ static int init_sdei_scs(void)
 	int cpu;
 	int err = 0;
 
-	if (!IS_ENABLED(CONFIG_SHADOW_CALL_STACK))
+	if (!scs_is_enabled())
 		return 0;
 
 	for_each_possible_cpu(cpu) {

index fea3223704b6..12cfe9d0d3fa 100644
@@ -30,6 +30,7 @@
 #include <linux/efi.h>
 #include <linux/psci.h>
 #include <linux/sched/task.h>
+#include <linux/scs.h>
 #include <linux/mm.h>
 
 #include <asm/acpi.h>

@@ -42,6 +43,7 @@
 #include <asm/cpu_ops.h>
 #include <asm/kasan.h>
 #include <asm/numa.h>
+#include <asm/scs.h>
 #include <asm/sections.h>
 #include <asm/setup.h>
 #include <asm/smp_plat.h>

@@ -312,6 +314,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	jump_label_init();
 	parse_early_param();
 
+	dynamic_scs_init();
+
 	/*
 	 * Unmask asynchronous aborts and fiq after bringing up possible
 	 * earlycon. (Report possible System Errors once we can report this

2.35.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Will, Catalin, what's left for this series? I'd really to get this landed -- it's reviewed and tested, and will be used on real devices.

Thanks!

-Kees

Kees Cook

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel